Job Description
The Data team builds and maintains the infrastructure responsible for consuming, exposing and creating product and product-derived (augmented) datasets. This team owns the pipelines that transport and process database data from all of BrightTALK’s product surfaces. The team is responsible for building and operating the infrastructure and services that ensure data accuracy and data availability for stakeholders in Data Science, Business and other Engineering feature teams.
We are looking for the ideal Data Engineer who will partner with business, analytics and engineering teams to design, build and maintain ease for use data structures to facilitate reporting and monitoring key performance indicators. Collaborating across disciplines, you will identify internal/external data sources to design table structure, define ETL strategy & automated QA checks and implement scalable ETL solutions.
Responsibilities
- Building highly reliable data services to integrate with dozens of consumers
- Creating ETL or ELT pipelines that transform and process petabytes of structured and unstructured data in real-time
- Developing data models for optimal storage and retrieval to support sub-second latency
- Deploying and monitoring large database clusters that are performant and highly-available
- Working cross-functionally with data scientists, backend engineers, and product managers to design and implement, and new data models to support the product
- Developing skills through exceptional training as well as frequent coaching and mentoring from colleagues
- Some upcoming technical challenges include scaling our data ingestion pipelines across a growing number of AWS and data center based data sources, reducing the latency of our product data ingestion pipelines through moving batch jobs into a streaming architecture, and extending our data lake architecture for the growing ecosystem of data ingestion and creation tools
Qualifications
- Bachelor's degree (or equivalent) in Computer Science or related field
- 5+ years of experience building real-time and distributed system architecture, from whiteboard to production
- Strong programming skills in Python, Scala and SQL.
- Experience across the entire spectrum of data engineering, including:
- Data stores (e.g., AWS RDS, AWS Athena, AWS Aurora, AWS Redshift)
- Data pipeline and workflow orchestration tools (e.g., Azkaban, Airflow)
- Data processing technologies (e.g., Spark, Pentaho)
- Deployment and monitoring large database clusters in public cloud platforms (e.g., Docker, Terraform, Datadog)
- Goals can change fast. You anticipate and react quickly.
- You own what you work on. You move fast and get things done.
- Excellent communication. You will need communicate complex ideas effectively to both technical and non-technical audiences, and both verbally and in writing
- You must work collaboratively in a cross-functional team and with people at all levels in an organization
- Industry experience building and productionizing innovative end-to-end Machine Learning systems is a plus.
We are dedicated to learning and growing, going after big dreams, taking ownership, remaining open and honest while recognizing we are in it together. We are committed to hiring individuals with a diversity of perspectives, backgrounds, and beliefs who can use their passion to help us build a world changing audience and thrive along the way. Join BrightTALK and experience a collaborative and engaged team who works together to provide learning and growth opportunities for businesses and professionals.
Company Info.
BrightTALK
BrightTALK brings professionals and businesses together to learn and grow.
Thousands of thought leaders are actively sharing their insights, their ideas and their most up-to-date knowledge with professionals all over the globe through the technologies that BrightTALK has created.
Get Similar Jobs In Your Inbox
BrightTALK is currently hiring Data Engineer Jobs in Bangalore, Karnataka, India with average base salary of ₹600,000 - ₹1,000,000 / Year.