Data Engineering Tech Lead (Chennai, IN)

Key Skills

Java Programming, Python Programming, SQL

Job Description

The Data team builds and maintains the infrastructure responsible for consuming, exposing and creating product and product-derived (augmented) datasets. This team owns the pipelines that transport and process database data from all of BrightTALK’s product surfaces. The team is responsible for building and operating the infrastructure and services that ensure data accuracy and data availability for stakeholders in Data Science, Business and other Engineering feature teams.

The Data Engineering Lead will provide technical leadership and strategic direction for our data platform. You will be responsible for the delivery and technical quality of our programs, taking on complex multi-year projects, and lead the work of several ICs. You’ll also partner with cross-functional stakeholders to drive initiatives forward, and mentor team members in technical design, project leadership, and running team processes. The Data Engineering Lead will collaborate with experienced data scientists, engineers, product managers and vendors to build and maintain scalable data infrastructure. You will be deeply involved in the technical details of building highly available infrastructure and ultimately work to build a safer financial system for billions of people

Responsibilities

• Building highly reliable data services to integrate with dozens of consumers
• Creating ETL or ELT pipelines that transform and process petabytes of structured and unstructured data in real-time
• Designing data models for optimal storage and retrieval to support sub-second latency
• Deploying and monitoring large database clusters that are performant and highly-available
• Working cross-functionally with data scientists, backend engineers, and product managers to design and implement, and new data models to support the product
• Developing your skills through exceptional training as well as frequent coaching and mentoring from colleagues
• Some upcoming technical challenges include scaling our data ingestion pipelines across a growing number of AWS and data center based data sources, reducing the latency of our product data ingestion pipelines through moving batch jobs into a streaming architecture, and extending our data lake architecture for the growing ecosystem of data ingestion and creation tools

Qualifications

• Bachelor's degree (or equivalent) in Computer Science or related field
• 5+ years of experience building real-time and distributed system architecture, from whiteboard to production
• Strong programming skills in Python, Java or Scala and SQL.
• Versatility. Experience across the entire spectrum of data engineering, including:
• Data stores (e.g., AWS RDS, AWS Athena, AWS Aurora, AWS Redshift)
• Data pipeline and workflow orchestration tools (e.g., Azkaban, Airflow)
• Data processing technologies (e.g., Spark, Pentaho)
• Deployment and monitoring large database clusters in public cloud platforms (e.g., Docker, Terraform, Datadog)
• Adaptable. Goals can change fast. You anticipate and react quickly.
• Autonomous. You own what you work on. You move fast and get things done.
• Excellent communication. You will need communicate complex ideas effectively to both technical and non-technical audiences, and both verbally and in writing
• Collaborative. You must work collaboratively in a cross-functional team and with people at all levels in an organization
• Industry experience building and productionizing innovative end-to-end Machine Learning systems is a plus.

We are dedicated to learning and growing, going after big dreams, taking ownership, remaining open and honest while recognizing we are in it together. We are committed to hiring individuals with a diversity of perspectives, backgrounds, and beliefs who can use their passion to help us build a world changing audience and thrive along the way. Join BrightTALK and experience a collaborative and engaged team who works together to provide learning and growth opportunities for businesses and professionals.