Lead Data Engineer

Key Skills

Java Programming, Python Programming, SQL, Azure, AWS, Big Data Technology, Apache Hadoop, Apache Kafka, Scala Programming, NoSQL, Google Cloud Platform (GCP), Data Modelling, SPARK Programming, Databricks, Data Engineering, Data Lake, Large scale data processing, Data Governance

Job Description

• Deep experience with complex data systems and good instincts around data modelling and usage.
• Knowledge of data engineering technologies, architecture, and processes. Specifically, Azure Data Lake, Hadoop ecosystem, Kafka, and common third-party integration and orchestration tools.
• Good knowledge of multi-cloud data ecosystem and build scalable solutions on cloud (Azure, GCP and AWS)
• Good knowledge of Big Data Ecosystem-Spark, Hadoop, Databricks
• Work across 3-4 teams to develop practices which lead to the highest quality products and contribute transformation change within the cloud
• Experience building large scale data processing ecosystems with real time and batch style data as input using big data technologies
• Experience in any programming language like Scala or Python.
• Exposure to agile methodology and proven ability to technically lead a team of engineers across geographies.
• Implement Data Quality, Data Governance on Azure Cloud ecosystem
• Good instincts around technical architecture, including metadata, Rest API Integrations, Data API and Solution design of NoSQL and File systems.
• Willingness and ability to invest in engineering growth.
• Strong communication skills and ability to coordinate across a diverse group of technical and non-technical stakeholders.

PREFERRED QUALIFICATIONS

Key Responsibilities

The Lead Data Engineer is responsible for guiding and mentoring number of (3-4) delivery squads (scrum teams), driving the backlog while being accountable for design and development of complex data & analytics products.

Specific responsibilities include:

• Guide and mentor 3 to 4 squads comprising data engineers/developers.
• Ensure high-quality output from engineering team, including setting coding standards and practices.
• Partner with the Data platform Product Owner to create a product roadmap and execute through agile processes.
• Collaborate with Principal Architect and software delivery teams to prevent overlapping work and coordinate on interoperability requirements.
• Work with other product engineering teams to ensure a consistent software ecosystem.
• Assist in architectural design and make decisions around technology and implementation decisions.
• Industry-level expert in data platform architecture and engineering for a variety of workloads from analytics, data science and AI/ML
• Able to conceptualize and able to implement the data stream ecosystems and drive requirements.
• Experience with engineering an AI / ML Workbench with Pre-processing, Training (Feature Engineering), Inferencing, Model Deployment (with model experimentation) and Ongoing MLOps
• Understand and able to solve canonical models, align across various product teams and get the common objects defined, responsible for plan and delivery.