Manager Data Engineering

Key Skills

Java Programming, Python Programming, C++, C Programming, SQL, Cloud computing, Scala Programming, Machine learning techniques, Data science techniques, MATLAB Programming, PyTorch, TensorFlow, R Programming

Job Description

The Manager Data Engineer role requires expertise with managing the requirements analysis, design, development and deployment of data-driven solutions for clients. This role will work across multiple clients and have responsibility for leading a team of onshore and offshore data engineers.

Responsibilities:

• Own the design and implementation of data platforms and pipelines to support advanced analytics models and decision making driven by our AI products
• Architect the (standard) reference architecture of data platforms that will ingest, transform and manage high volumes of data (in batch or near real-time) for consumption by our AI products
• Implement best practices for designing and developing scalable, parameterized and automated data pipelines leveraging distributed parallel processing capabilities employing Spark, Scala and Dask
• Guide teams on the implementation of workflows to automate data pipeline using python and Argo
• Develop standards for designing and developing data validation tests to assess the quality and integrity of the input data.
• Institute best practices for performance testing and profiling of code to validate scalability and sustainability of solutions
• Mentor Data Engineers on delivery teams with prioritizing their task backlogs and adopting best practices in deploying the data pipeline workflows
• Stay abreast of the latest technology trends and best practices as they relate to architecting and delivering best in class data driven solutions for our clients
• Note: The statements herein are intended to describe the general nature and level of work being performed by employees and are not to be construed as an exhaustive list of responsibilities, duties, abilities, and skills required of personnel so classified.

Qualifications:

Minimum Education:

• Bachelors, Master's or Ph.D. Degree in Computer Science or Engineering.
• Minimum Work Experience (years):
• 6 – 8 years overall work experience, with 2 - 3 years experience leading professional services delivery teams
• with progressively increasing scope of responsibilities up to a Team Lead/Supervisor-level equivalent
• 5 - 7 years of experience with SQL and developing data transformations / ETLs
• 3 - 5 years of experience programming with at least one of the following languages: Python, Scala or Go
• 3 - 5 years of experience developing distributed systems using open source technologies: Spark and Dask.
• 5 – 7 years of experience with relational databases or 3 – 5 years experience with NoSQL databases running
• in Linux environments (MySQL, MariaDB, PostgreSQL, MongoDB, Redis).
• Key Skills and Competencies:
• Experience working with AWS / Azure / GCP environment is highly desired.
• Experience with data orchestration tools like Azure Data Factory (ADF), Informatica and Airflow, is desired.
• Experience in data models in the Retail and Consumer products industry is desired.
• Experience working on agile projects and understanding of agile concepts is desired.
• Demonstrated ability to learn new technologies quickly and independently.
• Excellent verbal and written communication skills, especially in leading technical communications with
• clients and internal/direct reports.
• Ability to work and achieve stretch goals in a very innovative and fast-paced environment.
• Ability to work collaboratively in a diverse team environment.
• Licenses/Certifications, special qualifications:
  • AWS, Azure or GCP certification
  • Databricks/Spark certification (desirable)
• Travel Requirements: 10%
• Able to telework? Yes