Lead Data Engineer (P248)

Key Skills

Apache Cassandra, Apache Hadoop, AWS, Azure, Data Engineering, Data science techniques, Database, Google Cloud Platform (GCP), Leadership Skill, MongoDB, NoSQL, SQL

Job Description

As a Lead Data Engineer, you will have the opportunity to build and support solutions that ingest, store and distribute our big data to be consumed by our products. Our data engineers on this team use Databricks, Terraform, PySpark, Python, Snowflake and Angular along with other data engineering technologies and visualization tools, while working alongside our data scientists to deliver data capabilities and services to customers, both internal and external.

Responsibilities

Take ownership of systems, processes, and the tech stack while driving features to completion through all phases of the entire 84.51° SDLC. This includes internal and external facing applications as well as process improvement activities:

• Lead the design of and develop Cloud based solutions
• Perform unit and integration testing
• Participate in implementation of BI visualizations
• Collaborate with Architecture, Data Scientists, Product, and other Lead Engineers to ensure consistent development practices
• Provide mentoring to junior engineers
• Partner closely with Product colleagues to inform strategy and roadmap decisions
• Participate in Agile rituals (Sprint Reviews, Stand-Ups, Retros, etc.)
• Collaborate with other engineers to solve and bring new perspectives to complex problems
• Drive improvements in people, practices, and procedures
• Embrace new technologies and an ever-changing environment

Requirements:

Bachelor’s degree typically in Computer Science, Management Information Systems, Mathematics, Business Analytics or another STEM degree.

• 5+ years proven ability of professional data development experience
• 3+ years proven ability of developing within a data platform, with some preferably in Databricks and Snowflake
• 3+ years developing experience with either Java or Python
• Understanding and some experience with front-end development, preferably Angular
• Proficient in SQL
• Experience with PySpark/Spark/Airflow
• Cloud Native Architecture / ReUsable Capabilities (Microservices, Libraries, Engines)
• Experience with (Azure/GCP/AWS) Cloud Platforms
• Experience with Terraform
• Test Driven Development

 Preferred Skills – Experience in the following

• Experience working with Unity Catalog
• Exposure to NoSQL (Mongo, Cassandra)
• Exposure to Hadoop
• Exposure to Service Oriented Architecture
• Exposure to cloud platforms (Azure/GCP/AWS)
• Exposure to BI tooling e.g Tableau, PowerBI, Cognos
• Proficient with relational data modeling and/or data mesh principles
• Continuous Integration/Continuous Delivery