Lead Data Engineer

Key Skills

Amazon Simple Storage Service (S3), Apache Airflow, Apache Kafka, AWS, Data Modeling, Data Warehousing, Java Programming, Jupyter Notebook, Python Programming, Scala Programming, Scrum Agile Methodology, Snowflake, Spark Core

Job Description

About the Role

As the Lead Data Engineer, you will lead a team to architect, build and operate Brivo’s big data platform and data warehouse that are real-time, stable and scalable to support data analytics, reporting, data visualization and machine learning. In this role you will help to build the Data Engineering team, mentor and motivate team members on cutting edge big data technology and best practices. 

You will also be expected to:

• Distill big data strategy into tactical projects by collaborating with stakeholders, architecture team and product team to support data analytics, reporting, data visualization, and machine learning.
• Define quarterly project requirements by tracking milestones, communicating blockers and ensuring progress.
• Lead the team to build the new real-time big data platforms with open source technologies such as Kafka, Spark, Flink, and Presto etc.
• Lead the team to build the new data services with open source technologies such as Superset etc.
• Contribute up to 50% of your time coding in at least one of Python, Java, Scala, Node.js, or other mainstream programming languages.
• Maintain operational status of all data sources and pipelines including maintaining on call rotation.
• Display deep understanding of data warehousing concepts, data manipulation strategies, data modeling, data pipelines, data services, and workflows.
• Experience with leading projects of large data volume (approximately 10TB of data) stored in heterogeneous data sources.
• Refactor the existing data model into an easy-to-maintain data solution across the organization.
• Collaborate with QC team to assure product quality.
• Collaborate with the release team and operational team to automate build and deployment from DEV to PROD.
• Lead team projects and initiatives from inception to rollout managing cross-team dependencies while keeping overall customer SLA top of mind.
• Mentor team members to improve processes and promote the best practices.

About You

• 5+ years of work experience with a combination of data modeling, ETL, database operations, and data services..
• 5+ years of work experience with coding in one of mainstream programming languages: Python, Java, Scala, Node.js etc.
• 3+ years of experience working on data warehouses such as Snowflake.
• 3+ years of experience operating production workloads on PostgreSQL (preferred), MySQL databases.
• 3+ years of experience with open source technologies (Spark, Flink, Kafka, Presto, Hive, etc.)
• 2+ years of experience in architecting and building scalable data platforms processing data on a terabyte or petabyte scale.
• 3+ years of experience managing ETL pipelines with tools such as Fivetran, Airflow, Talend, etc.
• 2+ years of experience with AWS EMR, Athena, Lambda, Kafka, S3, EC2, Kubernetes.
• Experience leading teams of 3+ engineers.
• Familiarity with Agile principles including but not limited to Scrum meetings and mechanics.
• Good understanding of modern data structures and business intelligence reporting tools and track record of applying those to build a system.
• Intellectually curious about emerging technologies with the ability to turn that into reasonable working solutions.
• Ability to be efficient and to prioritize and deliver on time.
• Excellent communication skills and dynamic team player.

Preferred Skills

• Experience in real-time analytics applications.
• Experience in both batch and stream processing technologies, Python, Java or Scala.
• Machine learning experience with Spark, jupyter Notebook, or similar.