Sr/Staff Data Engineer

Key Skills

Apache Kafka, AWS, Data Engineering, Data pipelines, Data science techniques, Design, ETL frameworks, Google Cloud Platform (GCP), Grafana, Planning and organization, Python Programming, SQL, Streaming APIs

Job Description

What you’ll do:

• Independently understand all aspects of a business problem including those unrelated to their area of expertise, weigh pros and cons of different approaches and suggest ones likely to succeed

• Work with a cross-functional organization including engineering, delivering, subject-matter experts, product managers, as well as platform engineers to deliver a scalable framework.

• Map the customer data into Machinify canonical form. Identify and ingest non canonical fields and generalize the process to a minimal level of customization.

• Proactively design and adapt the canonical form to suit changing query patterns and needs.

• Ultimately own data availability and quality for the Data Science organization.

What You Bring:

• Deep experience as a hands-on Data Engineer building production data pipelines

• Experience managing the delivery of complex data

• Experience in ETL orchestration and workflow management tools with a strong preference for Apache Airflow

• Experience in Spark or other distributed computing frameworks

• SQL and Python

• Advanced SQL performance tuning

• Kubernetes and building Docker images

• AWS & GCP

• Experience working with APIs to collect or ingest data

• Manage SLA for all pipelines in allocated areas of ownership

• Streaming technologies like kafka , spark streaming etc

• ELK stack , Grafana etc

The base salary for this position will vary based on an array of factors unique to each candidate such as qualifications, years and depth of experience, skill set, certifications, etc. The base salary range for this role is $200k-250k. We are hiring for different seniorities, and our Recruiting team will let you know if you qualify for a different role/range. Salary is one component of the total compensation package, which includes meaningful equity, excellent healthcare, flexible time off, and other benefits and perks.