Lead Software Engineer, Data Pipeline

Key Skills

Apache Airflow, Apache Kafka, AWS, Data pipelines, DynamoDB, Kubernetes-K8s, Leadership, MySQL, Python Programming, RabbitMQ, Software Development, Spark Core

Job Description

At Klaviyo, we value the unique backgrounds, experiences and perspectives each Klaviyo (we call ourselves Klaviyos) brings to our workplace each and every day. We believe everyone deserves a fair shot at success and appreciate the experiences each person brings beyond the traditional job requirements. If you’re a close but not exact match with the description, we hope you’ll still consider applying. Want to learn more about life at Klaviyo?

At Klaviyo, we love tackling tough engineering problems and look for engineers who specialize in certain areas but are passionate about building, owning & scaling features end to end from scratch and breaking through any obstacle or technical challenge in their way. We push each other to move out of our comfort zone, learn new technologies and work hard to ensure each day is better than the last.

Team Overview:

Klaviyo operates a real-time data analytics platform coded primarily in Python that is built for massive scale and hosted on Amazon Web Services (AWS). The Event Pipeline team ingests billions of interactions per day to power core, Klaviyo functionality and generate data-driven insights. These interactions are ingested from disparate sources and efficiently normalized, filtered, validated and enriched prior to persistence.

As a Lead Engineer on the Events Pipeline Team, you will have ownership over defining the evolutionary, technical vision for our data ingestion pipeline. You will be responsible for the design, implementation, and tuning of highly visible mission critical stream processing systems. You will be constantly challenged to improve these systems from every angle, such as monitoring, failure recovery, data quality, tooling, and much more.

Team Tech Stack:

• Python
• Spark
• Airflow
• RabbitMQ, Kafka
• DynamoDB
• MySQL
• Kubernetes
• AWS

How You'll Make a Difference

• Responsible for coaching engineers, managing/reviewing technical documentation and articulating a phased approach to achieving the team's overall technical vision.
• Wholly owning features that 50,000+ customers rely on daily to reach 4B+ consumers to drive their business forward.
• Working on crafting our highly scalable real-time data pipeline that processes billions of events weekly across data sets growing to petabyte scale.
• Working on a deep analytics system that provides insight into hundreds of terabytes of data.
• Be responsible for the mission, technical direction and operations of projects that have immediate impact.
• Helping architect and design robust and high performance data processing and storage systems leveraging new databases to serve different internal and external use cases. Contribute to open source data processing technologies.
• Contributing to the company as a subject matter expert in multiple areas, constantly pushing yourself to be a better engineer and to level up all of your peers within your team and within Klaviyo.

Who You Are

• 10+ years of experience in software development, built and delivered at least one complex data processing system
• 2 years of experience in a people management role or team leadership role, coordinating the delivery of nontrivial projects.
• 5 years of experience testing and launching software products 
• Experience working with reliable, fault-tolerant, and highly performant distributed systems.
• Experience with high throughput orchestration tools like temporal or cadence.