ML Pipeline Architect

Key Skills

Apache Airflow, Azure, Data pipelines, Data science techniques, Design, Google Cloud Platform (GCP), Infrastructure as code, Kubeflow, Machine learning techniques, MLflow, Operations, Product Management, Python Programming

Job Description

We’re on a mission to democratize AI by building the definitive AI data development platform. The AI landscape has gone through incredible change between 2016, when Snorkel started as a research project in the Stanford AI Lab, to the generative AI breakthroughs of today. But one thing has remained constant: the data you use to build AI is the key to achieving differentiation, high performance, and production-ready systems. We work with some of the world’s largest organizations to empower scientists, engineers, financial experts, product creators, journalists, and more to build custom AI with their data faster than ever before. Excited to help us redefine how AI is built? Apply to be the newest Snorkeler!

As a ML Pipeline Architect you’ll lead the successful design and implementation of Snorkel Flow and associated model artifacts for our enterprise customers on premises or hosted cloud environments. This is a new role to Snorkel and will be its founding member responsible for shaping the approach we apply to future customer engagements. You’ll work closely with our internal engineering teams improving the design, architecture and implementation of Snorkel Flow, data integrations and model deployments, as well as represent our customer facing teams as Snorkel’s ML pipeline expert. This is a phenomenal opportunity to join our customer success team and drive its strategy for the future.

Main Responsibilities

• Design, prototype, and refine scalable infrastructure for operating Snorkel's machine learning pipeline at scale within customer on-premises and cloud environments
• Bridge the gap between data science and production, ensuring Snorkel Flow ML pipelines are deployed efficiently, reliably, and securely
• Work closely with our Product Management, Product Engineering, and Support teams to resolve issues that occur during the implementation
• Guide the team in adopting the best MLOps practices, ensuring a consistent and production-ready approach across all ML developments
• Assist in troubleshooting and resolving ML operations related issues
• Submit product feature requests to drive the platform forward
• Contribute to the Snorkel user documentation, including but not limited to design, testing, delivery and training
• Work closely with Snorkel’s field engineering teams and customers

Minimum Qualifications

• S. degree in Computer Science, Engineering, or comparable degree/experience
• 5+ years experience in a Platform Engineering, ML Operations or similar role
• Previous experience with cloud infrastructure providers such as Amazon Web Services, Microsoft Azure, or Google Cloud Platform
• Hands-on experience with common container and container orchestration solutions, e.g., Docker and Kubernetes
• Solid experience in developing data pipelines, deploying machine learning models as well as building model tracking, monitoring and online learning tools
• You are experienced with infrastructure-as-code tools such as terraform
• Experience with ML operation frameworks like Kubeflow, MLflow, and Airflow
• You’re deeply technical with a scrappy mindset

Preferred Qualifications

• Experience operating Kubernetes orchestration tools in a production setting
• Proficiency in Python, especially when setting up automation and system monitoring
• Exposure to authentication protocols like LDAP, SAML and OAuth2
• Experience implementing software products or solutions to large enterprise companies
• Comfortable working in a fast-paced, small-team environment where you’ll wear many hats

LI-JD1

Be Your Best At Snorkel

Snorkel AI is on a mission to make machine learning practical for everyone, and it starts with building a team that welcomes, represents and gives opportunity to all. We work at the frontier of AI and software engineering, and believe that underrepresented communities need to play a part in shaping the future of these fields. At Snorkel AI, we actively work to create an environment that values end-to-end ownership, diverse forms of impact, and opportunities for personal growth.

Snorkelers are supported by an amazing team and an amazing set of benefits. We offer comprehensive medical, dental, and vision plans for Snorkelers and their families, plus a yearly wellness stipend. Our 401k program lets Snorkelers plan for their future and our parental leave program lets new parents take up to 20 weeks of paid time off. Learn more about these benefits and more — like our workstation setup allowance — on our Careers page.

Snorkel AI is proud to be an Equal Employment Opportunity employer and is committed to building a team that represents a variety of backgrounds, perspectives, and skills. Snorkel AI embraces diversity and provides equal employment opportunities to all employees and applicants for employment. Snorkel AI prohibits discrimination and harassment of any type on the basis of race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state, or local law. All employment is decided on the basis of qualifications, performance, merit, and business need.