Spark Platform Engineer

Key Skills

Python Programming, Scala Programming, C++, Java Programming, Machine learning techniques, Data science techniques, SQL, Apache Hadoop, MapReduce, TensorFlow, PyTorch, R Programming

Job Description

Want to build cutting-edge infrastructure around Apache Spark? We’re looking for Spark Platform infrastructure engineers who thrive at the intersection of machine learning (ML), scalable infrastructure, and massive data flows.

Our Spark platform delivers cost-efficient, on-demand access to computing resources for hundreds of developers and analysts, and we build tools on top of Spark to support data-mining and ML workflows. We’re invested in the latest technology frameworks, and are contributing back to open-source projects to help us achieve our mission. Finally, we’re simplifying data workflows by building tooling and processes to centralize data products and feature stores for analysts and ML needs.

Come work with and learn from our passionate and diverse team of engineers with years of experience spanning machine learning modeling to systems engineering. We communicate across the company, inputting data-processing needs and outputting efficient tooling and systems. As data infrastructure evolves, we continue to ride the wave of innovation by combining industry best practices and state-of-the-art tooling to bolster Yelp’s machine learning platform. See this blog post for an overview of our Spark Platform.

We’d love to have you apply, even if you don't feel you meet every single requirement in this posting. At Yelp, we’re looking for great people, not just those who simply check off all the boxes.

This opportunity is fully remote and does not require you to be located in any particular region. We welcome applicants from throughout Ontario.

Where You Come In:

• Build out the infrastructure for Yelp’s Spark platform
• Build standard tooling for on-demand cluster creation with Kubernetes with Jupyter or native Spark integration
• Create frameworks to abstract feature transformation on top of Spark
• Collaborate with other engineering teams on building centralized data & feature stores
• Connect with different internal groups for input on their ML infra and data needs
• Mentor other engineers and share the skills you’ve learned
• Gain expertise in cutting-edge machine learning infrastructure

What it Takes to Succeed:

• An interest in building tooling around Spark infrastructure in support of machine learning, data ETL, and ML feature store uses cases
• Deep understanding of the programming languages and systems you’ve worked onFamiliarity with tooling, including Jupyter, Apache Spark, Docker and Kubernetes
• Ideally 2 years industry experience in data infrastructure
• Passion for architecting large systems with elegant interfaces that can scale easily
• A team player who lives the Yelp Values and thrives in a diverse and inclusive work culture

At Yelp, we believe that diversity is an expression of all the unique characteristics that make us human: race, age, sexual orientation, gender identity, religion, disability, and education — and those are just a few. We recognize that diverse backgrounds and perspectives strengthen our teams and our product. The foundation of our diversity efforts are closely tied to our core values, which include “Playing Well With Others” and “Authenticity.”

We’re proud to be an equal opportunity employer and consider qualified applicants without regard to race, color, religion, sex, national origin, ancestry, age, genetic information, sexual orientation, gender identity, marital or family status, veteran status, medical condition or disability.