Engineering Manager, Big Data Compute Spark
The Big Data Compute Spark (BDCS) team provides high-performance, scalable, distributed data processing at rest with the customized-for-Netflix fork of Apache Spark on Hadoop to meet Netflix’s large scale data analytics and machine learning needs.
This team of 7 people (and growing) is central to batch data processing and analytics in the Data Platform at Netflix. It provides support for Spark to ETL data into the Exabytes-scale data warehouse and a variety of ways (for casual and power users) to access that data using Spark.
The team is working on integrating Spark accelerators to further drive performance, evaluating containerization platforms to run Spark, building better batch and real time observability into Spark workflows and no/low touch ways to use Spark.
This is a dream team of highly passionate and intelligent engineers who work well together. The team works on solving challenging problems at scale that have a huge impact on the Data Platform. The team has PMC members and committers who shape and contribute to open-source projects.
Check out some of our talks on Spark.
What you will do:
What we are looking for:
A few more things about us:
As a team, we come from many different backgrounds and countries, and our fields of education range from the humanities to engineering to computer science and we strive to give people the opportunity to wear different hats, should they choose to. We strongly believe this diversity and agility has helped us build an inclusive and empathetic environment, and we are looking forward to adding your perspective to the mix!
Our culture is unique, and we tend to live by our values, so it’s worth learning more about Netflix here.
Blogs and Talks:
Maestro Blog
Incremental Processing Blog
Auto Remediation Blog
Evolving from Rule Based Classifier to Machine Learning Powered Auto Remediation
Maestro presentation at AWS Re:Invent
Robust Foundation for Data Pipelines at Scale
Building a Scalable Workflow Scheduler with Netflix Conductor
DataWorks summit: Iceberg table format
Big Data Storage: SubSurface
Trino enjoying the view (Common Views)
Make data fluid with Apache Druid
Iceberg Right Ahead!
Compensation and Benefits
Our compensation structure consists solely of an annual salary; we do not have bonuses. You
choose each year how much of your compensation you want in salary versus stock options. To determine your personal top-of-market compensation, we rely on market indicators and consider your specific job family, background, skills, and experience to determine your compensation in the market range. The range for this role is $190,000 - $920,000.
Netflix provides comprehensive benefits including Health Plans, Mental Health support, a 401(k) Retirement Plan with employer match, Stock Option Program, Disability Programs, Health Savings and Flexible Spending Accounts, Family-forming benefits, and Life and Serious Injury Benefits. We also offer paid leave of absence programs. Full-time hourly employees accrue 35 days annually for paid time off to be used for vacation, holidays, and sick paid time off. Full-time salaried employees are immediately entitled to flexible time off. See more detail about our Benefits here.
At Netflix, our mission is to entertain the world. With 200+ million paid members in over 190 countries enjoying TV series, documentaries, and feature films across a wide variety of genres and languages, Netflix is reinventing entertainment from end to end. We are revolutionizing how shows and movies are produced, pushing technological boundaries to efficiently deliver streaming video at a massive scale over the internet.
Los Gatos, CA, USA
4-6 year
Los Angeles, CA, USA
4-6 year
Los Gatos, CA, USA
2-4 year
Los Gatos, CA, USA
4-6 year
Los Gatos, CA, USA
2-4 year
Los Gatos, CA, USA
4-6 year
Los Gatos, CA, USA
4-6 year
Los Gatos, CA, USA
4-6 year
Los Gatos, CA, USA
6-8 year
Los Gatos, CA, USA
10-12 year
Los Gatos, CA, USA
10-12 year
Los Gatos, CA, USA
6-8 year
Los Gatos, CA, USA
4-6 year
Los Gatos, CA, USA
4-6 year
Los Gatos, CA, USA
6-8 year
Los Gatos, CA, USA
12-14 year
Los Gatos, CA, USA
10-12 year
United States
4-6 year
Los Gatos, CA, USA
4-6 year
Los Gatos, CA, USA
4-6 year
Los Angeles, CA, USA
4-6 year
Los Angeles, CA, USA
10-12 year
United States
4-6 year
Los Angeles, CA, USA
8-10 year
Los Gatos, CA, USA
10-12 year
Los Gatos, CA, USA
4-6 year