Software Reliability Engineer, Dojo

Key Skills

Ansible, API, C Programming, C++, Continuous Integration & Continuous Delivery - CI/CD, Docker, GoLang, Kubernetes-K8s, Linux Operating system, Networking technologies, Puppet, Python Programming, Rust, Terraform, TypeScript

Job Description

What to Expect

Are you ready to shape the future of AI supercomputing? Tesla is looking for a seasoned Site Reliability Engineer (SRE) with a software engineering background to join the team behind Dojo, our cutting-edge supercomputer. As a member of our highly motivated and multi-disciplinary team, you will be responsible for supporting engineers and users of our existing infrastructure while being a keystone in architecting future system reliability and control-plane components of our custom supercomputer.

Dive into all facets of the component lifecycle, from designing, building, deploying, and supporting systems that power our on-premises bare-metal server and network switch provisioning, configuration management, workload scheduling, and even the interfaces and APIs the end-users interact with. With a deep understanding of modern cloud platforms and their ground-up construction, you will play a critical role in supporting and maintaining internal and external customers' workloads on our custom hardware. Embrace the challenge of working alongside a cross-functional team that pushes the boundaries of hardware, software, and mechanical capabilities, providing scalable and timely solutions to meet our ongoing deployment efforts.

What You’ll Do

• Bachelor's degree in computer science, or related field, with 3 years of additional equivalent experience or evidence of exceptional ability related to the position.
• Demonstrable knowledge in L2/L3 ethernet or HPC fabric networking. From physical underlay to software-defined multi-tenant virtual overlay networks.
• Proficiency in practical software engineering and distributed systems architecture. Build new and improve existing APIs and services that power our control plane.
• Experience with change management, advanced hardware-in-the-loop CI/CD practices, and configuration-as-code systems such as Ansible, Puppet, Salt, or Terraform.
• Excellent time management and communication skills with the ability to both drive end-to-end solutions both independently and collaboratively.
• Self-motivated, technically well-rounded, and intellectually curious: willing to take on new challenges, evaluate new and emergent technologies, and dive into unfamiliar territory.

What You’ll Bring

• Highly familiar with distributed Linux operating system environments (Ubuntu/Debian, CentOS/RHEL, Alpine, etc.)
• Supporting and building bare-metal/cloud server provisioning components (DHCP, DNS, TFTP, iPXE, IPMI/OpenBMC/Redfish, BIOS Configuration.)
• Deploying and maintaining hardware and software monitoring and observability (log aggregation, metrics, distributed tracing, and alerting.)
• Administering and building service discovery and workload schedulers such as Kubernetes, Slurm, LSF, etc.
• On-call rotation management, alert triaging, and root-cause analysis. Following through with service events with post-mortems and recommended action items.
• Containerizing new and existing software components with Docker and deploying them at scale with Kubernetes or similar technologies.
• Contributing to and deploying production-ready software components in Rust, C/C++, Go, Python, and/or Typescript.
• Working with or building out software-defined networking in on-prem or cloud environments.
• Nice to have: Previous experience in all facets of datacenter management from rack planning, power management, cooling, and structured cabling.
• Nice to have: Experience with hardware accelerators (FPGA / ASIC based) or embedded systems firmware development.
• Nice to have: Experience with machine-learning training or inference workloads.

Compensation and Benefits

Benefits

Along with competitive pay, as a full-time Tesla employee, you are eligible for the following benefits at day 1 of hire:

• Aetna PPO and HSA plans > 2 medical plan options with $0 payroll deduction
• Family-building, fertility, adoption and surrogacy benefits
• Dental (including orthodontic coverage) and vision plans, both have options with a $0 paycheck contribution
• Company Paid (Health Savings Account) HSA Contribution when enrolled in the High Deductible Aetna medical plan with HSA
• Healthcare and Dependent Care Flexible Spending Accounts (FSA)
• LGBTQ+ care concierge services
• 401(k) with employer match, Employee Stock Purchase Plans, and other financial benefits
• Company paid Basic Life, AD&D, short-term and long-term disability insurance
• Employee Assistance Program
• Sick and Vacation time (Flex time for salary positions), and Paid Holidays
• Back-up childcare and parenting support resources
• Voluntary benefits to include: critical illness, hospital indemnity, accident insurance, theft & legal services, and pet insurance
• Weight Loss and Tobacco Cessation Programs
• Tesla Babies program
• Commuter benefits
• Employee discounts and perks program

Expected Compensation

$104,000 - $348,000/annual salary + cash and stock awards + benefits

Pay offered may vary depending on multiple individualized factors, including market location, job-related knowledge, skills, and experience. The total compensation package for this position may also include other elements dependent on the position offered. Details of participation in these benefit plans will be provided if an employee receives an offer of employment.