Staff Software Engineer, ML Infrastructure

Key Skills

Argo, CircleCI, Continuous Integration & Continuous Delivery - CI/CD, Data science techniques, DataDog, Design, DevOps, Docker, GPUs, GraphQL, Helm, Infrastructure as code, Java Programming, Jenkins, Kubernetes-K8s, Machine learning techniques, MLOps tools, Pandas, Python Programming, PyTorch, Site Reliability Engineering (SRE), Splunk, Spring Boot, TensorFlow

Job Description

Who we are

We’re looking for a self-motivated, highly driven Staff Software Engineer to join our Machine Learning Operations (MLOps) team. As a team, we enable Attentive’s Machine Learning (ML) practice to directly impact Attentive’s AI product suite through the tools to train, inference, and deploy ML models with higher velocity and performance, while maintaining reliability. We build and maintain a foundational ML platform spanning the full ML lifecycle for consumption by ML engineers and data scientists. This is an exciting opportunity to join a rapidly growing MLOps team at the ground floor with the ability to drive and influence the architectural roadmap enabling the entire ML organization at Attentive.

This team and role is responsible for building and operating the ML compute and orchestration architecture here at Attentive, which currently consists of a hosted notebook solution with Spark on AWS EMR, a multi-cluster CPU and GPU-enabled training and inference orchestrator leveraging Metaflow on Argo Workflows, and an ML feature store. We are excited to bring on more engineers to continue expanding this stack.

Why Attentive needs you

• Define and lead cross-functional ML infrastructure and ML platform projects
• Demonstrate the ability to analyze, troubleshoot, coordinate, and resolve complex ML infrastructure issues
• Orchestrate Kubernetes and ML training / inference infrastructure exposed as an ML platform
• Expose and manage environments, interfaces, and workflows to enable ML engineers to develop, build, and test ML models and services
• Manage and expand our feature store implementation that allows ML teams to self-service data labeling, feature engineering, and batch inferencing
• Close the latency gap on model inference to online, real-time model serving
• Develop automation workflows to improve team efficiency and ML stability
• Analyze and improve efficiency, scalability, and stability of various system resources
• Partner with other teams and business stakeholders to deliver business initiatives
• Help onboard new team members, provide mentorship and enable successful ramp up on your team's code bases

About you

• You have been working in the areas of MLOps / ML Platform / Data Platform / Site Reliability Engineering / DevOps / Infrastructure for 8+ years, and have an understanding of best practices for DevOps applied to ML
• You have successfully led major cross-functional, cross-team ML infrastructure or ML platform projects
• Your passion is infrastructure and exposing platform capabilities through interfaces that enable high performance ML practices, rather than designing ML experiments (this team does not directly develop ML models)
• You have deep experience in Kubernetes applied to ML use cases such as CPU & GPU training, hosting and exposing ML tools, and managing ML endpoints as web services
• You understand the key differences between online and offline ML inferences and can voice the critical elements to be successful with each
• You have a background in software development and are passionate about bringing that experience to bear on the world of ML infrastructure
• You have experience with Infrastructure as Code using Terraform and can’t imagine a world without it
• You understand the importance of CI/CD in building high-performing teams and have worked with tools like Jenkins, CircleCI, Argo Workflows, and ArgoCD
• You are passionate about observability and worked with tools such as Splunk, Nagios, Sensu, Datadog, New Relic
• You are very familiar with containers and container orchestration and have direct experience with vanilla Docker as well as Kubernetes as both a user and as an administrator.
• Some sample projects
• Design and lead implementation of an online inference pipeline with champion/challenger model testing
• Unite existing pipelines across data, ML, and platform teams to handle low-latency, high volume real-time streaming use cases in production inference workflows
• Define golden path build and release pipelines for better reliability and Python package management
• Identify opportunities to improve scalability, resiliency, and cost efficacy of GPU training and inference workflows
• Design and lead implementation of a low-touch, automated model CI/CD pipeline

Our scale

• 8,000 brands powered by Attentive sent over 2.2 billion text messages over Cyber Week 2023 (Black Friday/Cyber Monday) representing a growth of 31% from 2022
• We sent 32 billion SMS messages in 2023, up 32% YoY. That’s an average of 87 million per day
• Our production cluster contains over 18,000 containers which serve 200+ services
• Our streaming services process over 80 billion events per month

What we use

• Our infrastructure runs primarily in Kubernetes hosted in AWS’s EKS
• Infrastructure tooling includes Istio, Datadog, Terraform, CloudFlare, and Helm
• Our backend is Java / Spring Boot microservices, built with Gradle, coupled with things like DynamoDB, Pulsar, AirFlow, Postgres, Planetscale, and Redis, hosted via AWS
• Our frontend is built with React and TypeScript, and uses best practices like GraphQL, Storybook, Radix UI, Vite, esbuild, and Playwright
• Our automation is driven by custom and open source machine learning models, lots of data and built with Python, Metaflow, HuggingFace , PyTorch, TensorFlow, and Pandas
• You'll get competitive perks and benefits, from health & wellness to equity, to help you bring your best self to work.

For US based applicants:

• The US base salary range for this full-time position is $180,000 - $270,000 annually + equity + benefits
• Our salary ranges are determined by role, level and location