Sr Software Engineer - LLM - Search Relevance

Key Skills

Large Language Models - LLMs, Machine learning techniques, Natural Language Processing (NLP)

Job Description

While candidates in the listed locations are encouraged for this role, candidates in other locations will be considered.

At Databricks, we are passionate about enabling data teams to solve the world's toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world's best data and AI infrastructure platform so our customers can use deep data insights to improve their business. Founded by engineers — and customer-obsessed — we leap at every opportunity to solve technical challenges, from designing next-gen UI/UX for interfacing with data to scaling our services and infrastructure across millions of virtual machines. And we're only getting started.

As an engineer applying large language models (LLM) to search relevance at Databricks, you’ll work closely with the teams behind Databricks’ Dolly LLM and also product teams to improve search and discovery quality and experience. Our work is necessarily cross-functional, and successful individuals on our team embody an unusually high degree of empathy and ownership, demonstrating an intuitive ability to understand how individual technical decisions shape Databricks’ business strategy.

Databricks has a long-standing commitment to research and open source, and though our teams are focused first and foremost to business impact, we work hard to foster a creative, intellectually stimulating environment featuring visiting speakers, academic partnerships, and industrial collaborations.

The impact you'll have:

• Drive the development and deployment of LLM based search and discovery relevance models and systems integrated with Databricks' products and services.
• Design and implement automated ML and NLP pipelines for data preprocessing, query understanding and rewrite, ranking and retrieval, and model evaluation, enabling rapid experimentation and iteration.
• Collaborate with product managers and cross-functional teams to drive technology-first initiatives that enable novel business strategies and product roadmaps for the search and discovery experience.
• Contribute to the broader AI community by publishing research, presenting at conferences, and actively participating in open-source projects, enhancing Databricks' reputation as an industry leader.

What we look for:

• BS+ (M.S. or PhD preferred) in Computer Science, or a related field.
• 5+ years experience developing search relevance systems at scale in production or in high-impact research environments.
• Experience applying LLM to search relevance
• Experience in one or more of the following:
• Query understanding
• NLP
• Text mining
• Recommendations
• Personalization
• Discovery
• Conversational AI
• Strong understanding of computer science fundamentals.
• Contributions to well-used open-source projects..

Benefits

• Comprehensive health coverage including medical, dental, and vision
• 401(k) Plan
• Equity awards
• Flexible time off
• Paid parental leave
• Family Planning
• Gym reimbursement
• Annual personal development fund
• Employee Assistance Program (EAP)

About Databricks

Databricks is the data and AI company. More than 9,000 organizations worldwide — including Comcast, Condé Nast, and over 50% of the Fortune 500 — rely on the Databricks Lakehouse Platform to unify their data, analytics and AI. Databricks is headquartered in San Francisco, with offices around the globe. Founded by the original creators of Apache Spark™, Delta Lake and MLflow, Databricks is on a mission to help data teams solve the world’s toughest problems. To learn more, follow Databricks on Twitter, LinkedIn and Facebook.