Head of Evaluation

Key Skills

Aartificial intelligence, Effective communication skills, Large Language Models - LLMs, Teamwork

Job Description

Harvey is a secure AI platform for professionals in law, tax, and finance that augments productivity and automates complex workflows. Harvey uses algorithms with reasoning-adept LLMs that have been customized by our expert team of lawyers, engineers and research scientists. We’ve found product market fit and are scaling our team very quickly. Some reasons to join Harvey are:

• Exceptional product market fit: We have partnered with the largest law firms and professional service providers in the world like A&O, PwC, and many others.

• Strategic investors: Raised over $100 million from strategic investors including Sequoia, Kleiner Perkins, and the OpenAI Startup Fund.

• World-class team: Harvey is hiring the best technical and non-technical talent from DeepMind, Google Brain, Stripe, FAIR, Tesla Autopilot, Superhuman, Glean, etc.

• Partnerships: Our engineers and researchers work directly with OpenAI to build the future of generative AI and redefine professional services.

• Value: Top of market cash and equity compensation.

Role

We are looking for a technical lead who can own the development of our evaluation platform. In this role, you will:

• Build a team of 10-20 researchers and engineers with experience evaluating LLMs and large-scale AI systems.

• Lead research and development of novel model-based evaluation methods and language model programs for evaluating complex tasks in legal and professional services.

• Design and implement a red-teaming pipeline for our custom models and collaborate with other research teams to fine-tune models from human feedback.

• Train reward models that accurately reflect the preferences of top-tier domain experts.

• Experiment with synthetic data generation and LLM-based data augmentation to complement human-generated eval benchmarks.

Impact:

• Lead research and development of Harvey’s evaluation platform.

• Contribute to a product that transforms the nature of professional services.

• Help define what it means for LLMs to effectively perform complex knowledge work tasks.

• Work directly with our founders, research, and product teams, as well as foundation model providers like OpenAI.

• Tackle unsolved research and engineering problems, including the hardest in the world relevant to LLMs in production.

Qualifications

• 5+ years experience leading highly-technical teams composed of both researchers and engineers.

• Experience evaluating large-scale AI systems in high-stakes settings.

• Technical: can serve as a tech lead and contribute substantially to our codebase as necessary.

• Ability to communicate complex technical outcomes to diverse stakeholders.

• Strong conviction in setting technical direction.

Compensation

The expected range of compensation for this role is between $300,000 and $385,000. Additionally, this role is eligible to participate in our equity plan. The successful candidate’s starting salary will be determined based on non-discriminatory factors such as skills, experience, and geographic location.