Senior GenAI Data Scientist, Amazon Bedrock Service GTM - Model Evaluations and Choice

Key Skills

Aartificial intelligence, AWS, Data science techniques, Large Language Models - LLMs, Natural Language Processing (NLP), Python Programming

Job Description

You must have deep understanding of Generative AI models, including their strengths, limitations, and potential risks. You should have expertise in Responsible AI practices, such as bias mitigation, fairness evaluation, and ethical AI principles. In addition you should have hands on experience with AI security best practices, including vulnerability assessments, red teaming, and fine grained data access controls.

Candidates must have great communication skills and be very technical, with the ability to impress Amazon Web Services customers at any level, from executive to developer. Previous experience with Amazon Web Services is desired but not required, provided you have experience building large scale solutions. You will get the opportunity to work directly with senior ML engineers and Data Scientists at customers, partners and Amazon Web Services service teams, influencing their roadmaps and driving innovation.

Travel up to 40% may be possible.

Key job responsibilities

• Conduct in-depth evaluations of foundation models available on Amazon Bedrock.
• Assess model performance and latency for various industry-specific use cases and develop related technical assets.
• Act as a trusted advisor to our customers, helping them navigate the choice of LLMs, benchmarking methodologies to compare different models capabilities and efficiencies.
• Collaborate with sales and marketing teams to develop and execute go-to-market strategies and sales motions based on model performance insights.
• Assist Amazon Bedrock customers in selecting the most appropriate models for their specific needs and use cases.
• Demonstrate Model Choice, Evaluations, and Performance Benchmarking: Develop technical assets and content to educate customers on model choice, model evaluations, and performance benchmarking for various use cases.
• Collaborate with GenAI Product/Engineering and Customer-Facing Builder Teams: Work closely with the Amazon Bedrock product and engineering teams and customer-facing builders (Solution Architects and Technical Field Community members) to launch new services, support beta customers, and develop technical assets.
• Thought Leadership and External Representation: Serve as a thought leader in the Generative AI space, representing AWS at industry events and conferences, such as AWS re:Invent.
• Develop technical content, workshops, and thought leadership to enable the broader technical community, including Solution Architects, Data Scientists, and Technical Field Community members.

BASIC QUALIFICATIONS

• 5+ years of Data Scientist or Machine Learning Solutions Architect experience preferably with a focus on AI/ML model evaluations, benchmarking, and NLP
• 5+ years of experience with Python to analyze datasets, train , evaluate, deploy, fine-tune, and optimize models.
• Experience with open source frameworks for building applications powered by language models like LangChain, LlamaIndex, ragas.ai, and other model evaluations frameworks etc.
• Design, develop, and optimize high-quality prompts and templates that guide the behavior and responses of LLM.
• Experience with design, deployment, and evaluation of LLM-powered agents and tools and orchestration approaches.
• 3+ years of machine learning/statistical modeling data analysis tools and techniques, and parameters that affect their performance experience

PREFERRED QUALIFICATIONS

• Experience with open source frameworks for building applications powered by language models like LangChain, LlamaIndex, and Whylabs etc.
• Design, develop, and optimize high-quality prompts and templates that guide the behavior and responses of LLM.
• Experience with design, deployment, and evaluation of LLM-powered agents and tools and orchestration approaches
• Customer facing skills to represent AWS well within the customer’s environment and drive discussions with senior personnel regarding trade-offs, best practices, and risk mitigation.
• Master's degree in a quantitative field such as statistics, mathematics, data science, business analytics, economics, finance, engineering, or computer science
• Should be able to interact with Chief Data Science Officers, as well as the people within their organizations.
• Demonstrated ability to think strategically about business, product, and technical challenges in an enterprise environment.
• Track record of thought leadership and innovation around Machine Learning.
• Familiarity with AWS services and the cloud computing landscape is preferred.

Amazon is committed to a diverse and inclusive workplace. Amazon is an equal opportunity employer and does not discriminate on the basis of race, national origin, gender, gender identity, sexual orientation, protected veteran status, disability, age, or other legally protected status. For individuals with disabilities who would like to request an accommodation, please visit https://www.amazon.jobs/en/disability/us.

Our compensation reflects the cost of labor across several US geographic markets. The base pay for this position ranges from $143,300/year in our lowest geographic market up to $247,600/year in our highest geographic market. Pay is based on a number of factors including market location and may vary depending on job-related knowledge, skills, and experience. Amazon is a total compensation company. Dependent on the position offered, equity, sign-on payments, and other forms of compensation may be provided as part of a total compensation package, in addition to a full range of medical, financial, and/or other benefits. For more information, please visit https://www.aboutamazon.com/workplace/employee-benefits. This position will remain posted until filled. Applicants should apply via our internal or external career site.