Postdoctoral researcher on LLM auditing

Key Skills

Aartificial intelligence, Algorithms, Data science techniques, Large Language Models - LLMs, Machine learning techniques, Natural Language Processing (NLP), Python Programming, SPARK Programming, SQL

Job Description

The unprecedented ability of large language models (LLMs) to interpret text data with human-like reasoning is poised to transform many fields. Nevertheless, for LLMs to be safe and effective for use in high-risk domains like healthcare, it is crucial to understand biases embedded in this technology, as it has been shown to vary in performance across subgroups and even discriminate against minorities. This project aims to study and develop red-teaming solutions to audit LLMs and to understand the limits of current approaches. Methodologies developed in this project will be tested on real-world clinical data, including unstructured notes. This project is a supplement of our existing PCORI project “Diagnostic Tools for Quality Improvement of Machine Learning-Based Clinical Decision Support Systems” (see project description here).

We are seeking a postdoctoral researcher to join our lab. The primary responsibilities are:

• Rigorously analyze and evaluate existing red-teaming algorithms for LLMs
• Develop new statistical methods/frameworks for comprehensive red-teaming of LLMs
• Develop an explanation framework and statistical inference procedures to understand systematic limitations of LLMs
• Write, edit, and publish research manuscripts in collaboration with the team

Our team is highly collaborative and includes members with wide-ranging expertise:

• Jean Feng: PI of the lab
• Julian Hong: Assistant Professor and Medical Director of Radiation Oncology Informatics in the Department of Radiation Oncology. Led one of the first randomized controlled studies of clinical machine learning. Research interests include the development and implementation of computational methods for providing personalized cancer care for patients, natural language processing of clinical notes, and evaluation of AI-based tools.
• Fan Xia: Assistant Professor in the Department of Epidemiology and Biostatistics at UCSF. Research interests include causal inference, clinical trial design, and machine learning.
• Alexej Gossmann: Staff Fellow and mathematical statistician in the Division of Imaging, Diagnostics, and Software Reliability (CDRH/OSEL/DIDSR) at the FDA. Research interests include performance evaluation of AI/ML-enabled medical devices and software in medicine.
• And many others, including Berkman Sahiner, Gene Pennello, Adarsh Subbaswamy, Nicholas Petrick, and Romain Pirracchio!

The position:

We are looking to hire a postdoctoral researcher to join the team. The position (100% funded) will be for two years. Salary and benefits are competitive.

Qualifications:

The post-doctoral researcher position requires at least a PhD degree in data science, (bio)statistics, computer science, or another relevant field. We are looking for someone who:

• has experience in training and testing ML algorithms for large datasets
• has experience in natural language processing and working with LLMs
• has experience in methodological development and can perform independent research, with a strong and relevant publication record
• has strong software engineering background (e.g. python, torch, huggingface, git-based workflows, high-performance computing, SQL, spark)
• is able to work collaboratively with a team

Applying

If you are interested, please submit the following materials to jean.feng@ucsf.edu:

• A cover letter
• A CV summarizing your education and work experience so far
• The names and email addresses of three references
• A code sample
• One representative publication

Screening of applicants will begin immediately and will continue as needed throughout the recruitment period.