Healthcare AI Evaluator

About This Role

Mercor is seeking a Healthcare AI Evaluator to write prompts, evaluate large language model (LLM) responses for accuracy and clarity, and conduct fact-checking of medical claims.

Responsibilities

• Write and refine prompts to guide model behavior in healthcare scenarios
• Evaluate LLM-generated responses to healthcare queries for accuracy, reasoning, clarity, and completeness
• Conduct fact-checking of medical claims using trusted public sources and authoritative references
• Annotate model responses by identifying strengths, areas of improvement, and factual inaccuracies
• Assess tone, completeness, and appropriateness of responses for real-world healthcare use
• Apply consistent evaluation standards by following clear taxonomies, benchmarks, and detailed evaluation guidelines

Requirements

• 5+ years of real-world professional experience in Healthcare, supported by an associated expert degree (e.g., MD, DO, RN, NP, PA, PharmD, MPH, or equivalent)
• Experience in one or more of the following sub-domains: General Clinical Care, Specialty Medicine or Surgery, Diagnostics, Imaging & Laboratory Medicine, Public Health, Healthcare Systems & Administration
• Significant experience using large language models (LLMs)
• Excellent writing communication skills for complex medical topics
• Strong attention to detail and comfort in evaluating clinical reasoning and medical explanations

Qualifications

• Associated expert degree (e.g., MD, DO, RN, NP, PA, PharmD, MPH, or equivalent)
• 5+ years of real-world professional experience in Healthcare

Nice to Have

• Prior experience with RLHF, model evaluation, or data annotation work
• Experience writing or editing high-quality medical or healthcare-related content
• Experience in clinical documentation, charting, or patient communication
• Familiarity with evaluation rubrics, benchmarks, or quality scoring systems

Skills

* Required skills