AI Prompt & Data Quality Specialist

About This Role

This role is for an AI Prompt & Data Quality Specialist who will lead "Human-in-the-Loop" workflows, designing AI prompts, defining "Ground Truth" for model accuracy, and creating high-quality evaluation datasets.

Responsibilities

• Draft and develop complex system prompts for various use cases (e.g., summarization, extraction, persona-based chat)
• Implement advanced strategies (Chain-of-Thought, Few-Shot) to guide the model through complex reasoning tasks
• Iterate on prompt versions to reduce latency and token usage while maintaining quality
• Review and annotate model outputs to tag them for accuracy, tone, safety, and hallucinations
• Perform "A/B ranking" on model responses to support Reinforcement Learning from Human Feedback (RLHF)
• Create "ideal" human-generated responses to serve as Ground Truth for automated testing and fine-tuning
• Maintain guidelines, developing and updating the Annotation Rubric that defines correct and safe responses

Requirements

• Exceptional command of English grammar and semantics
• Familiarity with GPT-4, Claude, or similar Large Language Models (LLMs), including their limitations (hallucinations, bias)
• Extreme attention to detail and ability to perform repetitive data review tasks without losing focus or consistency

Qualifications

• Bachelor

Nice to Have

• Prior experience with RLHF, data labeling platforms (like Scale AI, Labelbox), or QA roles
• Proficiency with Excel/Google Sheets (for data organization) or basic Python (for handling JSON data)
• Background in Networking to fact-check complex model outputs

Skills

* Required skills

Benefits