AI Platform Engineer, Applied AI

About This Role

Design and implement infrastructure for large-scale AI systems, focusing on reliability and efficiency. Build evaluation frameworks, observability tools, and diagnostic infrastructure to optimize AI system performance.

Responsibilities

• Build evaluation infrastructure to measure AI system speed, accuracy, and reliability
• Develop observability tools and dashboards that surface quality metrics and track improvements over time
• Diagnose performance gaps across retrieval, ranking, prompting, and other AI workflows
• Prototype and validate improvements to AI pipelines, including RAG pipelines, chunking strategies, retrieval methods, and re-ranking approaches
• Analyze usage data to identify opportunities for enhancement and new AI features
• Collaborate with engineers to ensure production changes improve system quality and reliability
• Stay current with cutting-edge AI research, techniques, and tools to inform platform improvements

Requirements

• Proven experience building and evaluating AI systems in production, including RAG pipelines, retrieval/search systems, and LLM-powered applications
• Strong proficiency in Python for prototyping and experimentation
• Willingness to learn and work with Ruby on Rails for integrating with production systems
• Expertise in AI infrastructure, evaluation pipelines, observability tooling, and data processing
• Deep understanding of RAG architecture, embeddings, retrieval, re-ranking, and context management
• Strong experimentation and data analysis skills, including designing A/B tests, measuring results, and iterating quickly
• Ability to work autonomously in a fast-paced, dynamic environment while collaborating effectively with a team
• Strong alignment with company values and CEFR Level C2 / ILR Level 5 proficiency in English

Qualifications

• Proven experience building and evaluating AI systems in production

Nice to Have

• Experience with AI evaluation frameworks such as Braintrust or LangSmith

Skills

* Required skills

Benefits