Software Engineer Evaluator

About This Role

Turing is seeking a Software Engineering Evaluator to create cutting-edge datasets for training, benchmarking, and advancing large language models. The role involves curating code examples, providing precise solutions, and making corrections in various programming languages, while collaborating closely with researchers.

Responsibilities

• Create cutting-edge datasets for training, benchmarking, and advancing large language models
• Curate code examples, build solutions, and correct code in Python, JavaScript (including ReactJS), C/C++, Java, Rust, and Go
• Evaluate and refine AI-generated code for efficiency, scalability, and reliability
• Collaborate with cross-functional teams to enhance AI-driven coding solutions against industry performance benchmarks
• Build agents that can verify the quality of code and identify error patterns
• Hypothesize on steps in the software engineering cycle and evaluate model capabilities on them
• Design verification mechanisms that can automatically verify solutions to software engineering tasks

Requirements

• Several years of software engineering experience (+5 years)
• 2+ years of continuous full-time experience at a top-tier product company (e.g., Google, Stripe, Amazon, Apple, Meta, Netflix, Microsoft, Datadog, Dropbox, Shopify, PayPal, IBM Research)
• Strong expertise in building full-stack applications and deploying scalable, production-grade software using modern languages and tools
• Deep understanding of software architecture, design, development, debugging, and code quality/review assessment
• Excellent oral and written communication skills and structured evaluation rationales
• Ability to curate code examples and provide precise solutions in Python, JavaScript (including ReactJS), C/C++, Java, Rust, and Go

Qualifications

• 5+ years of software engineering experience, including 2+ years of continuous full-time experience at a top-tier product company (e.g., Google, Stripe, Amazon, Apple, Meta, Netflix, Microsoft, Datadog, Dropbox, Shopify, PayPal, IBM Research)

Skills

* Required skills