Job Title

About Turing:Turing is one of the world’s fastest-growing AI companies, accelerating the advancement and deployment of powerful AI systems. Turing helps customers in two ways: working with the world’s leading AI labs to advance frontier model capabilities in thinking, reasoning, coding, agentic behavior, multimodality, multilinguality, STEM, and frontier knowledge; and leveraging that work to build real-world AI systems that solve mission-critical priorities for companies.Role Overview:We are looking for strong, detail-oriented software practitioners to help evaluate and improve datasets for agentic coding models.This role involves working with realistic coding tasks in an agentic coding harness, reviewing model trajectories, verifying solutions, and producing high-quality annotations.Depending on the assignment, the work may include:Online evaluations: Manually interacting with blinded models on predefined tasks, then ranking and grading resulting trajectoriesOffline evaluations: Designing realistic coding tasks, calibrating them through user simulation, writing task-specific rubrics, and grading generated trajectoriesThis is not a basic annotation role. Candidates are expected to read and debug code, validate behavior, follow detailed process rules, and make consistent judgment calls across model runs.We are specifically looking for candidates with enough engineering maturity to independently work on realistic software tasks, not just toy problems or shallow code-review exercises.What does day-to-day look like:Execute realistic coding tasks within the assigned agentic coding harness while maintaining model blindness and session independenceFollow task instructions, milestones, planned interactions, and evaluation guardrails consistently across runsVerify model outputs by reading code, running commands, checking logs, and inspecting generated artifactsPerform targeted validation of outputs using tests, scripts, and manual checksWrite clear, specific, evidence-based rationales for trajectory rankings and assessmentsDesign multi-step, realistic coding tasks (offline work), including user intent and milestone structureCreate and refine task-specific rubrics and binary evaluation criteriaReview completed work for quality, completeness, consistency, and schema complianceIdentify and escalate broken environments, unclear instructions, or process gaps with clear supporting evidenceRequirements:Software Engineering Fluency (Mandatory)5+ years of experience in software engineering, QA, developer tooling, data/ML engineering, or similar code-heavy rolesStrong hands-on experience in at least 1–2 programming languages or ecosystemsRepresentative languages include: Python, JavaScript/TypeScript, Rust, Java, C/C++, Bash/CLI environments, Haskell, Swift, SQL, or other production-relevant ecosystemsAbility to: Read and understand unfamiliar codebases Run and interpret tests, scripts, and CLI tools Debug issues and reason about edge cases or partial fixes Evaluate whether an implementation is functionally correctAdditional Preferred Qualifications (Offline / Senior Candidates):Strong Docker skills and experience building/debugging reproducible environmentsExperience working in large, complex repositories (not just small or greenfield projects)Demonstrated originality and sound engineering judgment in defining technical problemsAbility to design realistic, non-trivial tasks that go beyond tutorials, README flows, or simple bug fixesPerks of Freelancing With Turing:Work on cutting-edge AI projects with leading foundation model companiesCollaborate on high-impact work at the frontier of LLM evaluation and reasoningRemote, flexible opportunities with global teamsCompetitive compensation based on experience and project scopeOffer Details:Commitments Required: 8 hours per day with a 4-hour overlap with PST.Employment Type: Contractor position (Note: this role does not include medical/paid leave).Duration of Contract: 5 weeks; [expected start date is next week].