← Back to all jobs
A senior software engineer reviewing AI-generated code on a laptop
AI Training & Data North America only

Senior Software Engineer – AI Evaluation / Coding Agents

Evaluate and improve AI coding models by reviewing AI-generated code and assessing coding-agent performance.

Posted 20 Nov 2026Last checked 20 Nov 2026Open

About this work

Experienced software engineers are wanted to evaluate and improve AI coding models by reviewing AI-generated code, assessing coding-agent performance and identifying technical weaknesses. Key responsibilities include evaluating code generated by AI coding agents; identifying technical errors and assessing code quality; designing evaluation criteria and rubrics; reviewing and labelling evaluation data; building and maintaining evaluation pipelines; and communicating findings to research and engineering teams. The 3-month contract offers 20–40 hours per week with at least 6 hours of daily Pacific Time overlap, starting as soon as possible. Selection is an AI interview (approximately 25 minutes), a practical code/AI evaluation (approximately 30 minutes) and a hiring manager interview (approximately 20 minutes). The practical exercise focuses on reviewing and evaluating AI-generated code, not competitive programming or algorithm puzzles.

What you need

  • 5+ years of hands-on software engineering experience
  • Strong proficiency in Python, TypeScript/JavaScript, Go or another major programming language
  • Experience working with substantial production codebases
  • Strong code-review skills and technical judgment
  • Familiarity with GitHub, CI/CD and open-source projects
  • Experience using LLMs or AI coding tools
  • Strong written communication

Good to know

Selection: AI interview, practical code/AI evaluation and hiring manager interview.

Report a problem with this listing