
Senior Software Engineer – AI Evaluation / Coding Agents
Evaluate and improve AI coding models by reviewing AI-generated code and assessing coding-agent performance.
About this work
Experienced software engineers are wanted to evaluate and improve AI coding models by reviewing AI-generated code, assessing coding-agent performance and identifying technical weaknesses. Key responsibilities include evaluating code generated by AI coding agents; identifying technical errors and assessing code quality; designing evaluation criteria and rubrics; reviewing and labelling evaluation data; building and maintaining evaluation pipelines; and communicating findings to research and engineering teams. The 3-month contract offers 20–40 hours per week with at least 6 hours of daily Pacific Time overlap, starting as soon as possible. Selection is an AI interview (approximately 25 minutes), a practical code/AI evaluation (approximately 30 minutes) and a hiring manager interview (approximately 20 minutes). The practical exercise focuses on reviewing and evaluating AI-generated code, not competitive programming or algorithm puzzles.
What you need
- 5+ years of hands-on software engineering experience
- Strong proficiency in Python, TypeScript/JavaScript, Go or another major programming language
- Experience working with substantial production codebases
- Strong code-review skills and technical judgment
- Familiarity with GitHub, CI/CD and open-source projects
- Experience using LLMs or AI coding tools
- Strong written communication
Good to know
Selection: AI interview, practical code/AI evaluation and hiring manager interview.
Report a problem with this listing