Coding for AI labs: what the work actually is

From a distance, coding work in AI training looks like software engineering. Up close, it is mostly judgment: deciding whether code is correct, proving it with tests, and building problems hard enough to teach a model something. This guide covers the task types you will meet, what reviewers check, the skills and languages in demand, where to apply and what tends to go wrong.

The work, task by task#

Coding projects tend to draw on six kinds of task. The first five revolve around a prompt or a conversation; the sixth happens inside a real software environment. For the wider field, see the complete guide to AI training work.

Reference solutions are the answers a model learns from or is graded against, so the bar is higher than code that merely works. Reviewers expect the idiomatic approach, sensible names, handled edge cases and an explanation that teaches rather than restates the code.

Ranking and grading is the bulk of many projects. You see a prompt and two or more model responses, decide which is better and say why. Run the code whenever the tooling allows. A response that reads well can still fail on an empty list, an off-by-one boundary or a library function that does not exist.

Tests turn your judgment into something a machine can apply thousands of times. Labs use them to grade model output automatically, and in reinforcement learning a passing test can serve as the reward.

Adversarial prompts ask you to find what the model gets wrong: the edge case it misses, the requirement it drops, the API it misremembers. The useful ones fail the way real requests fail. A prompt that trips the model only through trickery teaches it little.

Multi-turn debugging stretches grading across a conversation. You play the user, run each version of the model's code, report what broke and judge whether each revision fixes the problem or just moves it.

Agentic tasks: building the exam#

The most demanding work is building tasks for coding agents: systems that work inside a real environment, running commands, editing files and executing tests, rather than answering in a chat window.

Public benchmarks show the shape of these tasks. In SWE-bench, published at ICLR 2024, a model receives a codebase and a real GitHub issue and must produce a patch that resolves it. Tests decide whether it did, and since June 2024 the official harness has run them in Docker containers to make results more reproducible.

Terminal-Bench, released in 2025, widens the frame from one repository to the whole terminal. Its tasks range from compiling code to training models and setting up servers. Each task comes with an instruction in English, a test script that checks whether the agent succeeded, and a reference ("oracle") solution that shows the task can be done.

People build and check these tasks, and the checking is where much of the value lies. In August 2024, the SWE-bench team and OpenAI released SWE-bench Verified, a subset of 500 problems that software engineers had confirmed were solvable. The screening flagged, among other problems, issue descriptions too vague to define a correct fix and tests so specific they would reject valid solutions. ByteDance's Multi-SWE-bench, released in April 2025, used 68 expert annotators to curate 1,632 tasks from 2,456 candidates. The same team released Multi-SWE-RL, an open dataset for reinforcement learning research whose first release held 4,723 tasks.

Building one usually takes five steps:

1. Write the instruction. State everything a capable engineer would need to know, and nothing that gives away the solution.
2. Build the environment. A Dockerfile sets up the starting state: a repository at a particular commit, a broken build, a misconfigured service.
3. Write the reference solution. A script that solves the task from a clean start.
4. Write the verifier. Tests that check the end state the instruction asks for, not the route taken to reach it.
5. Prove it. The reference solution must pass, an agent that does nothing must fail, and repeated runs must agree.

What reviewers look for#

Rubrics differ by project, but reviewers of coding work tend to check the same things.

• Evidence, not confidence. Say what you ran and what happened. "Raises IndexError on an empty list, at line 12" is a finding. "Has edge-case issues" is a mood.
• Tests that match the specification. Not looser, so a wrong solution passes, and not stricter, so a right one fails. Overly strict tests were one of the flaws the SWE-bench Verified screening hunted for.
• Outcome over method. Grade whether code does what was asked, not whether it matches the solution you would have written.
• The format, followed exactly. If the project asks for a diff, a fixed justification structure or a particular test framework, deviating counts against you however good the code is.
• Honest difficulty. A task should be hard because the work is hard, not because the instruction hides a requirement or the environment leaks the answer.

Three skills sit underneath all of this. The first is reading unfamiliar code quickly: agentic tasks often start inside someone else's repository, and you have to find the relevant files before you can judge a fix. The second is testing discipline, the habit of writing the failing test before believing any claim, including your own. The third is clear writing, because a reviewer judges your reasoning through what you put on the page.

Languages and stacks in demand#

Python is the center of gravity. The original SWE-bench draws its tasks from Python repositories, and the ByteDance team behind Multi-SWE-bench describes existing benchmarks such as SWE-bench as Python-centric.

The range is widening. Multi-SWE-bench spans Java, TypeScript, JavaScript, Go, Rust, C and C++, and the SWE-bench project now offers a multilingual dataset alongside the original. Some platforms recruit by language: Alignerr has advertised AI-training roles for specific languages, down to Objective-C and LPC (see the Alignerr profile).

Agentic tasks also assume the tools around the code: a Linux shell, Git, Docker, and the build and test tooling of whatever stack a task uses. Systems knowledge counts as much as algorithms.

Depth beats breadth: one language you know well is worth more than ten you have touched. List only what you could be tested on today.

How to get in#

Most large platforms run coding work, but they differ in how much of it is code and in how they screen.

• Turing focuses on coding and technical reasoning, usually on hourly contracts that state the expected weekly hours.
• Outlier, Scale AI's contributor platform, runs coding projects alongside writing, STEM and language work. Eligibility depends on where you live.
• Mercor posts software-engineering contracts as individual listings. You apply to one, and most listings include a short AI interview.
• micro1 began as a way for companies to hire AI-vetted software engineers. Its AI interviewer can add a coding challenge for technical roles.
• DataAnnotation has a self-paced coding track with its own qualification test, though it reportedly accepts applicants only from the US, Canada, the UK, Ireland, Australia and New Zealand.
• Alignerr posts some software roles by programming language.

Pay and eligibility change often; current details are on each profile.

Expect a coding assessment, a review of model-written code, or both. How to pass AI training assessments covers the format, and how AI interviews work at micro1 and Mercor covers the spoken screens. Where we have a referral link, Apply buttons on ASV use it, as our disclosure page explains.

ASV runs this work in-house, too. Our own paid projects include software task authoring: contributors build hard, verifiable terminal tasks for coding agents, packaged in the format described above, with an instruction, a Dockerfile, a reference solution and tests. Before submitting, you confirm that the reference solution passes, that an agent which does nothing fails, that the task is deterministic and that no hidden answer is exposed to the agent. An admin reviews every submission, and the work a project selects is paid directly by ASV. For tasks graded by rubric rather than by tests, see writing rubrics that hold up.

Pitfalls#

Quality scores follow you. Reviews, audits and spot checks feed a record that shapes what you are offered next. A fast, careless week can cost more than it earns. Read the guidelines in full, and again whenever they change.

The clock is part of the job. Mercor requires its Insightful time tracker on hourly projects and does not allow manual time entries. Its documentation says the tool takes periodic screenshots while the timer runs, and that penalties for violations can escalate from reduced or reversed pay for the affected work to removal from projects and, in the most serious cases, suspension of platform access. At micro1, hourly projects are paid from Hubstaff time records, while pay-per-task projects pay for approved tasks, not hours tracked. On per-task work, a debugging session that runs twice as long halves your effective rate. Time-box investigations, and report a broken task rather than sinking an evening into it.

Confidentiality is strict. Mercor's conduct policy treats project materials, client information and prompts as confidential, and lists posting confidential material publicly or in third-party tools as a violation. Expect similar terms elsewhere, and read your agreement. For engineers, the risky habits are specific: pushing task code to a public repository, pasting it into an AI assistant or asking about it on a forum.

Work comes and goes. Projects end when a client's needs change, often with little notice. Keep more than one platform active, and treat any single project as temporary. Anyone who asks you to pay for access to coding work is running a scam; see how to spot AI training job scams.

The engineers who do well here treat every submission as evidence: run it, test it, explain it, and leave the reviewer nothing to guess.

Sources#

• SWE-bench (SWE-bench team, via GitHub)
• Introducing SWE-bench Verified (OpenAI)
• Terminal-Bench (Terminal-Bench team, via GitHub)
• Multi-SWE-bench (ByteDance Seed, via GitHub)
• Software Engineer, Objective-C - AI Training (Freelance, Remote) (Alignerr, via Built In)
• Account security and conduct (Mercor)
• Use Insightful for Time Tracking (Mercor)
• Deel (micro1)