Writing rubrics that hold up
What makes a rubric criterion atomic, objective and weighted to the stakes, the mistakes that break rubrics, and how reviewers judge yours.
On this page7 sections
A rubric is how an expert's judgment leaves the room. Written once, it can grade thousands of model responses, and in reinforcement learning its scores become the reward a model is trained to maximize. This guide covers what makes a criterion hold up, the mistakes that quietly break rubrics, and how reviewers decide whether yours is ready.
What a rubric is, and what it is for
In AI training work, a rubric is often a list of criteria written for one prompt and used to judge any response to it. Each criterion is a statement a grader can usually mark as met or not met: the response flags the drug interaction, cites the controlling clause, returns an empty list for empty input. Each carries a weight. The grader, a person or a model, checks every criterion, and the response's score is the points it earned as a share of the points available.
Two OpenAI benchmarks from 2025 show the form at scale. HealthBench, released in May, grades models on 5,000 health conversations, each with its own rubric: 48,562 criteria in all, written by 262 physicians who had practiced in 60 countries. Criteria for things a response should not do carry negative points. PaperBench, released in April, asks AI agents to replicate 20 papers from the ICML 2024 machine learning conference. Its rubrics, developed with the papers' own authors, break each replication into a tree of requirements, with 8,316 individually gradable tasks across the 20 papers.
Rubrics also train models. In reinforcement learning, a model produces many responses, a grader scores them, and the model is updated toward the higher scorers. Where answers can be checked mechanically, as with math or code that has tests, scoring is straightforward. Where they cannot, as with a clinical explanation or a legal memo, a rubric can stand in. In 2025, researchers at Scale AI used rubric scores as rewards to train models on medicine and science questions, an approach they called Rubrics as Rewards. In their experiments, rubric-based rewards outperformed rewards from a judge model that rated each response on a simple scale with no rubric.
That use is less forgiving. A rubric used as a reward is not read by a patient colleague who knows what you meant. It is optimized against, and any criterion that can be satisfied without doing the real work invites exactly that.
What makes a criterion hold up
None of this is new to teachers. In her book on classroom rubrics, the education researcher Susan Brookhart asks, among other things, that criteria be appropriate, definable, observable, distinct from one another and, taken together, complete. The Scale AI researchers set out a similar list for model training: criteria grounded in expert guidance, covering the main dimensions of quality including common mistakes, weighted by importance, and each checkable on its own. In practice, a criterion that holds up is:
- Atomic. One criterion, one verdict. "Identifies the bug and fixes it" is two checks; a response that does only one earns neither a clean pass nor a clean fail.
- Objective. Two careful graders reach the same verdict. If the verdict depends on taste, the criterion measures the grader.
- Verifiable. The grader can check it from the response and any reference the task supplies, without guessing what you meant. If checking takes steps, such as running a script or comparing against a drawing, write the steps down.
- Independent. One mistake fails one criterion. If "gives the correct total" and "the final figure matches the reference" both hinge on the same sum, one arithmetic slip costs twice.
- Weighted. Points track consequences. A wrong dose should outweigh a missing courtesy, however many courtesies the rubric lists.
- Grounded. Every criterion traces to the prompt, its reference material or a settled standard of the field. A response cannot fairly fail a requirement that neither the task nor the field imposes.
- About the outcome. Grade what the response delivers, not the method you would have used. A model may reach a correct result by a route you did not predict.
Atomic does not mean tiny. If four measurements come from one reference drawing, one criterion that checks all four against the drawing, weighted to match, beats four one-point items that fail together whenever the scale is wrong.
Taken together, the criteria also need coverage. Many first drafts cover only the happy path: they describe the ideal answer and stop. Strong rubrics also name the ways a response typically goes wrong: the plausible misdiagnosis, the off-by-one error, the confident citation to a case that does not exist. HealthBench handles this with negative points; Rubrics as Rewards labels such items as pitfalls.
Note
Assume your rubric will be applied by a model grader that sees only the prompt, the response and one criterion at a time. If a criterion needs your intent to be applied correctly, rewrite it.
Weak criteria, rewritten
Weak criteria tend to fail in a few predictable ways. The rewrites below are illustrative; each assumes a prompt that supplies the facts it relies on.
| Weak criterion | What goes wrong | Stronger version |
|---|---|---|
| The response is accurate | No grader can check "accurate" in the abstract | Flags that amoxicillin is a penicillin, which the patient says they are allergic to |
| Gives a comprehensive overview of the contract's risks | Rewards length and names no risk | Identifies that the indemnity in clause 7 has no liability cap |
| Mentions Rayleigh scattering | A keyword check that name-dropping passes | Explains that air molecules scatter short blue wavelengths more strongly than long red ones |
| Finds the error in the proof and fixes it | Two checks in one | Identifies that step 3 divides by x − 1, which is zero when x = 1 (a separate criterion checks the fix) |
| Handles edge cases well | Vague, and names no case | Returns an empty list, without raising an error, when the input is empty |
| Uses a hash set to remove duplicate IDs | Grades the method, not the result | Lists each customer ID exactly once |
| Is well written | Taste, not a test | Answers the question in the first sentence, before any background |
Every stronger version names something a stranger could find, or fail to find, in the response.
Mistakes that break rubrics
- Vague words. "Good," "clear," "comprehensive," "appropriate," "properly." Each hands the decision back to the grader's taste, which is what a rubric exists to replace. If you cannot say what "clear" looks like in this response, you do not have a criterion yet.
- Overlapping items. Two criteria that hinge on the same fact punish one error twice and make a single slip look like a pattern. For each pair, ask whether a response could fail one without failing the other.
- Rewarding length. Criteria that ask a response to "discuss," "consider" or "explain in detail" are satisfied by saying more, so a padded answer can collect every point a tight, correct one earns. Automated graders can lean the same way. A 2023 study of LLM judges documented verbosity bias, a tendency to favor longer answers, and a study led by researchers at the University of Texas at Austin found that, in the settings it examined, much of the reward gain in reinforcement learning from human feedback came from longer responses. Where brevity matters, make it a criterion, and give no points for material the prompt did not ask for.
- Leaking the answer. A rubric checks two kinds of things: requirements the task states, and judgments an expert should make unprompted. Requirements belong in the prompt. Judgments do not. If the prompt hints at the red-flag symptom, the hidden bug or the counterargument a criterion rewards, that criterion now measures reading comprehension.
- Unweighted critical errors. Take ten criteria worth one point each. A response passes nine and fails one: it recommends ten times the safe dose. It scores 90 percent. Weight that criterion so failing it sinks the score, attach negative points to the unsafe recommendation, or mark it as an automatic failure if the project allows one.
Important
The rubric should know the answer; the prompt should not. State every requirement in the prompt, and never the insight you are testing for.
How reviewers judge a rubric
Formats differ by project, but the questions a careful reviewer asks are much the same, and you can ask them first.
- Does every criterion trace to something? The prompt, a reference file, or a standard any expert in the field would apply. Criteria that encode a private preference get cut.
- Would two graders agree? Reviewers read each criterion as a stranger would. Benchmark builders test this directly: OpenAI compared HealthBench's model grader with physicians' own judgments, and built a separate benchmark to check PaperBench's judge against human grading.
- Does the score order responses correctly? A strong response should score near the top, a flawed one should lose points where it is flawed, and a padded one should gain nothing. A rubric that every response passes in full measures nothing.
- Do the weights match the stakes? Reviewers find the worst plausible error and check that it costs more than any pile of minor ones.
- Is the count right? Too few criteria miss failures; too many trivial ones add noise and double counting. Consolidate checks that stand or fall together.
- Is each criterion about the output? Grade what the response produced, not how you would have produced it.
Tip
Before you submit, grade three responses with your own rubric: your reference answer, a plausible wrong answer and a padded one. If the wrong answer lands within a few points of the reference, fix the weights before a reviewer does.
Where this work pays
Rubric writing now appears in job titles. Mercor has listed a role titled Rubrics Writer / Auditor, whose posting describes evaluating AI-generated written conversations against predefined rubrics. Our gigs directory profiles Mercor and the other platforms we cover, with current pay ranges, payouts and availability; where we have a referral link, Apply buttons on ASV use it, as our disclosure page explains.
ASV also pays for this work directly. Its own paid projects include task authoring, where contributors write a task and the rubric that grades it. There, each criterion carries a weight, the type of check it needs, the steps a reviewer follows to verify it and a short justification. The rubric editor flags vague words such as "properly" and "looks nice." Everything in this guide applies there. For the grader's side of the same discipline, see how to pass AI training assessments; for code-heavy tasks, see coding for AI labs.
The test of a rubric is simple to state and hard to meet: a stranger, or a model, should be able to apply it and reach the verdict you would.
Sources
- Introducing HealthBench (OpenAI)
- PaperBench: Evaluating AI's Ability to Replicate AI Research (OpenAI)
- Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains (Scale AI, via arXiv)
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (UC Berkeley and others, via arXiv)
- A Long Way to Go: Investigating Length Correlations in RLHF (University of Texas at Austin, via arXiv)
- How to Create and Use Rubrics for Formative Assessment and Grading, chapter 1 (ASCD)
- Rubrics Writer / Auditor (Mercor)