Skip to main content
An outcome is a session’s definition of done: an objective and a rubric. Each time the agent finishes its work, a grader checks it against every criterion in the rubric. It either accepts the work or sends the agent back with what fell short, and the agent keeps revising until the rubric is met or the passes run out. Use an outcome when “the agent stopped” isn’t proof enough that the work is right. How well this works depends mostly on the rubric, so this page starts with how to write one.
Launching a graded session and saving an agent’s default rubric need the Developer or Admin role in the organization. The User role can read outcomes. See Organizations and roles. You need an agent to do the work and an environment.

​
Write a rubric

A rubric is Markdown. Each list item is one criterion. The grader owes a verdict on every list item and on nothing else.
This rubric has six criteria in three sections. Every criterion counts the same: you can’t set weights, and a strong result on one criterion never makes up for a failure on another.

​
Make each criterion stand on its own

The agent receives the criteria, grouped by section, and nothing else from the rubric. The grader receives the objective and the whole rubric, but it owes one verdict for each criterion, so each criterion has to carry everything needed to meet it and to check it.
  • Make it checkable. “The CSV has a numeric price column” can be checked. “The data looks good” can’t, and a criterion like that sends the work back until the passes run out.
  • Give each list item one requirement. When one item holds two requirements, a failure doesn’t say which one the work missed.
  • Keep each criterion on one line. Text on a second line isn’t part of the criterion.
  • Say what counts as evidence in the criterion. Write “The test suite passes when run with make test”, not “Tests pass” followed by a paragraph about how to run them.
  • Ask for work that leaves evidence. A file, a passing test, or a record in another system can be verified. A claim in the agent’s last message can’t.
  • Put shared background in the objective. The opening message is the objective, and both the agent and the grader read it. Facts both need, such as which repository to use, belong there.
  • Start from a good example. Take a result you would accept, list what makes it good, and turn each point into a criterion.

​
Name the deliverable

Only files the agent saves as deliverables are kept with the session. Agents already know where to save them, so you don’t need to give a path. Name the file in the opening message and in the rubric, for example “Save the notes as release-notes.md” and ”- The notes are saved as a deliverable named release-notes.md.” Deliverables are saved before every grading pass, so the grader checks the version the agent just finished. See Deliverables and artifacts.

​
Name your criteria

A list item written as - key: text gives the criterion a name:
A key starts with a lowercase letter and uses lowercase letters and digits, with single hyphens or underscores between them. A name is for the people who write and review the rubric. The agent, the grader’s verdicts, and the result show the text after the colon without the name, so that text has to make sense by itself. Name every criterion or none: a rubric that mixes named and unnamed items is refused.
A list item that starts with a lowercase word, a colon, and a space is read as a named criterion. - output: saved as report.csv gets the name output, and the agent is shown “saved as report.csv” without it. If your other items have no name, the rubric is refused. To keep an item unnamed, start it with a capital letter or reword it.
A rubric is refused when:
  • It has no list item, or more than 200.
  • A list item is empty, such as a lone - or - [ ], or a named item has no text, such as - todo:.
  • A line starts with a dash and no space, such as -The file exists.
  • It mixes named and unnamed criteria, uses the same key twice, or has a key longer than 64 characters.
  • It is larger than 256 KiB, or one criterion is larger than 4 KiB.

​
Send criteria through the API

Through the API, send the criteria as a list rather than Markdown:
  • outcome.criteria when you start a session or add an outcome.
  • defaultCriteria on an agent.
Each criterion has a criterion_key, its criterion_text, and an optional section, and the same rules apply. Markdown rubrics are still accepted, but they are deprecated. A Markdown rubric is stored as written, and its criteria are read from it into the same list. A Markdown rubric reads back exactly as you sent it. A list of criteria reads back with the rubric written as Markdown: one - key: text line per criterion, with a ## section heading where its section starts and a bare # where it ends. Criteria read from a Markdown rubric written without names get the names c001, c002, and so on, in order. Read criteria for the list itself; each criterion verdict names its criterion by criterion_key.

​
Who grades

When the writer finishes a turn, a grader runs one pass in a linked session. The rubric is all you provide. The grader receives a captured copy of the writer’s workspace and a frozen point in its transcript. The writer waits for that pass before it revises or finishes.
  • It judges against the rubric. The grader has its own transcript. It reads the objective and rubric and can inspect the writer’s frozen transcript for evidence.
  • It checks a captured workspace. It can open files and run tests. Its changes to that copy do not alter the writer’s workspace. Don’t write a criterion that requires the grader to change the writer’s work.
  • Its guidance comes from the criteria and the objective. How strict to be and what counts as evidence belong in each criterion’s text; shared background belongs in the objective. Paragraphs in a rubric are not criteria, so the grader owes no verdict on them.
  • It is a managed agent. Grading runs on an agent named Outcome grader that the platform creates the first time an agent’s outcome is graded, one for each of your agents. It runs on the same model as the writer and the platform’s grading prompt, and it is hidden from your agent list but can be read by its ID. Verdicts and Quality name it as the evaluator. Writer versions that share a model share one grader version, so their pass rates are comparable. It is deleted only with the writer agent.
  • You can choose a different grader. Set evaluatorAgentId on an outcome, or defaultEvaluatorAgentId on an agent version, to grade with one of your own agents instead. The outcome’s value wins over the agent version’s, and without either the Outcome grader grades. Your agent keeps its own model and instructions, runs without tools, credentials, memory or integrations, and can only read the writer’s frozen snapshot and submit its verdicts. The agent must exist in your organization, cannot be an Outcome grader, and cannot be the agent that writes the outcome; otherwise the request returns 400.

​
Grade a session against a rubric

1

Open the launch form

In the sidebar, click Sessions, then click Launch session. In Launch a session, choose the Agent and Environment.
2

Turn on grading

Under Task, select Grade this session against a rubric. It’s already selected when the agent has a stored rubric.
3

Write the rubric

Write your criteria in Rubric. The field starts with the agent’s default rubric, or with a short template. Replace the template’s lines with your own. The paragraphs under its ## Evidence and ## Ignore headings are notes, not criteria.
4

Write the objective and launch

Write the task in Opening message. It’s the objective the grader checks the work against, and a graded session needs one. Click Launch session.
Grading is on from the first turn. Each pass has its own linked session and workspace copy. Its usage and compute time are included in the writer’s displayed total and shared spend limit. See Pricing.

​
Grade every session of an agent

Store a rubric on the agent, and its sessions are graded without anyone turning grading on.
  1. Open the agent and stay on the Configuration tab.
  2. In Outcome, click Add if the section is closed.
  3. Write the rubric in the text box.
  4. Click Save new version.
The rubric is saved with that version. A session that runs a version with a rubric and starts with an opening message is graded against it, however the session starts: from the launch form, through the API, or by an automation or Slack. The opening message is the objective, so a session that starts without one runs ungraded.
A stored rubric also grades sessions nobody launches by hand. Runs of an automation that uses the agent are graded for up to five passes, and each pass adds to that session’s cost. See Pricing.
In Launch a session, an agent with a stored rubric has Grade this session against a rubric already selected and shows the rubric. Clear the box to run that session ungraded. An edit to the rubric in the form applies to that session only. An automation keeps the agent version it was saved with, and its runs are graded against that version’s rubric. A version saved with the Outcome section empty runs ungraded. An older version that has no rubric setting at all is graded against the agent’s current rubric. After you add or change a rubric, move the automation to the new version so you know which rubric its runs use.

​
Read the result

  1. In the sidebar, click Sessions and open the session.
  2. Read the Outcome line above the transcript: the result so far, the number of unmet criteria if any, and the objective.
  3. Click Details to see the objective, the pass limit, the Rubric, and each pass’s criterion verdicts and reasons.
  4. In the writer’s activity timeline, open Outcome grading and select a linked grader session. It has its own activity lane on the writer’s timeline, and its transcript opens beside the writer’s. Each pass checks its own snapshot.
After a final result, the session still accepts messages, but those turns aren’t graded. To grade more work, launch a new session. Each grading pass that reached a verdict is listed in the outcome, one entry per pass. A pass that couldn’t grade the work has no verdict and isn’t listed. When it ends the outcome, the outcome’s result says so and the outcome’s explanation gives the reason; through the API these are terminal_result and terminal_explanation. Outcome grading is a loop on the session: the session’s Outcome tab shows each grader pass, the feedback the writer received and the revision that followed it.

​
How grading works

  1. The agent works until it ends a turn with nothing left to do. A turn you interrupt or cancel isn’t graded.
  2. The agent’s deliverables are saved, and a grading pass appears while its workspace copy is prepared. The grader runs in a linked session on that copy. It can inspect the frozen writer transcript and check files, tests, and other evidence.
  3. The grader gives every criterion one verdict: Pass, Fail, or Not applicable. One Fail is enough to send the work back.
  4. The agent receives the whole scorecard, with the grader’s reason for each failed criterion, and revises. Then the next pass starts with a new grader.
For a session launched from the console, grading stops after five passes, the first one included. The outcome is Satisfied when no criterion failed and at least one passed.

​
What can go wrong

The most common problems:
  • The work keeps coming back, then the outcome reads Max iterations reached. A criterion is vague or can’t be met. Read the grader’s reason on the failed criterion, rewrite it so it can be checked, and launch again.
  • A deliverable criterion fails. The agent saved the file under another name, or didn’t save it as a deliverable. Name the file in both the opening message and the rubric.
  • A session wasn’t graded. The box was cleared in the launch form, or the session started with no opening message. For a session an automation started, the automation may run an agent version saved without the rubric: move it to the new version.
  • The transcript says the outcome check returned no result. The grader didn’t return a usable verdict, so the work wasn’t judged. Send a message to continue. Grading runs again when the agent next finishes.

​
Limits

Other limits are on Limits.

​
Next steps

Deliverables and artifacts

Ask for named deliverables the grader can check.

Sessions

Launch, steer, and follow a session while it’s graded.

Agents

Store a default rubric on an agent.

Pricing

See what a session hour covers.