> ## Documentation Index
> Fetch the complete documentation index at: https://docs.labelbox.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Define outcomes
> Grade a session against a Markdown rubric, send the agent back to revise until every criterion passes, and read the verdict for each criterion.
An outcome is a session's definition of done: an objective and a rubric. Each time the agent finishes its work, an independent grader checks it against every criterion in the rubric and either accepts it or sends the agent back with the failed criteria and the reasons.
Use outcomes when "the agent stopped" is not proof enough that the work is right. A session with an outcome only reports `execution_state` `completed` after the grader accepts the work.
The TypeScript samples assume the `rl` client from [Client setup](/managed-agents/api#client-setup).
## Before you begin
* You need the organization developer or admin role to start a session or add an outcome to one. The organization user role can read outcomes and grading results. See [Organizations and roles](/managed-agents/organizations-and-roles) and [API keys](/managed-agents/api-keys).
* You need an agent and an environment. See [Agents](/managed-agents/agents) and [Environments](/managed-agents/environments).
* Decide what the agent must produce. If it is a file, pick its name now (for example `release-notes.md`). You will use that name in the task and in the rubric.
## How grading works
```mermaid theme={"theme":"css-variables"}
flowchart TD
start["You start a session with an outcome"] --> work["The agent works"]
work --> finish["The agent ends its turn with nothing left to do"]
finish --> keep["Deliverables are saved"]
keep --> grade["The grader checks every criterion"]
grade -->|"a criterion failed and passes remain"| feedback["The agent receives every verdict and the reasons for failures"]
feedback --> work
grade -->|"no criterion failed"| done["satisfied: session completed"]
grade -->|"a criterion failed on the last allowed pass"| capped["max_iterations_reached: session idle"]
grade -->|"the grader returned no usable verdict"| broken["grader_failed: session idle, outcome still open"]
```
1. The agent works until it ends a turn with nothing queued. That is its claim that the work is done. A turn you stop with an interrupt or a cancel is not graded.
2. Deliverables the agent saved are kept, then the grader starts a **grading pass**. The grader is a separate model run on the session's model, with its own context. It sees the objective and the rubric, never the agent's reasoning. It works in the same sandbox with the same tools and credentials, so it can open files, run the project's tests, and check external systems the agent changed.
3. The grader gives each criterion exactly one verdict. The result of the pass is computed from those verdicts, not chosen by the grader: one `fail` is enough to send the work back.
4. On `needs_revision`, the agent receives the full scorecard, with a reason for every failed criterion, and starts another turn. Then step 1 repeats.
**What success means:** the outcome's `status` is `terminal` with `terminal_result` `satisfied`, and the session's `execution_state` is `completed` with `stop_reason` `outcome_satisfied`. Nothing else marks a graded session as completed.
## Write a rubric
A rubric is Markdown. The grader owes a verdict on every list item and on nothing else.
| Markdown | How it is read |
| ------------------------------------- | ------------------------------------------------------------------------------------------ |
| List item (`-`, `*`, `+`, `1.`, `1)`) | One criterion. A task box such as `- [ ]` is stripped. Nested list items are criteria too. |
| Heading (`#` to `######`) | The section for the criteria below it, returned as `section`. |
| Paragraph | Context for the grader. Not scored, and not shown to the agent. |
| Fenced code block | An example. List items inside a fence are not criteria. |
| Horizontal rule (`---`) | Ignored. |
Write each criterion on one line. A wrapped second line is read as a paragraph, so the criterion loses that text.
```markdown theme={"theme":"css-variables"}
# Release notes for v4.12.0
The notes cover every pull request merged between the tags v4.11.0 and v4.12.0.
Customers read them, so internal tracker ids do not belong in the text.
## Deliverable
- The notes are saved as a deliverable named `release-notes.md`.
## Coverage
- Every merged pull request in the range appears exactly once.
- No entry describes a pull request outside the range.
## Format
- Breaking changes are listed first, under a heading named "Breaking changes".
- Each entry is one sentence and ends with the pull request number in parentheses.
```
This rubric has five criteria in three sections. The two paragraphs tell the grader how to find the range and who reads the notes.
All criteria count the same. You cannot set weights (every criterion reports `weight: 1`), and a strong result on one criterion never offsets a failure on another. `not_applicable` does not count as a failure.
### Name the deliverable
Only files the agent saves under `/workspace/.managed-agents/outputs` are kept as deliverables. Files anywhere else, including paths you invent in a prompt, are not kept. Agents already know where to save deliverables, so you do not need to give a path. Name the file in the task and in the rubric, for example "Save the notes as `release-notes.md`" and "- The notes are saved as a deliverable named `release-notes.md`." Deliverables are saved before every grading pass, so the grader checks the version the agent just finished. See [Deliverables and artifacts](/managed-agents/artifacts).
* **Make each criterion checkable on its own.** "The CSV has a numeric `price` column" can be checked. "The data looks good" cannot, and a criterion like that causes revision loops that do not end.
* **Give one requirement to each list item.** When one item holds two requirements, a failure does not say which one the work missed.
* **Put facts the agent needs where it can see them.** The agent sees the objective and the criteria, not the paragraphs. Put background in the task message.
* **Start from a good example.** Ask a model what makes a known-good result good, and turn the answer into criteria.
* **Cap a new rubric.** Set `max_iterations` until you know the rubric can be met. A criterion that can never pass keeps an uncapped session revising.
* **Keep the text stable.** Criterion ids stay the same only while the rubric text is identical. See [Follow a criterion across sessions](#follow-a-criterion-across-sessions).
## Outcome fields
These fields go in `outcome` on `startSession`, or in the body of `defineSessionOutcome`.
| Field | Required | Description |
| ---------------- | ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| `description` | Yes | The objective. The grader measures the work against it. If the start request has no `message`, the agent also starts from it. |
| `rubric` | One of `rubric` or `rubric_ref` | The rubric as one Markdown string, with 1 to 200 criteria. |
| `rubric_ref` | One of `rubric` or `rubric_ref` | The rubric as a file from your [library](/managed-agents/files): `{"type": "file", "file_id": "..."}`. |
| `max_iterations` | No | The largest number of grading passes. Omit it or send `0` to revise until the rubric is satisfied (the default). Negative values are rejected. |
To keep a long or shared rubric in one place, [upload it](/managed-agents/files#upload-a-file) as a text or YAML file of up to 256 KiB and send `rubric_ref` instead of `rubric`. Its text is read once, when the outcome is defined, and returned as the outcome's `rubric`, so later edits to the file don't change a running session. An unknown or expired file returns `404`; a file that isn't UTF-8 text, is too large, or holds no criterion returns `400`.
## Start a session with an outcome
Send `outcome` in the start request. Grading is on from the first turn.
1. Open the agent and click **Start session**, or click **Launch session** on the **Sessions** tab.
2. In **Launch a session**, choose the **Agent** and **Environment**.
3. Under **Task**, select **Grade this session against a rubric**.
4. Write the criteria in **Rubric**.
5. Write the task in **Opening message**. The console sends it as the objective.
6. Click **Launch session**.
The console never sets a cap, so the session revises until the rubric is satisfied.
```typescript theme={"theme":"css-variables"}
const { session_id } = await rl.managedAgents.startSession({
'Idempotency-Key': 'release-notes-v4-12-0',
body: {
agent_id: '5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55',
environment_id: '9d3e7b52-1a4c-4f80-b6e9-2c8a5d0f7e13',
message: 'The repository is at /workspace/app. Compare the tags v4.11.0 and v4.12.0.',
outcome: {
description: 'Release notes for v4.12.0, saved as release-notes.md.',
rubric: rubricMarkdown,
max_iterations: 4,
},
},
});
```
```bash theme={"theme":"css-variables"}
curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/sessions' \
-H "Authorization: Bearer $RECURSION_API_KEY" \
-H 'Idempotency-Key: release-notes-v4-12-0' \
-H 'Content-Type: application/json' \
-d '{
"agent_id": "5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55",
"environment_id": "9d3e7b52-1a4c-4f80-b6e9-2c8a5d0f7e13",
"message": "The repository is at /workspace/app. Compare the tags v4.11.0 and v4.12.0.",
"outcome": {
"description": "Release notes for v4.12.0, saved as release-notes.md.",
"rubric": "## Deliverable\n- The notes are saved as a deliverable named `release-notes.md`.\n\n## Format\n- Breaking changes are listed first, under a heading named \"Breaking changes\".",
"max_iterations": 4
}
}'
```
The API answers `202 Accepted`. The session is accepted, not finished. Keep `session_id` to [read the grading](#read-the-grading).
```json theme={"theme":"css-variables"}
{
"session_id": "b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38",
"status_path": "/v1/sessions/b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38"
}
```
With `max_iterations: 4`, the grader runs at most four passes, so the agent gets up to three revisions.
### Choose the message and the objective
`message` and `outcome.description` are separate. The agent starts from the message. The grader measures against the objective and never sees the message. Use the message for facts the agent needs that are not part of the bar, such as where the repository is.
| You send | The agent starts from | The grader measures against |
| ------------------------------------------- | --------------------- | ------------------------------------------ |
| `outcome` only | `outcome.description` | `outcome.description` |
| `outcome` and `message` | `message` | `outcome.description` |
| `message` only, agent has a default rubric | `message` | `message`, with the agent's default rubric |
| `message` only, agent has no default rubric | `message` | Nothing. The session is not graded. |
Keep the two consistent. If they disagree, the agent works toward one target and is graded against another. The agent can see the objective and the criteria for the whole session, not just in its first turn.
## Grade every session of an agent
Store a rubric on the agent as `default_rubric` to grade every session started from it. In the console, the field is **Default rubric** in the **Outcome** section of the agent's **Configuration** tab. Through the API, send `default_rubric` on `createAgent` or `createAgentVersion`. It must have 1 to 200 criteria. See [Agents](/managed-agents/agents).
| Start request | Result |
| ----------------------------- | ------------------------------------------------------------------------------------ |
| `message`, no `outcome` | Graded against the default rubric. The message is the objective. |
| No `message` and no `outcome` | Not graded. There is no task to grade. |
| An `outcome` | Graded against that outcome. The default rubric is ignored as a whole, never merged. |
| `skip_default_outcome: true` | Not graded. |
In **Launch a session**, an agent with a default rubric has **Grade this session against a rubric** selected and shows the stored rubric. Clear the box to run ungraded. An edit in the form applies to that session only and gives its criteria new ids.
## Add an outcome to a running session
Call `defineSessionOutcome` to grade a session that started without an outcome, or to set the next objective after an outcome ended. The body has the same fields as `outcome`. There is no console action for this.
A session runs one outcome at a time. While an outcome is not `terminal`, a second one is rejected.
```typescript theme={"theme":"css-variables"}
const outcome = await rl.managedAgents.defineSessionOutcome({
session_id: 'b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38',
body: {
description: 'A one-page summary of v4.12.0 for the support team, saved as support-summary.md.',
rubric:
'- The summary is saved as a deliverable named `support-summary.md`.\n- It lists every breaking change from release-notes.md.',
},
});
```
```bash theme={"theme":"css-variables"}
curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38/outcomes' \
-H "Authorization: Bearer $RECURSION_API_KEY" \
-H 'Content-Type: application/json' \
-d '{
"description": "A one-page summary of v4.12.0 for the support team, saved as support-summary.md.",
"rubric": "- The summary is saved as a deliverable named `support-summary.md`.\n- It lists every breaking change from release-notes.md."
}'
```
The response is the stored outcome. Some fields are omitted.
```json theme={"theme":"css-variables"}
{
"outcome_id": "4a9d2f70-6c1b-4e83-9f25-8b0e3a7c5d14",
"session_id": "b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38",
"agent_id": "5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55",
"description": "A one-page summary of v4.12.0 for the support team, saved as support-summary.md.",
"rubric": "- The summary is saved as a deliverable named `support-summary.md`.\n- It lists every breaking change from release-notes.md.",
"max_iterations": 0,
"status": "pending",
"created_at": "2026-09-17T11:40:02Z",
"updated_at": "2026-09-17T11:40:02Z"
}
```
The new objective and criteria join the agent's instructions, and a message naming the objective appears in the transcript. Defining an outcome does not start a turn by itself. If the session is idle, send a message so the agent starts working toward it. See [Session operations](/managed-agents/session-operations).
## Read the grading
`listSessionOutcomes` returns every outcome of a session, oldest first, each with all of its grading passes.
1. On the **Sessions** tab, open the session.
2. Read the **Outcome** line above the transcript: the current result, the number of unmet criteria, and the start of the objective.
3. Click **Details** to see the objective, the cap (**revises until satisfied**, or **at most N revisions** where N is `max_iterations`), the **Rubric**, and one block per pass, newest first (**Pass 1** is the first pass). Each pass shows its result, how many criteria were met, its cost and tokens, and every criterion with its verdict. A failed criterion also shows the grader's reason.
```typescript theme={"theme":"css-variables"}
const { outcomes } = await rl.managedAgents.listSessionOutcomes({
session_id: 'b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38',
});
const latest = outcomes?.at(-1);
console.log(latest?.status, latest?.terminal_result);
```
```bash theme={"theme":"css-variables"}
curl 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38/outcomes' \
-H "Authorization: Bearer $RECURSION_API_KEY"
```
In this response, the first pass sent the work back and the second accepted it. Some fields are omitted.
```json theme={"theme":"css-variables"}
{
"outcomes": [
{
"outcome_id": "e5c07a3b-1f94-4d26-b8a0-6d2c9e4f7b51",
"session_id": "b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38",
"description": "Release notes for v4.12.0, saved as release-notes.md.",
"rubric": "## Deliverable\n- The notes are saved as a deliverable named `release-notes.md`.\n\n## Format\n- Breaking changes are listed first, under a heading named \"Breaking changes\".",
"max_iterations": 4,
"status": "terminal",
"terminal_result": "satisfied",
"created_at": "2026-09-17T10:15:03Z",
"updated_at": "2026-09-17T10:31:47Z",
"ended_at": "2026-09-17T10:31:47Z",
"evaluations": [
{
"outcome_id": "e5c07a3b-1f94-4d26-b8a0-6d2c9e4f7b51",
"iteration": 0,
"result": "needs_revision",
"explanation": "1 of 2 criteria not met.",
"criteria": [
{
"criterion_id": "r3fa91c07b2de.c001",
"criterion_text": "The notes are saved as a deliverable named `release-notes.md`.",
"section": "Deliverable",
"weight": 1,
"verdict": "pass",
"rationale": "release-notes.md is in the outputs folder and has 41 lines.",
"evidence_event_ids": ["7b2e4c10-5a9d-4f63-8e01-3c6d9a2f5b74"]
},
{
"criterion_id": "r3fa91c07b2de.c002",
"criterion_text": "Breaking changes are listed first, under a heading named \"Breaking changes\".",
"section": "Format",
"weight": 1,
"verdict": "fail",
"rationale": "The removal of the v1 export endpoint appears under \"API\", after two feature sections. There is no \"Breaking changes\" heading.",
"evidence_event_ids": ["7b2e4c10-5a9d-4f63-8e01-3c6d9a2f5b74"]
}
],
"grader_thread_id": "a3d5e7f9-0b2c-4d6e-8f1a-3b5c7d9e1f20",
"input_tokens": 18420,
"output_tokens": 912,
"cost_micros": 68950,
"started_at": "2026-09-17T10:21:30Z",
"ended_at": "2026-09-17T10:23:02Z"
},
{
"outcome_id": "e5c07a3b-1f94-4d26-b8a0-6d2c9e4f7b51",
"iteration": 1,
"result": "satisfied",
"explanation": "All 2 criteria met.",
"grader_thread_id": "c8e2a4f6-1d3b-4e5f-9a7c-2b4d6f8a0c13",
"started_at": "2026-09-17T10:30:11Z",
"ended_at": "2026-09-17T10:31:47Z"
}
]
}
]
}
```
| Field | What it tells you |
| -------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `status`, `terminal_result` | Whether grading is finished and how it ended. Poll until `status` is `terminal`. |
| `evaluations` | The grading passes, oldest first. `iteration` starts at `0`. Empty until the first pass ends. |
| `evaluations[].result` | The result of that pass. See the tables below. |
| `evaluations[].explanation` | A one-line summary, such as `1 of 2 criteria not met.` The agent receives the full scorecard, with every verdict and the reason for each failure. |
| `criteria[].verdict`, `criteria[].rationale` | The verdict for one criterion and the grader's reason. A `fail` always has a reason. Criteria come back in rubric order. |
| `criteria[].evidence_event_ids` | The grader's own tool calls in that pass, such as commands it ran and files it read. They are attached by the service, not written by the grader, so they cannot be invented. |
| `grader_thread_id` | The grader's thread for that pass. It appears in the session tree, so you can read everything the grader did. See [Events](/managed-agents/events). |
| `cost_micros` and token fields | What the pass cost, in millionths of a US dollar, and its tokens. See [Cost of grading](#cost-of-grading). |
When a session has no outcome, `outcomes` is `null`. Each pass also writes an `outcome_evaluation` event to the session, so you can follow the loop live on the [event stream](/managed-agents/events).
### Results and statuses
| Criterion verdict | Meaning |
| ----------------- | --------------------------------------------------------- |
| `pass` | The work meets the criterion. |
| `fail` | The work does not meet it. `rationale` says why. |
| `not_applicable` | The criterion does not apply to this work. Not a failure. |
| Pass result | When | What happens next |
| ------------------------ | ------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------- |
| `needs_revision` | At least one criterion failed and passes remain. | The agent revises, then the grader runs another pass. |
| `satisfied` | No criterion failed. | The outcome ends. The session is `completed` with `stop_reason` `outcome_satisfied`. |
| `max_iterations_reached` | A criterion failed on the last pass `max_iterations` allows. | The outcome ends. The session goes `idle` with `stop_reason` `max_iterations`. The last pass keeps its verdicts. |
| `failed` | The rubric could not be graded. | The outcome ends. The session goes `idle` with `stop_reason` `outcome_unmet`. Fix the rubric and define a new outcome. |
Every result except `needs_revision` is final, and becomes the outcome's `terminal_result`.
| Outcome `status` | Meaning |
| ---------------- | -------------------------------------------- |
| `pending` | No grading pass has run yet. |
| `running` | The agent is working toward the outcome. |
| `evaluating` | A grading pass is in progress. |
| `terminal` | Grading is finished. Read `terminal_result`. |
If the grader cannot produce a usable verdict, the session goes `idle` with `stop_reason` `grader_failed`. The work is not marked as failed and the outcome stays open. Send a message to continue; the next time the agent finishes, grading runs again.
After a final result, the session still accepts follow-up messages. Those turns are not graded. To grade more work, [add a new outcome](#add-an-outcome-to-a-running-session).
### Follow a criterion across sessions
A criterion id looks like `r3fa91c07b2de.c001`. The first part identifies the rubric and the last part is the criterion's position in it. The ids are stable: the same rubric text gives the same ids in every session, so you can track how one criterion fares across runs and agent versions. Any edit to the rubric, even a typo fix, gives every criterion a new id. A default rubric stored on the agent keeps its ids for as long as you do not change it.
## Cost of grading
Each grading pass is a model run with its own tokens, and its cost is counted like the agent's own work. Every pass reports its `cost_micros` and token counts. Grading passes are child sessions of the graded session, so their cost appears in the `subtree` and `tree` scopes of the model-cost reads, not in `self`. See [Usage and cost](/managed-agents/usage-and-cost).
To keep grading costs down, write criteria the grader can check quickly, and set `max_iterations` on rubrics you have not tested yet. An uncapped outcome whose rubric can never pass keeps revising and grading.
## What can go wrong
| Symptom or code | Cause | Fix |
| --------------------------------------------------------------- | -------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `400 invalid_request` on `outcome.description` | The objective is empty. | Send `description`. |
| `400 invalid_request` on `outcome.rubric` | The rubric is missing, has no list item, or has more than 200 criteria. | Write each criterion as a Markdown list item. Paragraphs alone are not criteria. Split very large rubrics. |
| `400 invalid_request` on `outcome.max_iterations` | The value is negative. | Send `0`, a positive integer, or nothing. |
| `400 invalid_request` on `outcome` from `defineSessionOutcome` | The session already has an outcome that is not `terminal`. | Wait for `terminal_result`, then define the next one. |
| `403 forbidden` | Your role can only read sessions. | See [Organizations and roles](/managed-agents/organizations-and-roles). Use an account or key with the developer or admin role. |
| `404 not_found` | The session does not exist or is outside your organization. | Check the session id and your API key's organization. |
| The deliverable criterion fails because the file can't be found | The agent saved the file outside the outputs folder, or under another name. | Name the deliverable file in both the task and the rubric. See [Deliverables and artifacts](/managed-agents/artifacts). |
| The session keeps revising | A criterion is vague or impossible, so it never passes. | Read the failing `rationale`, rewrite the criterion, and set `max_iterations`. |
| `stop_reason` is `grader_failed` | The grader did not return a usable verdict. The work was not judged. | Send a message to resume; grading runs again when the agent finishes. |
| `outcomes` is `null` | The session was started without an outcome, or with a default rubric but no `message`. | Send `outcome`, or send a `message` when relying on the agent's default rubric. |
| Criterion history does not line up across sessions | The rubric text differed, so the ids differ. | Store the rubric as the agent's `default_rubric` and reuse it unchanged. |
| `429 rate_limit_exceeded` | Too many requests. | Wait for the time in `Retry-After`, then retry. |
The full list of codes is in [Errors](/managed-agents/errors).
## Limits
| Limit | Value |
| ---------------------------- | ------------------------------------------------- |
| Criteria in one rubric | 1 to 200 |
| Open outcomes in one session | 1 |
| `max_iterations` | `0` (no cap, the default) or any positive integer |
| Criterion weights | Not settable. Every criterion has weight `1`. |
Other limits are on [Limits](/managed-agents/limits).
## Next steps
Ask for named deliverables the grader can check.
Follow each grading pass live and read what the grader did.
Store a default rubric so every session is graded.
See what the agent and the grader cost.