> ## Documentation Index
> Fetch the complete documentation index at: https://docs.labelbox.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Evaluations
> Score finished sessions against a keyed rubric with an evaluator agent, read pass and fail verdicts with evidence, and track quality by agent version over time.
An **evaluation** scores sessions after they run. You define an **evaluator**, an agent whose job is to judge, with a rubric of keyed criteria. You pick the sessions to judge, and the evaluator reads each one's transcript and records a verdict: `pass`, `fail`, or `not_applicable`, overall and per criterion, with a rationale and the events it cites as evidence.
Evaluations differ from [outcomes](/managed-agents/outcomes). An outcome grades one session while it runs and can send the agent back to revise. An evaluation judges many finished sessions at once, without changing them, so you can compare agents and versions.
The samples assume the `rl` client from [Client setup](/managed-agents/api#client-setup).
## Before you begin
* Creating an evaluator and starting an evaluation need create permission. Reading verdicts needs read. Organization developers and admins have both. See [Organizations and roles](/managed-agents/organizations-and-roles).
* You need finished sessions to judge.
## How an evaluation runs
```mermaid theme={"theme":"css-variables"}
flowchart LR
pick["Pick sessions"] --> run["Evaluation run: one root session"]
run --> t1["Judge session 1"]
run --> t2["Judge session 2"]
run --> tn["Judge session N"]
t1 --> verdicts["Verdicts with rationale and evidence"]
t2 --> verdicts
tn --> verdicts
verdicts --> overview["Overview by agent version"]
```
* Each session is judged as it stood when the run started. Later messages don't change its verdict.
* A run is an ordinary root session of the evaluator, with one child per judged session, so you can read or stream it like any other. It isn't listed with your other sessions.
* Verdicts are immutable. Judging the same session again records a new verdict.
* An evaluator's version freezes its rubric, so verdicts from one version are comparable with each other.
## Create an evaluator
An evaluator is an agent with the evaluation toolset and a rubric. The rubric has one criterion per line, as `- key: what to judge`, with a lowercase key. Keep a key when you reword its criterion, so its history stays comparable.
1. In the sidebar, click **Evaluations**, then open **Evaluators**.
2. Start from a template, such as **Core quality** or **Memory use**, or from a blank definition.
3. Write the **Evaluation criteria**. The console shows the keys it parsed.
4. Under **Run limits**, set **Concurrent target evaluations**, from 1 to 25 (the default is 8), and an optional **Maximum evaluation run cost (USD)**.
5. Save the evaluator.
```typescript theme={"theme":"css-variables"}
const evaluator = await rl.managedAgents.createAgent({
'Idempotency-Key': 'release-notes-evaluator-v1',
body: {
name: 'Release notes evaluator',
model: 'anthropic/claude-sonnet-4-5',
system: 'Judge whether each session produced accurate, complete release notes.',
toolsets: [{ type: 'evaluation' }],
default_rubric: '- accurate: Every change listed matches the merged pull requests.\n- complete: No merged user-facing change is missing.',
multiagent: { type: 'coordinator', limits: { max_concurrent_threads: 8, max_tree_cost_usd: '5' } },
},
});
```
```bash theme={"theme":"css-variables"}
curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/agents' \
-H "Authorization: Bearer $RECURSION_API_KEY" \
-H 'Content-Type: application/json' \
-H 'Idempotency-Key: release-notes-evaluator-v1' \
-d '{
"name": "Release notes evaluator",
"model": "anthropic/claude-sonnet-4-5",
"system": "Judge whether each session produced accurate, complete release notes.",
"toolsets": [{"type": "evaluation"}],
"default_rubric": "- accurate: Every change listed matches the merged pull requests.\n- complete: No merged user-facing change is missing.",
"multiagent": {"type": "coordinator", "limits": {"max_concurrent_threads": 8, "max_tree_cost_usd": "5"}}
}'
```
`max_concurrent_threads` is how many sessions are judged at once. With `max_tree_cost_usd` set, the run stops starting new judgments once its recorded cost reaches that amount; sessions it didn't reach are reported as skipped. To see the template definitions, call `listAgentTemplates`; each is an ordinary create body you can copy.
## Start an evaluation
Choose sessions by id, or by status with a limit. Only root sessions that are visible to you and have an agent can be judged.
1. In the sidebar, click **Sessions** and select the sessions to judge, up to 100.
2. Click **Evaluate selected**, choose an evaluator, and start.
3. Follow the run under **Evaluations** > **Runs**.
```typescript theme={"theme":"css-variables"}
const run = await rl.managedAgents.startSession({
'Idempotency-Key': 'release-notes-eval-2026-09-25',
body: {
agent_id: '9e3b7c15-4a82-4d06-b1f9-6c2e8a5d3f70',
evaluation: { statuses: ['completed'], limit: 20 },
},
});
```
```bash theme={"theme":"css-variables"}
curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/sessions' \
-H "Authorization: Bearer $RECURSION_API_KEY" \
-H 'Content-Type: application/json' \
-H 'Idempotency-Key: release-notes-eval-2026-09-25' \
-d '{
"agent_id": "9e3b7c15-4a82-4d06-b1f9-6c2e8a5d3f70",
"evaluation": {"session_ids": ["e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53", "b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38"]}
}'
```
Set exactly one of `session_ids` (up to 100) or `statuses` with `limit` (1 to 100, newest matching sessions first). With `session_ids`, every session must be eligible, or no run is created. The response is the run's root session; its `session_id` is the `run_session_id` on the run and its verdicts.
**What success means:** the run session reaches `completed`, and `listEvaluationRuns` shows `verdicts_total` equal to the number of sessions it judged. A judged session that failed or was cancelled can still get a verdict; its `failure_class` says so.
## Read verdicts
* **Evaluations** > **Overview** charts pass rates by agent version and by criterion for one evaluator.
* **Evaluations** > **Runs** lists runs with their **Targets**, **Verdicts**, and **Status**.
* A session's **Evaluation** tab shows every verdict recorded for it.
```typescript theme={"theme":"css-variables"}
const { evaluations, next_page_token } = await rl.managedAgents.listManagedAgentEvaluations({
evaluator_agent_id: '9e3b7c15-4a82-4d06-b1f9-6c2e8a5d3f70',
limit: 50,
});
```
```bash theme={"theme":"css-variables"}
curl 'https://api.recursion.labelbox.com/managed-agents/v1/evaluations?evaluator_agent_id=9e3b7c15-4a82-4d06-b1f9-6c2e8a5d3f70&limit=50' \
-H "Authorization: Bearer $RECURSION_API_KEY"
```
```json theme={"theme":"css-variables"}
{
"evaluations": [
{
"evaluation_id": "0f6c3a92-8d14-4b57-a2e9-5c7b1d3f8e40",
"target_session_id": "e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53",
"target_agent_id": "5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55",
"target_agent_version_id": "c1a94e07-2f6b-4d18-b3a5-0e7d8c6f4a21",
"evaluator_agent_id": "9e3b7c15-4a82-4d06-b1f9-6c2e8a5d3f70",
"evaluator_agent_version_id": "4d8a2e61-7c39-4f05-b3e1-9a6c2d5f8b17",
"run_session_id": "6b2f9d47-1e83-4c50-a7d2-3f8e5b1c9a64",
"result": "fail",
"failure_class": "criterion_failure",
"summary": "The notes omit one merged user-facing change.",
"criteria": [
{ "criterion_key": "accurate", "verdict": "pass", "rationale": "Every listed change matches a merged pull request.", "evidence_event_ids": ["01929a7e-4c5b-7d31-9e8f-2a6b3c4d5e6f"] },
{ "criterion_key": "complete", "verdict": "fail", "rationale": "The retry fix is missing.", "evidence_event_ids": ["01929a7f-1b2c-7e43-8d9a-5b6c7d8e9f01"] }
],
"created_at": "2026-09-25T12:10:33Z"
}
],
"next_page_token": ""
}
```
| Read | Call | Notes |
| -------- | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Verdicts | `listManagedAgentEvaluations` | Newest first. Filter by `target_session_id`, `target_agent_id`, `evaluator_agent_id`, or `run_session_id`, combined with AND. 50 per page by default, up to 100. |
| Runs | `listEvaluationRuns` | Newest first, including active runs, with `targets_total`, `pass_count`, `fail_count`, `not_applicable_count`, and `verdicts_total`. |
| Overview | `getEvaluationOverview` | One evaluator version's dashboard: pass-rate tiles, per-agent rows, and series by agent version or criterion. Needs `evaluator_agent_version_id`. `range` is `7d`, `28d` (the default), or `90d`, compared with the window before it. |
Page tokens are tied to your filters and expire after an hour. Verdicts stay readable after the judged session or agent is deleted. For quality charts next to usage and cost, read the `managed_agent_quality` family in [Analytics](/managed-agents/analytics).
## Write a rubric that judges well
* **One thing per criterion.** A criterion that checks two things can't tell you which failed.
* **Make it checkable from the transcript.** The evaluator sees what the session did and said; name the file, command output, or claim to look for.
* **Keep keys stable.** Rewording under the same key keeps history comparable; a new key starts a new series.
* **Say what's not applicable.** If a criterion doesn't apply to some sessions, say when, so they're `not_applicable` rather than `fail`.
## What can go wrong
| Code or symptom | Cause | Fix |
| -------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
| `400 invalid_request` on `evaluation` | Both or neither of `session_ids` and `statuses` were sent, `limit` is missing or out of range, or more than 100 ids. | Send one selector within its limits. |
| `404 not_found` on start | A session id isn't visible to you, or isn't eligible. | Judge root sessions with an agent, from your organization. |
| A session can't be selected in the console | It isn't a root session, has no agent, or has no transcript yet. | Select its root session, or wait until it has run. |
| `400 invalid_request` on `toolsets` | A toolset other than `{"type": "evaluation"}`. | Send exactly that marker, or none for an ordinary agent. |
| Some sessions were never judged | The run reached `max_tree_cost_usd`. | Raise the cost cap, or judge fewer sessions per run. |
| `409 evaluation_cancellation_pending` or `409 evaluation_deletion_pending` | A run is still cleaning up. | Retry after it finishes. |
| Overview shows gaps | No verdicts in part of the window. | Coverage fields are `null` there rather than zero; widen the range or judge more sessions. |
For every error code, see [Errors](/managed-agents/errors).
## Limits
| Limit | Value |
| ---------------------------- | ------------------------------ |
| Sessions per run | 100 |
| Sessions judged at once | 1 to 25, default 8 |
| Verdicts per page | 1 to 100, default 50 |
| Overview agent rows per page | 1 to 100, default 25 |
| Overview comparison | Up to 5 agents and 12 versions |
| Page token lifetime | 1 hour |
See [Limits](/managed-agents/limits) for every other limit.
## Next steps
Grade a session while it runs and let the agent revise.
Chart quality with usage and cost.
Version agents so their quality is comparable.
Find the sessions to judge.