Evaluations

An evaluation scores sessions after they run. You define an evaluator, an agent whose job is to judge, with a rubric of keyed criteria. You pick the sessions to judge, and the evaluator reads each one’s transcript and records a verdict: pass, fail, or not_applicable, overall and per criterion, with a rationale and the events it cites as evidence. Evaluations differ from outcomes. An outcome grades one session while it runs and can send the agent back to revise. An evaluation judges many finished sessions at once, without changing them, so you can compare agents and versions. The samples assume the rl client from Client setup.

Before you begin

Creating an evaluator and starting an evaluation need create permission. Reading verdicts needs read. Organization developers and admins have both. See Organizations and roles.
You need finished sessions to judge.

How an evaluation runs

Each session is judged as it stood when the run started. Later messages don’t change its verdict.
A run is an ordinary root session of the evaluator, with one child per judged session, so you can read or stream it like any other. It isn’t listed with your other sessions.
Verdicts are immutable. Judging the same session again records a new verdict.
An evaluator’s version freezes its rubric, so verdicts from one version are comparable with each other.

Create an evaluator

An evaluator is an agent with the evaluation toolset and a rubric. The rubric has one criterion per line, as - key: what to judge, with a lowercase key. Keep a key when you reword its criterion, so its history stays comparable.

Console
TypeScript
cURL

In the sidebar, click Evaluations, then open Evaluators.
Start from a template, such as Core quality or Memory use, or from a blank definition.
Write the Evaluation criteria. The console shows the keys it parsed.
Under Run limits, set Concurrent target evaluations, from 1 to 25 (the default is 8), and an optional Maximum evaluation run cost (USD).
Save the evaluator.

const evaluator = await rl.managedAgents.createAgent({
  'Idempotency-Key': 'release-notes-evaluator-v1',
  body: {
    name: 'Release notes evaluator',
    model: 'anthropic/claude-sonnet-4-5',
    system: 'Judge whether each session produced accurate, complete release notes.',
    toolsets: [{ type: 'evaluation' }],
    default_rubric: '- accurate: Every change listed matches the merged pull requests.\n- complete: No merged user-facing change is missing.',
    multiagent: { type: 'coordinator', limits: { max_concurrent_threads: 8, max_tree_cost_usd: '5' } },
  },
});

curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/agents' \
  -H "Authorization: Bearer $RECURSION_API_KEY" \
  -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: release-notes-evaluator-v1' \
  -d '{
    "name": "Release notes evaluator",
    "model": "anthropic/claude-sonnet-4-5",
    "system": "Judge whether each session produced accurate, complete release notes.",
    "toolsets": [{"type": "evaluation"}],
    "default_rubric": "- accurate: Every change listed matches the merged pull requests.\n- complete: No merged user-facing change is missing.",
    "multiagent": {"type": "coordinator", "limits": {"max_concurrent_threads": 8, "max_tree_cost_usd": "5"}}
  }'

max_concurrent_threads is how many sessions are judged at once. With max_tree_cost_usd set, the run stops starting new judgments once its recorded cost reaches that amount; sessions it didn’t reach are reported as skipped. To see the template definitions, call listAgentTemplates; each is an ordinary create body you can copy.

Start an evaluation

Choose sessions by id, or by status with a limit. Only root sessions that are visible to you and have an agent can be judged.

Console
TypeScript
cURL

In the sidebar, click Sessions and select the sessions to judge, up to 100.
Click Evaluate selected, choose an evaluator, and start.
Follow the run under Evaluations > Runs.

const run = await rl.managedAgents.startSession({
  'Idempotency-Key': 'release-notes-eval-2026-09-25',
  body: {
    agent_id: '9e3b7c15-4a82-4d06-b1f9-6c2e8a5d3f70',
    evaluation: { statuses: ['completed'], limit: 20 },
  },
});

curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/sessions' \
  -H "Authorization: Bearer $RECURSION_API_KEY" \
  -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: release-notes-eval-2026-09-25' \
  -d '{
    "agent_id": "9e3b7c15-4a82-4d06-b1f9-6c2e8a5d3f70",
    "evaluation": {"session_ids": ["e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53", "b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38"]}
  }'

Set exactly one of session_ids (up to 100) or statuses with limit (1 to 100, newest matching sessions first). With session_ids, every session must be eligible, or no run is created. The response is the run’s root session; its session_id is the run_session_id on the run and its verdicts. What success means: the run session reaches completed, and listEvaluationRuns shows verdicts_total equal to the number of sessions it judged. A judged session that failed or was cancelled can still get a verdict; its failure_class says so.

Read verdicts

Console
TypeScript
cURL

Evaluations > Overview charts pass rates by agent version and by criterion for one evaluator.
Evaluations > Runs lists runs with their Targets, Verdicts, and Status.
A session’s Evaluation tab shows every verdict recorded for it.

const { evaluations, next_page_token } = await rl.managedAgents.listManagedAgentEvaluations({
  evaluator_agent_id: '9e3b7c15-4a82-4d06-b1f9-6c2e8a5d3f70',
  limit: 50,
});

curl 'https://api.recursion.labelbox.com/managed-agents/v1/evaluations?evaluator_agent_id=9e3b7c15-4a82-4d06-b1f9-6c2e8a5d3f70&limit=50' \
  -H "Authorization: Bearer $RECURSION_API_KEY"

{
  "evaluations": [
    {
      "evaluation_id": "0f6c3a92-8d14-4b57-a2e9-5c7b1d3f8e40",
      "target_session_id": "e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53",
      "target_agent_id": "5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55",
      "target_agent_version_id": "c1a94e07-2f6b-4d18-b3a5-0e7d8c6f4a21",
      "evaluator_agent_id": "9e3b7c15-4a82-4d06-b1f9-6c2e8a5d3f70",
      "evaluator_agent_version_id": "4d8a2e61-7c39-4f05-b3e1-9a6c2d5f8b17",
      "run_session_id": "6b2f9d47-1e83-4c50-a7d2-3f8e5b1c9a64",
      "result": "fail",
      "failure_class": "criterion_failure",
      "summary": "The notes omit one merged user-facing change.",
      "criteria": [
        { "criterion_key": "accurate", "verdict": "pass", "rationale": "Every listed change matches a merged pull request.", "evidence_event_ids": ["01929a7e-4c5b-7d31-9e8f-2a6b3c4d5e6f"] },
        { "criterion_key": "complete", "verdict": "fail", "rationale": "The retry fix is missing.", "evidence_event_ids": ["01929a7f-1b2c-7e43-8d9a-5b6c7d8e9f01"] }
      ],
      "created_at": "2026-09-25T12:10:33Z"
    }
  ],
  "next_page_token": "<opaque page token>"
}

Read	Call	Notes
Verdicts	`listManagedAgentEvaluations`	Newest first. Filter by `target_session_id`, `target_agent_id`, `evaluator_agent_id`, or `run_session_id`, combined with AND. 50 per page by default, up to 100.
Runs	`listEvaluationRuns`	Newest first, including active runs, with `targets_total`, `pass_count`, `fail_count`, `not_applicable_count`, and `verdicts_total`.
Overview	`getEvaluationOverview`	One evaluator version’s dashboard: pass-rate tiles, per-agent rows, and series by agent version or criterion. Needs `evaluator_agent_version_id`. `range` is `7d`, `28d` (the default), or `90d`, compared with the window before it.

Page tokens are tied to your filters and expire after an hour. Verdicts stay readable after the judged session or agent is deleted. For quality charts next to usage and cost, read the managed_agent_quality family in Analytics.

Write a rubric that judges well

One thing per criterion. A criterion that checks two things can’t tell you which failed.
Make it checkable from the transcript. The evaluator sees what the session did and said; name the file, command output, or claim to look for.
Keep keys stable. Rewording under the same key keeps history comparable; a new key starts a new series.
Say what’s not applicable. If a criterion doesn’t apply to some sessions, say when, so they’re not_applicable rather than fail.

What can go wrong

Code or symptom	Cause	Fix
`400 invalid_request` on `evaluation`	Both or neither of `session_ids` and `statuses` were sent, `limit` is missing or out of range, or more than 100 ids.	Send one selector within its limits.
`404 not_found` on start	A session id isn’t visible to you, or isn’t eligible.	Judge root sessions with an agent, from your organization.
A session can’t be selected in the console	It isn’t a root session, has no agent, or has no transcript yet.	Select its root session, or wait until it has run.
`400 invalid_request` on `toolsets`	A toolset other than `{"type": "evaluation"}`.	Send exactly that marker, or none for an ordinary agent.
Some sessions were never judged	The run reached `max_tree_cost_usd`.	Raise the cost cap, or judge fewer sessions per run.
`409 evaluation_cancellation_pending` or `409 evaluation_deletion_pending`	A run is still cleaning up.	Retry after it finishes.
Overview shows gaps	No verdicts in part of the window.	Coverage fields are `null` there rather than zero; widen the range or judge more sessions.

For every error code, see Errors.

Limits

Limit	Value
Sessions per run	100
Sessions judged at once	1 to 25, default 8
Verdicts per page	1 to 100, default 50
Overview agent rows per page	1 to 100, default 25
Overview comparison	Up to 5 agents and 12 versions
Page token lifetime	1 hour

See Limits for every other limit.

Next steps

Outcomes

Grade a session while it runs and let the agent revise.

Analytics

Chart quality with usage and cost.

Agents

Version agents so their quality is comparable.

Sessions

Find the sessions to judge.

Get started

Account and access

Build agents

Coordinate multiple agents

Configure environments

Run and observe sessions

Manage agent context

Automate

Tutorials

Before you begin

How an evaluation runs

Create an evaluator

Start an evaluation

Read verdicts

Write a rubric that judges well

What can go wrong

Limits

Next steps

Outcomes

Analytics

Agents

Sessions

​Before you begin

​How an evaluation runs

​Create an evaluator

​Start an evaluation

​Read verdicts

​Write a rubric that judges well

​What can go wrong

​Limits

​Next steps

Outcomes

Analytics

Agents

Sessions

Before you begin

How an evaluation runs

Create an evaluator

Start an evaluation

Read verdicts

Write a rubric that judges well

What can go wrong

Limits

Next steps