triage.md.
You need fine-tuning and evaluations enabled for your organization, an environment for the sessions, and the Developer or Admin role: it covers creating agents, starting sessions, creating and cancelling fine-tuning jobs, and starting evaluations. See Fine-tuning for how fine-tuning is enabled. Without evaluations, starting evaluations in step 4 returns
400 invalid_request.environmentId.
1. Run the task on the base model
Create an agent on the base model. Take the model id of Inkling 2 Small from the model list, under its maker; it is the same model the fine-tuning job in step 2 trains (the job names itthinkingmachines/Inkling-2-Small). See Agent setup.
- cURL
- CLI
- TypeScript
agentId (here 5058a0fe-b1c9-43f7-b9bb-8eb7ab7b712c). Start a session with the task and the environment.
- cURL
- CLI
- TypeScript
active; see Sessions. costState stays so_far until the sandbox stops at the environment’s idle stop (30 minutes by default) and its charges settle, which can take up to an hour more. Read the session again until it is final before you compare costs; see Usage and cost.
- cURL
- CLI
- TypeScript
triage.md with 1: bug, 2: billing, 3: how-to, 4: bug, 5: billing.
2. Fine-tune and read the scores
Create a short training run that saves a checkpoint after every step. A baseline job (steps: 0) scores the untuned model on its own; it is optional here, because a trained job already reports its untuned score as summary.evalReward.base. This cookbook creates both so you can see how much held-out scores vary between runs.
- cURL
- CLI
- TypeScript
- cURL
- CLI
- TypeScript
base is the untuned model’s score at step 0 and final the score after training.
- cURL
- CLI
- TypeScript
The separate baseline job (
776fd97b-c35c-54ef-aeda-506a39306036) scored the same untuned model on the same quick held-out set at 0.6317. That is 0.02 below the tuned job’s own base, with no training in between: held-out scores vary from run to run because the model samples its answers, and that variation is larger than this tune’s gain. Compare base with final inside one job, and treat a difference smaller than the run-to-run spread as noise.
Two steps move the held-out reward by about 0.005, well inside that spread: this tune barely moves results, and the cookbook shows the mechanics, not a gain. Longer runs on the full task sets can move the score more; the spread above was measured on the 10-task quick held-out set, so it does not carry over to the 120-task full set, and scores on quick and full are not comparable. These full runs each trained 40 to 50 steps on the full train and held-out sets and gained 0.0125 to 0.04:
Even these gains are modest, so compare against a baseline on the same held-out set before you rely on a tuned model. See Run a long job for how to run one.
3. Run the same task on the tuned checkpoint
List the checkpoints and copy thetunedModel of the one you want. Every saved checkpoint is a model; this run saved one after each step.
- cURL
- CLI
- TypeScript
- cURL
- CLI
- TypeScript
- cURL
- CLI
- TypeScript
triage.md; its session cost 0.004380146, final.
- cURL
- CLI
- TypeScript
4. Compare the two runs
Both sessions are finished, so an evaluator agent can grade them against the same criteria. Any agent in your organization can grade; give it a model and instructions that judge a finished session. Pick a model id from the model list.- cURL
- CLI
- TypeScript
agentId (here f02dc7ae-83ad-450a-bd7f-b68047528523). Start one evaluation per session in one batch. See Outcome rubrics for how criteria are written. Starting a batch takes no Idempotency-Key and is not safe to retry: a retry starts another batch. A verdict appears in the list only when its evaluation finishes, so before sending the request again, list sessions with evaluationTargetSessionId=<target> and the evaluation kind, or list evaluation runs. Check each item of the response too: an item with errorCode did not start.
- cURL
- CLI
- TypeScript
result, a summary, and a verdict with a rationale for each criterion.
- cURL
- CLI
- TypeScript
Compare more than pass or fail: the verdict rationales, the held-out scores from step 2, and cost. Tuned-model turns are charged per token like any model. Once both sessions are
final, their totals (here base 0.034016485, tuned 0.004380146) also include sandbox time, which differs between runs, so they do not isolate the model. To compare each session’s own token counts, use listSessionUsage; see Usage and cost. In the console, open Advanced, then Evaluations: Activity lists each evaluation run with its evaluating agent, progress, and result, and its Targets tab shows the verdict and the criteria it checked. To compare agents over time, use the Quality tab with the source set to Evaluations.
What can go wrong
400when starting evaluations. “Evaluations are not enabled for this organization.” (invalid_request) means evaluations are off. A404means an evaluator agent or session id is wrong. A session that has not finished comes back as an item witherrorCodeinvalid_request: wait for it to finish, then start that one again.400creating the tuned agent. The model id is not a tuned-model id: copytunedModelexactly.- The tuned session ends
failedwith a model error. The id is well formed but names no checkpoint. Copy it again from the checkpoint list; see Fine-tuning.
Clean up
Delete what you created when you are done: the sessions, the base, tuned, and evaluator agents, and the environment if you made one. Fine-tuning jobs and their checkpoints are kept; there is no delete for a job. Evaluation verdicts stay after you delete the sessions.Next steps
Fine-tuning
Every operation, with examples of the common ones.
Fine-tuning reference
Settings, states, and metrics.
Outcome rubrics
How criteria are written.
Limits
Every fine-tuning bound.