Skip to main content
Experimental. Fine-tuning’s operations, fields, and limits can change between releases.
This cookbook runs the whole loop once, end to end: an agent does a task on a base model, you fine-tune that model and score it against the untuned baseline, a second agent runs the same task on the tuned checkpoint, and you compare the two runs. Every request is shown as cURL, CLI, and TypeScript; the tab you pick stays selected. Set up your client as in Fine-tuning. Ids and numbers below come from a real run. The task is small on purpose, so a run takes minutes: classify five support tickets and save the result as triage.md.
You need fine-tuning and evaluations enabled for your organization, an environment for the sessions, and the Developer or Admin role: it covers creating agents, starting sessions, creating and cancelling fine-tuning jobs, and starting evaluations. See Fine-tuning for how fine-tuning is enabled. Without evaluations, starting evaluations in step 4 returns 400 invalid_request.
You need an environment for the sessions. Create one, or use any environment you already have, and keep its environmentId.

1. Run the task on the base model

Create an agent on the base model. Take the model id of Inkling 2 Small from the model list, under its maker; it is the same model the fine-tuning job in step 2 trains (the job names it thinkingmachines/Inkling-2-Small). See Agent setup.
The response carries agentId (here 5058a0fe-b1c9-43f7-b9bb-8eb7ab7b712c). Start a session with the task and the environment.
The session is accepted right away:
Read it until it leaves active; see Sessions. costState stays so_far until the sandbox stops at the environment’s idle stop (30 minutes by default) and its charges settle, which can take up to an hour more. Read the session again until it is final before you compare costs; see Usage and cost.
The agent saved triage.md with 1: bug, 2: billing, 3: how-to, 4: bug, 5: billing.

2. Fine-tune and read the scores

Create a short training run that saves a checkpoint after every step. A baseline job (steps: 0) scores the untuned model on its own; it is optional here, because a trained job already reports its untuned score as summary.evalReward.base. This cookbook creates both so you can see how much held-out scores vary between runs.
Follow each job as in Fine-tuning. Both ran in about seven minutes here. When the tuned job succeeds, read its scores: base is the untuned model’s score at step 0 and final the score after training.
The separate baseline job (776fd97b-c35c-54ef-aeda-506a39306036) scored the same untuned model on the same quick held-out set at 0.6317. That is 0.02 below the tuned job’s own base, with no training in between: held-out scores vary from run to run because the model samples its answers, and that variation is larger than this tune’s gain. Compare base with final inside one job, and treat a difference smaller than the run-to-run spread as noise. Two steps move the held-out reward by about 0.005, well inside that spread: this tune barely moves results, and the cookbook shows the mechanics, not a gain. Longer runs on the full task sets can move the score more; the spread above was measured on the 10-task quick held-out set, so it does not carry over to the 120-task full set, and scores on quick and full are not comparable. These full runs each trained 40 to 50 steps on the full train and held-out sets and gained 0.0125 to 0.04: Even these gains are modest, so compare against a baseline on the same held-out set before you rely on a tuned model. See Run a long job for how to run one.

3. Run the same task on the tuned checkpoint

List the checkpoints and copy the tunedModel of the one you want. Every saved checkpoint is a model; this run saved one after each step.
Create an agent whose model is that value, with the same instructions, and start the same task in the same environment.
Read the session as before. The tuned agent finished the task and saved the same triage.md; its session cost 0.004380146, final.

4. Compare the two runs

Both sessions are finished, so an evaluator agent can grade them against the same criteria. Any agent in your organization can grade; give it a model and instructions that judge a finished session. Pick a model id from the model list.
The response carries agentId (here f02dc7ae-83ad-450a-bd7f-b68047528523). Start one evaluation per session in one batch. See Outcome rubrics for how criteria are written. Starting a batch takes no Idempotency-Key and is not safe to retry: a retry starts another batch. A verdict appears in the list only when its evaluation finishes, so before sending the request again, list sessions with evaluationTargetSessionId=<target> and the evaluation kind, or list evaluation runs. Check each item of the response too: an item with errorCode did not start.
Each evaluation runs as its own session and takes a few minutes. List the verdicts for your evaluator agent, so other evaluations in the organization do not mix in; each item has the target session, the result, a summary, and a verdict with a rationale for each criterion.
Compare more than pass or fail: the verdict rationales, the held-out scores from step 2, and cost. Tuned-model turns are charged per token like any model. Once both sessions are final, their totals (here base 0.034016485, tuned 0.004380146) also include sandbox time, which differs between runs, so they do not isolate the model. To compare each session’s own token counts, use listSessionUsage; see Usage and cost. In the console, open Advanced, then Evaluations: Activity lists each evaluation run with its evaluating agent, progress, and result, and its Targets tab shows the verdict and the criteria it checked. To compare agents over time, use the Quality tab with the source set to Evaluations.

What can go wrong

  • 400 when starting evaluations. “Evaluations are not enabled for this organization.” (invalid_request) means evaluations are off. A 404 means an evaluator agent or session id is wrong. A session that has not finished comes back as an item with errorCode invalid_request: wait for it to finish, then start that one again.
  • 400 creating the tuned agent. The model id is not a tuned-model id: copy tunedModel exactly.
  • The tuned session ends failed with a model error. The id is well formed but names no checkpoint. Copy it again from the checkpoint list; see Fine-tuning.

Clean up

Delete what you created when you are done: the sessions, the base, tuned, and evaluator agents, and the environment if you made one. Fine-tuning jobs and their checkpoints are kept; there is no delete for a job. Evaluation verdicts stay after you delete the sessions.

Next steps

Fine-tuning

Every operation, with examples of the common ones.

Fine-tuning reference

Settings, states, and metrics.

Outcome rubrics

How criteria are written.

Limits

Every fine-tuning bound.