> ## Documentation Index
> Fetch the complete documentation index at: https://docs.labelbox.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Fine-tuning reference

> Every fine-tuning setting, state, phase, timeline event, metric, and paging rule, with defaults and limits.

> [!WARNING]
>
> **Experimental.** Fine-tuning's operations, fields, and limits can change between releases.

Use this page to look up values you send to or read from the fine-tuning operations. For how a run works, see [Fine-tuning](https://docs.labelbox.com/recursion/fine-tuning.md).

## Models and dataset

| Value | Meaning |
| - | - |
| `thinkingmachines/Inkling-2-Small` | Base model. Maximum context 262144 tokens. |
| `nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16` | Base model. Maximum context 262144 tokens. |
| `nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16` | Base model. Maximum context 262144 tokens. |
| `automationbench-public-600` | The one dataset: 600 scored automation tasks, 480 train and 120 held-out, split with a fixed seed. `datasetProvenance` on a job records the dataset revision, the split seed, and the task counts. |

## Settings

Every setting is optional on create and is validated against its bound. Omit a setting to take its default: an explicit `null` for `name`, `settings`, or a setting, an unknown setting, or a value of the wrong type returns `400 invalid_request`. A job reports the settings it runs with in `settings`.

| Setting | Default | Bounds | What it does |
| - | - | - | - |
| `steps` | 2 | 0 to 2147483647 | Training steps. 0 is a baseline: it trains nothing and saves no checkpoint. An attempt ends at its 24-hour limit and the run continues from its latest checkpoint, so a run ends when it finishes its steps, stops early, or is cancelled. An attempt that ends without a checkpoint newer than the one it started from is stalled; the third stalled attempt in a row fails the job. |
| `groupsPerBatch` | 4 | 1 to 32 | Task groups per step. |
| `groupSize` | 4 | 2 to 16 | Rollouts per task. Rewards are compared inside a group. |
| `learningRate` | 0.00004 | 1e-7 to 0.001 | Optimizer learning rate. |
| `loraRank` | 32 | 8, 16, 32, 64 | Rank of the adapter that is trained. |
| `contextLength` | the model's maximum, 262144 | 256 to 262144 | Maximum tokens of one rollout's context. The one budget that ends a long rollout. |
| `trainSet` | `quick` | `quick`, `full` | Train tasks: `quick` is the 8 easiest, `full` is all 480, cycled. |
| `evalSet` | `quick` | `quick`, `full` | Held-out tasks per check: `quick` is the 10 easiest, `full` is all 120. |
| `evalTasks` | none | 1 to 120 | The easiest N held-out tasks, in place of `evalSet`. |
| `evalEvery` | 1 | 1 to `steps` (1 when `steps` is 0) | Steps between held-out checks. |
| `evalAttempts` | 4 | 1 to 16 | Attempts per held-out task, for pass\@k and pass^k. |
| `checkpointEvery` | 10 | 1 to 2147483647 | Steps between saved checkpoints. The last step is always saved. |
| `earlyStopPatience` | none (never stop early) | 1 to 2147483647 | Stop after this many held-out checks in a row without an improvement. The job succeeds with `result.endReason` `early_stop`. |
| `earlyStopMinDelta` | 0 | 0 to 1 | The gain over the best score that counts as an improvement. Needs `earlyStopPatience`. |
| `vcpus` | 8 | 1 to 56 | vCPUs of the job's compute, with 4 GiB of memory each. A deployment may admit fewer than 56; a larger value is refused with an error that names the limit. |
| `maxInFlightRollouts` | 256 | 2 to 4096, and at least the larger of `groupSize` and `evalAttempts` | Rollouts running at once, training and held-out checks together. It does not follow `vcpus`. |
| `samplingStepsAhead` | 0 | 0 to 8 | Steps sampling may run ahead of training. 0 is on-policy; above 0 the run trains on slightly stale samples and goes faster. A baseline accepts only 0. |
| `sampleTimeoutMinutes` | 30 | 1 or more | Minutes one sampling call may take. A call that times out or fails with a connection error is issued again, three attempts in all, before its rollout fails and is left out of its group. A value over 30 also lengthens the no-progress limit after which an attempt ends and restarts. |
| `rolloutTimeoutMinutes` | 240 | 1 or more | Minutes one rollout may take before it fails and is left out of its group. |

A step or a held-out check that loses more than half of its rollouts is a bad round; three bad rounds in a row (one for a baseline) fail the attempt.

## Job status and phase

| `status` | Meaning |
| - | - |
| `queued` | Waiting for its first compute. `startedAt` is `null`. |
| `running` | An attempt works, or the job waits for its next compute after a restart or resume. |
| `succeeded` | Final. `result.endReason` is `steps_completed` or `early_stop`; `result.tunedModel` is set, except for a baseline, where it stays `null`. |
| `failed` | Settled. `failure` says why. `resumeFineTuningJob` runs it again. |
| `cancelled` | Settled by cancel. `resumeFineTuningJob` runs it again. |

| `progress.phase` | Meaning |
| - | - |
| `queued` | Waiting for compute, from creation and again between attempts. |
| `provisioning` | The compute is being created. |
| `installing`, `setup` | The compute is being prepared for training. |
| `sampling` | Rollouts of the current step are running. `progress.split` is `train`. |
| `training` | The step's update is running. |
| `eval` | The final held-out check is running (after the last step, or a baseline's check). Earlier checks run during `sampling` and `training` and do not change the phase. |
| `checkpoint` | A checkpoint is being saved. |
| `finalizing` | The final result is being collected. |
| `done`, `failed`, `cancelled` | The job settled in that state. |

`progress.lastEventAt` is the last time the job reported anything, heartbeats included. `progress.rollouts` counts the current step's rollouts by status. `progress.ingestErrors` counts progress updates from the run that could not be read and were skipped.

## Timeline events

`listFineTuningJobEvents` returns the timeline in order, filtered by `type`. Each event has `seq`, `occurredAt`, and `type`; the other fields apply by type and are `null` otherwise.

| `type` | Meaning | Fields set |
| - | - | - |
| `phase` | The job entered a phase. | `phase`, `step`, `split` |
| `heartbeat` | A periodic sign of life. | `phase`, `step`, `inFlight`, `done`, `total` |
| `step` | A training step finished, or the final held-out check was written. Training steps count from 0; the last `step` event, at `steps` (0 for a baseline), marks the final check. A checkpoint's `step` counts updates, so checkpoint 10 arrives beside `step` event 9. | `step` |
| `checkpoint` | A checkpoint was saved at `step`, or, with a `reason`, could not be saved and training went on. | `step`, `reason` |
| `resume` | A new attempt started from the checkpoint at `step`, or from the beginning when `step` is `null`. | `attempt`, `step`, `reason` (why the attempt before ended, or "The job was resumed on request." for a manual resume) |
| `ingest_error` | A progress update from the run could not be read and was skipped. The run continues. | `reason`, `offset` |

## Rollouts and groups

| Field | Meaning |
| - | - |
| `rolloutId` | `<split>-<step>-<group>-<sample>`. |
| `split` | `train` or `eval`. |
| `status` | `queued`, `running`, `done`, or `failed`. |
| `reward`, `pass` | The task's reward from 0 to 1 and whether every check passed; `null` before it is done. |
| `stopReason` | Why a finished rollout stopped: `completed` (the model finished), `context_full` (the context budget ran out), or `parse_failure` (its replies could not be parsed). Rollouts have no turn cap, so there is no turn-limit reason. `null` before it is done. |
| `error` | Why a failed rollout failed. |
| `turns`, `promptTokens`, `sampledTokens` | Model turns and tokens so far. |

A group is the rollouts of one task in one step. `listFineTuningJobRolloutGroups` needs `split` and `step`, and filters by `spread`, `status`, and task name `q`, sorted by `sortBy` (`group`, `task`, `meanReward`, `stdReward`) and `sortDirection`. `spread` is `flat` when every scored rollout got the same reward, so the group teaches nothing, and `mixed` otherwise. It is `null` while any rollout of the group is queued or running, or when none is scored, so `spread=flat` on a running step returns fewer groups than you might expect.

For `listFineTuningJobRollouts`, `step` needs `split`, `status` needs `split` and `step`, and `updatedAtGte` takes no other filter.

## Metrics

Metric names are exactly as reported. `train/` metrics come from a step's rollouts, `eval/` from a held-out check.

| Name | Meaning | Unit |
| - | - | - |
| `train/reward`, `eval/reward` | Mean reward over the rollouts. | 0 to 1 |
| `train/pass_rate`, `eval/pass_rate` | Share of rollouts passing every check. | 0 to 1 |
| `train/reward_min`, `train/reward_max` (also `eval/`) | Lowest and highest reward. | 0 to 1 |
| `train/reward_hist/b00` to `b09` (also `eval/`) | Rollouts with reward in \[0, 0.1), ..., \[0.9, 1.0]. | count |
| `train/flat_group_frac`, `train/mixed_group_frac` | Share of groups whose rollouts all scored the same, and share whose rewards differ. | 0 to 1 |
| `train/group_reward_std/mean` | Mean reward standard deviation inside groups. | reward |
| `train/reach` | Share of the step's groups with at least one reward above 0. | 0 to 1 |
| `train/coverage/seen`, `/reached` | Cumulative share of the selected train tasks sampled, and reached with any reward. | 0 to 1 |
| `train/stop/completed_frac`, `/context_full_frac`, `/parse_failure_frac` (also `eval/`) | How rollouts ended. | 0 to 1 |
| `train/failed_rollouts`, `train/failed_rollout_frac` (also `eval/`) | Rollouts that failed, and their share. | count, 0 to 1 |
| `train/turns_per_attempt` (also `eval/`) | Mean model turns per rollout. | turns |
| `train/tokens/sampled_total`, `/prompt_total` | Generated and prompt tokens in the step. | tokens |
| `train/pass_at/<a>/k<k>`, `train/pass_all/<a>/k<k>` (also `eval/`) | Pass\@k (at least one of k attempts succeeds) and pass^k (all k succeed), per task, averaged over tasks. `<a>` is the reward threshold for a success: `a100`, `a075`, or `a050`. | 0 to 1 |
| `eval/radar/reward`, `pass_at_1`, `pass_at_k`, `pass_all_k`, `completion`, `efficiency`, `cost_efficiency` | Held-out summary scores; k is `evalAttempts`. | 0 to 1 |
| `cost/sampling_usd`, `cost/training_usd`, `cost/eval_usd` | The step's cost by part. | USD |
| `cost/step_usd`, `cost/total_usd` | The step's cost, and the run's cost so far. The live estimate. | USD |
| `optim/loss`, `optim/entropy`, `optim/lr` | Loss, mean entropy of sampled tokens, and learning rate of the step. | |
| `time/total`, `time/sampling`, `time/train_step`, `time/eval` | Wall time of the step and its parts. | seconds |

A run can report further metrics; read any of them with the `name` parameter.

## Checkpoints

`listFineTuningJobCheckpoints` returns the checkpoints by step, and a job reports its latest as `latestCheckpoint`.

| Field | Meaning |
| - | - |
| `checkpointId` | The checkpoint's opaque id. Treat it as a string to copy. |
| `step` | The weights after this many training steps. |
| `attempt` | The attempt that last saved it. |
| `savedAt` | When it was saved. |
| `tunedModel` | The model id an agent selects to run these weights: `tuned/` followed by the `checkpointId`. |

## Cost states

| `billedCostStatus` | Meaning |
| - | - |
| `pending` | The cost is still being worked out and can change. Read the job again later; `billedCostReason` says why, and `estimatedCostUsd` is the best estimate meanwhile. |
| `final` | The cost will not change unless you resume the job: a resume clears it, and the resumed run's settlement covers every attempt. `billedCostUsd` is the amount to read. |
| `unavailable` | No reliable cost could be determined. `billedCostReason` says why. |
| `null` | The job has not settled, or it never ran. |

`billedCostUsd` is a usage estimate for the run, not a bill. The live estimate while a run works is the `cost/total_usd` metric.

## Paging, following, and idempotency

| Operation | Page size (default, maximum) | Follow with |
| - | - | - |
| `listFineTuningJobs` | 20, 100 | not followable: newest first |
| `listFineTuningJobMetrics` | 500, 2000 | `stepGte` from the step of the last held-out (`eval/`) point you read, since held-out points can land after later training steps; `name` up to 50 times |
| `listFineTuningJobRollouts` | 200, 1000 | `updatedAtGte` |
| `listFineTuningJobRolloutGroups` | 20, 200 | read the same page again |
| `listFineTuningJobEvents` | 200, 1000 | `seqGt` |
| `listFineTuningJobCheckpoints` | 100, 1000 | `stepGt`; when `latestCheckpoint.attempt` changes, read again from the start and replace rows by `step` |

Pass `nextCursor` back as `cursor` with the same filters and `limit` until it is `null`. A cursor expires after one hour, and a rollout-groups walk through a step that is still running can skip or repeat groups; it is complete only once the step has no queued or running rollouts. `total` is always `null` except for rollout groups.

`createFineTuningJob` requires `Idempotency-Key`. Renaming is safe to repeat. `cancelFineTuningJob` accepts an optional `Idempotency-Key`; send one so a retry that lands after a resume cannot cancel the resumed run. `resumeFineTuningJob` is not safe to retry blindly: a retry after the resumed run settles starts another attempt, which adds training cost, so read the job first.

## Errors

| Code | When |
| - | - |
| `400 invalid_request` | A field or query parameter that is missing, unknown, repeated, malformed, out of range, or in an invalid combination (`details.field` names it); a create or rename body with an unknown field or setting, an explicit `null` for `name`, `settings`, or a setting, a value of the wrong type, or a body that is not valid JSON; a `cursor` that is invalid or expired; or the organization is not enabled for fine-tuning. Fine-tuning rules: an unknown `model` or `dataset`, a `name` that is empty after trimming, over 120 characters or holds a control character, `earlyStopMinDelta` without `earlyStopPatience`, a `jobId` that is not a UUID, and a baseline (`steps: 0`) that sets `earlyStopPatience` or a `samplingStepsAhead` above 0. |
| `404 not_found` | No such job in the organization. |
| `409 conflict` | Cancelling a succeeded or failed job, resuming a succeeded job, or reusing a create `Idempotency-Key` after its receipt expired. |
| `409 fine_tuning_job_stopping` | Resuming a job whose last run is still being stopped. Retry in a minute. |
| `409 idempotency_conflict`, `409 idempotency_in_progress` | The `Idempotency-Key` rules of [API conventions](https://docs.labelbox.com/recursion/api.md#idempotent-mutations). |

A failed job's `failure.reason` is one line, such as `The training run reached its 24-hour limit.`, `The training compute stopped.`, or the run's own error. See [Troubleshooting](https://docs.labelbox.com/recursion/troubleshooting.md#fine-tuning).
