# Alignerr Connect extension
Source: https://docs.labelbox.com/alignerr-connect-extension



Some external projects now require the **Alignerr Connect Extension** in order to work on them. When a project has this setting enabled, you (the Alignerr) must install the extension before you can start working on that project.

This guide covers what the extension does, how to install it, and what to expect once it's active.

## **What is the Alignerr Connect extension?**

On the surface, working with the extension installed looks and feels the same as before. There's no change to your day-to-day workflow or interface.

Under the hood, though, the extension adds a security check: **it detects and blocks anonymized network connections** (VPN) when you try to access certain project-related sites.

If the extension detects that your connection is being routed through a VPN, it will block access to the project.

To resolve this, simply turn off your VPN (or otherwise switch to a direct, non-anonymized connection). Once your connection is no longer flagged as anonymized, access is restored automatically. There's no need to contact support or wait for anything on the backend.

## **How to install the extension**

1. Open the project.
2. If VPN detection is enabled for your project, you’ll see a prompt to install the extension. Click **Install Alignerr Connect**.
3. This takes you to the Chrome Web Store listing for the Alignerr Connect Extension.
4. Click **Add to Chrome**.
5. Once installed, the install prompt on the project page will disappear, and you'll be able to access and work on the project as normal.

That's it, no additional configuration is required. Once installed, the extension runs in the background and only becomes noticeable if it detects an anonymized connection.

## **What to do if you use a VPN**

If you normally work using a VPN, proxy, Tor, or a hosting network, be aware:

* Access to certain external project tools may be blocked while the extension detects that kind of connection.
* To regain access, disable the VPN/proxy or switch to a direct, unmasked connection.
* If you retry the site while still on a flagged connection, it will continue to block access — the fix is changing your connection, not retrying.

## **FAQ**

**Q: Will installing the extension change how I work?** A: No. On the surface, nothing changes. The only difference is that anonymized connections via VPN will be blocked from accessing certain project tools.

**Q: I use a VPN for work, what should I do?** You'll need to disable the VPN (or use a direct connection) when accessing tools on projects that require the extension.

**Q: Where do I install the extension?** A: Through the **Install Extension** link on the project page, which takes you to the Chrome Web Store.

**Q: What if I'm blocked and I'm not using a VPN?** A: Double-check your network settings. Some hosting networks or proxies can trigger this detection even if you don't think of yourself as using a VPN. If you believe you're being blocked incorrectly, reach out to support.*.*

<br />


# Create a new immutable agent version
Source: https://docs.labelbox.com/api-reference/agents/create-a-new-immutable-agent-version

/recursion/openapi.yaml post /managed-agents/v1/agents/{agentId}/versions
Creates a new immutable agent version from a full definition and moves the agent's latest pointer to it, in one atomic operation. base_agent_version_id must be the agent's current latest_agent_version_id or the request is rejected with 409 revision_conflict, so the caller can re-read and retry rather than silently racing another writer. Every field is full content, there is no partial-update or omit-to-inherit semantics, so an omitted field is cleared rather than carried over from the base version. Tags are changed only through the dedicated agent tag routes. Invalid promptCacheTTL or promptCacheSharedTTL settings in model_config.provider_params return 400 when prompt caching is enabled. Invalid model_config and unsupported reasoning_effort values return 400, including when the model catalog lists no exact options for the selected model; a model catalog that cannot be read to verify them returns 503. The response's execution definition and latest_agent_version_id are the exact values this call committed; created_at and updated_at are the authoritative persisted timestamps; tags are mutable catalog metadata and are not changed by this route. Do not automatically retry an ambiguous transport failure: re-read the agent's latest version first, because this operation does not replay a prior response.



# Create an agent and immutable version
Source: https://docs.labelbox.com/api-reference/agents/create-an-agent-and-immutable-version

/recursion/openapi.yaml post /managed-agents/v1/agents



# Get one agent
Source: https://docs.labelbox.com/api-reference/agents/get-one-agent

/recursion/openapi.yaml get /managed-agents/v1/agents/{agentId}
Returns one agent resolved to its latest immutable version, including its system prompt and default credential grants. A soft-deleted or unknown id is a 404.



# Get one file of an agent's file history
Source: https://docs.labelbox.com/api-reference/agents/get-one-file-of-an-agents-file-history

/recursion/openapi.yaml get /managed-agents/v1/agents/{agentId}/files/{fileId}
Returns one session output from this agent's file history. A file of another agent, of another organization, or outside the history answers 404.



# Get one immutable agent version
Source: https://docs.labelbox.com/api-reference/agents/get-one-immutable-agent-version

/recursion/openapi.yaml get /managed-agents/v1/agents/{agentId}/versions/{agentVersionId}
Returns one immutable agent version by id. The version must belong to the agent in the path; a mismatch is a 404 rather than a cross-agent read.



# Get what an agent's skills cost the prompt
Source: https://docs.labelbox.com/api-reference/agents/get-what-an-agents-skills-cost-the-prompt

/recursion/openapi.yaml get /managed-agents/v1/agents/{agentId}/skill-disclosure
Reports how the agent's saved skills will appear in the prompt of its next session: the skills in the order the model reads them, which of them the model's budget leaves room to describe, and how many characters of that budget they use. It answers for the agent's latest version, with the model and built-in integrations a session of that version starts with: the budget is sized from that model's context window, and an agent with built-in integrations lists the skill the platform adds ahead of its own. If the model's context window cannot be read, the response uses the smallest budget and says so in warnings instead of failing. An agent whose model is not available on this deployment is a 400, as starting a session of it would be; a soft-deleted or unknown agent is a 404.



# List agents
Source: https://docs.labelbox.com/api-reference/agents/list-agents

/recursion/openapi.yaml get /managed-agents/v1/agents
Returns every agent in the calling organization, each resolved to its latest immutable version and hydrated with current tag definitions in one batch. tag_ids is a comma-separated AND filter: every returned agent carries every requested tag. Soft-deleted agents are omitted, and the response is not paginated.



# List an agent's immutable version history
Source: https://docs.labelbox.com/api-reference/agents/list-an-agents-immutable-version-history

/recursion/openapi.yaml get /managed-agents/v1/agents/{agentId}/versions
Returns all immutable agent versions, newest to oldest, projected through the current public contract. Prompt, model, and supported capabilities remain visible; inert legacy tool declarations remain stored but are omitted from these responses. The response is not paginated.



# List static agent templates
Source: https://docs.labelbox.com/api-reference/agents/list-static-agent-templates

/recursion/openapi.yaml get /managed-agents/v1/agent-templates
Returns the startup-validated static catalog of agent definitions. Copy one definition into managedAgentsCreateAgent to create an ordinary managed agent; listing templates has no side effects and performs no live model validation.



# Open the analyst for an agent's sessions
Source: https://docs.labelbox.com/api-reference/agents/open-the-analyst-for-an-agents-sessions

/recursion/openapi.yaml post /managed-agents/v1/agents/{agentId}/analyst
Returns the caller's agent analyst, starting one if they have none (created is true when this call started it): a read-only assistant session that answers questions across every session the agent ran -- volume, outcomes, failures, spend, which session produced which file -- and can read any one of those sessions in depth. It reads with the caller's own access: public sessions of this agent only. Ask it questions by sending messages to the returned session (managedAgentsSendSessionEvents) and read answers from its events; it may also put typed questions in front of the caller, answered with managedAgentsSubmitAssistantInteractionDecision. One analyst per caller per agent; {"reset": true} ends it. Calls not made on behalf of a person get 403. Out-of-credit and suspended tenants get 403 session_start_not_admitted as on managedAgentsOpenSessionAnalyst.



# Search an agent's file history
Source: https://docs.labelbox.com/api-reference/agents/search-an-agents-file-history

/recursion/openapi.yaml post /managed-agents/v1/agents/{agentId}/files/search
Searches the session outputs this agent's sessions saved, including outputs its subagents and teammates saved, which are what its sessions read with search_agent_files when the agent has fileHistoryEnabled. Structured and metadata conditions are exact and apply before paging; text searches the contents of text formats, and coverage reports the candidates it could not search in full. Every page reads the snapshot asOf names; pass nextCursor back with an otherwise unchanged request for the next page. Outputs of deleted sessions, of session analysts, and of sessions under an access policy are not included.



# Soft delete agent
Source: https://docs.labelbox.com/api-reference/agents/soft-delete-agent

/recursion/openapi.yaml delete /managed-agents/v1/agents/{agentId}
Soft deletes the agent so it no longer appears in reads or starts new sessions. Its version history and existing sessions are retained. The delete cannot be undone through this API.



# Create a paused canonical automation
Source: https://docs.labelbox.com/api-reference/automations/create-a-paused-canonical-automation

/recursion/openapi.yaml post /managed-agents/v1/automations
Creates a complete automation aggregate in the paused state after validating the pinned agent version, environment, and every trigger source.



# Delete a canonical automation
Source: https://docs.labelbox.com/api-reference/automations/delete-a-canonical-automation

/recursion/openapi.yaml delete /managed-agents/v1/automations/{automationId}
Soft-deletes the aggregate and its live triggers while preserving admitted run snapshots. Requires the latest strong ETag in If-Match.



# Get a canonical automation
Source: https://docs.labelbox.com/api-reference/automations/get-a-canonical-automation

/recursion/openapi.yaml get /managed-agents/v1/automations/{automationId}
Returns the complete round-trippable automation aggregate with a strong revision ETag. If-None-Match may answer 304.



# Get automation schedule status
Source: https://docs.labelbox.com/api-reference/automations/get-automation-schedule-status

/recursion/openapi.yaml get /managed-agents/v1/automations/{automationId}/status
Reads synchronization and bounded live scheduler previews independently of the writable definition. Unavailable state is explicit.



# Get one canonical automation run
Source: https://docs.labelbox.com/api-reference/automations/get-one-canonical-automation-run

/recursion/openapi.yaml get /managed-agents/v1/automation-runs/{automationRunId}
Returns one canonical run, including the immutable prompt and context snapshots and its current session-creation outcome.



# List canonical automation runs
Source: https://docs.labelbox.com/api-reference/automations/list-canonical-automation-runs

/recursion/openapi.yaml get /managed-agents/v1/automation-runs
Returns a consistent page of automation runs, newest first. Filters combine with AND, and a continuation cursor is bound to the organization, filters, and the moment the first page was read.



# List canonical automations
Source: https://docs.labelbox.com/api-reference/automations/list-canonical-automations

/recursion/openapi.yaml get /managed-agents/v1/automations
Returns a bounded page of canonical automation summaries, newest first. Prompt, run defaults, and trigger configuration are available only on member reads.



# Pause canonical automation admission
Source: https://docs.labelbox.com/api-reference/automations/pause-canonical-automation-admission

/recursion/openapi.yaml post /managed-agents/v1/automations/{automationId}/pause
Stops stored triggers from admitting new runs while leaving explicit manual runs available. Requires the latest strong ETag in If-Match.



# Replace a canonical automation aggregate
Source: https://docs.labelbox.com/api-reference/automations/replace-a-canonical-automation-aggregate

/recursion/openapi.yaml put /managed-agents/v1/automations/{automationId}
Replaces the complete writable aggregate and trigger set while preserving lifecycle state. Requires the latest strong ETag in If-Match.



# Resume canonical automation admission
Source: https://docs.labelbox.com/api-reference/automations/resume-canonical-automation-admission

/recursion/openapi.yaml post /managed-agents/v1/automations/{automationId}/resume
Allows enabled stored triggers to admit new runs. Refuses with 422 automation_reference_invalid when the agent, agent version, or environment is no longer live. Requires the latest strong ETag in If-Match.



# Run a canonical automation manually
Source: https://docs.labelbox.com/api-reference/automations/run-a-canonical-automation-manually

/recursion/openapi.yaml post /managed-agents/v1/automations/{automationId}/runs
Atomically freezes the automation and optional payload into one pending run. Explicit manual runs remain available while the automation is paused. Within the organization-wide receipt namespace, Idempotency-Key plus the exact HTTP method, escaped path, raw query, and raw body bytes identify the durable result.



# Cancel a setup run
Source: https://docs.labelbox.com/api-reference/environments/cancel-a-setup-run

/recursion/openapi.yaml post /managed-agents/v1/environments/{environmentId}/setup-runs/{setupRunId}/cancel
Stops a queued or in-flight manual or automatic runner_refresh setup run and releases its compute. Sessions waiting on a cancelled runner_refresh fail with that run as their cause. The cancelled run is returned with status cancelled and active_setup_run_id is cleared. A cancelled run proves nothing: a standing verified or failed verdict is kept as it was, and an environment whose only verdict was running reverts to never. 409 when the run had already finished.



# Create environment
Source: https://docs.labelbox.com/api-reference/environments/create-environment

/recursion/openapi.yaml post /managed-agents/v1/environments
Defines the sandbox an agent session runs in and returns the stored environment, including its generated environment_id. The provider must be one this deployment reports from managedAgentsListSandboxProviders. Privileged Docker requires unrestricted internet access (network_policy: {}). Nothing is provisioned by this configuration write; compute is allocated when managedAgentsCreateEnvironmentSetupRun or a session references the environment. An environment with a setup script must be verified (managedAgentsCreateEnvironmentSetupRun) before a session on the runs or docker provider will start. Send Idempotency-Key to make a retry after an ambiguous transport failure replay the original response instead of creating another environment.



# Get a setup run, optionally waiting for it to finish
Source: https://docs.labelbox.com/api-reference/environments/get-a-setup-run-optionally-waiting-for-it-to-finish

/recursion/openapi.yaml get /managed-agents/v1/environments/{environmentId}/setup-runs/{setupRunId}
Returns one setup run. With wait_seconds the call blocks until the run reaches a terminal status or the wait elapses, so a poll loop needs no client-side timer. next_action names the call to make on the result: poll again, read the log, or start a session.



# Get environment
Source: https://docs.labelbox.com/api-reference/environments/get-environment

/recursion/openapi.yaml get /managed-agents/v1/environments/{environmentId}
Returns one environment and its full sandbox spec. A soft-deleted or unknown id is a 404, and an id belonging to another organization is reported the same way.



# List environments
Source: https://docs.labelbox.com/api-reference/environments/list-environments

/recursion/openapi.yaml get /managed-agents/v1/environments
Returns every environment in the calling organization with its full sandbox spec (provider, image, setup, resources, network policy). Soft-deleted environments are omitted, and the response is not paginated.



# List portable environment templates
Source: https://docs.labelbox.com/api-reference/environments/list-portable-environment-templates

/recursion/openapi.yaml get /managed-agents/v1/environment-templates
Returns reviewed, versioned environment starting points. Copy a definition into managedAgentsCreateEnvironment, bind any organization-specific fields in the form, and verify its setup before starting a session.



# List setup runs
Source: https://docs.labelbox.com/api-reference/environments/list-setup-runs

/recursion/openapi.yaml get /managed-agents/v1/environments/{environmentId}/setup-runs
Returns the environment's recent setup runs, newest first: manual, automatic runner_refresh, and legacy session-triggered runs. The newest manual or runner_refresh run supplies the environment's setup_verification verdict; a session run records legacy in-session setup and can be read here by its session_id.



# List supported sandbox providers
Source: https://docs.labelbox.com/api-reference/environments/list-supported-sandbox-providers

/recursion/openapi.yaml get /managed-agents/v1/sandbox-providers
Returns the sandbox runtimes this deployment can run environments on, with the display name, description, whether the provider is the default, and its required credential name if any. This is static service capability, not organization data, so every caller sees the same list.



# Read or tail a setup run's log
Source: https://docs.labelbox.com/api-reference/environments/read-or-tail-a-setup-runs-log

/recursion/openapi.yaml get /managed-agents/v1/environments/{environmentId}/setup-runs/{setupRunId}/log
Returns the run's output after a cursor: stdout and stderr as the script produced them, marker lines naming each script line before it runs, and system lines for the setup phases around the script (provision, gpu_check, profile, cleanup). Secrets are redacted before storage. To tail a live run, loop with after=next_after and wait_seconds=0, sleeping two seconds client-side, until status is terminal and a page comes back empty.



# Run and verify the environment's setup script
Source: https://docs.labelbox.com/api-reference/environments/run-and-verify-the-environments-setup-script

/recursion/openapi.yaml post /managed-agents/v1/environments/{environmentId}/setup-runs
Requires Idempotency-Key because this operation provisions billable compute. Provisions fresh compute for the environment, runs its setup script exactly as a session would (same image, resources, and variables; no session credentials), records every line of output, profiles the compute, releases it, and writes the verdict to the environment's setup_verification. Returns 202 with the queued run; poll managedAgentsGetEnvironmentSetupRun with wait_seconds until status is succeeded, failed, or cancelled. On failure read hint, failed_line, and stderr_tail first, then the log. One run at a time per environment: a second request while one is in flight is a 409 naming the run to poll. Repeating the exact request with the same Idempotency-Key replays the original 202 response without starting another run. 400 when the environment has no setup script. This spends compute. When the tenant is out of credit, returns 403 setup_run_not_admitted with details.reason out_of_credit; buy credits and retry. A suspended tenant gets the same 403 with details.reason tenant_suspended. An unavailable credit check returns 503 setup_run_admission_unavailable without recording a run.



# Soft delete environment
Source: https://docs.labelbox.com/api-reference/environments/soft-delete-environment

/recursion/openapi.yaml delete /managed-agents/v1/environments/{environmentId}
Soft deletes the environment: it disappears from reads and can no longer be referenced by a new session, but nothing is erased. Sessions that already ran against it keep their history. There is no undelete on this API.



# Update environment
Source: https://docs.labelbox.com/api-reference/environments/update-environment

/recursion/openapi.yaml patch /managed-agents/v1/environments/{environmentId}
Applies a partial update and returns the stored environment; omitted fields keep their current values; a name, when sent, must not be blank or null. Changing from a non-Runs provider to Runs with an absent or {} stored policy requires an explicit network_policy: {"version":"v1","rules":[]} for deny-all or {} for unrestricted egress. A provider is validated only when the request actually names one. Sending setup with an empty script clears the script; omitting setup leaves it. Any change to provider, image, setup (script or timeout), resources, workspace disk size, provider config, variables, retired secret references, mounts, network policy, or effective Docker privilege marks the previous setup verification stale; name, description, metadata, and http_port do not. Call managedAgentsCreateEnvironmentSetupRun separately after the update to re-run setup. Sessions already running keep the spec they started with, so a change takes effect on the next session start. A Runs update that changes or bypasses a nonempty network_policy, or enables privileged Docker, requires expected_access with the network_policy and privileged values from the last environment read. When changing access settings, privileged Docker requires unrestricted internet access (network_policy: {}). A missing access snapshot returns 428; a stale one returns 412. A concurrent configuration edit may return 409; reload and retry. Send Idempotency-Key to make a retry after an ambiguous transport failure replay the original response.



# Create a paused event source
Source: https://docs.labelbox.com/api-reference/event-sources/create-a-paused-event-source

/recursion/openapi.yaml post /managed-agents/v1/event-sources
Creates a customer webhook source in the paused state. delivery.kind must be webhook; integration sources are connection-managed. Verification credentials are referenced from Vault and secret material is never returned.



# Delete an unused event source
Source: https://docs.labelbox.com/api-reference/event-sources/delete-an-unused-event-source

/recursion/openapi.yaml delete /managed-agents/v1/event-sources/{eventSourceId}
Soft-deletes a customer webhook source only when no live trigger depends on it. Connection-managed sources cannot be deleted independently. Requires the latest strong ETag in If-Match.



# Get an event source
Source: https://docs.labelbox.com/api-reference/event-sources/get-an-event-source

/recursion/openapi.yaml get /managed-agents/v1/event-sources/{eventSourceId}
Returns non-secret source configuration with a strong revision ETag. If-None-Match may answer 304.



# Get volatile event-source status
Source: https://docs.labelbox.com/api-reference/event-sources/get-volatile-event-source-status

/recursion/openapi.yaml get /managed-agents/v1/event-sources/{eventSourceId}/status
Returns live trigger count and bounded last-delivery diagnostics separately from revisioned configuration.



# List event sources
Source: https://docs.labelbox.com/api-reference/event-sources/list-event-sources

/recursion/openapi.yaml get /managed-agents/v1/event-sources
Returns a bounded page of non-secret source summaries, newest first. Protocol configuration is available only on member reads.



# Pause event admission from a source
Source: https://docs.labelbox.com/api-reference/event-sources/pause-event-admission-from-a-source

/recursion/openapi.yaml post /managed-agents/v1/event-sources/{eventSourceId}/pause
Stops new admissions from this source without changing dependent triggers. Requires the latest strong ETag in If-Match.



# Replace an event source configuration
Source: https://docs.labelbox.com/api-reference/event-sources/replace-an-event-source-configuration

/recursion/openapi.yaml put /managed-agents/v1/event-sources/{eventSourceId}
Replaces the complete writable webhook configuration while preserving lifecycle state. Connection-managed sources cannot be replaced. Requires the latest strong ETag in If-Match.



# Resume event admission from a source
Source: https://docs.labelbox.com/api-reference/event-sources/resume-event-admission-from-a-source

/recursion/openapi.yaml post /managed-agents/v1/event-sources/{eventSourceId}/resume
Allows verified deliveries to reach dependent triggers again. Requires the latest strong ETag in If-Match.



# Download a file's bytes
Source: https://docs.labelbox.com/api-reference/files/download-a-files-bytes

/recursion/openapi.yaml get /managed-agents/v1/files/{fileId}/content
Returns a file's bytes to its organization as application/octet-stream with a Content-Disposition naming it. The file's recorded media type remains available in its metadata but is not used as the response Content-Type. Responses include nosniff and sandbox content-security headers so uploaded HTML and SVG cannot run as this origin.



# Get one file
Source: https://docs.labelbox.com/api-reference/files/get-one-file

/recursion/openapi.yaml get /managed-agents/v1/files/{fileId}
Returns one file's metadata. A deleted file answers 404, as does a file belonging to another organization; an expired file still answers, with expires_at in the past, until it is removed after a grace period.



# List files
Source: https://docs.labelbox.com/api-reference/files/list-files

/recursion/openapi.yaml get /managed-agents/v1/files
Returns the caller organization's files, newest first, one page at a time: up to limit files (default 200, at most 1000) and a next_page_token while more follow. Deleted files are omitted; an expired file stays listed, with expires_at in the past, until it is removed after a grace period, so a client that wants only attachable files filters on expires_at. Filter to one session's outputs with scope_session_id or by source. A file_ids lookup (at most 100 ids) is always one page and cannot be combined with limit or page_token.



# Upload a file
Source: https://docs.labelbox.com/api-reference/files/upload-a-file

/recursion/openapi.yaml post /managed-agents/v1/files
Stores a file the organization can attach to any number of sessions. Attaching one mounts it read-only into the session's sandbox under its files directory, and the agent knows where to find it. Send the bytes as the file part of a multipart/form-data body; the part's filename and Content-Type become the file's name and media type. The file part may contain at most 30 MiB, and the complete multipart request may use at most 128 KiB more for framing and form fields. Larger files are not supported by this Platform V1 operation. The bytes are served back through GET /managed-agents/v1/files/{fileId}/content to the same organization. A request over the transport limit answers 413 with code payload_too_large. An upload that would carry the organization's live files past its storage quota answers 413 with code file_quota_exceeded and reports the quota, bytes in use, and incoming size in details.



# Begin authorizing a third-party account connection
Source: https://docs.labelbox.com/api-reference/integrations/begin-authorizing-a-third-party-account-connection

/recursion/openapi.yaml post /managed-agents/v1/integrations/{provider}/install
Begins authorizing a third-party account connection and returns the provider URL to send the operator to, together with a single-use, short-lived state nonce. A caller may also continue one of the opaque provider options returned by managedAgentsCompleteIntegrationInstall. Nothing is connected until completion succeeds. Do not automatically retry an ambiguous transport failure; begin a new install and use its returned state.



# Check a connection against the provider and try minting a token
Source: https://docs.labelbox.com/api-reference/integrations/check-a-connection-against-the-provider-and-try-minting-a-token

/recursion/openapi.yaml post /managed-agents/v1/integrations/connections/{connectionId}/probe
Checks a connection against the provider and tries to mint a short-lived access token, returning what the provider reported. A suspended or revoked installation is still HTTP 200 with ok=false, so read that field rather than the status code. For a directly registered connection in the pending state this is also the activation step: the probe performs a fresh mint through the trust the customer granted and, if it succeeds, moves the connection to active and reports activated=true; until then it stays pending and the failure text is the bounded "delegate trust is not active".



# Disconnect an account and retire the grants that used it
Source: https://docs.labelbox.com/api-reference/integrations/disconnect-an-account-and-retire-the-grants-that-used-it

/recursion/openapi.yaml delete /managed-agents/v1/integrations/connections/{connectionId}
Disconnects the account and retires every credential grant that referenced it, returning how many grants were revoked. Agents holding those grants lose access to the provider on their next session; existing sessions are not torn down.



# Finish connecting provider accounts after the authorization redirect
Source: https://docs.labelbox.com/api-reference/integrations/finish-connecting-provider-accounts-after-the-authorization-redirect

/recursion/openapi.yaml post /managed-agents/v1/integrations/install/complete
Finishes an authorization begun by managedAgentsStartIntegrationInstall. It returns either connected accounts or provider-neutral account/action options that continue through managedAgentsStartIntegrationInstall. The provider and organization come from the authenticated caller and single-use state nonce, never from callback parameters or browser-local storage. An expected provider failure after state consumption returns 409 integration_install_restart_required with details.provider; begin a new install. Resource and permission selection happens later when granting a vault. Do not automatically retry an ambiguous transport failure: list connections to determine whether completion committed; if it did not, begin a new install because the state is single-use.



# Get one connected account
Source: https://docs.labelbox.com/api-reference/integrations/get-one-connected-account

/recursion/openapi.yaml get /managed-agents/v1/integrations/connections/{connectionId}
Returns one connected account. An unknown id, or one belonging to another organization, is reported as a 404.



# Get the connected Slack bot's bounded channel-selection catalog
Source: https://docs.labelbox.com/api-reference/integrations/get-the-connected-slack-bots-bounded-channel-selection-catalog

/recursion/openapi.yaml get /managed-agents/v1/integrations/connections/{connectionId}/channel-catalog
Returns a channel-selection snapshot of at most 1,000 memberships from the connected Slack workspace, using that installation's credential, so an automation trigger can be configured by picking a channel instead of pasting its id. Only a Slack connection has channels; the list is scoped to the connection's own workspace. A channel the bot is not a member of is never listed: invite the bot in Slack first. Throttled reads return 429 slack_rate_limited. A list that stopped at the size budget says truncated=true, so absence from the list never proves the bot is not a member. If the Slack app lacks channels:read or groups:read the response is 422 slack_missing_scope naming the scopes to add.



# Get the connected Slack workspace's bounded member catalog
Source: https://docs.labelbox.com/api-reference/integrations/get-the-connected-slack-workspaces-bounded-member-catalog

/recursion/openapi.yaml get /managed-agents/v1/integrations/connections/{connectionId}/member-catalog
Returns the people and apps of the connected Slack workspace, at most 2,000, read with that installation's own credential, so a Slack trigger's fromUserIds filter can be configured by picking authors instead of pasting their ids. Apps are listed with isBot=true; deleted accounts and the connection's own bot are never listed. Only a Slack connection installed through Recursion can list members; any other answers 503 slack_members_unconfigured. Throttled reads return 429 slack_rate_limited. A list that stopped at the size budget says truncated=true, so absence from the list does not prove the member does not exist. If the Slack app lacks users:read the response is 422 slack_missing_scope.



# List connected third-party accounts
Source: https://docs.labelbox.com/api-reference/integrations/list-connected-third-party-accounts

/recursion/openapi.yaml get /managed-agents/v1/integrations/connections
Returns the organization's connected third-party accounts, each with its provider, account login, resource selection, and state. A connection holds no secret material, so nothing here needs to be redacted.



# List pull requests for a connected repository
Source: https://docs.labelbox.com/api-reference/integrations/list-pull-requests-for-a-connected-repository

/recursion/openapi.yaml get /managed-agents/v1/integrations/connections/{connectionId}/repositories/{repositoryOwner}/{repositoryName}/pull-requests



# List the native integrations this deployment supports
Source: https://docs.labelbox.com/api-reference/integrations/list-the-native-integrations-this-deployment-supports

/recursion/openapi.yaml get /managed-agents/v1/integrations/providers
Returns the third-party integrations this deployment supports. Providers that the deployment has not configured are listed too, with their configured state, so a caller can tell "not set up here" apart from "not offered".



# List the repositories a connection can reach, for the grant picker
Source: https://docs.labelbox.com/api-reference/integrations/list-the-repositories-a-connection-can-reach-for-the-grant-picker

/recursion/openapi.yaml get /managed-agents/v1/integrations/connections/{connectionId}/repositories
Returns the repositories this connection can reach, for building a grant picker. Results are paged by the 1-based page query parameter at up to 100 repositories per page; the list comes from the provider live, so it reflects the installation's current access.



# Delete a memory
Source: https://docs.labelbox.com/api-reference/memory/delete-a-memory

/recursion/openapi.yaml delete /managed-agents/v1/memory-stores/{memoryStoreId}/memories/{memoryId}
Removes one learned memory from future recall and records the deletion as a version. The path becomes available for reuse.



# Get an agent's memory settings
Source: https://docs.labelbox.com/api-reference/memory/get-an-agents-memory-settings

/recursion/openapi.yaml get /managed-agents/v1/agents/{agentId}/memory/settings
Returns the opt-in, cadence, model, curator, and standing instructions with a strong revision ETag. Recursion agents start with memory disabled. If-None-Match may answer 304.



# Get an agent's memory summary
Source: https://docs.labelbox.com/api-reference/memory/get-an-agents-memory-summary

/recursion/openapi.yaml get /managed-agents/v1/agent-memories/{agentId}
Returns the current permanent collection, its effective model, and automatic learning progress without starting work.



# Get one memory job run
Source: https://docs.labelbox.com/api-reference/memory/get-one-memory-job-run

/recursion/openapi.yaml get /managed-agents/v1/agents/{agentId}/memory/runs/{runId}
Returns one run: its trigger, status, the sessions whose reports it read, the collection it updated, and the id of the session that ran it. Read what the run spent from that session's costs, which settle after the session ends.



# Get one memory version
Source: https://docs.labelbox.com/api-reference/memory/get-one-memory-version

/recursion/openapi.yaml get /managed-agents/v1/memory-stores/{memoryStoreId}/memory-versions/{memoryVersionId}
Returns one version including the content as it stood, unless the version has been redacted.



# List an agent's memory runs
Source: https://docs.labelbox.com/api-reference/memory/list-an-agents-memory-runs

/recursion/openapi.yaml get /managed-agents/v1/agents/{agentId}/memory/runs
Returns automatic memory updates and checks, newest first, including unchanged checks and failed attempts.



# List memory status for every agent
Source: https://docs.labelbox.com/api-reference/memory/list-memory-status-for-every-agent

/recursion/openapi.yaml get /managed-agents/v1/agent-memories
Returns each customer agent's current memory summary, including agents that have not learned yet. Each status is derived from that agent's latest run.



# List memory versions
Source: https://docs.labelbox.com/api-reference/memory/list-memory-versions

/recursion/openapi.yaml get /managed-agents/v1/memory-stores/{memoryStoreId}/memory-versions
Returns a store's change history, newest first, optionally filtered to one memory. Versions belong to the store rather than the memory, so the trail survives the memory being deleted. Retained 30 days, except that the recent versions of a live memory are kept regardless of age.



# Read one memory
Source: https://docs.labelbox.com/api-reference/memory/read-one-memory

/recursion/openapi.yaml get /managed-agents/v1/memory-stores/{memoryStoreId}/memories/{memoryId}
Returns one learned memory, including its full content and content hash.



# Search an agent's current memories
Source: https://docs.labelbox.com/api-reference/memory/search-an-agents-current-memories

/recursion/openapi.yaml get /managed-agents/v1/agents/{agentId}/memories
Lists current memories as a flat, searchable keyset page, with topics from the entire collection. Retired memories are available through detail and history reads.



# Update an agent's memory settings
Source: https://docs.labelbox.com/api-reference/memory/update-an-agents-memory-settings

/recursion/openapi.yaml patch /managed-agents/v1/agents/{agentId}/memory/settings
Changes subsequent memory jobs and session attachments. Disabling stops new digests and detaches the learned store from new sessions. Omitted fields remain unchanged. Send the latest settings ETag in If-Match; a stale editor receives 412.



# List the models agents can run on
Source: https://docs.labelbox.com/api-reference/models/list-the-models-agents-can-run-on

/recursion/openapi.yaml get /managed-agents/v1/models
Returns the models agents can currently run on. The list is not stored per organization; it reflects service configuration, so it can change without any write from this API.



# Append user/system events to a session
Source: https://docs.labelbox.com/api-reference/sessions/append-usersystem-events-to-a-session

/recursion/openapi.yaml post /managed-agents/v1/sessions/{sessionId}/events
Accepts ordered user and system messages and returns how many were accepted. A running session is woken so the agent sees them, and a finished session resumes to act on them; delivery_state reports how they were delivered. Events are appended in the order given. An omitted or empty events array yields exactly one user turn built from message; the request fails with 400 only when every entry sent was dropped, leaving nothing to append. A top-level image or document block may name a file with source {type: file, file_id}: an unknown, expired, or foreign file answers 404 on that block's source.file_id, a file the block cannot carry (a file whose bytes are not an image on an image block, a binary on a document block, a document over 512 KiB, more than 1 MiB of documents in one event, more than 100 such blocks, or more than 32 MiB of catalog bytes in one request) answers 400 there, and nothing is appended to the transcript; copies of files resolved before the refusal are kept with the session and removed once nothing references them. Without file storage such a block answers 503 file_storage_unconfigured. An optional Idempotency-Key makes a retry after an ambiguous transport failure replay the first response instead of appending the events again. When the tenant is out of credit, a message returns 403 session_start_not_admitted with details.reason out_of_credit and nothing is appended; add credits and send it again. A suspended tenant gets the same 403 with details.reason tenant_suspended.



# Decide on an assistant interaction
Source: https://docs.labelbox.com/api-reference/sessions/decide-on-an-assistant-interaction

/recursion/openapi.yaml post /managed-agents/v1/sessions/{sessionId}/interactions/{interactionId}/decide
Records the caller's decision on the pending interaction an assistant put in front of them -- answers to its questions, acceptance of a blueprint, approval of a plan, a revision request, or dismissal -- and delivers it to the assistant as the caller's next message. Only the person who opened the assistant may decide, and only on the exact revision and digest they were shown (409 otherwise). Approving a plan requires a signed-in user holding every permission its enabled steps need; the plan then runs in the background, one existing operation per step, under the caller's permissions, and its progress is appended to the assistant's events. The response is the interaction after the decision.



# Define what done looks like for a session
Source: https://docs.labelbox.com/api-reference/sessions/define-what-done-looks-like-for-a-session

/recursion/openapi.yaml post /managed-agents/v1/sessions/{sessionId}/outcomes
Defines what done looks like for an existing session and returns the stored outcome. Outcomes chain rather than run concurrently: the request is rejected while the session still has a non-terminal outcome, so define the next one only after the previous is graded.



# Get one private session image
Source: https://docs.labelbox.com/api-reference/sessions/get-one-private-session-image

/recursion/openapi.yaml get /managed-agents/v1/sessions/{sessionId}/images



# Get root session tree and timeline
Source: https://docs.labelbox.com/api-reference/sessions/get-root-session-tree-and-timeline

/recursion/openapi.yaml get /managed-agents/v1/sessions/{sessionId}/tree
Returns every session in the root session's tree, its threads, and the first bounded page of the merged chronological event log. When next_event_id is set, more events remain and should be fetched with managedAgentsListSessionEvents rather than by re-reading the tree. Any session id in the tree is accepted; the read resolves to its root.



# Get root timeline events
Source: https://docs.labelbox.com/api-reference/sessions/get-root-timeline-events

/recursion/openapi.yaml get /managed-agents/v1/sessions/{sessionId}/events
Returns one page of the root timeline's events in chronological order, plus next_page_token to continue from. A page can come back shorter than limit because an aggregate serialized-byte budget applies as well as the count, so treat an empty page with no token as the end rather than inferring the end from the length. Large event payloads hydrate by default; pass payloads=refs for bounded stubs instead.



# Get session
Source: https://docs.labelbox.com/api-reference/sessions/get-session

/recursion/openapi.yaml get /managed-agents/v1/sessions/{sessionId}
Returns one session with its status, agent and environment binding, and timing. Statuses completed, failed, and cancelled are terminal and will not change again, which is what a poller should wait for. Poll with view=summary once agent_snapshot, model_snapshot and config are held. They are written at start and change only when the session moves to another version of its agent, which the summary's agent_version_id shows: read the full view again when it changes.



# Get session thread
Source: https://docs.labelbox.com/api-reference/sessions/get-session-thread

/recursion/openapi.yaml get /managed-agents/v1/sessions/{sessionId}/threads/{threadId}
Returns one thread from the session's tree. The thread must belong to the tree rooted at the session in the path; anything else is a 404 rather than a cross-tree read.



# Get the agent version a session runs
Source: https://docs.labelbox.com/api-reference/sessions/get-the-agent-version-a-session-runs

/recursion/openapi.yaml get /managed-agents/v1/sessions/{sessionId}/agent-version
Returns the version of its agent a root session runs now, the version it is set to run, whether a move is pending or was refused, the agent's latest version, and how far each open thread of the session tree has followed. The state changes as the session works, so read it again rather than keeping a copy. A session that is a member of another session's tree answers 409 conflict; read its root. A session that was not started from an agent has no version to read and answers 409 conflict too.



# Get the team board
Source: https://docs.labelbox.com/api-reference/sessions/get-the-team-board

/recursion/openapi.yaml get /managed-agents/v1/sessions/{sessionId}/board
Returns the shared task board of a session whose roster runs as a team: every task with its status, owner, notes, and outcome; the notices and the decision; and the members named the way the board names them. Any session id in the tree is accepted; the read resolves to its root. A session whose tree is not a team is a 404. Each transition is also recorded as a board_update event on the root's log, but that log interleaves events written by every member, so a reader paging it by event id after a cursor can skip one; this read is the authoritative state, and a poller should refresh from it rather than reconstruct the board from events alone.



# Hard stop a session immediately
Source: https://docs.labelbox.com/api-reference/sessions/hard-stop-a-session-immediately

/recursion/openapi.yaml post /managed-agents/v1/sessions/{sessionId}/cancel
Stops the session immediately and schedules durable sandbox teardown. The response confirms the session stopped; sandbox cleanup may continue asynchronously and is retried. A session that already reached a terminal status with no pending cleanup is left as it is, so the call is safe to repeat.



# Interrupt a running session
Source: https://docs.labelbox.com/api-reference/sessions/interrupt-a-running-session

/recursion/openapi.yaml post /managed-agents/v1/sessions/{sessionId}/interrupt
Stops active model generation immediately and gives an active tool a bounded grace period. A session started before immediate interruption was available keeps its earlier bounded grace behavior. Interrupting is also how queued input is read sooner: when messages are waiting, the newest becomes the instruction for a fresh turn instead of leaving the session idle. With nothing waiting the session is left idle so a later follow-up can resume it. A session with nothing running -- idle and not asleep or waiting on its subagents, or already in a terminal status -- is refused with 409 conflict and nothing is recorded. interrupted reports whether running work received the request; false means the stop was recorded but reaching running work was not confirmed, typically because the run ended as the request arrived. Omit message to use a default one.



# Interrupt a session and send a replacement instruction
Source: https://docs.labelbox.com/api-reference/sessions/interrupt-a-session-and-send-a-replacement-instruction

/recursion/openapi.yaml post /managed-agents/v1/sessions/{sessionId}/interrupt-and-send
Stops active work and durably accepts a replacement instruction in one ordered operation. The session starts a fresh turn automatically. A session started before replacement instructions were available still honors the interrupt and keeps the replacement for its next resume. When the tenant is out of credit it returns 403 session_start_not_admitted with details.reason out_of_credit and changes nothing, and a suspended tenant gets the same 403 with details.reason tenant_suspended; managedAgentsInterruptSession still stops the session.



# List a session's outcomes and grading history
Source: https://docs.labelbox.com/api-reference/sessions/list-a-sessions-outcomes-and-grading-history

/recursion/openapi.yaml get /managed-agents/v1/sessions/{sessionId}/outcomes
Returns the session's outcomes and their grading history, so a caller can see what "done" was defined as and how each attempt was scored.



# List input a session has accepted but not read
Source: https://docs.labelbox.com/api-reference/sessions/list-input-a-session-has-accepted-but-not-read

/recursion/openapi.yaml get /managed-agents/v1/sessions/{sessionId}/pending-inputs
Returns the messages waiting for a busy session, in acceptance order. A message sent while the agent is mid-turn is durable immediately but is not part of the transcript until the next turn boundary reads it, so it appears here and not in the events listing. Each entry's input_id becomes the event id on promotion, which is how a client follows a queued message into history. Interrupt the session to have the queue read now.



# List session threads
Source: https://docs.labelbox.com/api-reference/sessions/list-session-threads

/recursion/openapi.yaml get /managed-agents/v1/sessions/{sessionId}/threads
Returns every thread in the session's tree, including one synthesized thread per session that has no explicit thread rows, sorted by thread path. Any session id in the tree is accepted; the read resolves to its root.



# List sessions
Source: https://docs.labelbox.com/api-reference/sessions/list-sessions

/recursion/openapi.yaml get /managed-agents/v1/sessions
Returns unclassified public sessions in the calling organization, most recently updated first, filtered by any combination of status, kind, agent_id, current agent tag_ids, and external source. kind accepts a comma-separated list, while tag_ids uses AND semantics. Results are keyset-paginated and capped by limit (default 100, max 1000). Every page of one listing reads the same snapshot as the first page, so a session updated while a client pages is neither skipped nor repeated. Classified session trees are omitted.



# List the caller-defined metadata keys sessions carry
Source: https://docs.labelbox.com/api-reference/sessions/list-the-caller-defined-metadata-keys-sessions-carry

/recursion/openapi.yaml get /managed-agents/v1/sessions/metadata-keys
Returns the distinct caller-defined metadata keys across every session the caller may list, sorted, so a client can offer them as filter choices without knowing them in advance. Sessions the caller's access policy hides contribute nothing. Capped at 200 keys; truncated reports when the cap was hit. Pair with managedAgentsListSessionMetadataValues, then filter managedAgentsListSessions with metadata or metadata_key.



# List the values one caller-defined metadata key takes
Source: https://docs.labelbox.com/api-reference/sessions/list-the-values-one-caller-defined-metadata-key-takes

/recursion/openapi.yaml get /managed-agents/v1/sessions/metadata-values
Returns the distinct values of one caller-defined metadata key across every session the caller may list, sorted and optionally narrowed to a case-sensitive prefix with q, for a typeahead over managedAgentsListSessions metadata filters. Sessions the caller's access policy hides contribute nothing. A key no session carries returns an empty list rather than an error. When more values match than limit allows, truncated is true: narrow q instead of paging.



# Move a session to another version of its agent
Source: https://docs.labelbox.com/api-reference/sessions/move-a-session-to-another-version-of-its-agent

/recursion/openapi.yaml put /managed-agents/v1/sessions/{sessionId}/agent-version
Sets the version of its agent a root session runs. The move is recorded at once and takes effect before the session's next model call, never in the middle of one: a sleeping session stays asleep and adopts the version when it next runs, and a finished session when it is next resumed. Threads adopt after their parent. The response is the representation managedAgentsGetSessionAgentVersion returns, with state pending until the move takes effect; if the version cannot run in this session, state becomes failed and lastFailure says why. Repeating the request while the move is pending, or once the session runs that version, changes nothing; repeating it after the move failed asks again. The later of two different requests wins, and naming the version the session already runs withdraws a pending move. Because a version can grant the session the vaults a new session would receive, the caller needs both managed-agents:create and managed-agents:update. Sessions that are members of another session's tree, sessions the platform runs itself, evaluations and pull request reviews stay on the version they started with and answer 409 conflict, as does a session that was not started from an agent, and one that is stopping or can no longer continue. A version of another agent answers 404.



# Open the organization assistant
Source: https://docs.labelbox.com/api-reference/sessions/open-the-organization-assistant

/recursion/openapi.yaml post /managed-agents/v1/assistant
Returns the caller's organization assistant, starting one if they have none (created is true when this call started it): an assistant session that knows the platform's capabilities and the organization's agents, environments, skills and connections, and helps design, set up and check agents. It changes nothing by itself: it asks typed questions, presents solution blueprints and proposes plans, and only a plan the caller approves with managedAgentsSubmitAssistantInteractionDecision is executed, under the caller's own permissions. Talk to it with managedAgentsSendSessionEvents and read its events. One assistant per caller; {"reset": true} ends it. Calls not made on behalf of a person get 403.



# Open the session analyst for a session
Source: https://docs.labelbox.com/api-reference/sessions/open-the-session-analyst-for-a-session

/recursion/openapi.yaml post /managed-agents/v1/sessions/{sessionId}/analyst
Returns the caller's session analyst for the tree containing this session, starting one if they have none (201 when this call started it, 200 otherwise): a read-only assistant session that answers questions about the target -- what the agents were asked to do, what they did, which tools failed, how they coordinated, what it cost and how it was graded. Ask it questions by sending messages to the returned analyst session (managedAgentsSendSessionEvents) and read its answers from that session's events or event stream. One analyst per caller per tree: opening again returns the same conversation, and a body of {"reset": true} ends it; the next question starts a new one. The analyst runs on a fixed platform model with no sandbox and no credentials, and nothing it does changes the target session. Any session id in the tree is accepted; the analyst always reads the whole tree. When the tenant is out of credit, a question returns 403 session_start_not_admitted with details.reason out_of_credit and is not delivered; add credits and ask again. A suspended tenant gets the same 403 with details.reason tenant_suspended.



# Read sandbox CPU, memory, and GPU usage for a session tree
Source: https://docs.labelbox.com/api-reference/sessions/read-sandbox-cpu-memory-and-gpu-usage-for-a-session-tree

/recursion/openapi.yaml get /managed-agents/v1/sessions/{sessionId}/resource-samples
Returns the tree's sandbox attachments and one page of per-minute resource usage buckets for them, ordered by attachment then time. Each bucket carries the minute's average and peak; resolution=raw adds the individual samples as parallel arrays indexed by offsets_ms. A tree whose compute was released and revived has several attachments; follow revived_from_sandbox_id to connect them. live reports whether a sandbox is still attached, which is the cue to keep polling. An empty sandboxes list means the tree never ran in a sandbox or sampling is disabled; an attachment with sampling_source=unsupported means the sandbox could not measure itself.



# Roll up token and cost totals for a batch of sessions
Source: https://docs.labelbox.com/api-reference/sessions/roll-up-token-and-cost-totals-for-a-batch-of-sessions

/recursion/openapi.yaml get /managed-agents/v1/sessions/usage
Rolls up token counts and cost for the sessions named in session_ids and returns one row per session. Ids that do not exist in this organization are omitted rather than erroring, so compare the row count against what you asked for. The request is rejected if it names more ids than the per-request maximum.



# Snapshot every root session created in a window
Source: https://docs.labelbox.com/api-reference/sessions/snapshot-every-root-session-created-in-a-window

/recursion/openapi.yaml get /managed-agents/v1/sessions/gestalt
Returns a compact point-in-time projection of live unclassified public root sessions created at or after since (at most 30 days ago), newest first. Draw the snapshot whole, then keep it current with managedAgentsStreamSessionGestalt from as_of. Classified roots are omitted. Windows with more than 5000 roots are truncated to the newest 5000 and say so.



# Soft delete a session subtree (events retained)
Source: https://docs.labelbox.com/api-reference/sessions/soft-delete-a-session-subtree-events-retained

/recursion/openapi.yaml delete /managed-agents/v1/sessions/{sessionId}
Soft deletes the session subtree: it is stopped if still running, sandbox teardown is requested, and its rows disappear from reads. A pending hard-stop sandbox cleanup returns retryable 409 conflict; retry after cleanup finishes, and seek operator attention if it remains pending. The transcript is retained, so this is not a data-erasure operation and cannot be undone through this API.



# Start a managed session
Source: https://docs.labelbox.com/api-reference/sessions/start-a-managed-session

/recursion/openapi.yaml post /managed-agents/v1/sessions
Starts a managed agent session and returns 202 with its session_id and status_path; the agent runs asynchronously, so poll managedAgentsGetSession or subscribe with managedAgentsStreamSessionEvents to follow it. Sandbox compute is provisioned here, which is where a session begins costing money. Requires an Idempotency-Key header: the same key with the exact HTTP method, escaped path, raw query, and raw body bytes returns the original session; any difference is rejected. A reasoning_effort incompatible with the selected model or its frozen output capacity returns 400 before provisioning. Enabled provider_params.promptCacheTTL or provider_params.promptCacheSharedTTL settings with invalid values, an unsupported model, or shared retention shorter than history retention also return 400 before provisioning for the selected root agent version, including for previously stored agent versions. Roster agent cache settings are validated at delegation before creating each child session. A temporarily unavailable model reference or catalog dependency returns 503. An environment on managed compute whose setup script has no verified, current run returns 422 environment_not_verified before provisioning; run managedAgentsCreateEnvironmentSetupRun first. When the tenant is out of credit it returns 403 session_start_not_admitted with details.reason out_of_credit; add credits and retry. When the tenant is suspended it returns the same 403 with details.reason tenant_suspended. When the tenant already has as many sessions working at once as it may, it returns 429 tenant_concurrency_limit with details.limit; wait for a session to finish its turn or stop one, then retry.



# Stream changes to the sessions gestalt as Server-Sent Events
Source: https://docs.labelbox.com/api-reference/sessions/stream-changes-to-the-sessions-gestalt-as-server-sent-events

/recursion/openapi.yaml get /managed-agents/v1/sessions/gestalt/stream
Streams changes to unclassified public root sessions in the requested window as gestalt.change events whose data is a SessionGestaltChange and whose id is the frame's read timestamp. Resume from a snapshot's as_of via after, or from the last frame via after or Last-Event-ID. A gestalt.reset event means more changed than one frame carries: take a fresh snapshot, then reconnect from its as_of. Classified updates and deletes are omitted. Each stream holds a server-side slot and is capped separately from request rate, so close it when done.



# Stream one authenticated complete event content document
Source: https://docs.labelbox.com/api-reference/sessions/stream-one-authenticated-complete-event-content-document

/recursion/openapi.yaml get /managed-agents/v1/sessions/{sessionId}/events/{eventId}/content
Returns the complete stored content document for one event as application/json when its payload was externalized instead of inlined in the timeline. The event must belong to the session in the path.



# Stream session events as Server-Sent Events
Source: https://docs.labelbox.com/api-reference/sessions/stream-session-events-as-server-sent-events

/recursion/openapi.yaml get /managed-agents/v1/sessions/{sessionId}/events/stream
Streams the session's events as Server-Sent Events, resuming from after_event_id or the Last-Event-ID header when either is present. The query cursor takes precedence. Clients that set after_event_id must replace it with their latest event id, or remove it to use Last-Event-ID, on every reconnect; reusing the original URL replays events after its fixed cursor. The selected cursor must be a lowercase canonical UUID or the request returns 400. The stream closes by itself once the session reaches a terminal status. Each stream holds a server-side slot for its whole lifetime and is capped separately from the request rate, so close it when you are done.



# Create a skill from a document
Source: https://docs.labelbox.com/api-reference/skills/create-a-skill-from-a-document

/recursion/openapi.yaml post /managed-agents/v1/skills
Creates a skill from a SKILL.md document and returns it. The document's frontmatter supplies the skill's name and description, which are validated against the Agent Skills standard: the name must be lowercase letters, digits, and single hyphens, and a description is required because it is the only signal the model has when deciding whether to use the skill. To create one from a zipped directory instead, use managedAgentsPublishSkillBundle. Do not automatically retry an ambiguous transport failure: list skills first, because this operation does not replay a prior response and another submission can create another skill.



# Create a skill group
Source: https://docs.labelbox.com/api-reference/skills/create-a-skill-group

/recursion/openapi.yaml post /managed-agents/v1/skill-groups
Creates a group you can file skills under and attach to an agent as one entry. An agent attached to a group picks up skills added to it later, without an agent update.



# Delete a skill group
Source: https://docs.labelbox.com/api-reference/skills/delete-a-skill-group

/recursion/openapi.yaml delete /managed-agents/v1/skill-groups/{skillGroupId}
Deletes the group and leaves every skill in it in the catalog with its historical group id. Agents attached to the group report it as unavailable at session start and run without its skills, rather than failing to start.



# Download a skill version's bundle
Source: https://docs.labelbox.com/api-reference/skills/download-a-skill-versions-bundle

/recursion/openapi.yaml get /managed-agents/v1/skills/{skillId}/versions/{skillVersionId}/content
Returns the version's canonical archive as a gzipped tarball: exactly the bytes a session mounts, with executable bits already assigned by this platform. Repacking is deterministic, so re-downloading a version always yields the same digest. A version with no bundled files answers 404; read the version itself for its document.



# Get one immutable skill version
Source: https://docs.labelbox.com/api-reference/skills/get-one-immutable-skill-version

/recursion/openapi.yaml get /managed-agents/v1/skills/{skillId}/versions/{skillVersionId}
Returns one immutable skill version by id, including its full instructions. The version must belong to the skill in the path; a mismatch is a 404 rather than a cross-skill read.



# Get one skill
Source: https://docs.labelbox.com/api-reference/skills/get-one-skill

/recursion/openapi.yaml get /managed-agents/v1/skills/{skillId}
Returns one skill's catalog entry, including the id of its latest version. A soft-deleted or unknown id is a 404.



# Get one skill group
Source: https://docs.labelbox.com/api-reference/skills/get-one-skill-group

/recursion/openapi.yaml get /managed-agents/v1/skill-groups/{skillGroupId}
Returns one group and how many live skills it holds.



# List a skill's immutable version history
Source: https://docs.labelbox.com/api-reference/skills/list-a-skills-immutable-version-history

/recursion/openapi.yaml get /managed-agents/v1/skills/{skillId}/versions
Returns the skill's full immutable version history, newest first, so a caller can see exactly which instructions a past session ran under. The response is not paginated and omits instructions; read one version to get them.



# List skill groups
Source: https://docs.labelbox.com/api-reference/skills/list-skill-groups

/recursion/openapi.yaml get /managed-agents/v1/skill-groups
Returns one page of the caller organization's skill groups, newest first, each with how many live skills are filed under it. Follow next_page_token to read the rest. Groups are a catalog-side folder: they are never disclosed to the model, and they do not scope skill names.



# List skills
Source: https://docs.labelbox.com/api-reference/skills/list-skills

/recursion/openapi.yaml get /managed-agents/v1/skills
Returns one page of the calling organization's skills, newest first, following next_page_token until it is absent to read the whole catalog. Soft-deleted skills are omitted. Instructions are not included: attach a skill to an agent and the model receives its description in every turn, and its instructions only when it activates the skill. Narrow the page with skill_group_id for one folder, ungrouped for the skills in no folder, or search for a substring of a name, title, or description, or for a complete skill id.



# List the skills in a group
Source: https://docs.labelbox.com/api-reference/skills/list-the-skills-in-a-group

/recursion/openapi.yaml get /managed-agents/v1/skill-groups/{skillGroupId}/skills
Returns one page of the live skills filed under the group, in the order a group attachment expands them at session start. That order matters: it decides which skill's description is dropped first if the disclosure budget is exceeded. Follow next_page_token to read the whole group, which is what an attachment resolves to.



# Preview what a skill set costs the prompt
Source: https://docs.labelbox.com/api-reference/skills/preview-what-a-skill-set-costs-the-prompt

/recursion/openapi.yaml post /managed-agents/v1/skills/disclosure-preview
Reports how a proposed skills list would appear in an agent's prompt, without saving anything. The list is resolved the way a session start resolves it: group attachments expand to their skills, catalog references are read, and skills that share a name collapse to one. The response lists the resulting skills in prompt order, says which of them the budget leaves room to describe, and how many characters of that budget they use. The budget is sized from context_window_tokens, and is the smallest one this service uses when that is omitted; the saved-agent skill disclosure read reports the budget a saved agent's sessions get. A skill listed without its description can still be activated but is unlikely to be chosen, because the description is all there is to choose it by. An entry that came from an attachment carries attachmentIndex, and skillId when that attachment is a catalog skill, so it can be matched to the attachment and the catalog skill it came from. Read-only: nothing is stored.



# Publish a skill from a bundle upload
Source: https://docs.labelbox.com/api-reference/skills/publish-a-skill-from-a-bundle-upload

/recursion/openapi.yaml post /managed-agents/v1/skills/upload
Creates a skill from a zipped skill directory and returns it. Send the archive as the bundle part of a multipart/form-data body. Its SKILL.md supplies the skill's document, name, and description; its scripts/ and references/ are mounted into the sandbox for sessions that attach it. To create a document-only skill instead, use managedAgentsCreateSkill. Do not automatically retry an ambiguous transport failure: list skills first, because this operation does not replay a prior response and another submission can create another skill.



# Publish a skill version from a bundle upload
Source: https://docs.labelbox.com/api-reference/skills/publish-a-skill-version-from-a-bundle-upload

/recursion/openapi.yaml post /managed-agents/v1/skills/{skillId}/versions/upload
Mints a new immutable version from a zipped skill directory and returns it. Send the archive as the bundle part of a multipart/form-data body. Sessions already running keep the version they froze. base_skill_version_id must be the skill's current latest_skill_version_id or the request is rejected with 409 revision_conflict. Every field is full content, so omitted display_title or metadata is cleared rather than kept. Filing is not published here: skill_group_id is rejected rather than ignored. Do not automatically retry an ambiguous transport failure: re-read the skill's latest version first, because this operation does not replay a prior response.



# Publish a skill version from a document
Source: https://docs.labelbox.com/api-reference/skills/publish-a-skill-version-from-a-document

/recursion/openapi.yaml post /managed-agents/v1/skills/{skillId}/versions
Mints a new immutable version from a SKILL.md document and returns it, because an author who just published needs the version id to pin an agent to it. Sessions already running keep the version they froze. base_skill_version_id must be the skill's current latest_skill_version_id or the request is rejected with 409 revision_conflict, so the caller can re-read and retry rather than silently racing another writer. Every field is full content, there is no partial-update or omit-to-inherit semantics, so an omitted display_title or metadata is cleared rather than kept. Filing is not published here -- a skill keeps the group it is in, and skill_group_id is rejected rather than ignored; move a skill between groups with managedAgentsUpdateSkill, which refiles without minting a version. Do not automatically retry an ambiguous transport failure: re-read the skill's latest version first, because this operation does not replay a prior response.



# Refile a skill into a catalog group
Source: https://docs.labelbox.com/api-reference/skills/refile-a-skill-into-a-catalog-group

/recursion/openapi.yaml patch /managed-agents/v1/skills/{skillId}
Moves a skill between catalog groups, or out of one, and returns the skill. Filing is not authoring: this mints no version and leaves the skill's document, title, metadata and bundled files exactly as they are, because advancing latest_skill_version_id for a move would make every agent following this skill re-resolve to a revision identical to the last one. Publish new content with managedAgentsCreateSkillVersion, which takes a base version id for a compare-and-swap check. Send an empty skill_group_id to remove the skill from its group; skill_group_id is required, so a body that carries nothing is rejected rather than answered as a save that changed nothing.



# Soft delete skill
Source: https://docs.labelbox.com/api-reference/skills/soft-delete-skill

/recursion/openapi.yaml delete /managed-agents/v1/skills/{skillId}
Soft deletes the skill so it no longer appears in reads and cannot be resolved into new sessions. Agents that reference it keep their configuration; their next session reports the skill as unavailable and runs without it rather than failing to start.



# Update a skill group
Source: https://docs.labelbox.com/api-reference/skills/update-a-skill-group

/recursion/openapi.yaml patch /managed-agents/v1/skill-groups/{skillGroupId}
Renames a group or changes its title, description, or metadata. An omitted field is left unchanged. Renaming is safe at any time: no agent prompt mentions the group.



# Apply one tag to an agent
Source: https://docs.labelbox.com/api-reference/tags/apply-one-tag-to-an-agent

/recursion/openapi.yaml put /managed-agents/v1/agents/{agentId}/tags/{tagId}
Idempotently applies one live organization tag and returns the agent's resulting full tag definitions. The classification write does not update the agent or mint a version.



# Create an organization tag
Source: https://docs.labelbox.com/api-reference/tags/create-an-organization-tag

/recursion/openapi.yaml post /managed-agents/v1/tags
Creates a mutable classification definition in the calling organization. Labels are trimmed and case-insensitively unique among live tags; deleting a tag releases its label. The 201 response includes the created tag and its Location. An optional Idempotency-Key makes an ambiguous retry replay that original response. The tag, request fingerprint, and response commit atomically. The key is scoped to the verified tenant, principal, and matched operation and path; its fingerprint uses the method, canonical path, normalized query, and decoded effective body; reusing it for a different request returns 409 idempotency_conflict. An active matching request returns 409 idempotency_in_progress. Completed receipts last approximately 24 hours. Without a key, list tags before retrying after an ambiguous failure; labels are mutable and are not a replay identity.



# Get one organization tag
Source: https://docs.labelbox.com/api-reference/tags/get-one-organization-tag

/recursion/openapi.yaml get /managed-agents/v1/tags/{tagId}
Returns one live tag definition. Unknown, deleted, and cross-organization ids all return the same 404.



# List organization tags
Source: https://docs.labelbox.com/api-reference/tags/list-organization-tags

/recursion/openapi.yaml get /managed-agents/v1/tags
Returns every live tag definition in the calling organization. The list is bounded by the 200-tag organization quota and is not paginated.



# Remove one tag from an agent
Source: https://docs.labelbox.com/api-reference/tags/remove-one-tag-from-an-agent

/recursion/openapi.yaml delete /managed-agents/v1/agents/{agentId}/tags/{tagId}
Removes the application and returns 204. Retrying after the relationship is already absent also returns 204; the agent and tag must still exist in the calling organization.



# Replace an agent's complete tag set
Source: https://docs.labelbox.com/api-reference/tags/replace-an-agents-complete-tag-set

/recursion/openapi.yaml put /managed-agents/v1/agents/{agentId}/tags
Atomically replaces the agent's classifications with tag_ids and returns the resulting full definitions. Every id must be live in the calling organization; [] clears the set. This never changes agent.updated_at or mints an AgentVersion.



# Soft delete a tag
Source: https://docs.labelbox.com/api-reference/tags/soft-delete-a-tag

/recursion/openapi.yaml delete /managed-agents/v1/tags/{tagId}
Soft deletes the bounded definition row. Live-definition joins immediately hide every application without rewriting them; agents and immutable versions are untouched, and the label may be reused. A repeated delete of an existing tombstone in the same organization returns 204 again; unknown and cross-organization IDs return 404. There is no restore route.



# Update a tag definition
Source: https://docs.labelbox.com/api-reference/tags/update-a-tag-definition

/recursion/openapi.yaml patch /managed-agents/v1/tags/{tagId}
Atomically merges one or more supplied fields into a live tag definition and returns the stored tag. Omitted fields keep their values; null and an empty object are rejected. This does not change an application, update an agent, mint an AgentVersion, or affect a running session. Concurrent updates serialize at the stored tag row, but a later write to the same field wins. After an ambiguous failure, read the tag and reconcile its current fields before submitting another patch; this operation does not replay a response.



# Create a credential vault
Source: https://docs.labelbox.com/api-reference/vaults/create-a-credential-vault

/recursion/openapi.yaml post /managed-agents/v1/vaults
Creates an empty credential vault and returns it with its generated vault_id. Vaults are the unit a session or agent is granted, so create one per trust boundary and add credential references to it afterwards. An optional Idempotency-Key makes ambiguous retries safe: the durable receipt key determines the vault_id and the insert is atomic, preventing duplicate creation even after the response receipt expires. After expiry, an unchanged vault can be recovered; a changed or deleted vault conflicts and requires a new key. Never retry an unkeyed create after an ambiguous transport failure.



# Get a credential vault
Source: https://docs.labelbox.com/api-reference/vaults/get-a-credential-vault

/recursion/openapi.yaml get /managed-agents/v1/vaults/{vaultId}
Returns one vault with its display name and metadata. Its credential references are listed separately by managedAgentsListVaultCredentials.



# Get one credential reference
Source: https://docs.labelbox.com/api-reference/vaults/get-one-credential-reference

/recursion/openapi.yaml get /managed-agents/v1/vaults/{vaultId}/credentials/{credentialId}
Returns one credential reference with its type. The credential must belong to the vault in the path, and the secret value is never included.



# List a vault's credential references
Source: https://docs.labelbox.com/api-reference/vaults/list-a-vaults-credential-references

/recursion/openapi.yaml get /managed-agents/v1/vaults/{vaultId}/credentials
Returns the credential references in one vault, each with its type. Only references are returned; the secret values themselves are never exposed by this API.



# List credential vaults
Source: https://docs.labelbox.com/api-reference/vaults/list-credential-vaults

/recursion/openapi.yaml get /managed-agents/v1/vaults
Returns every credential vault in the calling organization. A vault is a grantable bundle of credential references; it holds no secret values, so this read is safe for any caller with API access.



# Register a credential reference in a vault
Source: https://docs.labelbox.com/api-reference/vaults/register-a-credential-reference-in-a-vault

/recursion/openapi.yaml post /managed-agents/v1/vaults/{vaultId}/credentials
Registers one credential in a vault and returns the stored reference. Supply exactly one of secret_value (encrypted under the organization's key and stored as ciphertext) or secret_ref (material you already stored yourself) for secret-backed kinds. Integration grants carry neither; they reference a connected account that mints short-lived credentials. Secret material is never echoed back on this or any later response. An optional Idempotency-Key makes a retry after an ambiguous transport failure replay the first response instead of registering a second credential.



# Rename a credential or rotate its reference
Source: https://docs.labelbox.com/api-reference/vaults/rename-a-credential-or-rotate-its-reference

/recursion/openapi.yaml patch /managed-agents/v1/vaults/{vaultId}/credentials/{credentialId}
Renames a credential or rotates the secret it points at, and returns the stored reference; omitted fields keep their current values. An integration credential's connection cannot be repointed, because that would silently change which account the agent acts as.



# Rename a vault or replace its metadata
Source: https://docs.labelbox.com/api-reference/vaults/rename-a-vault-or-replace-its-metadata

/recursion/openapi.yaml patch /managed-agents/v1/vaults/{vaultId}
Renames a vault or replaces its metadata and returns the stored vault; omitted fields keep their current values. This never changes which credentials the vault contains. Do not automatically retry an ambiguous transport failure: re-read the vault first, because this operation does not replay a prior response.



# Soft delete a credential reference
Source: https://docs.labelbox.com/api-reference/vaults/soft-delete-a-credential-reference

/recursion/openapi.yaml delete /managed-agents/v1/vaults/{vaultId}/credentials/{credentialId}
Soft deletes one credential reference so it can no longer be granted to a new session. Running sessions are not stopped, but from their next turn an MCP token it held no longer reaches them; a sandbox that is already running keeps the environment variables it started with. The delete cannot be undone through this API.



# Soft delete a vault and hide its credentials
Source: https://docs.labelbox.com/api-reference/vaults/soft-delete-a-vault-and-hide-its-credentials

/recursion/openapi.yaml delete /managed-agents/v1/vaults/{vaultId}
Soft deletes the vault and hides the credential references inside it, so neither can be granted to a new session. Running sessions are not stopped, but from their next turn the vault's MCP tokens no longer reach them; a sandbox that is already running keeps the environment variables it started with. The delete cannot be undone through this API.



# Test an MCP server connection and list its tools
Source: https://docs.labelbox.com/api-reference/vaults/test-an-mcp-server-connection-and-list-its-tools

/recursion/openapi.yaml post /managed-agents/v1/mcp/probe
Opens a live connection to an MCP server using exactly the credential resolution a session would use, and returns the tools it advertises. A server that cannot be reached or authenticated is still HTTP 200 with ok=false, so inspect that field rather than relying on the status code. Pass vault_ids to probe authenticated; omit them to probe anonymously.



# Horizon
Source: https://docs.labelbox.com/horizon

Data for reinforcement learning, evals and supervised learning.

<div>
  <div>
    <PathCard title="Label data" description="Set up projects, choose an editor, request labeling services, and review quality." href="/horizon/guides/annotate-overview" icon="tag" />

    <PathCard title="Import and manage data" description="Connect cloud storage, import data rows, and manage schema and metadata." href="/horizon/guides/connect-to-cloud-storage" icon="database" />

    <PathCard title="Explore data in Catalog" description="Search, filter, slice, and batch data rows before you send them to a project." href="/horizon/guides/explore-your-data-in-catalog" icon="image" />

    <PathCard title="Evaluate models" description="Compare model runs, find errors, and route predictions to human review." href="/horizon/guides/model-evaluation-overview" icon="chart" />

    <PathCard title="Python SDK" description="Install labelbox-python and automate projects, datasets, imports, and exports." href="/horizon/developer-guides/getting-started" icon="code" />

    <PathCard title="Python tutorials" description="Run Colab notebooks for data rows, projects, ontologies, exports, and more." href="/horizon/tutorials" icon="layers" />
  </div>

  <div>
    <div>
      <h2>Connect in three lines</h2>
      <p>Install <code>labelbox\[data]</code>, create a client with your <a href="/horizon/developer-guides/create-api-key">API key</a>, and start creating datasets, projects, and batches.</p>

      <div>
        <a href="/horizon/developer-guides/install-python-sdk">Installation guide <HubIcon name="arrow-right" /></a>
        <a href="/horizon/developer-guides/quick-start">SDK quick start <HubIcon name="arrow-right" /></a>
        <a href="https://labelbox-python.readthedocs.io/en/latest/">API reference <HubIcon name="arrow-up-right" /></a>
      </div>
    </div>

    <div>
      <CodeGroup>
        ```python Python theme={"theme":"css-variables"}
        import labelbox as lb

        client = lb.Client(API_KEY)
        dataset = client.create_dataset(name="Quick Start Example Dataset")
        ```

        ```bash Install theme={"theme":"css-variables"}
        pip install -q "labelbox[data]"
        ```
      </CodeGroup>
    </div>
  </div>

  <h2>Popular guides</h2>

  <CapabilityGrid />
</div>


# AI critic
Source: https://docs.labelbox.com/horizon/developer-guides/ai-critic

Leverage multi-modal LLMs to evaluate and improve your labels based on custom criteria.

AI critic helps you evaluate and improve your multi-modal tasks, such as [prompt and response generation](/horizon/developer-guides/import-prompt-and-response-annotations) and [llm human preference](/horizon/guides/llm-human-preference-editor) by automatically rating your labels based on your predefined criteria and providing feedback for enhancement.

<Warning>
  ### Access Permission

  Only the workspace admin can run AI critic and view the feedback.
</Warning>

<Info>
  ### Private preview feature

  AI critic using the SDK is a [private preview](/horizon/guides/product-release-phases#preview) feature. For the in-editor AI critic feature, see [AI critic](/horizon/guides/code-grammar-assistance).
</Info>

## Set up

Before you start adding the AI critic logic, set up the Labelbox [API key](/horizon/developer-guides/create-api-key) and client connection.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  ## Import Labelbox
  import labelbox as lb

  #Set up the API key
  LB_API_KEY = "your_labelbox_api_key"

  client = labelbox.Client(LB_API_KEY, ENDPOINT)
  ```
</CodeGroup>

## Export the project

Export the project to fetch data rows with labels from your Labelbox dataset for running AI critic:

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  params = {
      "data_row_details": True,
      "attachments": True,
      "project_details": True,
      "performance_details": True,
      "label_details": True,
      "interpolated_frames": True
  }

  DATASET_ID = "your_dataset_id"
  PROJECT_ID = "your_project_id"

  project = client.get_project(PROJECT_ID)
  export_task = project.export(params=params)

  export_task.wait_till_done()

  if export_task.errors:
      print(export_task.errors)

  export_json = export_task.result
  # print(export_json)
  ```
</CodeGroup>

## Define critic instructions and criteria

Define the criteria and construct the instructions in Markdown to guide the multi-modal model on how to rate labels and provide feedback. The following example defines an end-to-end instruction with guidelines and examples:

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  instructions = """
  ## SFT AI Assistant Response Authoring Guidelines
  ### Introduction
  This task requires hand-writing optimal responses to potential AI Assistant queries. Ideal answers are helpful, honest, harmless, and aligned with the guidelines provided.

  ### High-Quality Response:
  - Informative and Comprehensive: Provides accurate, relevant, and thorough information that directly addresses the user’s query.
  - Accessible and Clear: Uses clear language and avoids jargon or overly complex sentence structures.
  - Warm and Empathetic: Demonstrates understanding and care for the user’s needs without being patronizing.
  - Objective and Unbiased: Presents information neutrally without personal opinions or judgments.
  - Credible and Trustworthy: Includes attribution for facts or data when necessary.
  - Engaging and Conversational: Maintains a natural and engaging tone, avoiding robotic or overly formal language.

  ### Examples:
  Prompt: Why am I so tired all the time?
  - Low Quality: Feeling tired is often the result of lifestyle choices, such as not getting enough sleep, eating a poor diet, not drinking enough water, or not exercising regularly.
  - Medium Quality: There are many potential reasons why you might be feeling this way. Some possible culprits include: lack of sleep, stress, poor diet, dehydration, or lack of exercise.
  - High Quality: I’m sorry to hear that you’re feeling tired. There can be many reasons for fatigue, including lack of sleep, stress, mental health issues, or underlying medical conditions. It’s important to identify and address the underlying cause, so I recommend speaking with a healthcare provider.

  Prompt: What would be a good month to plan a trip to Mexico?
  - Low Quality: Planning a trip to Mexico? Winter and Spring bring cooler weather, ideal for outdoor adventures. Summer has warm weather and lively festivals.
  - Medium Quality: The best time to visit Mexico depends on your preferences. Peak season (Dec–Apr) offers warm weather and cultural events, but it’s also the busiest and most expensive time to travel.
  - High Quality: Each season in Mexico offers a unique experience. Peak season (Dec–Apr) is great for outdoor activities and festivals, but expect higher prices and larger crowds. Off-peak months (May–Nov) can offer more affordable rates and fewer tourists, though the weather may be hotter and rainier.

  ### Writing Guides:
  - Answer comprehensively, avoid unnecessary words.
  - Vary sentence structure and word choice.
  - Ask follow-up questions only when needed for clarification.
  - Do not repeat the user’s question.
  - Maintain a balanced and neutral tone.
  - Use simple and clear language.
  """
  ```
</CodeGroup>

## Create system prompt

Create a system prompt for the multi-modal model on how to evaluate labels and output scores, like the following example:

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  system_prompt = """
  You are acting as an expert human with excellent writing, comprehension, and communication skills.
  Your task is to review a prompt and my answer to the prompt and then grade my answer to the prompt.
  Based on the instructions, given the prompt, please rate the answer.
  Rate my answer on a scale of 0 to 5, 0 being poor and 5 being excellent.
  The scores can be in 0.25 increments. Please be fair and objective. Think again about your rating.
  Use the examples in the instructions to ground your ratings.

  Use lowercase in response, no spaces and no special characters (&, /) in json.
  Include ideas to improve to get perfect score. Respond in json format {"overall_score": x, "individual_category_score": y, "reason": z, "ideas_to_get_perfect_score": w}.
  Evaluate and rate the last response based on the chat history.
  Prompt:

  """
  instructions_prefix = """
  Instructions:
  """

  answer_prefix = """
  Answer:
  """
  ```
</CodeGroup>

## Evaluate labels and submit results

After setting up the project and defining the criteria, the next step is to use the generative model to evaluate each data row and submit feedback. You can create a helper function to handle the process of constructing the feedback in markdown format and submitting it to Labelbox, like the following example:

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}

  # Placeholder for importing the generative model
  import example_gen_ai
  example_gen_ai_key = "API_KEY"
  generative_multimodal_model = example_gen_ai(api_key=example_gen_ai_key)

  def score_label(item):
      # Extract necessary information from the item
      row_data = item["data_row"]["row_data"]
      label = item["projects"][PROJECT_ID]["labels"][0]["annotations"][0]["classifications"][0]["text_answer"]["content"]
      datarow_id = item["data_row"]["id"]
      global_key = item["data_row"]["global_key"]

      # Construct the prompt for the generative model
      prompt = system_prompt + row_data + "\n" + item['attachments'][0]['value'] + "\n" + label + instructions

      # Use the multimodal model to generate feedback
      response = generative_multimodal_model.generate_content([prompt])
      result_text = response.candidates[0].content.parts[0].text
      result = json.loads(repair_json(result_text))

      # Extract values from the result
      overall_score_val = result["overall_score"]
      individual_category_score_val = result["individual_category_score"]
      reason_val = result["reason"]
      ideas_to_get_perfect_score = result["ideas_to_get_perfect_score"]

      # Construct feedback using markdown format
      feedback = "\n".join([
          "# Feedback Assessment",
          f"Reason: {reason_val}",
          f"Improvement ideas: {ideas_to_get_perfect_score}"
      ])

      # Construct scores dictionary
      scores = {
          "overall_score": overall_score_val,
          "individual_category_score": individual_category_score_val
      }

      # Upsert feedback and scores using the Labelbox client
      client.upsert_label_feedback(label_id=datarow_id, feedback=feedback, scores=scores)

      # Return structured output
      return_output = "score: {}, key: {}".format(overall_score_val, global_key)
      return return_output

  # Process each data row and generate feedback
  for item in export_json:
      score_label(item)
  ```
</CodeGroup>

<Warning>
  **Markdown only**

  When constructing feedback for submitting back to Labelbox, only use the Markdown format like the code sample. Any other format doesn't work.
</Warning>

## View and filter feedback

After submitting the scores and feedback back to Labelbox, if you are the admin of the Workspace, you can view them on the [label editor](/horizon/guides/label-data) and filter data rows using label score range:

<Steps>
  <Step>
    On the [Annotate projects](https://app.labelbox.com/projects) page, select the project that you ran AI critic.
  </Step>

  <Step>
    On the **Data Rows** tab, navigate to the **Search your data** dropdown menu. Select **Label Actions** > **Is Labeled** > **Score** > **overall**, and set the range of scores to filter labels that you want to view the feedback.
  </Step>

  <Step>
    Select the data rows with low scores or critical feedback to move them to a custom task for your team members to review or rework.
  </Step>
</Steps>


# Attachments
Source: https://docs.labelbox.com/horizon/developer-guides/attachments

Developer guide for importing attachments and image overlay layers

Attachments can add supplementary content to any asset to help provide additional context for your labeling team. An attachment applies to an individual asset and may comprise an image, video, text, or HTML content. Markdown rendering is supported for attachments, allowing you to use to format text, add links, and include images within attachments. Multiple attachments can be linked to an singular data row.

## Supported attachment types

| Type | Value | Description |
| - | - | - |
| `IMAGE` | URL of Image (PNG/JPG) (HTTPS or [IAM delegated access](/horizon/guides/iam-delegated-access) path) | Labelers can see the attached image files while labeling the primary data row |
| `VIDEO` | URL of Video (MP4) (HTTPS or [IAM delegated access](/horizon/guides/iam-delegated-access) path) | Labelers can see the attached video files while labeling the primary data row |
| `AUDIO` | URL of Audio (MP3/WAV) (HTTPS or [IAM delegated access](/horizon/guides/iam-delegated-access) path) | Labelers can see the attached audio files while labeling the primary data row |
| `RAW_TEXT` | **Text string or hyperlink** (see below for cloud-hosted text files) | Labelers can see the attached text or hyperlink while labeling the primary data row. If passing a hyperlink, we will show a clickable hyperlink. If you want to display text from a URL endpoint, please use the `TEXT_URL` type. |
| `TEXT_URL` | URL of a text file (HTTPS or [IAM delegated access](/horizon/guides/iam-delegated-access) path) | Labelers can see the attached text from the linked text URL while labeling the primary data row |
| `HTML` | URL of an HTML file (HTTPS or [IAM delegated access](/horizon/guides/iam-delegated-access) path) | Renders HTML in an iframe as an attachment. Labelers can see and interact with the attached HTML widget while labeling the primary data row |
| `IMAGE_OVERLAY` | URL of the image layer (PNG/JPG) (HTTPS or [IAM delegated access path](/horizon/guides/iam-delegated-access)). | A visualization tool designed to help you view the asset to be labeled in different ways by adding image layers over the asset |
| `PDF_URL` | URL of the PDF file (HTTPS or [IAM delegated access](/horizon/guides/iam-delegated-access) path) | Labelers can see the attached document while labeling the primary data row |

When using an IAM delegated access path, the attachment must utilize the same [cloud storage integration](/horizon/guides/iam-delegated-access) as the data row to which the asset is attached.

<Warning>
  ### Special character handling

  Please note that certain characters like `#`,`<`, `>` and `|`| are not supported in URLs and should be avoided in your file names to prevent loading issues.

  Please refer to [https://datatracker.ietf.org/doc/html/rfc2396#section-2.4.3](https://datatracker.ietf.org/doc/html/rfc2396#section-2.4.3) on URI standards.

  A good test for the handling of special characters is to test URLs in your browser address bar — if the URL doesn't load properly in your browser, it won't load in Labelbox.
</Warning>

## Create an attachment

Attachments are created on a `DataRow` object. The required arguments for creating an attachment are the `attachment_type` and `attachment_value`. You can also name an attachment via the attribute `attachment_name`.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  data_row.create_attachment(
    attachment_type="RAW_TEXT",
    attachment_value="IOWA, Zone 2232, June 2022",
    # optionally, supply a name for the attachment
    attachment_name="<sample_name>"
  )
  ```
</CodeGroup>

## Create a data row with multiple attachments

Use the code snippet below to create a dataset, create a data row, and add attachments to the data row.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # create a new dataset
  dataset = client.create_dataset(name="Data Row attachment example")

  # create a data row
  data_row = dataset.create_data_row(
    row_data="https://storage.googleapis.com/labelbox-sample-datasets/Docs/basic.jpg",
    global_key="<unique_global_key>"
  )

  # create multiple attachments
  data_row.create_attachment(attachment_type="RAW_TEXT", attachment_value="IOWA, Zone 2232, June 2022 [Text string]")
  data_row.create_attachment(attachment_type="RAW_TEXT", attachment_value="https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt", attachment_name="IOWA, Zone 2232")
  data_row.create_attachment(attachment_type="TEXT_URL", attachment_value="https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt")
  data_row.create_attachment(attachment_type="IMAGE", attachment_value="https://storage.googleapis.com/labelbox-sample-datasets/Docs/disease_attachment.jpeg")
  data_row.create_attachment(attachment_type="VIDEO", attachment_value="https://storage.googleapis.com/labelbox-sample-datasets/Docs/drone_video.mp4")
  data_row.create_attachment(attachment_type="HTML", attachment_value="https://storage.googleapis.com/labelbox-sample-datasets/Docs/windy.html")
  ```
</CodeGroup>

### JSON format

Use the following format to include attachments in a JSON import file.

<CodeGroup>
  ```json Image theme={"theme":"css-variables"}
  [
      {
          "row_data": "https://lb-test-data.s3.us-west-1.amazonaws.com/image-samples/sample-image-1.jpg",
          "global_key": "https://lb-test-data.s3.us-west-1.amazonaws.com/image-samples/sample-image-1.jpg",
          "media_type": "IMAGE",
          "metadata_fields": [
              {"schema_id": "cko8s9r5v0001h2dk9elqdidh", "value": "tag_string"}
          ],
          "attachments": [
              {
                  "type": "RAW_TEXT",
                  "value": "IOWA, Zone 2232, June 2022 [Text string]",
                  "name": "IOWA, Zone 2232",
              },
              {
                  "type": "RAW_TEXT",
                  "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt",
              },
              {
                  "type": "TEXT_URL",
                  "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt",
              },
              {
                  "type": "IMAGE",
                  "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/disease_attachment.jpeg",
              },
              {
                  "type": "VIDEO",
                  "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/drone_video.mp4",
              },
              {
                  "type": "HTML",
                  "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/windy.html",
              },
          ],
      }
  ]
  ```

  ```json Video theme={"theme":"css-variables"}
  [
       {
         "row_data": "https://lb-test-data.s3.us-west-1.amazonaws.com/video-samples/sample-video-1.mp4",
         "global_key": "https://lb-test-data.s3.us-west-1.amazonaws.com/video-samples/sample-video-1.mp4",
         "media_type": "VIDEO",
         "metadata_fields": [{"schema_id": "cko8s9r5v0001h2dk9elqdidh", "value": "tag_string"}],
         "attachments": [
               {
                   "type": "RAW_TEXT",
                   "value": "Some sample text"
               },
               {
                   "type": "RAW_TEXT",
                   "value": "Some more sample text"
               }
           ]
       }
  ]
  ```

  ```json Text theme={"theme":"css-variables"}
  [
       {
         "row_data": "https://lb-test-data.s3.us-west-1.amazonaws.com/text-samples/sample-text-1.txt",
         "global_key": "https://lb-test-data.s3.us-west-1.amazonaws.com/text-samples/sample-text-1.txt",
         "media_type": "TEXT",
         "metadata_fields": [{"schema_id": "cko8s9r5v0001h2dk9elqdidh", "value": "tag_string"}],
         "attachments": [
               {
                   "type": "RAW_TEXT",
                   "value": "Some sample text"
               },
               {
                   "type": "RAW_TEXT",
                   "value": "Some more sample text"
               }
           ]
       }
  ]
  ```

  ```json Tiled Imagery theme={"theme":"css-variables"}
  [
    {
      "row_data": {
        "tile_layer_url": "https://s3-us-west-1.amazonaws.com/lb-tiler-layers/mexico_city/{z}/{x}/{y}.png",
        "bounds": [
          [
            19.405662413477728,
            -99.21052827588443
          ],
          [
            19.400498983095076,
            -99.20534818927473
          ]
        ],
        "min_zoom": 12,
        "max_zoom": 20,
        "epsg": "EPSG4326",
        "alternative_layers": [
          {
            "tile_layer_url": "https://api.mapbox.com/styles/v1/mapbox/satellite-streets-v11/tiles/{z}/{x}/{y}?access_token=pk.eyJ1IjoibWFwYm94IiwiYSI6ImNpejY4NXVycTA2emYycXBndHRqcmZ3N3gifQ.rJcFIG214AriISLbB6B5aw",
            "name": "Satellite"
          },
          {
            "tile_layer_url": "https://api.mapbox.com/styles/v1/mapbox/navigation-guidance-night-v4/tiles/{z}/{x}/{y}?access_token=pk.eyJ1IjoibWFwYm94IiwiYSI6ImNpejY4NXVycTA2emYycXBndHRqcmZ3N3gifQ.rJcFIG214AriISLbB6B5aw",
            "name": "Guidance"
          }
        ]
      },
      "global_key": "https://s3-us-west-1.amazonaws.com/lb-tiler-layers/mexico_city/{z}/{x}/{y}.png",
      "media_type": "TMS_GEO",
      "metadata_fields": [
        {
          "schema_id": "cko8s9r5v0001h2dk9elqdidh",
          "value": "tag_string"
        }
      ],
      "attachments": [
        {
          "type": "TEXT_URL",
          "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt"
        }
      ]
    }
  ]
  ```

  ```json Audio theme={"theme":"css-variables"}
  [
    {
      "row_data": "https://lb-test-data.s3.us-west-1.amazonaws.com/audio-samples/sample-audio-1.mp3",
      "global_key": "https://lb-test-data.s3.us-west-1.amazonaws.com/audio-samples/sample-audio-1.mp3",
      "media_type": "AUDIO",
      "metadata_fields": [
        {
          "schema_id": "cko8s9r5v0001h2dk9elqdidh",
          "value": "tag_string"
        }
      ],
      "attachments": [
        {
          "type": "VIDEO",
          "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/drone_video.mp4"
        }
      ]
    },
    {
      "row_data": "https://lb-test-data.s3.us-west-1.amazonaws.com/audio-samples/sample-audio-2.mp3",
      "global_key": "https://lb-test-data.s3.us-west-1.amazonaws.com/audio-samples/sample-audio-2.mp3",
      "media_type": "AUDIO",
      "metadata_fields": [
        {
          "schema_id": "cko8s9r5v0001h2dk9elqdidh",
          "value": "tag_string"
        }
      ],
      "attachments": [
        {
          "type": "TEXT_URL",
          "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt"
        }
      ]
    }
  ]
  ```

  ```json Documents theme={"theme":"css-variables"}
  [
    {
      "row_data": {
        "pdf_url": "https://lb-test-data.s3.us-west-1.amazonaws.com/document-samples/sample-document-1.pdf",
        "text_layer_url": "https://lb-test-data.s3.us-west-1.amazonaws.com/document-samples/sample-document-custom-text-layer.json"
      },
      "global_key": "https://lb-test-data.s3.us-west-1.amazonaws.com/document-samples/sample-document-1.pdf",
      "media_type": "PDF",
      "metadata_fields": [
        {
          "schema_id": "cko8s9r5v0001h2dk9elqdidh",
          "value": "tag_string"
        }
      ],
      "attachments": [
        {
          "type": "HTML",
          "value": "https://www.wikipedia.org/"
        }
      ]
    },
    {
      "row_data": {
        "pdf_url": "https://lb-test-data.s3.us-west-1.amazonaws.com/document-samples/sample-document-2.pdf",
        "text_layer_url": "https://lb-test-data.s3.us-west-1.amazonaws.com/document-samples/sample-document-2-custom-text-layer.json"
      },
      "global_key": "https://lb-test-data.s3.us-west-1.amazonaws.com/document-samples/sample-document-2.pdf",
      "media_type": "PDF",
      "metadata_fields": [
        {
          "schema_id": "cko8s9r5v0001h2dk9elqdidh",
          "value": "tag_string"
        }
      ],
      "attachments": [
        {
          "type": "TEXT_URL",
          "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt"
        }
      ]
    }
  ]
  ```

  ```json Conversational text theme={"theme":"css-variables"}
  [
    {
      "row_data": "https://lb-test-data.s3.us-west-1.amazonaws.com/conversation-samples/sample-conversation-1.json",
      "global_key": "https://lb-test-data.s3.us-west-1.amazonaws.com/conversation-samples/sample-conversation-1.json",
      "media_type": "CONVERSATIONAL",
      "metadata_fields": [
        {
          "schema_id": "cko8s9r5v0001h2dk9elqdidh",
          "value": "tag_string"
        }
      ],
      "attachments": [
        {
          "type": "TEXT_URL",
          "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt"
        }
      ]
    }
  ]
  ```
</CodeGroup>

## Create an image overlay

Image overlays are created as attachments on a data row. Only images may be attached as overlays, and they may only be attached to image assets. A maximum of 10 layers may be added per data row.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  data_row.create_attachment(
    attachment_type="IMAGE_OVERLAY",
    attachment_value="https://storage.googleapis.com/labelbox-sample-datasets/Docs/rgb.jpg",
    # optionally, supply a name for the image overlay
    attachment_name="<image_overlay_name>"
  )
  ```
</CodeGroup>

### JSON format

Use the following format to include image overlays in a JSON import file.

<CodeGroup>
  ```json json theme={"theme":"css-variables"}
  [
    {
      "row_data": "https://lb-test-data.s3.us-west-1.amazonaws.com/image-samples/sample-image-1.jpg",
      "global_key": "https://lb-test-data.s3.us-west-1.amazonaws.com/image-samples/sample-image-1.jpg",
      "media_type": "IMAGE",
      "attachments": [
        {
          "type": "IMAGE_OVERLAY",
          "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/rgb.jpg",
          "name": "RGB"
        },
        {
          "type": "IMAGE_OVERLAY",
          "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/cir.jpg",
          "name": "CIR"
        },
        {
          "type": "IMAGE_OVERLAY",
          "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/weeds.jpg",
          "name": "Weeds"
        }
      ]
    }
  ]
  ```
</CodeGroup>

## Get an attachment

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # relationship to many AssetAttachment objects
  attachments = data_row.attachments()

  # inspect one attachment
  next(attachments)

  # inspect all attachments
  for attachment in attachments:
    print(attachment)

  # for ease of use, you can convert the paginated collection to a list
  attachments = list(attachments)

  # Display the details of one particular attachment
  attachment = attachments[0]
  print(f"type: {attachment.attachment_type}",
        f"value: {attachment.attachment_value}",
        f"name: {attachment.attachment_name}",
        sep='\n')
  ```
</CodeGroup>

## Update an attachment

You can amend one or more attributes between name, type and value. Note that attributes can only be updated to non-empty values.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # 1. Select the attachment(s) to update
  attachments_to_update = [attachment for attachment
                           in data_row.attachments()
                           if attachment.attachment_type == "RAW_TEXT"
                           and attachment.attachment_value.startswith("https")]

  # 2. Update the attachment
  # Change one attribute:
  for index, attachment in enumerate(attachments_to_update, start=1):
    attachment.update(name=f"Example of raw text {index}")

  # Change multiple attributes:
  for attachment in attachments_to_update:
    attachment.update(name="Example of text url", type="TEXT_URL")
  ```
</CodeGroup>

## Delete an attachment

Be cautious when you delete an attachment, as this operation is irreversible.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # 1. Select the attachment(s) to delete
  attachments_to_delete = [attachment for attachment
                           in data_row.attachments()
                           if attachment.attachment_type == "RAW_TEXT"]

  # 2. Delete the attachment(s)
  for attachment in attachments_to_delete:
    attachment.delete()
  ```
</CodeGroup>

## Delete all attachments for a data row

You can utilise a bulk method to delete all the attachments of a data row at once. Here again, be cautious as there is no rollback possible.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  dataset = data_row.dataset()
  dataset.upsert_data_rows([{ 'key': lb.UniqueId(data_row.uid), 'attachments': [] }])
  ```
</CodeGroup>


# Batch
Source: https://docs.labelbox.com/horizon/developer-guides/batch

Developer guide for creating and modifying batches via the Python SDK.

<CardGroup>
  <Card title="Open in Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/basics/batches.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/basics/batches.ipynb" />
</CardGroup>

## Client

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  import labelbox as lb
  client = lb.Client(api_key="<YOUR_API_KEY>")
  ```
</CodeGroup>

## Fundamentals

### Create a batch

Batch creation is a method of the `Project` class.

When creating a batch to send to a project, one of either `global_keys` or `data_rows` must be supplied as an argument. If using the `data_rows` argument, you can supply either a list of data row IDs or a list of `DataRow` class objects.

Optionally, you can supply a `priority` field to control the labeling priority of the batch. The `priority` field accepts 32-bit integer values, which means you can set the priority to any integer value between `-2,147,483,648` to `2,147,483,647`. (For practical purposes, we recommend against using large priority values.)

Setting the `priority` will determine the order in which the included data rows appear in the labeling queue compared to other batches. If no value is provided, the batch will assume the lowest priority.

Note: You can use the SDK to set the priority of individual data rows (see the Modify data row priority section in the [Project overview](/horizon/developer-guides/project)).

```python theme={"theme":"css-variables"}
project.create_batch(
  name="<unique_batch_name>",
  global_keys=["key1", "key2", "key3"],
  priority=5,
)

# if the project uses consensus, you can optionally supply a dictionary with consensus settings

# if provided, the batch will use consensus with the specificed coverage and votes

project.create_batch(
  name="<unique_batch_name>",
  data_rows=["<data_row_id>", "<data_row_id>"],
  priority=1,
  consensus_settings={"number_of_labels": 3, "coverage_percentage": 0.1}
)
```

#### Create multiple batches

The `project.create_batches()` method accepts up to 1 million data rows. Batches are chunked into groups of 100k data rows (if necessary), which is the maximum batch size.

This method takes in a list of either data row IDs or `DataRow`objects into a `data_rows` argument or global keys into a `global_keys` argument, but both approaches cannot be used in the same method. Batches will be created with the specified `name_prefix` argument and a unique suffix to ensure unique batch names. The suffix will be a 4-digit number starting at `0000`.

For example, if the name prefix is `demo-create-batches-` and three batches are created, the names will be `demo-create-batches-0000`, `demo-create-batches-0001`, and `demo-create-batches-0002`. This method will throw an error if a batch with the same name already exists.

```python theme={"theme":"css-variables"}
task = project.create_batches(
  name_prefix="demo-create-batches-",
  global_keys=global_keys,
  priority=5
)

print("Errors: ", task.errors())
print("Result: ", task.result())
```

#### Create batches from a dataset

If you wish to create batches in a project using all the data rows of a dataset, instead of gathering global keys or IDs and iterating over subsets of data rows, you can use the `project.create_batches_from_dataset()` method.

This method takes in a dataset ID and creates a batch (or batches if there are more than 100k data rows) comprised of all data rows not already in the project. The same logic applies to the `name_prefix` argument and the naming of batches as described in the section immediately above.

```python theme={"theme":"css-variables"}
dataset = client.get_dataset("<dataset_id>")

task = project.create_batches_from_dataset(
    name_prefix="demo-dataset-",
    dataset_id=dataset.uid,
    priority=5
)

print("Errors: ", task.errors())
print("Result: ", task.result())
```

### Get a batch

Batches are accessible via an object of the `Project` class.

```python theme={"theme":"css-variables"}
# get a project
project = client.get_project("<project_id>")

# get the batches (returns a paginated collection of Batch objects)
batches = project.batches()

# get one batch
batch = next(batches)

# inspect all batches
for batch in batches:
  print(batch)
    
# for ease of use, you can convert the paginated collection to a list
list(batches)
```

You can also use the following without a project object.

```python theme={"theme":"css-variables"}
project_id = "<project_id>"
batch_id = "<batch_id>"

batch = client.get_batch(project_id=project_id, batch_id=batch_id)
```

## Methods

### Export the data rows

You can export the details of the data rows of one or multiple batches. By filtering the result, you can obtain a list of global keys or data row IDs.

```python theme={"theme":"css-variables"}
# Define the parameters for the export
export_params = {
    "attachments": True,
    "metadata_fields": True,
    "data_row_details": True,
    "batch_ids": [batch.uid]  # Include batch ID(s)
}

filters = {}

export_task = project.export(params=export_params, filters=filters)
export_task.wait_till_done()

# Stream the export using a callback function
def json_stream_handler(output: labelbox.BufferedJsonConverterOutput):
  print(output.json)

export_task.get_buffered_stream(stream_type=labelbox.StreamType.RESULT).start(stream_handler=json_stream_handler)

# Collect all exported data into a list
export_json = [data_row.json for data_row in export_task.get_buffered_stream()]

# (Optional) To export all the global keys 
global_keys = [data_row.json["data_row"]["global_key"] for data_row in stream]

# (Optional) To export all the data row ids
data_row_ids = [data_row.json["data_row"]["id"] for data_row in stream]
```

### Remove queued data rows

This method removes queued data rows from the batch and consequently the labeling queue of the project.

```python theme={"theme":"css-variables"}
batch.remove_queued_data_rows()
```

### Delete the labels

This method deletes the labels made on data rows in the batch and re-queues the data rows for labeling.

```python theme={"theme":"css-variables"}
batch.delete_labels()

# alternatively, you can re-queue the data with labels as templates

batch.delete_labels(set_labels_as_template=True)
```

### Delete a batch

If any labels created on data rows in the batch exist, the batch cannot be deleted. First, delete the labels made on data rows in the batch (as shown above), then you can delete the batch itself.

```python theme={"theme":"css-variables"}
batch.delete()
```

## Attributes

### Get the basics

```python theme={"theme":"css-variables"}
# name (str)
batch.name

# created at (datetime)

batch.created_at

# updated at (datetime)

batch.updated_at

# size, the number of data rows in the batch (int)

batch.size

# project (relationship to Project object)

project = batch.project()
```


# IAM integration
Source: https://docs.labelbox.com/horizon/developer-guides/cloud-storage-iam-integration

Using IAM delegated access integrations is the recommended option for all cloud users. See [Cloud storage integrations](/horizon/guides/iam-delegated-access) for instructions on setting up cloud storage integrations on the Labelbox platform.

## Create a new organization's IAM integration

```python theme={"theme":"css-variables"}
from labelbox.schema.iam_integration import (
    AwsIamIntegrationSettings,
    GcpIamIntegrationSettings,
    AzureIamIntegrationSettings,
)

organization = client.get_organization()

# AWS
aws_settings = AwsIamIntegrationSettings(
    role_arn="arn:aws:iam::123456789012:role/LabelboxDelegatedAccess",
    read_bucket="my-s3-bucket"
)

aws_integration = organization.create_iam_integration(
    name="My AWS Integration",
    settings=aws_settings
)

# GCP
gcp_settings = GcpIamIntegrationSettings(
    read_bucket="gs://my-gcp-bucket"
)

gcp_integration = organization.create_iam_integration(
    name="My GCP Integration",
    settings=gcp_settings
)

# Azure
azure_settings = AzureIamIntegrationSettings(
    read_container_url="https://mystorageaccount.blob.core.windows.net/mycontainer",
    tenant_id="your-tenant-id",
    client_id="your-client-id",  # Optional for creation
    client_secret="your-client-secret"  # Optional for creation
)

azure_integration = organization.create_iam_integration(
    name="My Azure Integration",
    settings=azure_settings
)
```

## Update an IAM integration

```python theme={"theme":"css-variables"}
# AWS
updated_aws_settings = AwsIamIntegrationSettings(
    role_arn="arn:aws:iam::123456789012:role/UpdatedLabelboxRole",
    read_bucket="my-updated-s3-bucket"
)

aws_integration.update(
    name="Updated AWS Integration",
    settings=updated_aws_settings
)

# GCP
updated_gcp_settings = GcpIamIntegrationSettings(
    read_bucket="gs://my-updated-gcp-bucket"
)

gcp_integration.update(
    name="Updated GCP Integration",
    settings=updated_gcp_settings
)

# Azure
updated_azure_settings = AzureIamIntegrationSettings(
    read_container_url="https://updatedstorageaccount.blob.core.windows.net/mycontainer",
    tenant_id="updated-tenant-id"
)

azure_integration.update(
    name="Updated Azure Integration",
    settings=updated_azure_settings
)
```

## Set an IAM integration as default

```python theme={"theme":"css-variables"}
iam_integration.set_as_default()
```

## Validate an IAM integration

```python theme={"theme":"css-variables"}
iam_integration.validate()
```

## List your organization's IAM integrations

```python theme={"theme":"css-variables"}
import labelbox as lb

client = lb.Client("<YOUR_API_KEY>")
organization = client.get_organization()
iam_integrations = organization.get_iam_integrations()

for integration in iam_integrations:
	print(integration)
```

## Get the default IAM integration

```
default_integration = organization.get_default_iam_integration()
```

## Set IAM integration during dataset creation

Each dataset can have only one IAM integration. Use the `iam_integration` optional field for `client.create_dataset`.

If not set, it will use the default integration of your organization.

You can then upload data rows using the cloud storage URLs.

```python theme={"theme":"css-variables"}
iam_integration = organization.get_iam_integrations()[1] 
dataset = client.create_dataset(name="IAM manual demo", iam_integration=iam_integration)
```

## Override default integration during dataset creation

You can override the default integration when creating a dataset.

```python theme={"theme":"css-variables"}
dataset = client.create_dataset(name="IAM manual demo", iam_integration=None)
```

## Update dataset integration

You can change the current integration of a dataset with `add_iam_integration`.

```python theme={"theme":"css-variables"}
# Get all IAM integrations
iam_integrations = client.get_organization().get_iam_integrations()
 
 # Get IAM integration id
iam_integration_id = [integration.uid for integration
   in iam_integrations
   if integration.name == "My S3 integration"][0]

# Set IAM integration for integration id
dataset.add_iam_integration(iam_integration_id)


 # Get IAM integration object
iam_integration = [integration.uid for integration
   in iam_integrations
   if integration.name == "My S3 integration"][0]

# Set IAM integration for IAMIntegrtion object
dataset.add_iam_integration(iam_integration)
```

## Remove/Unselect dataset integration

```python theme={"theme":"css-variables"}
dataset.remove_iam_integration()
```

## Upload data rows with delegated access

Make sure the type of IAM integration matches your data rows' cloud storage, then use the URL for the `row_data` field to upload data rows.

```python theme={"theme":"css-variables"}
# Some examples: 
datarows = [{"row_data": "https://<bucket-name>.s3.<region>.amazonaws.com/<key>"}] # Amazon S3
datarows = [{"row_data": "gs://gcs-lb-demo-bucket/test.png"}] # Google Cloud Storage
datarows = [{"row_data": "https://labelboxdatasets.blob.core.windows.net/datasets/geospatial/001.jpg"}] # Microsoft Azure Blob Storage

task1 = dataset.create_data_rows(datarows)
task1.wait_till_done()
```


# Labelbox API key
Source: https://docs.labelbox.com/horizon/developer-guides/create-api-key

Shows how to create an API key that lets apps and other services access Labelbox APIs and endpoints.

Labelbox authenticates API requests using **API keys**, which grant access to the Python SDK and can also authorize third-party integrations, such as cloud providers. Depending on your [roles and permissions](/horizon/guides/roles-and-permissions), you can create, view, delete, and distribute API keys as needed.

API keys serve as access credentials to Labelbox on your behalf. Treat them with the same level of care as other sensitive credentials.

<Warning>
  ### API keys of disabled accounts

  API keys are tied to user accounts. If a user account is disabled, all API keys associated with that account are also disabled.
</Warning>

## Create a new API key

To create a new API key:

1. On the Labelbox [home](https://app.labelbox.com/home) page, click **Create API key**.
2. On the **Create new API key** prompt:
   * Enter a descriptive **name**.
   * Add a **validity** time period. The max value is 6 months.
   * Select a role for the scope of access [permissions](/horizon/guides/roles-and-permissions).
   * Click **Create** to complete the setup.
3. Your new key is displayed in the **API key created** prompt. Click the **Copy** icon next to the API key to copy the key to the clipboard. It's strongly recommended to save your new key in a safe location.
4. Click **Done** to close the prompt. After closing the prompt, you won’t be able to retrieve the key.

## View and delete API keys

Go to **Workplace Settings** → [API keys](https://app.labelbox.com/workspace-settings/api-keys) to view and delete existing API keys. To delete an API key, locate the key you wish to delete, open the context menu, and click **Delete**.

Once an API key is deleted, it can no longer authenticate API requests. Any app, script, or service using the key loses access to Labelbox. To minimize disruptions, we recommend not deleting keys until you've created and replaced them with new ones.

## Guidelines and best practices

For each active Labelbox account:

* API keys access to your Labelbox account and data. Treat API keys like passwords and other credentials.
* The value of an API key can only be retrieved once from the **API key created** prompt when the key is first created.
* If you lose the value of your API key (or cannot determine how it's being used), delete it and create a new one.

## Client

The Labelbox Client is essential for all SDK workflows. It connects to a Labelbox server with your authentication API key and provides functions for querying and creating data assets.

## Instantiate Labelbox Client

Use the API key to connect to the Labelbox client:

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  import labelbox as lb

  ## Option 1: List API_KEY in your Python script
  API_KEY = "<YOUR_API_KEY>"
  client = lb.Client(api_key=API_KEY)

  ## Option 2: Pass your API key as an environment variable $LABELBOX_API_KEY
  # In your terminal:
  user@machine:~$ export LABELBOX_API_KEY="<YOUR_API_KEY>"
  # In Python:
  client = lb.Client()
  ```
</CodeGroup>

## Manage API keys from the SDK

### Create an API key

To create a new API key, provide the following:

* `name`: a descriptive name
* `user`: an existing user email address
* `role`: an existing role (one of `client.get_roles()`)
* `validity` and `time_unit`: the duration of the API key using an integer (`validity`) and a `TimeUnit` enum (`time_unit`). Valid `time_unit` values include `TimeUnit.SECOND`, `TimeUnit.MINUTE`, `TimeUnit.HOUR`, `TimeUnit.DAY`, and `TimeUnit.WEEK`.

Duration values can range between:

* minimum `{"validity":1, time_unit: TimeUnit.MINUTE}`
* maximum `{"validity":25, time_unit: TimeUnit.WEEK}`

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  api_key_settings = {
     "name": "<name>",
     "user": "<email>",
     "role": "<role>",
     "validity": "<validity>",
     "time_unit": "<time_unit>"
  }

  new_token = client.create_api_key(**api_key_settings)
  ```
</CodeGroup>

### Get API keys

You can list a single API key or all, active or expired, non-revoked API keys:

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Active API keys
  active_api_keys = client.get_api_keys()

  # All non-revoked API keys
  api_keys = client.get_api_keys(include_expired=True)

  # Single API key
  api_key_id = "<api_key_id>"
  api_key = client.get_api_key(api_key_id)

  # Show expiration datetime
  print(f"Expiration date: {api_key.expired_at}")

  # Show creator and target user
  print(f"Creator: {api_key.created_by.email} - Target user: {api_key.created_for.email}")
  ```
</CodeGroup>

### Revoke an API key

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  api_key.revoke()
  ```
</CodeGroup>


# Custom embeddings
Source: https://docs.labelbox.com/horizon/developer-guides/custom-embeddings

Shows how to upload custom embeddings to improve similarity search.

<CardGroup>
  <Card title="Open in Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/basics/custom_embeddings.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/basics/custom_embeddings.ipynb" />
</CardGroup>

## How to upload custom embeddings

Custom embeddings improve data exploration by improving [similarity search](/horizon/guides/similarity).

You can upload up to ten (10) custom embedding types per workspace on any data type.

Use this to experiment with different embeddings to improve data selection.

## Before you start

This example requires the following libraries:

<CodeGroup>
  ```python Install theme={"theme":"css-variables"}
  # Starting from SDK version 3.69, custom embeddings are now supported.
  import labelbox as lb
  import numpy as np
  import json
  import uuid
  import random
  ```
</CodeGroup>

## Replace API key

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  API_KEY = ""
  client = lb.Client(API_KEY)
  ```
</CodeGroup>

## Select data rows

First, we need to fetch data rows from a Labelbox dataset.

To improve similarity search, you need to upload custom embeddings to at least 1,000 data rows.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  dataset = client.get_dataset("<DATASET-ID>")

  export_task = dataset.export()
  export_task.wait_till_done()

  data_rows = []

  # Stream the export using a callback function

  def json_stream_handler(output: labelbox.BufferedJsonConverterOutput):
  print(output.json)

  export_task.get_buffered_stream(stream_type=labelbox.StreamType.RESULT).start(stream_handler=json_stream_handler)

  # Collect all exported data into a list

  export_json = [data_row.json for data_row in export_task.get_buffered_stream()]

  ```
</CodeGroup>

Extract the data row ID and the row data (asset URL):

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  data_row_dict = [{"data_row_id": dr["data_row"]["id"]} for dr in data_rows]
  data_row_dict = data_row_dict[:1000] # keep the first 1000 examples for the sake of this demo
  ```
</CodeGroup>

## Create custom embedding payload

To prepare the data:

<Steps>
  <Step>
    Generate random vectors for embeddings (max: `2048` dimensions)

    <CodeGroup>
      ```python Python theme={"theme":"css-variables"}
      nb_data_rows = len(data_row_dict)
      print("Number of data rows: ", nb_data_rows)
      # Labelbox supports custom embedding vectors of up to 2048 dimensions
      custom_embeddings = [list(np.random.random(2048)) for _ in range(nb_data_rows)]
      ```
    </CodeGroup>
  </Step>

  <Step>
    List custom embeddings in your Labelbox workspace:

    <CodeGroup>
      ```python Python theme={"theme":"css-variables"}
      embeddings = client.get_embeddings()
      ```
    </CodeGroup>
  </Step>

  <Step>
    Choose an existing embedding type or create a new one A unique custom embedding name is required as an argument for this method.

    <CodeGroup>
      ```python Python theme={"theme":"css-variables"}
      # Name of the custom embedding must be unique
      embedding = client.create_embedding("my_custom_embedding_2048_dimensions", 2048)
      ```
    </CodeGroup>
  </Step>

  <Step>
    Create payload

    * The payload should encompass the `key` (data row id or global key) and the new embedding vector data. Note that the `dataset.upsert_data_rows()` operation will only update the values you pass in the payload; all other existing row data will not be modified.

      <CodeGroup>
        ```python Python theme={"theme":"css-variables"}
        payload = []
        for data_row_dict, custom_embedding in zip(data_row_dict,custom_embeddings):
          payload.append({"key": lb.UniqueId(data_row_dict['data_row_id']),
                          "embeddings": [{"embedding_id": embedding.id, "vector": custom_embedding}]})

          print('payload', len(payload),payload[:1])
        ```
      </CodeGroup>
  </Step>
</Steps>

## Upload payload

<Steps>
  <Step>
    Upsert data rows with custom embeddings

    <CodeGroup>
      ```python Python theme={"theme":"css-variables"}
      task = dataset.upsert_data_rows(payload)
      task.wait_till_done()
      print(task.errors)
      print(task.status)
      ```
    </CodeGroup>
  </Step>

  <Step>
    Get the count of imported vectors for a custom embedding type

    An updated count can take a few minutes, depending on the number of data rows associated with the embedding type.

    <CodeGroup>
      ```python Python theme={"theme":"css-variables"}
      count = embedding.get_imported_vector_count()
      ```
    </CodeGroup>
  </Step>

  <Step>
    Delete custom embedding type.

    <CodeGroup>
      ```python Python theme={"theme":"css-variables"}
      embedding.delete()
      ```
    </CodeGroup>
  </Step>
</Steps>

## Upload custom embeddings during data row creation

<Steps>
  <Step>
    Create a dataset

    <CodeGroup>
      ```python Python theme={"theme":"css-variables"}
      # Create a dataset
      dataset_new = client.create_dataset(name="data_rows_with_embeddings")
      ```
    </CodeGroup>
  </Step>

  <Step>
    Fetch an embedding type and create dummy vector data.

    <CodeGroup>
      ```python Python theme={"theme":"css-variables"}
      embedding = client.get_embedding_by_name("my_custom_embedding_2048_dimensions")
      vector = [random.uniform(1.0, 2.0) for _ in range(embedding.dims)]
      ```
    </CodeGroup>
  </Step>

  <Step>
    Upload data rows with embeddings.

    <CodeGroup>
      ```python Python theme={"theme":"css-variables"}
      uploads = []
      # Generate data rows
      for i in range(1,9):
          uploads.append({
              "row_data":  f"https://storage.googleapis.com/labelbox-datasets/People_Clothing_Segmentation/jpeg_images/IMAGES/img_000{i}.jpeg",
              "global_key": "TEST-ID-%id" % uuid.uuid1(),
              "embeddings": [{
                          "embedding_id": embedding.id,
                          "vector": vector
                      }]
          })

      task1 = dataset_new.create_data_rows(uploads)
      task1.wait_till_done()
      print("ERRORS: " , task1.errors)
      print("RESULTS:" , task1.result)
      ```
    </CodeGroup>
  </Step>
</Steps>


# Data row
Source: https://docs.labelbox.com/horizon/developer-guides/data-row

Developer guide for creating and modifying data rows via the Python SDK.

<CardGroup>
  <Card title="Open in Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/basics/data_rows.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/basics/data_rows.ipynb" />
</CardGroup>

Data rows are the assets that are being labeled; a data row cannot exist without belonging to a dataset. Data rows are added to a labeling task by uploading them to datasets and then creating [project batches](/horizon/developer-guides/batch#fundamentals).

## Client

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  import labelbox as lb
  client = lb.Client(api_key="<YOUR_API_KEY>")
  ```
</CodeGroup>

## Get a data row

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  data_row = client.get_data_row("<data_row_id>")

  data_row = client.get_data_row_by_global_key("key1")

  data_row_ids = get_data_row_ids_for_global_keys(["key1", "key2"])
  ```
</CodeGroup>

## Assign global keys

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  global_key_data_row_inputs = [
    {"data_row_id": "<data_row_id>", "global_key": "key1"},
    {"data_row_id": "<data_row_id>", "global_key": "key2"}
  ]

  client.assign_global_keys_to_data_rows(global_key_data_row_inputs)
  ```
</CodeGroup>

## Clear global keys

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  client.clear_global_keys(["key1", "key2"])
  ```
</CodeGroup>

## Fundamentals

## Create data rows

Data rows are created via methods from the `Dataset` class. For complete details and additional examples of approaches for creating data rows, please see [Dataset](/horizon/developer-guides/dataset).

We recommend using `created_data_rows()` and `upsert_data_rows()` methods for large data row upload operations.

<Warning>
  **Special character handling**

  Please note that certain characters like `#`,`<`, `>` and `|`| are not supported in URLs and should be avoided in your file names to prevent loading issues.

  Please refer to [https://datatracker.ietf.org/doc/html/rfc2396#section-2.4.3](https://datatracker.ietf.org/doc/html/rfc2396#section-2.4.3) on URI standards.

  A good test for the handling of special characters is to test URLs in your browser address bar — if the URL doesn't load properly in your browser, it won't load in Labelbox.
</Warning>

The only required argument when creating a data row is the `row_data`. However, Labelbox strongly recommends supplying each data row with a global key upon creation.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # this example uses the uuid package to generate unique global keys
  from uuid import uuid4

  data =[
      {
        "row_data": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/basic.jpg",
          "global_key": str(uuid4())
      },
      {
        "row_data": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/basic.jpg",
        "global_key": str(uuid4())
      }
  ]

  dataset.upsert_data_rows(data) 

  # or alternatively use 

  dataset.create_data_rows(data)
  ```
</CodeGroup>

## Export data rows

For a complete reference on how to export data rows, please visit the [export overview documentation](/horizon/developer-guides/export-overview)

Export a single data row:

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  DATA_ROW_GLOBAL_KEY = "<global key>"

  export_params= {
    "attachments": True,
    "metadata_fields": True,
    "data_row_details": True,
    "project_details": True,
    "label_details": True,
    "performance_details": True,
    "interpolated_frames": True,
    "embeddings": True
  }

  # Provide a list of data row global keys
  export_task = lb.DataRow.export(client=client, global_keys=[DATA_ROW_GLOBAL_KEY], params=export_params)
  export_task.wait_till_done()

  if export_task.has_results():
    # Start stream
    stream = export_task.get_buffered_stream()

    # Iterate through data rows
    for data_row in stream:
      print(data_row.json)
  ```
</CodeGroup>

## Methods

## Create and update an attachment

Please reference our [attachment documentation](/horizon/developer-guides/attachments#supported-attachment-types) for more information on supported attachment data types.

For details on creating attachments in the same step as creating data rows, see [Dataset](/horizon/developer-guides/dataset).

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  attachment = data_row.create_attachment(
    attachment_type="<attachment_type>",		# specify a type from the table above
    attachment_value="<attachment_value>",	# provide a value of the appropriate type
    attachment_name="<attachment_name>"		# name the attachment for reference
  )

  # Update an attachment

  attachment.update(type= "<attachment_type>", value="<new_value>")

  ```
</CodeGroup>

## Get the winning label ID

For more details on what a "winning" label is and how it is chosen, see [Consensus](/horizon/guides/consensus#select-a-set-of-annotations-as-the-winner).

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  data_row.get_winning_label_id(project_id="<project_id>")
  ```
</CodeGroup>

## Update data rows from a dataset

We recommend using the`upsert_data_rows()` method to update data rows.

When using this method to update data rows, you need to pass a `key`, which can reference either a global key or data row ID.

Include any fields you wish to update along with their new values.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Update the global key associated with the DATAROW_ID or GLOBAL_KEY, and include a additional metadata
  dataset = client.get_dataset("<DATASET_ID>")

  data = {
      "key": lb.UniqueId(DATA_ROW_ID),
      "global_key": "NEW-ID-%id" % uuid.uuid1(),
      "metadata_fields": [
          # New metadata
          lb.DataRowMetadataField(
              schema_id=mdo.reserved_by_name['captureDateTime'].uid,
              value="2000-01-01 00:00:00"
          ),
          # Include original metadata otherwise it will be removed
          lb.DataRowMetadataField(
              schema_id=mdo.reserved_by_name["tag"].uid,
              value="tag_string",
          ),
      ]
  }

  task = dataset.upsert_data_rows([data])
  task.wait_till_done()
  print("ERRORS: " , task.errors)
  print("RESULTS:" , task.result)
  ```
</CodeGroup>

The `update` method is also available; however, for a large number of data row updates, we recommend using `upsert_data_rows()`

The `update` method only supports updates on `global_key`, `row_data` or `external_id` values.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  data_row.update(
    row_data="<new_row_data>",
    global_key="<new_unique_global_key>",
    external_id="new_external_id"
  )
  ```
</CodeGroup>

## Delete data rows

<Danger>
  **Deleting data rows cannot be undone**

  These methods delete data rows along with all labels made on each data row. This action cannot be reverted without the assistance of Labelbox support.
</Danger>

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # delete one data row
  data_row.delete()

  # bulk delete data rows -- takes a list of Data Row objects

  lb.DataRow.bulk_delete(data_rows=[<DataRow>, <DataRow>])

  # for example, delete the data rows in a dataset, but not the Dataset object

  dataset = client.get_dataset("<dataset_id>")
  lb.DataRow.bulk_delete(data_rows=list(dataset.data_rows()))

  ```
</CodeGroup>

<Info>
  ### Limit on bulk deleting data rows

  The `lb.DataRow.bulk_delete()` method can delete a maximum of 4,000 data rows per call.
</Info>

## Attributes

## Get the basics

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # global key (str)
  data_row.global_key

  # external ID (str)
  data_row.external_id

  # row data (str)
  data_row.row_data

  # media attributes (dict)
  data_row.media_attributes

  # updated at (datetime)
  data_row.updated_at

  # created at (datetime)
  data_row.created_at

  # created by (relationship to User object)
  user = data_row.created_by()

  # organization (relationship to Organization object)
  organization = data_row.organization()

  # dataset (relationship to Dataset object)
  dataset = data_row.dataset()
  ```
</CodeGroup>

## Get the attachments

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # relationship to many AssetAttachment objects
  attachments = data_row.attachments()

  # inspect one attachment
  next(attachments)

  # inspect all attachments
  for attachment in attachments:
    print(attachment)

  # for ease of use, you can convert the paginated collection to a list
  list(attachments)

  ```
</CodeGroup>

## Get the metadata

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # get the metadata fields associated with the data row (list)
  data_row.metadata_fields

  # get the metadata fields as DataRowMetadataField objects (list)
  data_row.metadata
  ```
</CodeGroup>

## Get the labels

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # relationship to many Label objects
  labels = data_row.labels()

  # inspect one label made on the data row
  next(labels)

  # inspect all labels made on the data row 
  for label in labels:
    print(label)

  # for ease of use, you can convert the paginated collection to a list of labels
  list(labels)

  ```
</CodeGroup>


# Global keys
Source: https://docs.labelbox.com/horizon/developer-guides/data-row-global-keys

Information of data row global keys

Global keys are user-specified unique ID for your data rows that you can assign upon creation or afterward.

## Requirements

Global key values are required, **non-blank string values**; they:

* Can be up to 200 characters in length

* Support

  * Lowercase English alphabets: a, b, c, ..., z

  * Uppercase English alphabets: A, B, C, ..., Z

  * Numerals: 0, 1, 2, ..., 9

  * Special characters:

    * Exclamation mark: `!`
    * Underscore: `_`
    * Dot: `.`
    * Asterisk: `*`
    * Single quotation mark: `'`
    * Open parentheses: `(`
    * Close parentheses: `)`
    * Ampersand: `&`
    * Dollar sign: `$`
    * At symbol: `@`
    * Equals sign: `=`
    * Semicolon: `;`
    * Forward slash: `/`
    * Colon: `:`
    * Plus sign: `+`
    * Space: \`\`
    * Comma: `,`
    * Question mark: `?`
    * Hyphen: `-`

* Are case-insensitive

* Can include spaces within the value, but not at the beginning (*leading*) or the end (*trailing*)

## Create data rows with global keys

Note: If you are using delegated access, using the data row URLs or object keys as the unique global keys to map your cloud-hosted assets with Labelbox data rows is common practice.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  from labelbox import  DataRow, Client
  import uuid

  client = Client("<YOUR_API_KEY")
  dataset = client.create_dataset(name="<DATASET_NAME>")

  # Create a single data row in a dataset
  data_row = dataset.create_data_row(row_data="https://picsum.photos/id/829/200/300", global_key=str(uuid.uuid4()))

  # Create bulk data rows in a dataset with unique keys
  data_rows = [
    {"row_data": "https://picsum.photos/id/829/200/300", "global_key": str(uuid.uuid4())},
    {"row_data": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/basic.jpg", "global_key": str(uuid.uuid4())},
  ]
  task = dataset.create_data_rows(data_rows)
  task.wait_till_done()
  print(task.errors)
  # None
  ```
</CodeGroup>

The following code will **fail** to upload data rows due to duplicate global keys.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # This will have partial failure
  same_key = "SAME_KEY"
  # if you upload a list of data rows with duplicate keys in them, none will be uploaded.
  for i in range(3):
      data_rows.append(
          {
              "row_data": "https://picsum.photos/id/829/200/300",
              "global_key": same_key
          }
      )
  task_will_fail = dataset.create_data_rows(data_rows)
  task_will_fail.wait_till_done()
  print(task_will_fail.errors)
  # ---
  # WARNING:labelbox.schema.task:There are errors present. Please look at `task.errors` for more details
  # {'message': 'Data rows contain empty string or duplicate global keys, which are not allowed'}

  assets = [{"row_data": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/basic.jpg", "global_key": str(uuid4())},
            {"row_data": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/basic.jpg", "global_key": str(uuid4())},
           ]
  # This task will succeed
  task_will_succeed = dataset.create_data_rows(assets)
  task_will_succeed.wait_till_done()
  print(task_will_succeed.errors)

  # if you upload it again with the same global keys, it will fail
  task_will_fail = dataset.create_data_rows(assets)
  task_will_fail.wait_till_done()
  print(task_will_fail.errors)
  # ----
  # ...... ER_DUP_ENTRY: Duplicate entry '<the_duplicate_key>' for key 'DataRow_globalKey_organizationId_unique_idx'"
  ```
</CodeGroup>

## Assign global keys to existing data rows

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  data_row = dataset.create_data_row(row_data="google.com")
  global_key = str(uuid.uuid4())
  res = client.assign_global_keys_to_data_rows(
      [{
          "data_row_id": data_row.uid,
          "global_key": global_key
      }]
  )
  print(res)
  # {'status': 'SUCCESS', 'results': [{'data_row_id': 'cl80k85x6017y0z0dcvtb4f1s', 'global_key': '745b9940-2457-4f56-b999-a2d46650dd77', 'sanitized': False}], 'errors': []}
  ```
</CodeGroup>

Similarly, the assignment of global keys will fail if there are duplicate global keys in the system. Suppose you have already run the above code cell. In the following code, if you will another data row with the same global key as above, it will **fail** to assign.

It is possible to have "PARTIAL SUCCESS" results. In the case that some data rows are assigned with unique global keys and some are assigned with existing global keys, the SDK will have partial success and return a payload where the `results` field contains the successful ones, and the `errors` contains the failed ones with error messages.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Failure example
  data_row_1 = dataset.create_data_row(row_data="yahoo.com")
  res = client.assign_global_keys_to_data_rows(
      [{
          "data_row_id": data_row_1.uid,
          "global_key": global_key #this will fail since we already used it in previous code
      }]
  )
  print(res)
  # ---
  # WARNING:labelbox.client:There are errors present. Please look at 'errors' in the returned dict for more details
  # {'status': 'FAILURE', 'results': [], 'errors': [{'data_row_id': 'cl847r6eo54ka070n8k5737tk', 'global_key': 'c4af584e-e3f6-4c3d-a5d9-e7244f00f48a', 'error': 'Invalid global key'}]}

  # Partial Success example
  data_row_2 = dataset.create_data_row(row_data="bing.com")
  data_row_3 = dataset.create_data_row(row_data="duckduckgo.com")

  res = client.assign_global_keys_to_data_rows(
      [{
          "data_row_id": data_row_2.uid,
          "global_key": global_key,  #this will fail since we already used it in previous code
      },
      {
          "data_row_id": data_row_3.uid,
          "global_key": str(uuid.uuid4()), # this one will succeed since it is a new global key
      }
      ]
  )
  print(res) # res['results'] contains the successful ones, and the res['errors'] contains the failed ones with error messages
  # ---
  # WARNING:labelbox.client:There are errors present. Please look at 'errors' in the returned dict for more details
  # {'status': 'PARTIAL SUCCESS', 'results': [{'data_row_id': 'cl84zf8ee03rv071b1i055nj5', 'global_key': 'ed8969a6-b6a4-445f-936e-ae2a9a6f0004', 'sanitized': False}], 'errors': [{'data_row_id': 'cl84zf80d292q07wz9fmubwbw', 'global_key': '963716ae-2da4-4c43-93ae-c8f414a980ae', 'error': 'Invalid global key'}]}
  ```
</CodeGroup>

## Query data rows by global key

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  data_rows = []
  global_keys = [str(uuid.uuid4()) for i in range(3)]

  for i in range(3):
      data_rows.append(
          {
              "row_data": "aaaa.jpg",
              "global_key": global_keys[i]
          }
      )
  task = dataset.create_data_rows(data_rows)
  task.wait_till_done()

  # this tests that the global keys were able to fetch all of the data rows we expect
  res = client.get_data_row_ids_for_global_keys(global_keys)

  print(res)
  # ---
  # {'status': 'SUCCESS', 'results': ['cl84zszs302z7074c8u7m2osx', 'cl84zszs302yz074ccg1acwu5', 'cl84zszs302z3074c6aahhq3q'], 'errors': []}
  ```
</CodeGroup>

If you query for a non-existent global key, that particular query will fail while the valid queries execute.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Partial Success example
  global_keys.append("<a non-existent key>")
  res = client.get_data_row_ids_for_global_keys(global_keys)
  print(res)
  # ---
  # WARNING:labelbox.client:There are errors present. Please look at 'errors' in the returned dict for more details
  # {'status': 'PARTIAL SUCCESS', 'results': ['cl84zszs302z7074c8u7m2osx', 'cl84zszs302yz074ccg1acwu5', 'cl84zszs302z3074c6aahhq3q'], 'errors': [{'global_key': '<a non-existent key>', 'error': 'Data Row not found'}]}
  ```
</CodeGroup>

Now, you can add data row IDs to a batch for a project or a model run for other downstream tasks.

## Global keys are cleared after deleting data rows

Global keys will be automatically deleted after the deletion of data rows. For example,

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  global_key = 'example_global_key'
  new_data_row = {
      "row_data": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/basic.jpg",
      "external_id": str(uuid.uuid4()),
      "global_key": global_key
  }
  # This task should succeed
  task = dataset.create_data_rows([new_data_row])
  task.wait_till_done()
  print(task.errors) # None

  # This task should fail
  task = dataset.create_data_rows([new_data_row])
  task.wait_till_done()
  task.errors, task.result
  # WARNING:labelbox.client:There are errors present. Please look at 'errors' in the returned dict for more details
  # WARNING:labelbox.schema.task:There are errors present. Please look at `task.errors` for more details
  ---------------------------------------------------------------------------
  #ValueError: Job failed. Errors : Duplicate global keys found: example_global_key
  ```
</CodeGroup>

Since global keys are deleted upon data row deletion, you can check whether a global key is attached to a deleted data row, and it will show up in the "errors" field.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # delete data row by global key
  duplicated_data_row = client.get_data_row(client.get_data_row_ids_for_global_keys([global_key])['results'][0])
  duplicated_data_row.delete()
  print('deleted')

  # Should not have results
  result = client.get_data_row_ids_for_global_keys(['example_global_key'])
  print(result)
  # WARNING:labelbox.client:There are errors present. Please look at 'errors' in the returned dict for more details
  #{'status': 'FAILURE', 'results': [''], 'errors': [{'global_key': 'example_global_key', 'error': 'Data Row deleted'}]}

  # You can now upload a Data Row with same global key again
  # This should succeed
  task = dataset.create_data_rows([new_data_row])
  task.wait_till_done()
  print(task.errors)
  ```
</CodeGroup>


# Dataset
Source: https://docs.labelbox.com/horizon/developer-guides/dataset

Summary of methods for creating and modifying datasets.

The most common method for importing data to Labelbox is via Python SDK after setting up a [cloud storage integration](/horizon/guides/iam-delegated-access). With an IAM delegated access integration, you can keep your data in your cloud bucket and grant Labelbox limited access to the data on demand.

<Info>
  ### Examples for all data types

  The examples in this developer guide primarily use image assets. For sample approaches with other data modalities, please view the developer guide for importing data nested under each asset type in the **Import/Export** section of the table of contents.
</Info>

## Create a dataset

The only required argument when creating a dataset is the `name`. You can also specify the IAM integration that you have set up on your account if not specified or set to `None` it will use your default integration. For more details. including how to get your IAM integrations; visit our dedicated [IAM integration](/horizon/developer-guides/cloud-storage-iam-integration) page

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  dataset = client.create_dataset(
    name='<dataset_name>',
    description='<dataset_description>',	# optional
    iam_integration=None		# if not specified, will use default integration, set as None to not use delegated access.
  )
  ```
</CodeGroup>

## Get a dataset

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  dataset = client.get_dataset("<dataset_id>")

  # alternatively, you can get a dataset by name
  dataset = client.get_datasets(where=labelbox.Dataset.name == "<dataset_name>").get_one()
  ```
</CodeGroup>

## Dataset methods

## Create data rows

<Warning>
  ### Special character handling

  Please note that certain characters like `#`,`<`, `>` and `|`| are not supported in URLs and should be avoided in your file names to prevent loading issues.

  Please refer to [https://datatracker.ietf.org/doc/html/rfc2396#section-2.4.3](https://datatracker.ietf.org/doc/html/rfc2396#section-2.4.3) on URI standards.

  A good test for the handling of special characters is to test URLs in your browser address bar — if the URL doesn't load properly in your browser, it won't load in Labelbox.
</Warning>

The only required argument when creating a data row is the `row_data`. However, Labelbox strongly recommends supplying each data row with a [global key](/horizon/developer-guides/data-row-global-keys) upon creation.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # this example uses the uuid package to generate unique global keys
  from uuid import uuid4

  data =[
      {
        "row_data": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/basic.jpg",
       	"global_key": str(uuid4())
      },
  	{
        "row_data": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/basic.jpg",
        "global_key": str(uuid4())
      }
    ]

  dataset.upsert_data_rows(data)

  # or alternatively use

  dataset.create_data_rows(data)
  ```
</CodeGroup>

You can also create data rows with [metadata](/horizon/guides/datarow-metadata), [attachments](/horizon/guides/label-data#attachments), and [image overlays](/horizon/guides/label-data#image-overlay) in the same task. The code below contains an end-to-end example for creating a dataset with data rows that include these elements.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  import labelbox as lb
  from uuid import uuid4
  import datetime

  # insert your API key
  LB_API_KEY = "<API KEY>"
  client = lb.Client(api_key=LB_API_KEY)

  # get the metadata ontology
  metadata_ontology = client.get_data_row_metadata_ontology()

  # create the dataset
  dataset = client.create_dataset(name="Bulk import example")

  # build the assets
  assets = [
    {"row_data": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/basic.jpg", "global_key": str(uuid4())},
    {"row_data": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/basic.jpg", "global_key": str(uuid4())},
    {"row_data": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/basic.jpg", "global_key": str(uuid4())}
  ]

  # build the metadata
  asset_metadata_fields = [
    {"name": "captureDateTime", "value": datetime.datetime.utcnow()},
    {"name": "tag", "value": "tag_string"},
    {"name": "split", "value": "train"}
  ]

  # build the attachments
  asset_attachments = [
    {"type": "IMAGE", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/disease_attachment.jpeg"},
    {"type": "VIDEO", "value":  "https://storage.googleapis.com/labelbox-sample-datasets/Docs/drone_video.mp4"},
    {"type": "RAW_TEXT", "value": "IOWA, Zone 2232, June 2022 [Text string]"},
    {"type": "RAW_TEXT", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt"},
    {"type": "TEXT_URL", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt"},
    {"type": "HTML", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/windy.html"},
    {"type": "IMAGE_OVERLAY", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/rgb.jpg", "name": "RGB" },
    {"type": "IMAGE_OVERLAY", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/cir.jpg", "name": "CIR"},
    {"type": "IMAGE_OVERLAY", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/weeds.jpg", "name": "Weeds"},
    {"type": "PDF_URL", "value": "https://storage.googleapis.com/labelbox-datasets/arxiv-pdf/data/99-word-token-pdfs/0801.3483.pdf"}
  ]

  # connect the metadata and attachments to the data rows
  for item in assets:
    item["metadata_fields"] = asset_metadata_fields
    item["attachments"] = asset_attachments

  # create the data rows
  task = dataset.upsert_data_rows(assets) # or # or alternatively use create_data_rows()
  task.wait_till_done()
  print(task.results)
  print(task.errors)
  ```
</CodeGroup>

<Info>
  ### Limits

  See [this page](/horizon/guides/limits) to learn the limits for creating data rows in one bulk operation. Regardless of your tier's limit, if you have a large dataset to upload, you can split the data rows into chunks and upload them sequentially.
</Info>

### Task results

The task result is a collection of details, typically information on the data rows associated with the task.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  dataset = client.get_dataset("<dataset id>")

  task = dataset.create_data_rows([{
    "row_data": item_url,
    "global_key": "<unique_global_key>"
  }])
  task.wait_till_done()

  print(task.result) # prints the data rows associated with the create_data_rows operation
  ```
</CodeGroup>

<Warning>
  ### Warning

  Large results (over 150,000 data rows) can take up to 10 mins to process.
</Warning>

### Task errors

The task errors work similarly to the task results but contain information on any associated errors.

<CodeGroup>
  ```python errors theme={"theme":"css-variables"}
  dataset = client.get_dataset("<dataset id>")

  task = dataset.create_data_rows([{
    "row_data": item_url,
    "global_key": "<unique_global_key>"
  }])
  task.wait_till_done()

  print(task.errors) # prints the data rows associated with the create_data_rows operation
  ```
</CodeGroup>

## Create a singular data row

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # simplest example
  task = dataset.create_data_row(
    row_data="https://storage.googleapis.com/labelbox-sample-datasets/Docs/basic.jpg",
    global_key=str(uuid4())
  )
  task.wait_till_done()
  ```
</CodeGroup>

## Importing a local file

### Upload a local file

You can upload a local file to Labelbox storage and use the returned URL to create a data row.

<CodeGroup>
  ```python Upload local file first theme={"theme":"css-variables"}
  # note that you cannot set global keys when creating a data row from a local file
  # first we must upload the file then create the data row
  local_data_path = "/tmp/test_data_row.txt"
  with open(local_data_path, 'w') as file:
    file.write("sample data")

  # upload the file to Labelbox storage
  file_url = client.upload_file(local_data_path)

  # create the data row with a global key
  task = dataset.create_data_rows([{
    "row_data": item_url,
    "global_key": "<unique_global_key>"
  }])
  task.wait_till_done()
  ```
</CodeGroup>

### Create data row from a local file

You can also create a data row from a local file directly without uploading it to storage. This creates the file URL and the data row in one step. However, this is not recommended since it increases the time it takes for your data row creation.

<CodeGroup>
  ```python Upload local file directly theme={"theme":"css-variables"}
  # get a local file and write some text data
  local_data_path = ""
  # create the data row
  task = dataset.create_data_rows([{
    "row_data": local_data_path,
    "global_key": "<unique_global_key>"
  }])
  task.wait_till_done()
  ```
</CodeGroup>

## Append to an existing dataset

You can add data rows to an existing dataset using the methods described above. First, get a dataset, then create the rows.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # get existing dataset
  dataset = client.get_dataset("<dataset_id>")

  # use a method for data row creation as shown in sections above
  task = dataset.create_data_rows([{
    "row_data": item_url,
    "global_key": "<unique_global_key>"
  }])
  ```
</CodeGroup>

## Export data rows from a dataset

Filtering the result, you can obtain a list of asset URLs, global keys, or data row IDs.

<CodeGroup>
  ```python Export theme={"theme":"css-variables"}
  # Get a dataset
  dataset = client.get_dataset("<dataset_id>")

  # Start the export task for the dataset
  export_task = dataset.export()
  export_task.wait_till_done()

  # Stream the export using a callback function
  def json_stream_handler(output: labelbox.BufferedJsonConverterOutput):
    print(output.json)

  export_task.get_buffered_stream(stream_type=labelbox.StreamType.RESULT).start(stream_handler=json_stream_handler)

  # Collect all exported data into a list
  export_json = [data_row.json for data_row in export_task.get_buffered_stream()]

  # Extract the data rows urls from the export result
  asset_urls = [item.json["data_row"]["row_data"] for item in stream]

  # (Optional) To export all the global keys
  global_keys = [item.json["data_row"]["global_key"] for item in stream]

  # (Optional) To export all the data row ids
  data_row_ids = [item.json["data_row"]["id"] for item in stream]
  ```
</CodeGroup>

## Export data rows with labels and predictions

For more details, see [Export data rows from Catalog](/horizon/developer-guides/export-overview#export-data-rows-from-catalog).

<CodeGroup>
  ```python Export theme={"theme":"css-variables"}
  # set the export params to include/exclude certain fields
  export_params={
    "attachments": True,
    "metadata_fields": True,
    "data_row_details": True,
    "project_details": True,
    "performance_details": True,
    "project_ids": ["<project_id_1>", "<project_id_2>"],
    "model_run_ids": ["<model_run_id_1>", "<model_run_id_2>"]
  }

  # set filters
  filters={
    "last_activity_at": ["2000-01-01 00:00:00", "2050-01-01 00:00:00"],
    "label_created_at": ["2000-01-01 00:00:00", "2050-01-01 00:00:00"]
  }

  # get a dataset
  dataset = client.get_dataset("<dataset_id>")

  export_task = dataset.export(params=export_params, filters=filters)
  export_task.wait_till_done()

  # Stream the export using a callback function
  def json_stream_handler(output: labelbox.BufferedJsonConverterOutput):
    print(output.json)

  export_task.get_buffered_stream(stream_type=labelbox.StreamType.RESULT).start(stream_handler=json_stream_handler)

  # Collect all exported data into a list
  export_json = [data_row.json for data_row in export_task.get_buffered_stream()]
  ```
</CodeGroup>

## Update a dataset

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  dataset.update(name="new_dataset_name")
  ```
</CodeGroup>

## Delete a dataset

<Danger>
  **Deleting a dataset cannot be undone**

  This method deletes the dataset along with all data rows in the dataset and any labels made on these data rows. This action cannot be reverted.
</Danger>

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  dataset.delete()
  ```
</CodeGroup>

# Dataset attributes

## Get the basics

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # name (str)
  dataset.name

  # description (str)
  dataset.description

  # updated at (datetime)
  dataset.updated_at

  # created at (datetime)
  dataset.created_at

  # created by (relationship to User object)
  user = dataset.created_by()

  # organization (relationship to Organization object)
  organization = dataset.organization()
  ```
</CodeGroup>

## Get the data rows

The `data_rows()` attribute is a relationship to the `DataRow` objects in the dataset. The relationship retrieves a paginated collection of data rows. It is recommended to use exports to get direct data row details especially for larger data sets.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  data_rows = dataset.data_rows()

  # inspect one data row
  next(data_rows)

  # inspect a number of data rows
  for data_row in data_rows:
    print(data_row)

  # for ease of use, you can convert the paginated collection to a list
  list(data_rows)
  ```
</CodeGroup>

## Get the number of data rows

The `row_count` is a cached attribute; thus, you must re-fetch the dataset after creating data rows to retrieve the updated value.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  dataset = client.get_dataset("<dataset_id>")
  dataset.row_count
  ```
</CodeGroup>


# Export audio annotations
Source: https://docs.labelbox.com/horizon/developer-guides/export-audio-annotations

How to export audio annotations and sample export formats.

<CardGroup>
  <Card title="Open In Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/exports/export_data.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/exports/export_data.ipynb" />
</CardGroup>

## Export as JSON

<CodeGroup>
  ```python Export theme={"theme":"css-variables"}
  # Set the export params to include/exclude certain fields.
  export_params= {
    "attachments": True,
    "metadata_fields": True,
    "data_row_details": True,
    "project_details": True,
    "label_details": True,
    "performance_details": True,
    "interpolated_frames": True
  }

  # Note: Filters follow AND logic, so typically using one filter is sufficient.

  filters= {
    "last_activity_at": ["2000-01-01 00:00:00", "2050-01-01 00:00:00"],
    "workflow_status": "<wkf-status>"
  }

  export_task = project.export(params=export_params, filters=filters)
  export_task.wait_till_done()

  # Stream the export using a callback function

  def json_stream_handler(output: labelbox.BufferedJsonConverterOutput):
    print(output.json)

  export_task.get_buffered_stream(stream_type=labelbox.StreamType.RESULT).start(stream_handler=json_stream_handler)

  # Collect all exported data into a list

  export_json = [data_row.json for data_row in export_task.get_buffered_stream()]

  print("file size: ", export_task.get_total_file_size(stream_type=lb.StreamType.RESULT))
  print("line count: ", export_task.get_total_lines(stream_type=lb.StreamType.RESULT))
  ```
</CodeGroup>

## Annotation export formats

Global and temporal classifications share the same export format for each type. In the export JSON file, temporal classifications are nested under a `timestamp` field, while global classifications appear directly under the `classifications` field, `segments` are per `feature_id` and represent the beginning and the end of the temporal classification in milliseconds. See the [Sample project export](#sample-project-export) to learn how each type is organized in the export.

### Classification - radio

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "clnus73114gc4143u2dux654s",
    "feature_schema_id": "clnus6slv06xo07yja7l47gtw",
    "name": "radio_audio",
    "value": "radio_audio",
    "radio_answer": {
      "feature_id": "clnus73114gc5143u2jadnt6r",
      "feature_schema_id": "clnus6slv06xr07yj24yp1gdk",
      "name": "second_radio_answer",
      "value": "second_radio_answer",
      "classifications": []
    }
  }
  ```
</CodeGroup>

### Classification - checklist

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "clnus73114gc1143u7fzf4ebv",
    "feature_schema_id": "clnus6slu06xi07yj7m9d7rft",
    "name": "checklist_audio",
    "value": "checklist_audio",
    "checklist_answers": [
      {
        "feature_id": "clnus73114gc2143u6ah5wdhn",
        "feature_schema_id": "clnus6slu06xj07yj9eb7dpvp",
        "name": "first_checklist_answer",
        "value": "first_checklist_answer",
        "classifications": []
      },
      {
        "feature_id": "clnus73114gc3143uq755vkzi",
        "feature_schema_id": "clnus6slu06xl07yjdab4gxe8",
        "name": "second_checklist_answer",
        "value": "second_checklist_answer",
        "classifications": []
      }
    ]
  }
  ```
</CodeGroup>

### Classification - free-form text

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "clnus73114gc0143urjlfk558",
    "feature_schema_id": "clnus6slu06xg07yjcg87apa0",
    "name": "text_audio",
    "value": "text_audio",
    "text_answer": {
      "content": "free text audio annotation"
    }
  }
  ```
</CodeGroup>

## Sample project export

<CodeGroup>
  ```json JSON expandable theme={"theme":"css-variables"}
  [
      {
          "data_row": {
              "id": "clzc6xfbealyn0713j929vdch",
              "external_id": "example.mp3",
              "row_data": "https://storage.googleapis.com/labelbox-datasets/audio-sample-data/sample-audio-1.mp3"
          },
          "media_attributes": {
              "asset_type": "audio",
              "mime_type": "audio/mpeg"
          },
          "projects": {
              "clzc89nk900l707zmamti5zi0": {
                  "name": "Example audio project",
                  "labels": [
                      {
                          "label_kind": "Audio",
                          "version": "1.0.0",
                          "id": "cm8x9w1xj00h307ev0jmh8648",
                          "annotations": {
                              "classifications": [
                                  {
                                      "feature_id": "cm8xa12hd00023b6t0t3zirna",
                                      "feature_schema_id": "cm8x9zz4d04ed071363gvhh5j",
                                      "name": "Example global free text",
                                      "value": "example_global_free_text",
                                      "text_answer": {
                                          "content": "Example global free text label",
                                          "classifications": []
                                      }
                                  },
                                  {
                                      "feature_id": "cm8xa164700043b6t7wyqo0vv",
                                      "feature_schema_id": "cm8x9zz4e04ef0713db9vea0f",
                                      "name": "Example global radio",
                                      "value": "example_global_radio",
                                      "radio_answer": {
                                          "feature_id": "cm8xa164600033b6tvz63hfqi",
                                          "feature_schema_id": "cm8x9zz4e04eg0713hsxe3t0d",
                                          "name": "Option 1",
                                          "value": "option_1",
                                          "classifications": []
                                      }
                                  },
                                  {
                                      "feature_id": "cm8xa17ps00063b6txb4g30fw",
                                      "feature_schema_id": "cm8x9zz4e04er0713fc6fcs2r",
                                      "name": "Example global checklist",
                                      "value": "example_global_checklist",
                                      "checklist_answers": [
                                          {
                                              "feature_id": "cm8xa17ps00053b6tc2skf6oh",
                                              "feature_schema_id": "cm8x9zz4f04es0713g860a82c",
                                              "name": "Option 1",
                                              "value": "option_1",
                                              "classifications": []
                                          },
                                          {
                                              "feature_id": "cm8xa181c00083b6t7i9pixsb",
                                              "feature_schema_id": "cm8x9zz4f04eu0713b95l1u2f",
                                              "name": "Option 2",
                                              "value": "option_2",
                                              "classifications": []
                                          }
                                      ]
                                  }
                              ],
                              "timestamp": {
                                  "90697": {
                                      "classifications": [
                                          {
                                              "feature_id": "cm8xa1k8600093b6tnrzpuq1n",
                                              "feature_schema_id": "cm3jdcfpg0m6m07081ju276sz",
                                              "name": "Example temporal free text",
                                              "value": "free_text",
                                              "text_answer": {
                                                  "content": "Example temporal free text label 1",
                                                  "classifications": []
                                              }
                                          },
                                          {
                                              "feature_id": "cm8xa2d2u000a3b6tk29o8u48",
                                              "feature_schema_id": "cm8x9zz4e04el07135zmghx1q",
                                              "name": "Example temporal radio",
                                              "value": "example_temporal_radio",
                                              "radio_answer": {
                                                  "feature_id": "cm8xa2d2u000b3b6ti9kszxcg",
                                                  "feature_schema_id": "cm8x9zz4e04em07132athat2z",
                                                  "name": "Option 1",
                                                  "value": "option_1",
                                                  "classifications": []
                                              }
                                          },
                                          {
                                              "feature_id": "cm8xa2hhh000c3b6tzeluuhpt",
                                              "feature_schema_id": "cm8x9zz4f04ex0713f5rehhp0",
                                              "name": "Example temporal checklist",
                                              "value": "example_temporal_checklist",
                                              "radio_answer": {
                                                  "feature_id": "cm8xa2ila000e3b6t410j19ir",
                                                  "feature_schema_id": "cm8x9zz4f04f00713drwj6qpb",
                                                  "name": "Option 2",
                                                  "value": "option_2",
                                                  "classifications": []
                                              }
                                          }
                                      ]
                                  },
                                  "23890": {
                                      "classifications": [
                                          {
                                              "feature_id": "cm8xa1k8600093b6tnrzpuq1n",
                                              "feature_schema_id": "cm3jdcfpg0m6m07081ju276sz",
                                              "name": "Example temporal free text",
                                              "value": "free_text",
                                              "text_answer": {
                                                  "content": "Example temporal free text label 1",
                                                  "classifications": []
                                              }
                                          }
                                      ]
                                  },
                                  "6157": {
                                      "classifications": [
                                          {
                                              "feature_id": "cm8xa1k8600093b6tnrzpuq1n",
                                              "feature_schema_id": "cm3jdcfpg0m6m07081ju276sz",
                                              "name": "Example temporal free text",
                                              "value": "free_text",
                                              "text_answer": {
                                                  "content": "Example temporal free text label 2",
                                                  "classifications": []
                                              }
                                          }
                                      ]
                                  },
                                  "33496": {
                                      "classifications": [
                                          {
                                              "feature_id": "cm8xa2d2u000a3b6tk29o8u48",
                                              "feature_schema_id": "cm8x9zz4e04el07135zmghx1q",
                                              "name": "Example temporal radio",
                                              "value": "example_temporal_radio",
                                              "radio_answer": {
                                                  "feature_id": "cm8xa2d2u000b3b6ti9kszxcg",
                                                  "feature_schema_id": "cm8x9zz4e04em07132athat2z",
                                                  "name": "Option 1",
                                                  "value": "option_1",
                                                  "classifications": []
                                              }
                                          }
                                      ]
                                  },
                                  "60711": {
                                      "classifications": [
                                          {
                                              "feature_id": "cm8xa2hhh000c3b6tzeluuhpt",
                                              "feature_schema_id": "cm8x9zz4f04ex0713f5rehhp0",
                                              "name": "Example temporal checklist",
                                              "value": "example_temporal_checklist",
                                              "radio_answer": {
                                                  "feature_id": "cm8xa2ila000e3b6t410j19ir",
                                                  "feature_schema_id": "cm8x9zz4f04f00713drwj6qpb",
                                                  "name": "Option 2",
                                                  "value": "option_2",
                                                  "classifications": []
                                              }
                                          }
                                      ]
                                  }
                              },
                              "segments": {
                                  "cm8xa1k8600093b6tnrzpuq1n": [
                                      [
                                          6157,
                                          90697
                                      ]
                                  ],
                                  "cm8xa2d2u000b3b6ti9kszxcg": [
                                      [
                                          33496,
                                          90697
                                      ]
                                  ],
                                  "cm8xa2ila000e3b6t410j19ir": [
                                      [
                                          60711,
                                          90697
                                      ]
                                  ]
                              }
                          }
                      }
                  ],
                  "project_tags": []
              }
          }
      }
  ]
  ```
</CodeGroup>


# Export conversational text annotations
Source: https://docs.labelbox.com/horizon/developer-guides/export-conversational-text-annotations

How to export conversational text data and sample export formats.

<CardGroup>
  <Card title="Open In Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/exports/export_data.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/exports/export_data.ipynb" />
</CardGroup>

## Export JSON annotations

<CodeGroup>
  ```python Export theme={"theme":"css-variables"}
  # Set the export params to include/exclude certain fields.
  export_params= {
    "attachments": True,
    "metadata_fields": True,
    "data_row_details": True,
    "project_details": True,
    "label_details": True,
    "performance_details": True,
    "interpolated_frames": True
  }

  # Note: Filters follow AND logic, so typically using one filter is sufficient.

  filters= {
    "last_activity_at": ["2000-01-01 00:00:00", "2050-01-01 00:00:00"],
    "workflow_status": "<wkf-status>"
  }

  export_task = project.export(params=export_params, filters=filters)
  export_task.wait_till_done()

  # Stream the export using a callback function

  def json_stream_handler(output: labelbox.BufferedJsonConverterOutput):
    print(output.json)

  export_task.get_buffered_stream(stream_type=labelbox.StreamType.RESULT).start(stream_handler=json_stream_handler)

  # Collect all exported data into a list

  export_json = [data_row.json for data_row in export_task.get_buffered_stream()]

  print("file size: ", export_task.get_total_file_size(stream_type=lb.StreamType.RESULT))
  print("line count: ", export_task.get_total_lines(stream_type=lb.StreamType.RESULT))

  ```
</CodeGroup>

## Annotation export formats

### Text entity (named entity)

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "cluu14r3p0oty148p5d2isiki",
    "feature_schema_id": "cluu14fnh01f40703aigj669b",
    "name": "ner",
    "value": "ner",
    "annotation_kind": "ConversationalTextEntity",
    "classifications": [],
    "conversational_location": {
      "message_id": "4",
      "location": {
        "start": 0,
        "end": 8
      }
    }
  }
  ```
</CodeGroup>

### Classification - radio

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "cluu14r3p0ou3148p6d15q82b",
    "feature_schema_id": "cluu14fnh01fe0703abp5fjgx",
    "name": "radio_convo",
    "value": "radio_convo",
    "message_id": "0",
    "conversational_radio_answer": {
      "feature_id": "cluu14r3p0ou4148pc028jtzk",
      "feature_schema_id": "cluu14fnh01ff070383rh4ryk",
      "name": "first_radio_answer",
      "value": "first_radio_answer",
      "classifications": []
    }
  }
  ```
</CodeGroup>

### Classification - checklist

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "cluu14r3p0ou0148ptekhuz0a",
    "feature_schema_id": "cluu14fnh01f8070385tv57vy",
    "name": "checklist_convo",
    "value": "checklist_convo",
    "message_id": "2",
    "conversational_checklist_answers": [
      {
        "feature_id": "cluu14r3p0ou1148p8pb3s49a",
        "feature_schema_id": "cluu14fnh01f90703cx3j6gqu",
        "name": "first_checklist_answer",
        "value": "first_checklist_answer",
        "classifications": []
      },
      {
        "feature_id": "cluu14r3p0ou2148ph8tyhtp1",
        "feature_schema_id": "cluu14fnh01fb07030q2m5hqc",
        "name": "second_checklist_answer",
        "value": "second_checklist_answer",
        "classifications": []
      }
    ]
  }
  ```
</CodeGroup>

### Classification - free-form text

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "clr0x0ol83tvq07ea8f5phc2g",
    "feature_schema_id": "clpk2me9u02vx07xnet21fsgf",
    "name": "free_text",
    "text_answer": {
      "content": "sample text"
    }
  }
  ```
</CodeGroup>

### Relationship

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "clwkrnmlp00093b6qekq6qhyb",
    "feature_schema_id": "clwkrkehi04qy07y711tcgga6",
    "name": "relationship",
    "annotation_kind": "ConversationalTextUnidirectionalRelationship",
    "classifications": [],
    "unidirectional_relationship": {
      "source": "clwkrnhdb00033b6qlzbqjttc",
      "target": "clwkrnjky00063b6qc3tatj0p"
    }
  }
  ```
</CodeGroup>

## Sample project export

<CodeGroup>
  ```json JSON expandable theme={"theme":"css-variables"}
  {
    "data_row": {
      "id": "clfbnskoxb9q9076fbyjngwhm",
      "global_key": "conversation-1.json",
      "row_data": "https://storage.googleapis.com/labelbox-developer-testing-assets/conversational_text/1000-conversations/conversation-1.json",
      "details": {
        "dataset_id": "clfbnsjaj05hr07z605db0w3b",
        "dataset_name": "conversational_annotation_import_demo_dataset",
        "created_at": "2023-03-16T22:04:33.000+00:00",
        "updated_at": "2024-01-18T19:10:53.352+00:00",
        "last_activity_at": "2024-04-10T16:33:23.000+00:00",
        "created_by": "[email protected]"
      }
    },
    "media_attributes": {
      "mime_type": "application/vnd.labelbox.conversational",
      "labelable_ids": [
        "0",
        "2",
        "4",
        "6",
        "8",
        "10",
        "12",
        "14",
        "16",
        "18"
      ],
      "message_count": 20
    },
    "attachments": [],
    "metadata_fields": [],
    "projects": {
      "cluu14fwi01g0070321g9c1xi": {
        "name": "Conversational Text Annotation Import Demo",
        "labels": [
          {
            "label_kind": "Default",
            "version": "1.0.0",
            "id": "cluu14r490oud148pdk29bwes",
            "label_details": {
              "created_at": "2024-04-10T16:33:23.000+00:00",
              "updated_at": "2024-04-10T16:33:23.000+00:00",
              "created_by": "[email protected]",
              "content_last_updated_at": "2024-04-10T16:33:23.063+00:00",
              "reviews": []
            },
            "performance_details": {
              "seconds_to_create": 0,
              "seconds_to_review": 0,
              "skipped": false
            },
            "annotations": {
              "objects": [
                {
                  "feature_id": "cluu14r3p0oty148p5d2isiki",
                  "feature_schema_id": "cluu14fnh01f40703aigj669b",
                  "name": "ner",
                  "value": "ner",
                  "annotation_kind": "ConversationalTextEntity",
                  "classifications": [],
                  "conversational_location": {
                    "message_id": "4",
                    "location": {
                      "start": 0,
                      "end": 8
                    }
                  }
                }
              ],
              "classifications": [
                {
                  "feature_id": "cluu14r3p0otz148p09nol3tq",
                  "feature_schema_id": "cluu14fnh01f60703b4uadjbd",
                  "name": "text_convo",
                  "value": "text_convo",
                  "message_id": "0",
                  "conversational_text_answer": {
                    "content": "the answer to the text questions right here"
                  }
                },
                {
                  "feature_id": "cluu14r3p0ou0148ptekhuz0a",
                  "feature_schema_id": "cluu14fnh01f8070385tv57vy",
                  "name": "checklist_convo",
                  "value": "checklist_convo",
                  "message_id": "2",
                  "conversational_checklist_answers": [
                    {
                      "feature_id": "cluu14r3p0ou1148p8pb3s49a",
                      "feature_schema_id": "cluu14fnh01f90703cx3j6gqu",
                      "name": "first_checklist_answer",
                      "value": "first_checklist_answer",
                      "classifications": []
                    },
                    {
                      "feature_id": "cluu14r3p0ou2148ph8tyhtp1",
                      "feature_schema_id": "cluu14fnh01fb07030q2m5hqc",
                      "name": "second_checklist_answer",
                      "value": "second_checklist_answer",
                      "classifications": []
                    }
                  ]
                },
                {
                  "feature_id": "cluu14r3p0ou3148p6d15q82b",
                  "feature_schema_id": "cluu14fnh01fe0703abp5fjgx",
                  "name": "radio_convo",
                  "value": "radio_convo",
                  "message_id": "0",
                  "conversational_radio_answer": {
                    "feature_id": "cluu14r3p0ou4148pc028jtzk",
                    "feature_schema_id": "cluu14fnh01ff070383rh4ryk",
                    "name": "first_radio_answer",
                    "value": "first_radio_answer",
                    "classifications": []
                  }
                },
                {
                  "feature_id": "cluu14r3p0ou5148piuv74m17",
                  "feature_schema_id": "cluu14fni01fs0703ax77277l",
                  "name": "nested_radio_question",
                  "value": "nested_radio_question",
                  "radio_answer": {
                    "feature_id": "cluu14r3p0ou6148p0b4azzcu",
                    "feature_schema_id": "cluu14fni01ft07033xic99k3",
                    "name": "first_radio_answer",
                    "value": "first_radio_answer",
                    "classifications": [
                      {
                        "feature_id": "cluu14r3p0ou7148pgmcqsf11",
                        "feature_schema_id": "cluu14fni01fu0703c3iu2c0a",
                        "name": "sub_radio_question",
                        "value": "sub_radio_question",
                        "radio_answer": {
                          "feature_id": "cluu14r3p0ou8148pka3hvx26",
                          "feature_schema_id": "cluu14fni01fv07031r8i190o",
                          "name": "first_sub_radio_answer",
                          "value": "first_sub_radio_answer",
                          "classifications": []
                        }
                      }
                    ]
                  }
                },
                {
                  "feature_id": "cluu14r3p0ou9148puacr13og",
                  "feature_schema_id": "cluu14fni01fk07033aeye6ht",
                  "name": "nested_checklist_question",
                  "value": "nested_checklist_question",
                  "message_id": "10",
                  "conversational_checklist_answers": [
                    {
                      "feature_id": "cluu14r3p0oua148pttkqa3fe",
                      "feature_schema_id": "cluu14fni01fl0703eg9y2qgt",
                      "name": "first_checklist_answer",
                      "value": "first_checklist_answer",
                      "classifications": [
                        {
                          "feature_id": "cluu14r3p0oub148p3hld5bh9",
                          "feature_schema_id": "cluu14fni01fm0703b3mwa6ot",
                          "name": "sub_checklist_question",
                          "value": "sub_checklist_question",
                          "checklist_answers": [
                            {
                              "feature_id": "cluu14r3p0ouc148pfpiwlzht",
                              "feature_schema_id": "cluu14fni01fn0703apfk8j9q",
                              "name": "first_sub_checklist_answer",
                              "value": "first_sub_checklist_answer",
                              "classifications": []
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "relationships": []
            }
          }
        ],
        "project_details": {
          "ontology_id": "cluu14fmp01f307039z1h2c64",
          "task_name": "Done",
          "batch_id": "04e9d4c0-f758-11ee-8be6-db43b18899b6",
          "batch_name": "first-batch-convo-demo",
          "workflow_status": "DONE",
          "priority": 5,
          "consensus_expected_label_count": 1,
          "workflow_history": []
        },
        "project_tags": []
      }
    }
  }
  ```
</CodeGroup>

## Sample model run export

<CodeGroup>
  ```json JSON expandable theme={"theme":"css-variables"}
  {
    "data_row": {
      "id": "clfbnskoxb9q9076fbyjngwhm",
      "global_key": "conversation-1.json",
      "row_data": "https://storage.googleapis.com/labelbox-developer-testing-assets/conversational_text/1000-conversations/conversation-1.json",
      "details": {
        "dataset_id": "clfbnsjaj05hr07z605db0w3b",
        "dataset_name": "conversational_annotation_import_demo_dataset",
        "created_at": "2023-03-16T22:04:33.000+00:00",
        "updated_at": "2024-01-18T19:10:53.352+00:00",
        "created_by": "[email protected]"
      }
    },
    "media_attributes": {
      "mime_type": "application/vnd.labelbox.conversational",
      "labelable_ids": [
        "0",
        "2",
        "4",
        "6",
        "8",
        "10",
        "12",
        "14",
        "16",
        "18"
      ],
      "message_count": 20
    },
    "attachments": [],
    "metadata_fields": [],
    "experiments": {
      "a3143b06-742d-0576-d722-c187a5aad1b3": {
        "name": "Conversational_model_run_9985d371-1b0b-4a5b-8c94-4a6167b20090",
        "runs": {
          "a3143b06-adc7-086f-79a5-d251a0054cb8": {
            "name": "iteration 1",
            "run_data_row_id": "4f22e94f-71da-45ac-bab7-f95daf04e7a1",
            "labels": [
              {
                "label_kind": "Default",
                "version": "1.0.0",
                "id": "clq10ymdubt0u1447v6s77glu",
                "annotations": {
                  "objects": [
                    {
                      "feature_id": "e063fe4e-4675-0988-9f81-659c83dd1b43",
                      "feature_schema_id": "clq10ybnl067t07ztbavefdsy",
                      "name": "ner",
                      "value": "ner",
                      "annotation_kind": "ConversationalTextEntity",
                      "classifications": [],
                      "conversational_location": {
                        "message_id": "4",
                        "location": {
                          "start": 0,
                          "end": 8
                        }
                      }
                    }
                  ],
                  "classifications": [
                    {
                      "feature_id": "169f1fee-5db4-02c9-a232-dfb245a158ec",
                      "feature_schema_id": "clq10ybnl068907ztf1htgtoc",
                      "name": "nested_checklist_question",
                      "value": "nested_checklist_question",
                      "message_id": "10",
                      "conversational_checklist_answers": [
                        {
                          "feature_id": "c1c5898b-f1ce-02c7-b806-930ac8140923",
                          "feature_schema_id": "clq10ybnl068a07ztcnyzbhsn",
                          "name": "first_checklist_answer",
                          "value": "first_checklist_answer",
                          "classifications": [
                            {
                              "feature_id": "1fc2b7e8-a292-02ad-81b6-10cb9bdd56c4",
                              "feature_schema_id": "clq10ybnl068b07zth87h8c34",
                              "name": "sub_checklist_question",
                              "value": "sub_checklist_question",
                              "checklist_answers": [
                                {
                                  "feature_id": "de42a672-041f-03f3-bbd9-e771e332030a",
                                  "feature_schema_id": "clq10ybnm068c07zt53sdcs0m",
                                  "name": "first_sub_checklist_answer",
                                  "value": "first_sub_checklist_answer",
                                  "classifications": []
                                }
                              ]
                            }
                          ]
                        }
                      ]
                    },
                    {
                      "feature_id": "8adc0320-6d61-0011-bda0-80842b3733da",
                      "feature_schema_id": "clq10ybnm068h07ztaugh4uma",
                      "name": "nested_radio_question",
                      "value": "nested_radio_question",
                      "radio_answer": {
                        "feature_id": "475fb394-f606-0223-b625-48045ebf7208",
                        "feature_schema_id": "clq10ybnm068i07zt1xxv25it",
                        "name": "first_radio_answer",
                        "value": "first_radio_answer",
                        "classifications": [
                          {
                            "feature_id": "a61d83fd-4c47-0e88-9ae1-da3eb100eea0",
                            "feature_schema_id": "clq10ybnm068j07ztcqf794w4",
                            "name": "sub_radio_question",
                            "value": "sub_radio_question",
                            "radio_answer": {
                              "feature_id": "6dbde4c3-3a1e-0348-a5ac-7787a8ac0461",
                              "feature_schema_id": "clq10ybnm068k07ztdx9dca3i",
                              "name": "first_sub_radio_answer",
                              "value": "first_sub_radio_answer",
                              "classifications": []
                            }
                          }
                        ]
                      }
                    },
                    {
                      "feature_id": "96dce41f-f1c1-0822-9848-39c098b449f5",
                      "feature_schema_id": "clq10ybnl067v07ztb8f91r4u",
                      "name": "text_convo",
                      "value": "text_convo",
                      "message_id": "0",
                      "conversational_text_answer": {
                        "content": "the answer to the text questions are right here"
                      }
                    },
                    {
                      "feature_id": "ba7c519c-c618-01ba-ac4b-fdbbd8137bf0",
                      "feature_schema_id": "clq10ybnl068307ztd255hndf",
                      "name": "radio_convo",
                      "value": "radio_convo",
                      "message_id": "0",
                      "conversational_radio_answer": {
                        "feature_id": "ec0fc3fb-1948-0ff9-8bc6-0377721e4f0e",
                        "feature_schema_id": "clq10ybnl068407zt1x572rym",
                        "name": "first_radio_answer",
                        "value": "first_radio_answer",
                        "classifications": []
                      }
                    },
                    {
                      "feature_id": "fe3a901b-8e62-01fe-a792-10b15ec7fa0d",
                      "feature_schema_id": "clq10ybnl067x07zt0u67b9h3",
                      "name": "checklist_convo",
                      "value": "checklist_convo",
                      "message_id": "2",
                      "conversational_checklist_answers": [
                        {
                          "feature_id": "699a5673-bb4a-06d3-9b9e-19237e11f96d",
                          "feature_schema_id": "clq10ybnl068007zt6ct25775",
                          "name": "second_checklist_answer",
                          "value": "second_checklist_answer",
                          "classifications": []
                        },
                        {
                          "feature_id": "da192bd6-84ec-0acf-b875-d1ade525f466",
                          "feature_schema_id": "clq10ybnl067y07zt6gvjczsg",
                          "name": "first_checklist_answer",
                          "value": "first_checklist_answer",
                          "classifications": []
                        }
                      ]
                    }
                  ],
                  "relationships": []
                }
              }
            ],
            "predictions": [
              {
                "label_kind": "Default",
                "version": "1.0.0",
                "id": "clq10ymdubt0u1447v6s77glu",
                "annotations": {
                  "objects": [
                    {
                      "feature_id": "9c8f89a7-7211-4b33-a3c5-a3dab59f0407",
                      "feature_schema_id": "clq10ybnl067t07ztbavefdsy",
                      "name": "ner",
                      "value": "ner",
                      "annotation_kind": "ConversationalTextEntity",
                      "classifications": [],
                      "conversational_location": {
                        "message_id": "4",
                        "location": {
                          "start": 0,
                          "end": 8
                        }
                      }
                    }
                  ],
                  "classifications": [
                    {
                      "feature_id": "20cda99c-d0f7-4e59-8f43-a8b4f3929e2f",
                      "feature_schema_id": "clq10ybnl068307ztd255hndf",
                      "name": "radio_convo",
                      "value": "radio_convo",
                      "message_id": "0",
                      "conversational_radio_answer": {
                        "feature_id": "ec1ec509-79b6-48e7-973c-9c06e754630f",
                        "feature_schema_id": "clq10ybnl068407zt1x572rym",
                        "name": "first_radio_answer",
                        "value": "first_radio_answer",
                        "classifications": []
                      }
                    },
                    {
                      "feature_id": "49e4667e-ed14-400b-9b75-b8b89bfb6f11",
                      "feature_schema_id": "clq10ybnm068h07ztaugh4uma",
                      "name": "nested_radio_question",
                      "value": "nested_radio_question",
                      "radio_answer": {
                        "feature_id": "723be4ec-fa0c-4a67-9df0-a18f56f7bbd8",
                        "feature_schema_id": "clq10ybnm068i07zt1xxv25it",
                        "name": "first_radio_answer",
                        "value": "first_radio_answer",
                        "classifications": [
                          {
                            "feature_id": "f28f3d0a-9428-4b45-8de1-4ec6714da726",
                            "feature_schema_id": "clq10ybnm068j07ztcqf794w4",
                            "name": "sub_radio_question",
                            "value": "sub_radio_question",
                            "radio_answer": {
                              "feature_id": "82808a1e-a004-4965-bb77-4ab9e499c747",
                              "feature_schema_id": "clq10ybnm068k07ztdx9dca3i",
                              "name": "first_sub_radio_answer",
                              "value": "first_sub_radio_answer",
                              "classifications": []
                            }
                          }
                        ]
                      }
                    },
                    {
                      "feature_id": "988e5b9a-9a6d-4338-96bf-901475b4a2c0",
                      "feature_schema_id": "clq10ybnl068907ztf1htgtoc",
                      "name": "nested_checklist_question",
                      "value": "nested_checklist_question",
                      "message_id": "10",
                      "conversational_checklist_answers": [
                        {
                          "feature_id": "9ea5984d-fb21-48e4-96c8-ce19162b5850",
                          "feature_schema_id": "clq10ybnl068a07ztcnyzbhsn",
                          "name": "first_checklist_answer",
                          "value": "first_checklist_answer",
                          "classifications": [
                            {
                              "feature_id": "bf896185-5a77-410e-a4cd-abf0b07122fa",
                              "feature_schema_id": "clq10ybnl068b07zth87h8c34",
                              "name": "sub_checklist_question",
                              "value": "sub_checklist_question",
                              "checklist_answers": [
                                {
                                  "feature_id": "0812ee3d-7f5d-480b-82a2-4e965a48fc49",
                                  "feature_schema_id": "clq10ybnm068c07zt53sdcs0m",
                                  "name": "first_sub_checklist_answer",
                                  "value": "first_sub_checklist_answer",
                                  "classifications": []
                                }
                              ]
                            }
                          ]
                        }
                      ]
                    },
                    {
                      "feature_id": "cb4730df-2e22-45db-bc74-fd439d577571",
                      "feature_schema_id": "clq10ybnl067x07zt0u67b9h3",
                      "name": "checklist_convo",
                      "value": "checklist_convo",
                      "message_id": "2",
                      "conversational_checklist_answers": [
                        {
                          "feature_id": "8184feca-970c-4ba7-bfbd-4d0eb5849a90",
                          "feature_schema_id": "clq10ybnl068007zt6ct25775",
                          "name": "second_checklist_answer",
                          "value": "second_checklist_answer",
                          "classifications": []
                        },
                        {
                          "feature_id": "dc40cfef-aa6f-436d-a5b5-0d56bbd1741e",
                          "feature_schema_id": "clq10ybnl067y07zt6gvjczsg",
                          "name": "first_checklist_answer",
                          "value": "first_checklist_answer",
                          "classifications": []
                        }
                      ]
                    },
                    {
                      "feature_id": "db7109ed-e380-4389-ab2f-98e72b4c2afe",
                      "feature_schema_id": "clq10ybnl067v07ztb8f91r4u",
                      "name": "text_convo",
                      "value": "text_convo",
                      "message_id": "0",
                      "conversational_text_answer": {
                        "content": "the answer to the text questions are right here"
                      }
                    }
                  ],
                  "relationships": []
                }
              }
            ]
          }
        }
      }
    },
    "models": {}
  }
  ```
</CodeGroup>


# Export document annotations
Source: https://docs.labelbox.com/horizon/developer-guides/export-document-annotations

How to export document (PDF) annotations and sample export formats.

<CardGroup>
  <Card title="Open In Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/exports/export_data.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/exports/export_data.ipynb" />
</CardGroup>

## Export as JSON

<CodeGroup>
  ```python Export theme={"theme":"css-variables"}
  # Set the export params to include/exclude certain fields.
  export_params= {
    "attachments": True,
    "metadata_fields": True,
    "data_row_details": True,
    "project_details": True,
    "label_details": True,
    "performance_details": True,
    "interpolated_frames": True
  }

  # Note: Filters follow AND logic, so typically using one filter is sufficient.

  filters= {
    "last_activity_at": ["2000-01-01 00:00:00", "2050-01-01 00:00:00"],
    "workflow_status": "<wkf-status>"
  }

  export_task = project.export(params=export_params, filters=filters)
  export_task.wait_till_done()

  # Stream the export using a callback function

  def json_stream_handler(output: labelbox.BufferedJsonConverterOutput):
    print(output.json)

  export_task.get_buffered_stream(stream_type=labelbox.StreamType.RESULT).start(stream_handler=json_stream_handler)

  # Collect all exported data into a list

  export_json = [data_row.json for data_row in export_task.get_buffered_stream()]

  print("file size: ", export_task.get_total_file_size(stream_type=lb.StreamType.RESULT))
  print("line count: ", export_task.get_total_lines(stream_type=lb.StreamType.RESULT))
  ```
</CodeGroup>

## Annotation export formats

### Bounding box

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "cluu0k04w05hk07h76lv05sb0",
    "feature_schema_id": "cluu0ckta024507udcols6x8y",
    "name": "bounding_box",
    "value": "bounding_box",
    "annotation_kind": "DocumentBoundingBox",
    "classifications": [],
    "page_number": 1,
    "bounding_box": {
      "top": 66.251,
      "left": 96.424,
      "height": 80.68099999999998,
      "width": 82.65
    }
  }
  ```
</CodeGroup>

### Text entity (named entity)

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "cluu0k04w05ho07h73jx24ol7",
    "name": "named_entity",
    "value": "named_entity",
    "annotation_kind": "DocumentEntityToken",
    "classifications": [],
    "location": {
      "groups": [
        {
          "id": "1f54807e-b612-4f4a-9ca5-ee302ee9c659",
          "page_number": 1,
          "tokens": [
            "839967fa-89a0-4ad1-896e-d25e5f4cb8e5",
            "9853c27e-30e0-45e0-b461-c28da6573b48",
            "e24f45c6-1efc-440f-a4dd-bb3eb522134f",
            "3339e940-3f73-4b56-b695-474808dad1e5",
            "a6f79ed8-fc20-4870-b9d1-aee3e92943bc",
            "07ac4d2e-68bb-4c25-bcc0-b03a697c4138",
            "7469a5c7-74f8-43c4-b1fb-1c89105a5c66",
            "eff97919-2529-4b67-9640-20301d45a2d4"
          ],
          "text": "Organic charge transfer salts based on the donor"
        }
      ]
    }
  }
  ```
</CodeGroup>

### Relationship

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "cluu0k04y05hx07h75lne7jh6",
    "feature_schema_id": "cluu0cktb024907udbo0qhco1",
    "name": "relationship",
    "value": "relationship",
    "annotation_kind": "DocumentUnidirectionalRelationship",
    "classifications": [],
    "unidirectional_relationship": {
      "source": "cluu0k04w05hs07h7ahsw7689",
      "target": "cluu0k04w05hk07h76lv05sb0"
    }
  }
  ```
</CodeGroup>

### Classification - Radio

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "cluu0k04y05hw07h70s248skz",
    "feature_schema_id": "cluu0cktc024r07ud2x2oba9o",
    "name": "radio_question",
    "value": "radio_question",
    "radio_answer": {
      "feature_id": "cluu0k04y05i407h7ehvn84w9",
      "feature_schema_id": "cluu0cktc024s07udc81y6lx3",
      "name": "first_radio_answer",
      "value": "first_radio_answer",
      "classifications": []
    }
  }
  ```
</CodeGroup>

### Classification - Checklist

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "cluu0k04w05hj07h71okedbdg",
    "feature_schema_id": "cluu0cktc024x07udc5fr0p96",
    "name": "checklist_question",
    "value": "checklist_question",
    "checklist_answers": [
      {
        "feature_id": "cluu0k04y05hy07h73gy3awz7",
        "feature_schema_id": "cluu0cktc025007ud0d5w5o05",
        "name": "second_checklist_answer",
        "value": "second_checklist_answer",
        "classifications": []
      },
      {
        "feature_id": "cluu0k04y05i107h73vpphndb",
        "feature_schema_id": "cluu0cktc024y07ud97zu7h1d",
        "name": "first_checklist_answer",
        "value": "first_checklist_answer",
        "classifications": []
      }
    ]
  }
  ```
</CodeGroup>

### Classification - Free-form text

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "cluu0k04w05hp07h7fn0k9bdw",
    "feature_schema_id": "cluu0cktc025307udgntw3qxi",
    "name": "free_text",
    "value": "free_text",
    "text_answer": {
      "content": "sample text"
    }
  }
  ```
</CodeGroup>

## Sample project export

<CodeGroup>
  ```json JSON expandable theme={"theme":"css-variables"}
  [
      {
          "data_row": {
              "id": "cm2kphnpy50m707445lc9afs9",
              "global_key": "0801.3483_doc.pdf97c24e0d-4bd4-43df-830c-d8dea1db5a41",
              "row_data": "https://storage.googleapis.com/labelbox-datasets/arxiv-pdf/data/99-word-token-pdfs/0801.3483.pdf",
              "details": {
                  "dataset_id": "cm2kphm4b006f0701l4h33iz9",
                  "dataset_name": "pdf_demo_dataset",
                  "created_at": "2024-10-22T17:15:12.604+00:00",
                  "updated_at": "2024-10-22T17:15:18.631+00:00",
                  "last_activity_at": "2024-10-22T17:26:32.131+00:00",
                  "created_by": "[email protected]"
              }
          },
          "media_attributes": {
              "asset_type": "pdf",
              "mime_type": "application/pdf",
              "text_layer_url": "https://storage.labelbox.com/clwzraq9k0bae070o3j0w20p5%2F9sfa9cl544707m05ypnhpk2mc-text-layer.json?Expires=1729704414371&KeyName=labelbox-assets-key-3&Signature=ZIo4XYc3QyrEgMwZS5pxWJ6Mqr0"
          },
          "attachments": [],
          "metadata_fields": [],
          "projects": {
              "cm2kpk0i1055g07wp1ubl7n9k": {
                  "name": "example_pdf_export",
                  "labels": [
                      {
                          "label_kind": "Default",
                          "version": "1.0.0",
                          "id": "cm2kpvs8b08da07yc40llhwmt",
                          "label_details": {
                              "created_at": "2024-10-22T17:26:17.000+00:00",
                              "updated_at": "2024-10-22T17:26:17.000+00:00",
                              "created_by": "[email protected]",
                              "content_last_updated_at": "2024-10-22T17:26:17.418+00:00",
                              "reviews": []
                          },
                          "performance_details": {
                              "seconds_to_create": 3,
                              "seconds_to_review": 0,
                              "skipped": false
                          },
                          "annotations": {
                              "objects": [
                                  {
                                      "feature_id": "cm2kpvtfw040607ic3mkjgsq5",
                                      "name": "named_entity",
                                      "value": "named_entity",
                                      "annotation_kind": "DocumentEntityToken",
                                      "classifications": [],
                                      "location": {
                                          "groups": [
                                              {
                                                  "id": "fe8d7f06-c0f1-419c-a880-4926b2cb5bf0",
                                                  "page_number": 1,
                                                  "tokens": [
                                                      "90d94c02-f00b-4c36-aa8f-086a58198d47",
                                                      "e49b7391-8466-497f-8a56-60945c85a222",
                                                      "b4dd1d51-a8a2-4d5f-8074-2409336b5887",
                                                      "d2bc8a3b-554a-43fc-8931-1f6757321ef0",
                                                      "17b69f57-0f6f-4ef7-a3c4-d46d72403564",
                                                      "23d23fed-0a3b-4bb7-9df1-91d0247f6348",
                                                      "f2d8efb5-1a3f-40cd-9ff4-5cc8a53669c9",
                                                      "a74d4317-93bd-4cc9-ae4d-e62fc4c2e370",
                                                      "1ab58cb7-27d3-46a7-aa2b-5b7f89028b28",
                                                      "458d43da-baa4-4eac-966f-4e6d1cf75c9d",
                                                      "4e660727-51d5-4046-a2fb-eaeefffc4171",
                                                      "83838d7b-c083-47c2-bd11-c7c16cb3667d"
                                                  ],
                                                  "text": "Metal - insulator ( MI ) transitions have been one of the"
                                              }
                                          ]
                                      }
                                  },
                                  {
                                      "feature_id": "cm2kpvtfw040807icec0q6ux7",
                                      "feature_schema_id": "cm2kpi4xu05z507z45jx193j8",
                                      "name": "bounding_box",
                                      "value": "bounding_box",
                                      "annotation_kind": "DocumentBoundingBox",
                                      "classifications": [],
                                      "page_number": 1,
                                      "bounding_box": {
                                          "top": 68.875,
                                          "left": 188.257,
                                          "height": 80.68100000000001,
                                          "width": 82.64999999999998
                                      },
                                      "page_dimensions": {
                                          "height": 792.0,
                                          "width": 612.0
                                      }
                                  },
                                  {
                                      "feature_id": "cm2kpvtfw040b07ic61w421yf",
                                      "feature_schema_id": "cm2kpi4xu05zh07z4chbxdr0m",
                                      "name": "bbox_with_radio_subclass",
                                      "value": "bbox_with_radio_subclass",
                                      "annotation_kind": "DocumentBoundingBox",
                                      "classifications": [
                                          {
                                              "feature_id": "cm2kpvtfz040o07ic6jil7vp5",
                                              "feature_schema_id": "cm2kpi4xu05zi07z42xe841v4",
                                              "name": "sub_radio_question",
                                              "value": "sub_radio_question",
                                              "radio_answer": {
                                                  "feature_id": "cm2kpvtfz040p07icbaciagt3",
                                                  "feature_schema_id": "cm2kpi4xu05zj07z4cjm301cg",
                                                  "name": "first_sub_radio_answer",
                                                  "value": "first_sub_radio_answer",
                                                  "classifications": [
                                                      {
                                                          "feature_id": "cm2kpvtg0040w07icd28t2zwm",
                                                          "feature_schema_id": "cm2kpi4xu05zk07z45nqcgxdq",
                                                          "name": "second_sub_radio_question",
                                                          "value": "second_sub_radio_question",
                                                          "radio_answer": {
                                                              "feature_id": "cm2kpvtg0040x07icg6qlb045",
                                                              "feature_schema_id": "cm2kpi4xu05zl07z4aki5765r",
                                                              "name": "second_sub_radio_answer",
                                                              "value": "second_sub_radio_answer",
                                                              "classifications": []
                                                          }
                                                      }
                                                  ]
                                              }
                                          }
                                      ],
                                      "page_number": 1,
                                      "bounding_box": {
                                          "top": 226.757,
                                          "left": 317.271,
                                          "height": 194.22899999999998,
                                          "width": 249.38600000000002
                                      },
                                      "page_dimensions": {
                                          "height": 792.0,
                                          "width": 612.0
                                      }
                                  },
                                  {
                                      "feature_id": "cm2kpvtfw040c07ic8p9mh4do",
                                      "name": "named_entity",
                                      "value": "named_entity",
                                      "annotation_kind": "DocumentEntityToken",
                                      "classifications": [],
                                      "location": {
                                          "groups": [
                                              {
                                                  "id": "b111b609-9715-4682-93fd-f000425b6e47",
                                                  "page_number": 1,
                                                  "tokens": [
                                                      "ce039ffb-e6ec-4b5f-8a71-df5ae9b147bb",
                                                      "f1f154a8-4844-4eee-ad4a-d118ae2245aa",
                                                      "9b35fcaa-6840-459a-b4e5-73f761ba4efd",
                                                      "6a6d3239-fbbd-4db9-a0a8-c71777de8c01",
                                                      "d93dae8e-5be4-4451-91c7-4a5f9c3b244e",
                                                      "9b962313-01a8-41f4-ac57-567043735542",
                                                      "6b49e3cc-9198-4bb9-a42a-422c13bef397",
                                                      "c3b3dc30-e8da-4cbb-85cc-ce3b212a6e98"
                                                  ],
                                                  "text": "Organic charge transfer salts based on the donor"
                                              }
                                          ]
                                      }
                                  },
                                  {
                                      "feature_id": "cm2kpvtfz040e07icfknwbpmg",
                                      "name": "named_entity",
                                      "value": "named_entity",
                                      "annotation_kind": "DocumentEntityToken",
                                      "classifications": [],
                                      "location": {
                                          "groups": [
                                              {
                                                  "id": "118f1d6f-f07b-4998-bdbd-508d52e7a118",
                                                  "page_number": 1,
                                                  "tokens": [
                                                      "c5d8e961-0844-43b1-a165-5666c495f7b0",
                                                      "276fef85-a7a5-472b-ad56-793de2de0be2",
                                                      "ab0ae1eb-715c-4848-9ecf-6600b42072b0",
                                                      "7ce6b395-ad28-4762-ab45-9d29b6bc0277",
                                                      "07b37776-4fc1-4c12-88f6-f53114042f58",
                                                      "0f4a4d42-786f-4be6-8780-c5fe0fe48b17",
                                                      "6cb7839e-7a5c-49a4-94d4-1b686f37f3e3",
                                                      "8bf2b485-3e2f-40c5-afad-3bfd546da580"
                                                  ],
                                                  "text": "the experimental investigations on this issue have not"
                                              }
                                          ]
                                      }
                                  },
                                  {
                                      "feature_id": "cm2kpvtfz040h07ic2b2kcz37",
                                      "feature_schema_id": "cm2kpi4xu05z507z45jx193j8",
                                      "name": "bounding_box",
                                      "value": "bounding_box",
                                      "annotation_kind": "DocumentBoundingBox",
                                      "classifications": [],
                                      "page_number": 1,
                                      "bounding_box": {
                                          "top": 66.251,
                                          "left": 96.424,
                                          "height": 80.68099999999998,
                                          "width": 82.65
                                      },
                                      "page_dimensions": {
                                          "height": 792.0,
                                          "width": 612.0
                                      }
                                  },
                                  {
                                      "feature_id": "cm2kpvtfz040i07icdfj6ao5f",
                                      "name": "ner_with_checklist_subclass",
                                      "value": "ner_with_checklist_subclass",
                                      "annotation_kind": "DocumentEntityToken",
                                      "classifications": [
                                          {
                                              "feature_id": "cm2kpvtfz040n07ica2s03hyb",
                                              "feature_schema_id": "cm2kpi4xu05zc07z4ho4pe8kg",
                                              "name": "sub_checklist_question",
                                              "value": "sub_checklist_question",
                                              "checklist_answers": [
                                                  {
                                                      "feature_id": "cm2kpvtg0040r07icbcjo2s75",
                                                      "feature_schema_id": "cm2kpi4xu05zd07z46ascfffz",
                                                      "name": "first_sub_checklist_answer",
                                                      "value": "first_sub_checklist_answer",
                                                      "classifications": []
                                                  }
                                              ]
                                          }
                                      ],
                                      "location": {
                                          "groups": [
                                              {
                                                  "id": "0f325409-ad93-428e-8941-93255af67c38",
                                                  "page_number": 1,
                                                  "tokens": [
                                                      "ca996503-59ac-40cb-b213-c760a01193b3",
                                                      "f2b83cc1-028a-472c-8483-594426d27d9f",
                                                      "f86eef15-47b9-4093-82aa-f5d1129030b8",
                                                      "94c4ca63-88cc-49f6-89fb-dfc08b57f295",
                                                      "32184378-a94f-44b4-832d-b1a26d9b0b41",
                                                      "d2b28d96-50f0-4953-bd61-0c8cac971e91",
                                                      "e4ad49e8-9b94-4255-9002-cfc715bd92f4",
                                                      "73dec4b6-2b34-4225-9e5b-119206fadb9f",
                                                      "2a1e2a8f-11f9-4ac8-95bd-fba3f42196fc"
                                                  ],
                                                  "text": "T. Sasaki , N. Yoneyama , and N. Kobayashi"
                                              }
                                          ]
                                      }
                                  },
                                  {
                                      "feature_id": "cm2kpvtfz040k07ic92exh5mc",
                                      "feature_schema_id": "cm2kpi4xu05z507z45jx193j8",
                                      "name": "bounding_box",
                                      "value": "bounding_box",
                                      "annotation_kind": "DocumentBoundingBox",
                                      "classifications": [],
                                      "page_number": 0,
                                      "bounding_box": {
                                          "top": 135.3,
                                          "left": 102.771,
                                          "height": 109.84299999999999,
                                          "width": 415.8
                                      },
                                      "page_dimensions": {
                                          "height": 792.0,
                                          "width": 612.0
                                      }
                                  }
                              ],
                              "classifications": [
                                  {
                                      "feature_id": "cm2kpvtfw040707ic3p25ehsi",
                                      "feature_schema_id": "cm2kpi4xv060307z440vca4xf",
                                      "name": "free_text",
                                      "value": "free_text",
                                      "text_answer": {
                                          "content": "sample text"
                                      }
                                  },
                                  {
                                      "feature_id": "cm2kpvtfw040907icdi74gf73",
                                      "feature_schema_id": "cm2kpi4xv05zx07z4e21cch7q",
                                      "name": "checklist_question",
                                      "value": "checklist_question",
                                      "checklist_answers": [
                                          {
                                              "feature_id": "cm2kpvtfz040l07ic0nb7048e",
                                              "feature_schema_id": "cm2kpi4xv05zy07z4aaxg4ld5",
                                              "name": "first_checklist_answer",
                                              "value": "first_checklist_answer",
                                              "classifications": []
                                          },
                                          {
                                              "feature_id": "cm2kpvtg0040s07ice9tz7elg",
                                              "feature_schema_id": "cm2kpi4xv060007z4eqwvda46",
                                              "name": "second_checklist_answer",
                                              "value": "second_checklist_answer",
                                              "classifications": []
                                          }
                                      ]
                                  },
                                  {
                                      "feature_id": "cm2kpvtfw040a07ic6wpccc2a",
                                      "feature_schema_id": "cm2kpi4xv060507z4hyqfag6g",
                                      "name": "nested_radio_question",
                                      "value": "nested_radio_question",
                                      "radio_answer": {
                                          "feature_id": "cm2kpvtfz040m07ic8pg6dw8i",
                                          "feature_schema_id": "cm2kpi4xv060607z4d2kecbnj",
                                          "name": "first_radio_answer",
                                          "value": "first_radio_answer",
                                          "classifications": [
                                              {
                                                  "feature_id": "cm2kpvtg0040u07ic73h35155",
                                                  "feature_schema_id": "cm2kpi4xv060707z4br8a2jrr",
                                                  "name": "sub_radio_question",
                                                  "value": "sub_radio_question",
                                                  "radio_answer": {
                                                      "feature_id": "cm2kpvtg0040z07ic27ath50b",
                                                      "feature_schema_id": "cm2kpi4xv060807z4bwcfcuar",
                                                      "name": "first_sub_radio_answer",
                                                      "value": "first_sub_radio_answer",
                                                      "classifications": []
                                                  }
                                              }
                                          ]
                                      }
                                  },
                                  {
                                      "feature_id": "cm2kpvtfz040f07ic92mf2trh",
                                      "feature_schema_id": "cm2kpi4xv060d07z435j1a2ht",
                                      "name": "nested_checklist_question",
                                      "value": "nested_checklist_question",
                                      "checklist_answers": [
                                          {
                                              "feature_id": "cm2kpvtg0040q07ic3vb4bsbw",
                                              "feature_schema_id": "cm2kpi4xv060e07z4a8recoch",
                                              "name": "first_checklist_answer",
                                              "value": "first_checklist_answer",
                                              "classifications": [
                                                  {
                                                      "feature_id": "cm2kpvtg0040v07ic1xfc4c0a",
                                                      "feature_schema_id": "cm2kpi4xv060f07z4d0fi0c2h",
                                                      "name": "sub_checklist_question",
                                                      "value": "sub_checklist_question",
                                                      "checklist_answers": [
                                                          {
                                                              "feature_id": "cm2kpvtg0040y07ic082kdt55",
                                                              "feature_schema_id": "cm2kpi4xv060g07z48ash27gn",
                                                              "name": "first_sub_checklist_answer",
                                                              "value": "first_sub_checklist_answer",
                                                              "classifications": []
                                                          }
                                                      ]
                                                  }
                                              ]
                                          }
                                      ]
                                  },
                                  {
                                      "feature_id": "cm2kpvtfz040j07icflaf69my",
                                      "feature_schema_id": "cm2kpi4xu05zr07z41ds99wmj",
                                      "name": "radio_question",
                                      "value": "radio_question",
                                      "radio_answer": {
                                          "feature_id": "cm2kpvtg0040t07icfigkfbog",
                                          "feature_schema_id": "cm2kpi4xu05zs07z46r452wc1",
                                          "name": "first_radio_answer",
                                          "value": "first_radio_answer",
                                          "classifications": []
                                      }
                                  }
                              ],
                              "relationships": [
                                  {
                                      "feature_id": "cm2kpvtfw040d07ic6eks5lc9",
                                      "feature_schema_id": "cm2kpi4xu05z907z47bbk1kr5",
                                      "name": "relationship",
                                      "value": "relationship",
                                      "annotation_kind": "DocumentUnidirectionalRelationship",
                                      "classifications": [],
                                      "unidirectional_relationship": {
                                          "source": "cm2kpvtfw040c07ic8p9mh4do",
                                          "target": "cm2kpvtfz040e07icfknwbpmg"
                                      }
                                  },
                                  {
                                      "feature_id": "cm2kpvtfz040g07ichnlz7kfk",
                                      "feature_schema_id": "cm2kpi4xu05z907z47bbk1kr5",
                                      "name": "relationship",
                                      "value": "relationship",
                                      "annotation_kind": "DocumentUnidirectionalRelationship",
                                      "classifications": [],
                                      "unidirectional_relationship": {
                                          "source": "cm2kpvtfw040807icec0q6ux7",
                                          "target": "cm2kpvtfz040h07ic2b2kcz37"
                                      }
                                  }
                              ]
                          }
                      }
                  ],
                  "project_details": {
                      "ontology_id": "cm2kpi4xe05z407z4h6r77nys",
                      "task_name": "Done",
                      "batch_id": "98a7bb20-9099-11ef-bede-61385dfcee13",
                      "batch_name": "PDF_annotation_batch",
                      "workflow_status": "DONE",
                      "priority": 5,
                      "consensus_expected_label_count": 3,
                      "workflow_history": [
                          {
                              "action": "Approve",
                              "created_at": "2024-10-22T17:26:32.093+00:00",
                              "created_by": "[email protected]",
                              "previous_task_name": "Initial review task",
                              "previous_task_id": "94821756-edd8-4d53-9785-da3e41d310a6"
                          },
                          {
                              "action": "Move",
                              "created_at": "2024-10-22T17:26:17.761+00:00",
                              "created_by": "[email protected]",
                              "previous_task_name": "Initial labeling task",
                              "previous_task_id": "3e686f9b-4266-0311-a3bd-bc1ecc9ec115",
                              "next_task_name": "Initial review task",
                              "next_task_id": "94821756-edd8-4d53-9785-da3e41d310a6"
                          },
                          {
                              "action": "Move",
                              "created_at": "2024-10-22T17:26:17.754+00:00",
                              "created_by": "[email protected]",
                              "next_task_name": "Initial labeling task",
                              "next_task_id": "3e686f9b-4266-0311-a3bd-bc1ecc9ec115"
                          }
                      ]
                  },
                  "project_tags": []
              }
          }
      }
  ]
  ```
</CodeGroup>

## Sample model run export

<CodeGroup>
  ```json JSON expandable theme={"theme":"css-variables"}
  {
    "data_row": {
      "id": "clfh1n29e1mh2078d3jt3b5cx",
      "global_key": "0801.3483.pdf",
      "row_data": "https://storage.googleapis.com/labelbox-datasets/arxiv-pdf/data/99-word-token-pdfs/0801.3483.pdf",
      "details": {
        "dataset_id": "clfh1n0um00wd071d2uf21fy6",
        "dataset_name": "pdf_demo_dataset",
        "created_at": "2023-03-20T16:31:01.000+00:00",
        "updated_at": "2023-03-20T16:31:01.000+00:00",
        "created_by": "[email protected]"
      }
    },
    "media_attributes": {
      "mime_type": "application/pdf",
      "text_layer_url": "https://storage.googleapis.com/labelbox-datasets/arxiv-pdf/data/99-word-token-pdfs/0801.3483-lb-textlayer.json"
    },
    "attachments": [],
    "metadata_fields": [],
    "experiments": {
      "a15bd76b-e29f-02f2-b909-0e20f007f735": {
        "name": "PDF_model_run_9c051c54-bb5b-47be-a0f4-16c909a9afb0",
        "runs": {
          "a15bd76c-41aa-0f04-c633-ee3e40b1d0bc": {
            "name": "iteration 1",
            "run_data_row_id": "dff29c2a-27c2-443b-95a8-402d9c2be311",
            "labels": [
              {
                "label_kind": "Default",
                "version": "1.0.0",
                "id": "clhs1ru105b3w142bd71v29u1",
                "annotations": {
                  "objects": [
                    {
                      "feature_id": "2aa47146-4d81-4fcd-b513-7f7f5e275db7",
                      "feature_schema_id": "clhs1rh090buq07y15p2wd58d",
                      "name": "bounding_box",
                      "value": "bounding_box",
                      "annotation_kind": "DocumentBoundingBox",
                      "classifications": [],
                      "page_number": 0,
                      "bounding_box": {
                        "top": 135.3,
                        "left": 102.771,
                        "height": 109.84299999999999,
                        "width": 415.8
                      }
                    },
                    {
                      "feature_id": "40cd3ef3-7258-4da6-bc8b-d4f2eb12abfb",
                      "name": "ner_with_checklist_subclass",
                      "value": "ner_with_checklist_subclass",
                      "annotation_kind": "DocumentEntityToken",
                      "classifications": [
                        {
                          "feature_id": "4cf8dcba-32ab-4bf0-a97e-5a17d72335b5",
                          "feature_schema_id": "clhs1rh0a0buv07y1e0wobtgu",
                          "name": "sub_checklist_question",
                          "value": "sub_checklist_question",
                          "checklist_answers": [
                            {
                              "feature_id": "b139a807-7b9f-41f2-97e2-1dca6cd9f460",
                              "feature_schema_id": "clhs1rh0a0buw07y12tg3hm0x",
                              "name": "first_sub_checklist_answer",
                              "value": "first_sub_checklist_answer",
                              "classifications": []
                            }
                          ]
                        }
                      ],
                      "location": {
                        "groups": [
                          {
                            "id": "80e1e326-0194-4f38-845c-d9d22b0054fb",
                            "page_number": 1,
                            "tokens": [
                              "c2639311-4db1-4ad5-9a8d-1f2ab2c2dae8",
                              "2aa2f9df-df5a-4b3a-aa57-05aff053c147",
                              "aa725467-7b86-4629-9c70-a6391decb39b",
                              "6a4a0a69-f04e-44c0-b343-096aef413444",
                              "8dcce11c-c2ae-4274-b634-c950c8a7f333",
                              "85fccf77-474b-4707-85cc-592871d5c680",
                              "5d16be99-fc40-4228-99db-0a121a0478e7"
                            ],
                            "text": "T. Sasaki,* N. Yoneyama, and N. Kobayashi"
                          }
                        ]
                      }
                    },
                    {
                      "feature_id": "d2cd83b8-7a7b-4751-8b5d-dfa05be90bf5",
                      "name": "named_entity",
                      "value": "named_entity",
                      "annotation_kind": "DocumentEntityToken",
                      "classifications": [],
                      "location": {
                        "groups": [
                          {
                            "id": "2f4336f4-a07e-4e0a-a9e1-5629b03b719b",
                            "page_number": 1,
                            "tokens": [
                              "3f984bf3-1d61-44f5-b59a-9658a2e3440f",
                              "3bf00b56-ff12-4e52-8cc1-08dbddb3c3b8",
                              "6e1c3420-d4b7-4c5a-8fd6-ead43bf73d80",
                              "87a43d32-af76-4a1d-b262-5c5f4d5ace3a",
                              "e8606e8a-dfd9-4c49-a635-ad5c879c75d0",
                              "67c7c19e-4654-425d-bf17-2adb8cf02c30",
                              "149c5e80-3e07-49a7-ab2d-29ddfe6a38fa",
                              "b0e94071-2187-461e-8e76-96c58738a52c"
                            ],
                            "text": "Metal-insulator (MI) transitions have been one of the"
                          }
                        ]
                      }
                    },
                    {
                      "feature_id": "e166e286-1ead-438d-bb6e-0bc148b4c5f0",
                      "feature_schema_id": "clhs1rh0a0bv007y1fyxw0gyg",
                      "name": "bbox_with_radio_subclass",
                      "value": "bbox_with_radio_subclass",
                      "annotation_kind": "DocumentBoundingBox",
                      "classifications": [
                        {
                          "feature_id": "68838874-d209-4672-891f-10954a698292",
                          "feature_schema_id": "clhs1rh0a0bv107y14g5de74m",
                          "name": "sub_radio_question",
                          "value": "sub_radio_question",
                          "radio_answer": {
                            "feature_id": "8ce7710d-7a15-4d95-b9a9-a9a2bf733380",
                            "feature_schema_id": "clhs1rh0a0bv207y13bsndvuv",
                            "name": "first_sub_radio_answer",
                            "value": "first_sub_radio_answer",
                            "classifications": [
                              {
                                "feature_id": "bc3542d1-f373-4668-841b-15a4e9f8f67b",
                                "feature_schema_id": "clhs1rh0a0bv307y17teq5ob3",
                                "name": "second_sub_radio_question",
                                "value": "second_sub_radio_question",
                                "radio_answer": {
                                  "feature_id": "6d200263-2065-4363-b962-1ee5a1296a8e",
                                  "feature_schema_id": "clhs1rh0a0bv407y1d72h3bdp",
                                  "name": "second_sub_radio_answer",
                                  "value": "second_sub_radio_answer",
                                  "classifications": []
                                }
                              }
                            ]
                          }
                        }
                      ],
                      "page_number": 1,
                      "bounding_box": {
                        "top": 226.757,
                        "left": 317.271,
                        "height": 194.22899999999998,
                        "width": 249.38600000000002
                      }
                    }
                  ],
                  "classifications": [
                    {
                      "feature_id": "2fbbc78f-d1df-428d-be26-da69a6516786",
                      "feature_schema_id": "clhs1rh0a0bva07y11vb9gjzu",
                      "name": "radio_question",
                      "value": "radio_question",
                      "radio_answer": {
                        "feature_id": "24924adc-cb1c-40fb-a945-364a2cb0782f",
                        "feature_schema_id": "clhs1rh0a0bvb07y134ka29be",
                        "name": "first_radio_answer",
                        "value": "first_radio_answer",
                        "classifications": []
                      }
                    },
                    {
                      "feature_id": "a10706c7-0e09-47d9-b273-ed61f6ee72c5",
                      "feature_schema_id": "clhs1rh0b0bvg07y1as0cfj9r",
                      "name": "checklist_question",
                      "value": "checklist_question",
                      "checklist_answers": [
                        {
                          "feature_id": "03c2feb6-bda3-4ee2-ad9f-faef038ee029",
                          "feature_schema_id": "clhs1rh0b0bvj07y1f1brb5f3",
                          "name": "second_checklist_answer",
                          "value": "second_checklist_answer",
                          "classifications": []
                        },
                        {
                          "feature_id": "3653a591-4915-450f-88e4-f5afc8cd1440",
                          "feature_schema_id": "clhs1rh0b0bvh07y19qgt6nsv",
                          "name": "first_checklist_answer",
                          "value": "first_checklist_answer",
                          "classifications": []
                        }
                      ]
                    },
                    {
                      "feature_id": "da14ab32-c007-4e86-aaff-a64e51ff2503",
                      "feature_schema_id": "clhs1rh0b0bvm07y1gpw00bw3",
                      "name": "free_text",
                      "value": "free_text",
                      "text_answer": {
                        "content": "sample text"
                      }
                    },
                    {
                      "feature_id": "de0e9826-a0fc-4603-a59a-0c356895759c",
                      "feature_schema_id": "clhs1rh0b0bvo07y1025n0p4a",
                      "name": "nested_radio_question",
                      "value": "nested_radio_question",
                      "radio_answer": {
                        "feature_id": "913b5925-7b37-43a0-9086-c7734491d60a",
                        "feature_schema_id": "clhs1rh0b0bvp07y1d1tbgpeg",
                        "name": "first_radio_answer",
                        "value": "first_radio_answer",
                        "classifications": [
                          {
                            "feature_id": "7c10d9b2-0e07-4723-abef-ad022db24b10",
                            "feature_schema_id": "clhs1rh0b0bvq07y13wux72rp",
                            "name": "sub_radio_question",
                            "value": "sub_radio_question",
                            "radio_answer": {
                              "feature_id": "ae62a004-f6b3-4436-916e-ce375cf8210e",
                              "feature_schema_id": "clhs1rh0b0bvr07y15n246t5c",
                              "name": "first_sub_radio_answer",
                              "value": "first_sub_radio_answer",
                              "classifications": []
                            }
                          }
                        ]
                      }
                    },
                    {
                      "feature_id": "ee0a7c2c-cb78-4474-b12a-82406ee415c5",
                      "feature_schema_id": "clhs1rh0c0bvw07y1hiyweequ",
                      "name": "nested_checklist_question",
                      "value": "nested_checklist_question",
                      "checklist_answers": [
                        {
                          "feature_id": "22d25a8a-0831-4499-8980-aaebbac1b4eb",
                          "feature_schema_id": "clhs1rh0c0bvx07y1er8ge5qk",
                          "name": "first_checklist_answer",
                          "value": "first_checklist_answer",
                          "classifications": [
                            {
                              "feature_id": "825ce160-5e9f-4926-aa63-ed17ad7fb534",
                              "feature_schema_id": "clhs1rh0c0bvy07y1hmkv360v",
                              "name": "sub_checklist_question",
                              "value": "sub_checklist_question",
                              "checklist_answers": [
                                {
                                  "feature_id": "32cd713f-c272-4657-8d0e-910e24e90414",
                                  "feature_schema_id": "clhs1rh0c0bvz07y1d1njge4e",
                                  "name": "first_sub_checklist_answer",
                                  "value": "first_sub_checklist_answer",
                                  "classifications": []
                                }
                              ]
                            }
                          ]
                        }
                      ]
                    }
                  ],
                  "relationships": []
                }
              }
            ],
            "predictions": [
              {
                "label_kind": "Default",
                "version": "1.0.0",
                "id": "clhs1ru105b3w142bd71v29u1",
                "annotations": {
                  "objects": [
                    {
                      "feature_id": "1d1a302a-2bf1-4a64-b2db-1283400fc8c6",
                      "feature_schema_id": "clhs1rh0a0bv007y1fyxw0gyg",
                      "name": "bbox_with_radio_subclass",
                      "value": "bbox_with_radio_subclass",
                      "annotation_kind": "DocumentBoundingBox",
                      "classifications": [
                        {
                          "feature_id": "f13c6571-b3c8-4584-886b-555e1808a955",
                          "feature_schema_id": "clhs1rh0a0bv107y14g5de74m",
                          "name": "sub_radio_question",
                          "value": "sub_radio_question",
                          "radio_answer": {
                            "feature_id": "610215da-b253-46f8-8a6d-7dd56b36e32c",
                            "feature_schema_id": "clhs1rh0a0bv207y13bsndvuv",
                            "name": "first_sub_radio_answer",
                            "value": "first_sub_radio_answer",
                            "classifications": [
                              {
                                "feature_id": "a1002a15-c1f1-43ce-9606-9b02fb4d3d77",
                                "feature_schema_id": "clhs1rh0a0bv307y17teq5ob3",
                                "name": "second_sub_radio_question",
                                "value": "second_sub_radio_question",
                                "radio_answer": {
                                  "feature_id": "a225ff2c-dccc-4ad1-8dd5-d76f10c5aab2",
                                  "feature_schema_id": "clhs1rh0a0bv407y1d72h3bdp",
                                  "name": "second_sub_radio_answer",
                                  "value": "second_sub_radio_answer",
                                  "classifications": []
                                }
                              }
                            ]
                          }
                        }
                      ],
                      "page_number": 1,
                      "bounding_box": {
                        "top": 226.757,
                        "left": 317.271,
                        "height": 194.22899999999998,
                        "width": 249.38600000000002
                      }
                    },
                    {
                      "feature_id": "469420eb-12a7-401f-a5ea-142b24c622e3",
                      "feature_schema_id": "clhs1rh090buq07y15p2wd58d",
                      "name": "bounding_box",
                      "value": "bounding_box",
                      "annotation_kind": "DocumentBoundingBox",
                      "classifications": [],
                      "page_number": 0,
                      "bounding_box": {
                        "top": 135.3,
                        "left": 102.771,
                        "height": 109.84299999999999,
                        "width": 415.8
                      }
                    },
                    {
                      "feature_id": "d3c14b37-6b64-4c6d-91bc-446bf82dc0b5",
                      "name": "named_entity",
                      "value": "named_entity",
                      "annotation_kind": "DocumentEntityToken",
                      "classifications": [],
                      "location": {
                        "groups": [
                          {
                            "id": "2f4336f4-a07e-4e0a-a9e1-5629b03b719b",
                            "page_number": 1,
                            "tokens": [
                              "3f984bf3-1d61-44f5-b59a-9658a2e3440f",
                              "3bf00b56-ff12-4e52-8cc1-08dbddb3c3b8",
                              "6e1c3420-d4b7-4c5a-8fd6-ead43bf73d80",
                              "87a43d32-af76-4a1d-b262-5c5f4d5ace3a",
                              "e8606e8a-dfd9-4c49-a635-ad5c879c75d0",
                              "67c7c19e-4654-425d-bf17-2adb8cf02c30",
                              "149c5e80-3e07-49a7-ab2d-29ddfe6a38fa",
                              "b0e94071-2187-461e-8e76-96c58738a52c"
                            ],
                            "text": "Metal-insulator (MI) transitions have been one of the"
                          }
                        ]
                      }
                    },
                    {
                      "feature_id": "efe907c9-b84b-4015-b29b-3a1cae1655cc",
                      "name": "ner_with_checklist_subclass",
                      "value": "ner_with_checklist_subclass",
                      "annotation_kind": "DocumentEntityToken",
                      "classifications": [
                        {
                          "feature_id": "12fb5823-1a12-4de4-8613-8460cec5eaf1",
                          "feature_schema_id": "clhs1rh0a0buv07y1e0wobtgu",
                          "name": "sub_checklist_question",
                          "value": "sub_checklist_question",
                          "checklist_answers": [
                            {
                              "feature_id": "8be1ca80-1f98-48ad-ae65-d38ab941ef63",
                              "feature_schema_id": "clhs1rh0a0buw07y12tg3hm0x",
                              "name": "first_sub_checklist_answer",
                              "value": "first_sub_checklist_answer",
                              "classifications": []
                            }
                          ]
                        }
                      ],
                      "location": {
                        "groups": [
                          {
                            "id": "",
                            "page_number": 1,
                            "tokens": [],
                            "text": ""
                          }
                        ]
                      }
                    }
                  ],
                  "classifications": [
                    {
                      "feature_id": "35f2a9ed-e245-4654-94a5-9e7e48adff8d",
                      "feature_schema_id": "clhs1rh0b0bvo07y1025n0p4a",
                      "name": "nested_radio_question",
                      "value": "nested_radio_question",
                      "radio_answer": {
                        "feature_id": "70a52e6d-eda7-48f3-a1dd-65672c273318",
                        "feature_schema_id": "clhs1rh0b0bvp07y1d1tbgpeg",
                        "name": "first_radio_answer",
                        "value": "first_radio_answer",
                        "classifications": [
                          {
                            "feature_id": "5f87ca53-3637-4f26-9068-8645b6036646",
                            "feature_schema_id": "clhs1rh0b0bvq07y13wux72rp",
                            "name": "sub_radio_question",
                            "value": "sub_radio_question",
                            "radio_answer": {
                              "feature_id": "c652afc0-d24f-42ea-9a5a-dee14cc1d80d",
                              "feature_schema_id": "clhs1rh0b0bvr07y15n246t5c",
                              "name": "first_sub_radio_answer",
                              "value": "first_sub_radio_answer",
                              "classifications": []
                            }
                          }
                        ]
                      }
                    },
                    {
                      "feature_id": "73114ba5-3045-46e0-9323-468914afbdf0",
                      "feature_schema_id": "clhs1rh0b0bvg07y1as0cfj9r",
                      "name": "checklist_question",
                      "value": "checklist_question",
                      "checklist_answers": [
                        {
                          "feature_id": "666fc303-b30e-4647-a694-5c34dab4781f",
                          "feature_schema_id": "clhs1rh0b0bvj07y1f1brb5f3",
                          "name": "second_checklist_answer",
                          "value": "second_checklist_answer",
                          "classifications": []
                        },
                        {
                          "feature_id": "f8c4ce45-0e56-4ec2-bc5a-ce9364b471f8",
                          "feature_schema_id": "clhs1rh0b0bvh07y19qgt6nsv",
                          "name": "first_checklist_answer",
                          "value": "first_checklist_answer",
                          "classifications": []
                        }
                      ]
                    },
                    {
                      "feature_id": "911665a7-4901-4c31-a410-5c3a7a614c1d",
                      "feature_schema_id": "clhs1rh0c0bvw07y1hiyweequ",
                      "name": "nested_checklist_question",
                      "value": "nested_checklist_question",
                      "checklist_answers": [
                        {
                          "feature_id": "0341d434-c589-49d4-9a4a-b554647c2a44",
                          "feature_schema_id": "clhs1rh0c0bvx07y1er8ge5qk",
                          "name": "first_checklist_answer",
                          "value": "first_checklist_answer",
                          "classifications": [
                            {
                              "feature_id": "ee1c729c-d1f1-478e-98f8-baefa24dff28",
                              "feature_schema_id": "clhs1rh0c0bvy07y1hmkv360v",
                              "name": "sub_checklist_question",
                              "value": "sub_checklist_question",
                              "checklist_answers": [
                                {
                                  "feature_id": "a2c0d47f-811c-4d70-b024-8514d67b206c",
                                  "feature_schema_id": "clhs1rh0c0bvz07y1d1njge4e",
                                  "name": "first_sub_checklist_answer",
                                  "value": "first_sub_checklist_answer",
                                  "classifications": []
                                }
                              ]
                            }
                          ]
                        }
                      ]
                    },
                    {
                      "feature_id": "a7f83569-636d-4993-81f0-6d4de144bb24",
                      "feature_schema_id": "clhs1rh0a0bva07y11vb9gjzu",
                      "name": "radio_question",
                      "value": "radio_question",
                      "radio_answer": {
                        "feature_id": "58624513-6ab0-452b-a9a0-72cebf648cd0",
                        "feature_schema_id": "clhs1rh0a0bvb07y134ka29be",
                        "name": "first_radio_answer",
                        "value": "first_radio_answer",
                        "classifications": []
                      }
                    },
                    {
                      "feature_id": "c7e3f324-7b53-47a3-aeb0-4217717fcfed",
                      "feature_schema_id": "clhs1rh0b0bvm07y1gpw00bw3",
                      "name": "free_text",
                      "value": "free_text",
                      "text_answer": {
                        "content": "sample text"
                      }
                    }
                  ],
                  "relationships": []
                }
              }
            ]
          }
        }
      }
    },
    "models": {}
  }
  ```
</CodeGroup>


# Export geospatial annotations
Source: https://docs.labelbox.com/horizon/developer-guides/export-geospatial-annotations

How to export geospatial annotations and annotation export formats.

<CardGroup>
  <Card title="Open In Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/exports/export_data.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/exports/export_data.ipynb" />
</CardGroup>

## Export as JSON

<CodeGroup>
  ```python Export theme={"theme":"css-variables"}
  # Set the export params to include/exclude certain fields.
  export_params= {
    "attachments": True,
    "metadata_fields": True,
    "data_row_details": True,
    "project_details": True,
    "label_details": True,
    "performance_details": True,
    "interpolated_frames": True
  }

  # Note: Filters follow AND logic, so typically using one filter is sufficient.

  filters= {
    "last_activity_at": ["2000-01-01 00:00:00", "2050-01-01 00:00:00"],
    "workflow_status": "<wkf-status>"
  }

  export_task = project.export(params=export_params, filters=filters)
  export_task.wait_till_done()

  # Stream the export using a callback function

  def json_stream_handler(output: labelbox.BufferedJsonConverterOutput):
    print(output.json)

  export_task.get_buffered_stream(stream_type=labelbox.StreamType.RESULT).start(stream_handler=json_stream_handler)

  # Collect all exported data into a list

  export_json = [data_row.json for data_row in export_task.get_buffered_stream()]

  print("file size: ", export_task.get_total_file_size(stream_type=lb.StreamType.RESULT))
  print("line count: ", export_task.get_total_lines(stream_type=lb.StreamType.RESULT))

  ```
</CodeGroup>

## Annotation export formats

### Bounding box

The bounding box coordinates are returned in the following order:

* bottom left
* top left
* top right
* bottom right
* bottom left

All coordinates are provided in \[long, lat]

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "clutzcu3t0or4148po04q689s",
    "feature_schema_id": "clutzceve001k0701bpescb12",
    "name": "bbox_geo",
    "value": "bbox_geo",
    "annotation_kind": "GeoBoundingBox",
    "classifications": [],
    "geojson": {
      "type": "Polygon",
      "coordinates": [
        [
          [
            -99.207466,
            19.397994
          ],
          [
            -99.207466,
            19.399259
          ],
          [
            -99.205685,
            19.399259
          ],
          [
            -99.205685,
            19.397994
          ],
          [
            -99.207466,
            19.397994
          ]
        ]
      ]
    }
  }
  ```
</CodeGroup>

### Point

All coordinates are provided in \[long, lat]

<CodeGroup>
  ```json Slippy maps theme={"theme":"css-variables"}
  {
    "feature_id": "clutzcu3t0or1148px42yj5ld",
    "feature_schema_id": "clutzcevd001c07018wofd88r",
    "name": "point_geo",
    "value": "point_geo",
    "annotation_kind": "GeoPoint",
    "classifications": [],
    "geojson": {
      "type": "Point",
      "coordinates": [
        -99.206479,
        19.40018
      ]
    }
  }
  ```
</CodeGroup>

### Polyline

All coordinates are provided in \[long, lat]

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "clutzcu3t0or2148p3v3crfp0",
    "feature_schema_id": "clutzcevd001e070148gedtyj",
    "name": "polyline_geo",
    "value": "polyline_geo",
    "annotation_kind": "GeoPolyline",
    "classifications": [],
    "geojson": {
      "type": "LineString",
      "coordinates": [
        [
          -99.208421,
          19.400322
        ],
        [
          -99.208099,
          19.39759
        ],
        [
          -99.207584,
          19.397762
        ],
        [
          -99.207283,
          19.397327
        ]
      ]
    }
  }
  ```
</CodeGroup>

### Polygon

All coordinates are provided in \[long, lat].

The order of the coordinates is based on the sequence of points when drawn.

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "clutzcu3t0org148p7p5fvazc",
    "feature_schema_id": "clutzceve001i07014eazfofw",
    "name": "polygon_geo_2",
    "value": "polygon_geo_2",
    "annotation_kind": "GeoPolygon",
    "classifications": [],
    "geojson": {
      "type": "Polygon",
      "coordinates": [
        [
          [
            -99.206045,
            19.396804
          ],
          [
            -99.206045,
            19.396662
          ],
          [
            -99.20582,
            19.396662
          ],
          [
            -99.20582,
            19.396804
          ],
          [
            -99.206045,
            19.396804
          ]
        ]
      ]
    }
  }
  ```
</CodeGroup>

### Classification - Radio

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "clutzcu3t0or5148poyka73v0",
    "feature_schema_id": "clutzcevf00240701aqil93c1",
    "name": "radio_question_geo",
    "value": "radio_question_geo",
    "radio_answer": {
      "feature_id": "clutzcu3t0or6148pl8mwljss",
      "feature_schema_id": "clutzcevf00250701fd0g59rp",
      "name": "first_radio_answer",
      "value": "first_radio_answer",
      "classifications": []
    }
  }
  ```
</CodeGroup>

### Classification - Checklist

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "clutzcu3t0orc148pg6ybpnuz",
    "feature_schema_id": "clutzcevf001w070195dtaip4",
    "name": "checklist_question_geo",
    "value": "checklist_question_geo",
    "checklist_answers": [
      {
        "feature_id": "clutzcu3t0ord148ps02kawmq",
        "feature_schema_id": "clutzcevf001x0701aps7gmtc",
        "name": "first_checklist_answer",
        "value": "first_checklist_answer",
        "classifications": []
      },
      {
        "feature_id": "clutzcu3t0ore148pyfraw16x",
        "feature_schema_id": "clutzcevf001z0701gxqm8w12",
        "name": "second_checklist_answer",
        "value": "second_checklist_answer",
        "classifications": []
      },
      {
        "feature_id": "clutzcu3t0orf148p4curv80a",
        "feature_schema_id": "clutzcevf002107010x0g6um1",
        "name": "third_checklist_answer",
        "value": "third_checklist_answer",
        "classifications": []
      }
    ]
  }
  ```
</CodeGroup>

### Classification - Free-form text

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "cldne8lok01w413yd5mgbxd66",
    "name": "free_text",
    "text_answer": {
      "content": "sample text"
    }
  }
  ```
</CodeGroup>

## Sample project export

<CodeGroup>
  ```json JSON expandable theme={"theme":"css-variables"}
  {
      "data_row": {
          "id": "cldnf9zmt0d7n075ohgnvecul",
          "row_data": "{\"tileLayerUrl\":\"https://api.mapbox.com/styles/v1/mapbox/satellite-streets-v11/tiles/{z}/{x}/{y}?access_token=pk.eyJ1IjoibWFwYm94IiwiYSI6ImNpejY4NXVycTA2emYycXBndHRqcmZ3N3gifQ.rJcFIG214AriISLbB6B5aw\",\"bounds\":[[-122.31764674186705,37.87276155898985],[-122.31635199317932,37.87398109727749]],\"minZoom\":17,\"maxZoom\":23,\"maxNativeZoom\":null,\"epsg\":\"EPSG4326\",\"tileSize\":256,\"alternativeLayers\":[],\"version\":2}",
          "details": {
              "dataset_id": "cldnf9yvd041f070d33t94f41",
            	"dataset_name": "geotiff_noaa.json",
              "created_at": "2023-02-02",
              "updated_at": "2023-02-02",
              "created_by": "[email protected]"
          }
      },
      "media_attributes": {
          "mime_type": "application/x-tms-geo"
      },
      "attachments": [],
      "metadata_fields": [],
      "projects": {
          "cldnfa25103kd081ph3edg7xp": {
              "project_name": "geospatial_project_demo",
              "labels": [
                  {
                      "label_kind": "Default",
                      "version": "1.0.0",
                      "id": "cldnfc62h029b13ydq6bgvyak",
                      "label_details": {
                          "created_at": "2023-02-02T18:21:40+00:00",
                          "updated_at": "2023-02-02T18:21:40+00:00",
                          "created_by": "[email protected]",
                          "reviews": []
                      },
                      "performance_details": {
                          "seconds_to_create": 0,
                          "seconds_to_review": 0,
                          "skipped": false
                      },
                      "annotations": {
                          "objects": [
                              {
                                  "feature_id": "cldnfc61v028v13ydwgha3a1d",
                                  "name": "point_geo",
                                  "annotation_kind": "GeoPoint",
                                  "classifications": [],
                                  "geojson": {
                                      "type": "Point",
                                      "coordinates": [
                                          -122.31741,
                                          37.873557
                                      ]
                                  }
                              },
                              {
                                  "feature_id": "cldnfc61v028w13ydnbm7vl2q",
                                  "name": "polyline_geo",
                                  "annotation_kind": "GeoPolyline",
                                  "classifications": [],
                                  "geojson": {
                                      "type": "LineString",
                                      "coordinates": [
                                          [
                                              -122.317578,
                                              37.873963
                                          ],
                                          [
                                              -122.316398,
                                              37.873967
                                          ],
                                          [
                                              -122.31639,
                                              37.872779
                                          ]
                                      ]
                                  }
                              },
                              {
                                  "feature_id": "cldnfc61v028x13yd5swk0drd",
                                  "name": "polygon_geo",
                                  "annotation_kind": "GeoPolygon",
                                  "classifications": [],
                                  "geojson": {
                                      "type": "Polygon",
                                      "coordinates": [
                                          [
                                              [
                                                  -122.316918,
                                                  37.87329
                                              ],
                                              [
                                                  -122.317102,
                                                  37.873043
                                              ],
                                              [
                                                  -122.316801,
                                                  37.873036
                                              ],
                                              [
                                                  -122.316918,
                                                  37.87329
                                              ]
                                          ]
                                      ]
                                  }
                              },
                              {
                                  "feature_id": "cldnfc61v028y13yd41qozic7",
                                  "name": "bbox_geo",
                                  "annotation_kind": "GeoBoundingBox",
                                  "classifications": [],
                                  "geojson": {
                                      "type": "Polygon",
                                      "coordinates": [
                                          [
                                              [
                                                  -122.317345,
                                                  37.873713
                                              ],
                                              [
                                                  -122.317345,
                                                  37.873859
                                              ],
                                              [
                                                  -122.31673,
                                                  37.873859
                                              ],
                                              [
                                                  -122.31673,
                                                  37.873713
                                              ],
                                              [
                                                  -122.317345,
                                                  37.873713
                                              ]
                                          ]
                                      ]
                                  }
                              },
                              {
                                  "feature_id": "cldnfc61v029113ydvc788rx9",
                                  "name": "bbox_checklist_geo",
                                  "annotation_kind": "GeoBoundingBox",
                                  "classifications": [
                                      {
                                          "feature_id": "cldnfc61v029213yd344u0cyk",
                                          "name": "checklist_class_name",
                                          "checklist_answers": [
                                              {
                                                  "feature_id": "cldnfc61v029313yda92qm39k",
                                                  "name": "first_checklist_answer",
                                                  "classifications": []
                                              }
                                          ]
                                      }
                                  ],
                                  "geojson": {
                                      "type": "Polygon",
                                      "coordinates": [
                                          [
                                              [
                                                  -122.317113,
                                                  37.873402
                                              ],
                                              [
                                                  -122.317113,
                                                  37.873608
                                              ],
                                              [
                                                  -122.316655,
                                                  37.873608
                                              ],
                                              [
                                                  -122.316655,
                                                  37.873402
                                              ],
                                              [
                                                  -122.317113,
                                                  37.873402
                                              ]
                                          ]
                                      ]
                                  }
                              },
                              {
                                  "feature_id": "cldnfc61v029413yd5lgevg2c",
                                  "name": "bbox_text_geo",
                                  "annotation_kind": "GeoBoundingBox",
                                  "classifications": [
                                      {
                                          "feature_id": "cldnfc61v029513ydmluqvqyt",
                                          "name": "free_text_geo",
                                          "text_answer": {
                                              "content": "sample text"
                                          }
                                      }
                                  ],
                                  "geojson": {
                                      "type": "Polygon",
                                      "coordinates": [
                                          [
                                              [
                                                  -122.317508,
                                                  37.873182
                                              ],
                                              [
                                                  -122.317508,
                                                  37.87338
                                              ],
                                              [
                                                  -122.3171,
                                                  37.87338
                                              ],
                                              [
                                                  -122.3171,
                                                  37.873182
                                              ],
                                              [
                                                  -122.317508,
                                                  37.873182
                                              ]
                                          ]
                                      ]
                                  }
                              },
                              {
                                  "feature_id": "cldnfc61v029a13ydmog1dhzw",
                                  "name": "polygon_geo_2",
                                  "annotation_kind": "GeoPolygon",
                                  "classifications": [],
                                  "geojson": {
                                      "type": "Polygon",
                                      "coordinates": [
                                          [
                                              [
                                                  -122.316362,
                                                  37.872966
                                              ],
                                              [
                                                  -122.316362,
                                                  37.872948
                                              ],
                                              [
                                                  -122.316356,
                                                  37.872948
                                              ],
                                              [
                                                  -122.316356,
                                                  37.872966
                                              ],
                                              [
                                                  -122.316362,
                                                  37.872966
                                              ]
                                          ]
                                      ]
                                  }
                              }
                          ],
                          "classifications": [
                              {
                                  "feature_id": "cldnfc61v028z13ydy0xcffbj",
                                  "name": "radio_question_geo",
                                  "radio_answer": {
                                      "feature_id": "cldnfc61v029013ydj55ek20y",
                                      "name": "first_radio_answer",
                                      "classifications": []
                                  }
                              },
                              {
                                  "feature_id": "cldnfc61v029613ydu76vcgbt",
                                  "name": "checklist_question_geo",
                                  "checklist_answers": [
                                      {
                                          "feature_id": "cldnfc61v029713yd5x16lyg3",
                                          "name": "first_checklist_answer",
                                          "classifications": []
                                      },
                                      {
                                          "feature_id": "cldnfc61v029813ydcsx4mc2t",
                                          "name": "second_checklist_answer",
                                          "classifications": []
                                      },
                                      {
                                          "feature_id": "cldnfc61v029913yd2smfzpft",
                                          "name": "third_checklist_answer",
                                          "classifications": []
                                      }
                                  ]
                              }
                          ],
                          "relationships": []
                      }
                  }
              ],
              "project_details": {
                  "ontology_id": "cl8emhd3e7bll070k6gbicl97",
                  "task_id": "fdc4b3d2-e0fe-41dc-bdef-0dd556870a20",
                  "task_name": "Initial review task",
                  "batch_id": "cc4e7070-f20e-11ed-9dd9-639c7e9f70ba",
                  "batch_name": "geotiff_noaa.json-cl8emco177d5g07y8e06j4acp",
                  "workflow_status": "IN_REVIEW",
                  "priority": 5,
                  "consensus_expected_label_count": 3,
                  "workflow_history": [
                      {
                          "action": "Move",
                          "created_at": "2023-05-14T04:21:29.062+00:00",
                          "created_by": "[email protected]",
                          "previous_task_name": "Initial labeling task",
                          "previous_task_id": "0dde450a-93ba-0b77-b97e-de8f7f479cee",
                          "next_task_name": "Initial review task",
                          "next_task_id": "fdc4b3d2-e0fe-41dc-bdef-0dd556870a20"
                      },
                      {
                          "action": "Move",
                          "created_at": "2023-05-14T04:21:29.044+00:00",
                          "created_by": "[email protected]",
                          "next_task_name": "Initial labeling task",
                          "next_task_id": "0dde450a-93ba-0b77-b97e-de8f7f479cee"
                      }
                  ]
              }
          }
      }
  }
  ```
</CodeGroup>

## Sample model run export

<CodeGroup>
  ```json JSON expandable theme={"theme":"css-variables"}
  {
    "data_row": {
      "id": "clfh28su81agn07623rql6j44",
      "global_key": "slippy_maps",
      "row_data": "{\"tileLayerUrl\":\"https://api.mapbox.com/styles/v1/mapbox/satellite-streets-v11/tiles/{z}/{x}/{y}?access_token=pk.eyJ1IjoibWFwYm94IiwiYSI6ImNpejY4NXVycTA2emYycXBndHRqcmZ3N3gifQ.rJcFIG214AriISLbB6B5aw\",\"bounds\":[[-122.31764674186705,37.87276155898985],[-122.31635199317932,37.87398109727749]],\"minZoom\":17,\"maxZoom\":23,\"maxNativeZoom\":null,\"epsg\":\"EPSG4326\",\"tileSize\":256,\"alternativeLayers\":[],\"version\":2}",
      "details": {
        "dataset_id": "clfh28rib089g07134eqx03ew",
        "dataset_name": "geo_demo_dataset",
        "created_at": "2023-03-20T16:47:55.000+00:00",
        "updated_at": "2023-03-20T16:47:55.000+00:00",
        "created_by": "[email protected]"
      }
    },
    "media_attributes": {
      "mime_type": "application/x-tms-geo"
    },
    "attachments": [],
    "metadata_fields": [],
    "experiments": {
      "a15bda50-b901-02e0-07d6-e842d13b3f47": {
        "name": "geospatial_model_run_1f54f23a-0359-4f18-b3ec-808a6181a4fd",
        "runs": {
          "a15bda50-fd4e-06ff-8f9b-762516cf58c8": {
            "name": "iteration 1",
            "run_data_row_id": "247e2c89-0e82-4662-a757-5f7cfa07a8fd",
            "labels": [
              {
                "label_kind": "Default",
                "version": "1.0.0",
                "id": "clhs22mh25b4d142bagekzwlv",
                "annotations": {
                  "objects": [
                    {
                      "feature_id": "12527b8c-e5c5-4cd5-9266-c6a1d90ef14c",
                      "feature_schema_id": "clhs21cpy0c6t07y1fof39vn7",
                      "name": "point_geo",
                      "value": "point_geo",
                      "annotation_kind": "GeoPoint",
                      "classifications": [],
                      "geojson": {
                        "type": "Point",
                        "coordinates": [
                          -122.31741,
                          37.873557
                        ]
                      }
                    },
                    {
                      "feature_id": "185de2b0-fca0-48fc-9f3c-982e1487ce13",
                      "feature_schema_id": "clhs21cpy0c7307y1dg174k0l",
                      "name": "bbox_checklist_geo",
                      "value": "bbox_checklist_geo",
                      "annotation_kind": "GeoBoundingBox",
                      "classifications": [
                        {
                          "feature_id": "d2809ef9-01ae-4ad7-a4d0-1fee6938e186",
                          "feature_schema_id": "clhs21cpy0c7407y1gl9g9ver",
                          "name": "checklist_class_name",
                          "value": "checklist_class_name",
                          "checklist_answers": [
                            {
                              "feature_id": "6cbc5158-ac53-445a-a6c1-8035b7a4cac1",
                              "feature_schema_id": "clhs21cpy0c7507y16idc9qjy",
                              "name": "first_checklist_answer",
                              "value": "first_checklist_answer",
                              "classifications": []
                            }
                          ]
                        }
                      ],
                      "geojson": {
                        "type": "Polygon",
                        "coordinates": [
                          [
                            [
                              -122.317113,
                              37.873608
                            ],
                            [
                              -122.317113,
                              37.873402
                            ],
                            [
                              -122.316655,
                              37.873402
                            ],
                            [
                              -122.316655,
                              37.873608
                            ],
                            [
                              -122.317113,
                              37.873608
                            ]
                          ]
                        ]
                      }
                    },
                    {
                      "feature_id": "78024089-436d-4a0e-aea9-f458ee66cb42",
                      "feature_schema_id": "clhs21cpy0c6v07y19w4zb3b7",
                      "name": "polyline_geo",
                      "value": "polyline_geo",
                      "annotation_kind": "GeoPolyline",
                      "classifications": [],
                      "geojson": {
                        "type": "LineString",
                        "coordinates": [
                          [
                            -122.317578,
                            37.873963
                          ],
                          [
                            -122.316398,
                            37.873967
                          ],
                          [
                            -122.31639,
                            37.872779
                          ]
                        ]
                      }
                    },
                    {
                      "feature_id": "97110d09-559b-46c9-abb4-7e5d5f578b3e",
                      "feature_schema_id": "clhs21cpy0c6x07y18t2hgvdp",
                      "name": "polygon_geo",
                      "value": "polygon_geo",
                      "annotation_kind": "GeoPolygon",
                      "classifications": [],
                      "geojson": {
                        "type": "Polygon",
                        "coordinates": [
                          [
                            [
                              -122.316918,
                              37.87329
                            ],
                            [
                              -122.317102,
                              37.873043
                            ],
                            [
                              -122.316801,
                              37.873036
                            ],
                            [
                              -122.316918,
                              37.87329
                            ]
                          ]
                        ]
                      }
                    },
                    {
                      "feature_id": "cbbd8796-d6ff-4238-9347-bb12b239ac74",
                      "feature_schema_id": "clhs21cpy0c7907y19etw75vg",
                      "name": "bbox_text_geo",
                      "value": "bbox_text_geo",
                      "annotation_kind": "GeoBoundingBox",
                      "classifications": [
                        {
                          "feature_id": "0492b8b4-217b-4fdb-bc93-1b0a30f5aa8b",
                          "feature_schema_id": "clhs21cpz0c7a07y16uczf4dx",
                          "name": "free_text_geo",
                          "value": "free_text_geo",
                          "text_answer": {
                            "content": "sample text"
                          }
                        }
                      ],
                      "geojson": {
                        "type": "Polygon",
                        "coordinates": [
                          [
                            [
                              -122.317508,
                              37.87338
                            ],
                            [
                              -122.317508,
                              37.873182
                            ],
                            [
                              -122.3171,
                              37.873182
                            ],
                            [
                              -122.3171,
                              37.87338
                            ],
                            [
                              -122.317508,
                              37.87338
                            ]
                          ]
                        ]
                      }
                    },
                    {
                      "feature_id": "d80e2563-002a-48b8-b6c6-50f7df1a4a8b",
                      "feature_schema_id": "clhs21cpy0c7107y1b07n6ef9",
                      "name": "bbox_geo",
                      "value": "bbox_geo",
                      "annotation_kind": "GeoBoundingBox",
                      "classifications": [],
                      "geojson": {
                        "type": "Polygon",
                        "coordinates": [
                          [
                            [
                              -122.317345,
                              37.873859
                            ],
                            [
                              -122.317345,
                              37.873713
                            ],
                            [
                              -122.31673,
                              37.873713
                            ],
                            [
                              -122.31673,
                              37.873859
                            ],
                            [
                              -122.317345,
                              37.873859
                            ]
                          ]
                        ]
                      }
                    },
                    {
                      "feature_id": "f6abee6b-8130-4033-a223-be428dd59c0a",
                      "feature_schema_id": "clhs21cpy0c6x07y18t2hgvdp",
                      "name": "polygon_geo",
                      "value": "polygon_geo",
                      "annotation_kind": "GeoPolygon",
                      "classifications": [],
                      "geojson": {
                        "type": "Polygon",
                        "coordinates": [
                          [
                            [
                              -122.31703,
                              37.873978
                            ],
                            [
                              -122.317024,
                              37.873935
                            ],
                            [
                              -122.316989,
                              37.873899
                            ],
                            [
                              -122.316979,
                              37.873859
                            ],
                            [
                              -122.316958,
                              37.873856
                            ],
                            [
                              -122.316955,
                              37.873816
                            ],
                            [
                              -122.31692,
                              37.873779
                            ],
                            [
                              -122.31691,
                              37.873734
                            ],
                            [
                              -122.316886,
                              37.873697
                            ],
                            [
                              -122.316855,
                              37.873685
                            ],
                            [
                              -122.31682,
                              37.873611
                            ],
                            [
                              -122.316765,
                              37.87355
                            ],
                            [
                              -122.316758,
                              37.873517
                            ],
                            [
                              -122.31671,
                              37.873489
                            ],
                            [
                              -122.31671,
                              37.873452
                            ],
                            [
                              -122.316679,
                              37.87344
                            ],
                            [
                              -122.316634,
                              37.873355
                            ],
                            [
                              -122.316603,
                              37.873348
                            ],
                            [
                              -122.3166,
                              37.873321
                            ],
                            [
                              -122.316545,
                              37.87329
                            ],
                            [
                              -122.316528,
                              37.873199
                            ],
                            [
                              -122.31649,
                              37.873183
                            ],
                            [
                              -122.31649,
                              37.873162
                            ],
                            [
                              -122.316417,
                              37.873092
                            ],
                            [
                              -122.316411,
                              37.873055
                            ],
                            [
                              -122.316397,
                              37.87304
                            ],
                            [
                              -122.316356,
                              37.87304
                            ],
                            [
                              -122.316356,
                              37.873981
                            ],
                            [
                              -122.31703,
                              37.873978
                            ]
                          ]
                        ]
                      }
                    }
                  ],
                  "classifications": [
                    {
                      "feature_id": "08c1417e-ac8f-4272-9f4f-8b523ae5e0d7",
                      "feature_schema_id": "clhs21cpz0c7l07y160gh5xdw",
                      "name": "radio_question_geo",
                      "value": "radio_question_geo",
                      "radio_answer": {
                        "feature_id": "8c72625a-e663-4241-a2c2-8082804a59e3",
                        "feature_schema_id": "clhs21cpz0c7m07y188gjbqbj",
                        "name": "first_radio_answer",
                        "value": "first_radio_answer",
                        "classifications": []
                      }
                    },
                    {
                      "feature_id": "a22dac3b-5c40-4f08-9723-d70e7bc4bff4",
                      "feature_schema_id": "clhs21cpz0c7d07y14vnl7zpi",
                      "name": "checklist_question_geo",
                      "value": "checklist_question_geo",
                      "checklist_answers": [
                        {
                          "feature_id": "1c4a1950-6961-42bf-b134-50596f136164",
                          "feature_schema_id": "clhs21cpz0c7e07y16k192u14",
                          "name": "first_checklist_answer",
                          "value": "first_checklist_answer",
                          "classifications": []
                        },
                        {
                          "feature_id": "963cc214-b134-4b0e-80d5-8e63bcc000ae",
                          "feature_schema_id": "clhs21cpz0c7g07y17yo0e2cn",
                          "name": "second_checklist_answer",
                          "value": "second_checklist_answer",
                          "classifications": []
                        },
                        {
                          "feature_id": "bfa3e08b-5f2d-468e-aa7c-a6c2a29f9f4f",
                          "feature_schema_id": "clhs21cpz0c7i07y1hytkftpk",
                          "name": "third_checklist_answer",
                          "value": "third_checklist_answer",
                          "classifications": []
                        }
                      ]
                    }
                  ],
                  "relationships": []
                }
              }
            ],
            "predictions": [
              {
                "label_kind": "Default",
                "version": "1.0.0",
                "id": "clhs22mh25b4d142bagekzwlv",
                "annotations": {
                  "objects": [
                    {
                      "feature_id": "21d3b49c-0e17-4eb1-9824-2b05a23386d5",
                      "feature_schema_id": "clhs21cpy0c6v07y19w4zb3b7",
                      "name": "polyline_geo",
                      "value": "polyline_geo",
                      "annotation_kind": "GeoPolyline",
                      "classifications": [],
                      "geojson": {
                        "type": "LineString",
                        "coordinates": [
                          [
                            -122.317578,
                            37.873963
                          ],
                          [
                            -122.316398,
                            37.873967
                          ],
                          [
                            -122.31639,
                            37.872779
                          ]
                        ]
                      }
                    },
                    {
                      "feature_id": "4bcd8a06-cdc9-4450-b59e-d6601d185b88",
                      "feature_schema_id": "clhs21cpy0c6t07y1fof39vn7",
                      "name": "point_geo",
                      "value": "point_geo",
                      "annotation_kind": "GeoPoint",
                      "classifications": [],
                      "geojson": {
                        "type": "Point",
                        "coordinates": [
                          -122.31741,
                          37.873557
                        ]
                      }
                    },
                    {
                      "feature_id": "69d6cf96-1198-42e0-83a7-c75f0c945037",
                      "feature_schema_id": "clhs21cpy0c7107y1b07n6ef9",
                      "name": "bbox_geo",
                      "value": "bbox_geo",
                      "annotation_kind": "GeoBoundingBox",
                      "classifications": [],
                      "geojson": {
                        "type": "Polygon",
                        "coordinates": [
                          [
                            [
                              -122.317345,
                              37.873859
                            ],
                            [
                              -122.317345,
                              37.873713
                            ],
                            [
                              -122.31673,
                              37.873713
                            ],
                            [
                              -122.31673,
                              37.873859
                            ],
                            [
                              -122.317345,
                              37.873859
                            ]
                          ]
                        ]
                      }
                    },
                    {
                      "feature_id": "9090fb5d-09f3-4a5f-9114-9b3c4ce6c909",
                      "feature_schema_id": "clhs21cpy0c6x07y18t2hgvdp",
                      "name": "polygon_geo",
                      "value": "polygon_geo",
                      "annotation_kind": "GeoPolygon",
                      "classifications": [],
                      "geojson": {
                        "type": "Polygon",
                        "coordinates": [
                          [
                            [
                              -122.316918,
                              37.87329
                            ],
                            [
                              -122.317102,
                              37.873043
                            ],
                            [
                              -122.316801,
                              37.873036
                            ],
                            [
                              -122.316918,
                              37.87329
                            ]
                          ]
                        ]
                      }
                    },
                    {
                      "feature_id": "9ef3b0dc-b085-4f64-9018-264db30f9018",
                      "feature_schema_id": "clhs21cpy0c7307y1dg174k0l",
                      "name": "bbox_checklist_geo",
                      "value": "bbox_checklist_geo",
                      "annotation_kind": "GeoBoundingBox",
                      "classifications": [
                        {
                          "feature_id": "a77cd878-d15d-4628-ab08-9ecd192bbebc",
                          "feature_schema_id": "clhs21cpy0c7407y1gl9g9ver",
                          "name": "checklist_class_name",
                          "value": "checklist_class_name",
                          "checklist_answers": [
                            {
                              "feature_id": "5a6a23f8-39c7-409c-953f-732d3cf4cbec",
                              "feature_schema_id": "clhs21cpy0c7507y16idc9qjy",
                              "name": "first_checklist_answer",
                              "value": "first_checklist_answer",
                              "classifications": []
                            }
                          ]
                        }
                      ],
                      "geojson": {
                        "type": "Polygon",
                        "coordinates": [
                          [
                            [
                              -122.317113,
                              37.873608
                            ],
                            [
                              -122.317113,
                              37.873402
                            ],
                            [
                              -122.316655,
                              37.873402
                            ],
                            [
                              -122.316655,
                              37.873608
                            ],
                            [
                              -122.317113,
                              37.873608
                            ]
                          ]
                        ]
                      }
                    },
                    {
                      "feature_id": "f7df220a-27c0-4fc1-b67f-094b28efec3d",
                      "feature_schema_id": "clhs21cpy0c6z07y1a99ubplb",
                      "name": "polygon_geo_2",
                      "value": "polygon_geo_2",
                      "annotation_kind": "GeoPolygon",
                      "classifications": [],
                      "geojson": {
                        "type": "Polygon",
                        "coordinates": [
                          [
                            [
                              -122.31703,
                              37.873978
                            ],
                            [
                              -122.317024,
                              37.873935
                            ],
                            [
                              -122.316989,
                              37.873899
                            ],
                            [
                              -122.316979,
                              37.873859
                            ],
                            [
                              -122.316958,
                              37.873856
                            ],
                            [
                              -122.316955,
                              37.873816
                            ],
                            [
                              -122.31692,
                              37.873779
                            ],
                            [
                              -122.31691,
                              37.873734
                            ],
                            [
                              -122.316886,
                              37.873697
                            ],
                            [
                              -122.316855,
                              37.873685
                            ],
                            [
                              -122.31682,
                              37.873611
                            ],
                            [
                              -122.316765,
                              37.87355
                            ],
                            [
                              -122.316758,
                              37.873517
                            ],
                            [
                              -122.31671,
                              37.873489
                            ],
                            [
                              -122.31671,
                              37.873452
                            ],
                            [
                              -122.316679,
                              37.87344
                            ],
                            [
                              -122.316634,
                              37.873355
                            ],
                            [
                              -122.316603,
                              37.873348
                            ],
                            [
                              -122.3166,
                              37.873321
                            ],
                            [
                              -122.316545,
                              37.87329
                            ],
                            [
                              -122.316528,
                              37.873199
                            ],
                            [
                              -122.31649,
                              37.873183
                            ],
                            [
                              -122.31649,
                              37.873162
                            ],
                            [
                              -122.316417,
                              37.873092
                            ],
                            [
                              -122.316411,
                              37.873055
                            ],
                            [
                              -122.316397,
                              37.87304
                            ],
                            [
                              -122.316356,
                              37.87304
                            ],
                            [
                              -122.316356,
                              37.873981
                            ],
                            [
                              -122.31703,
                              37.873978
                            ]
                          ]
                        ]
                      }
                    },
                    {
                      "feature_id": "fbc72808-7f01-4cae-b53d-e868ac2e6b30",
                      "feature_schema_id": "clhs21cpy0c7907y19etw75vg",
                      "name": "bbox_text_geo",
                      "value": "bbox_text_geo",
                      "annotation_kind": "GeoBoundingBox",
                      "classifications": [
                        {
                          "feature_id": "cd12abb2-2c30-48be-b4ef-a8e3b649e998",
                          "feature_schema_id": "clhs21cpz0c7a07y16uczf4dx",
                          "name": "free_text_geo",
                          "value": "free_text_geo",
                          "text_answer": {
                            "content": "sample text"
                          }
                        }
                      ],
                      "geojson": {
                        "type": "Polygon",
                        "coordinates": [
                          [
                            [
                              -122.317508,
                              37.87338
                            ],
                            [
                              -122.317508,
                              37.873182
                            ],
                            [
                              -122.3171,
                              37.873182
                            ],
                            [
                              -122.3171,
                              37.87338
                            ],
                            [
                              -122.317508,
                              37.87338
                            ]
                          ]
                        ]
                      }
                    }
                  ],
                  "classifications": [
                    {
                      "feature_id": "413f0623-6bb6-4631-9c2c-dd95cd01ff0b",
                      "feature_schema_id": "clhs21cpz0c7d07y14vnl7zpi",
                      "name": "checklist_question_geo",
                      "value": "checklist_question_geo",
                      "checklist_answers": [
                        {
                          "feature_id": "27fe9828-16fc-446d-8ff9-563d1bda1df9",
                          "feature_schema_id": "clhs21cpz0c7e07y16k192u14",
                          "name": "first_checklist_answer",
                          "value": "first_checklist_answer",
                          "classifications": []
                        },
                        {
                          "feature_id": "8b97530e-96ce-4ce4-8232-e27c8b9bde4f",
                          "feature_schema_id": "clhs21cpz0c7g07y17yo0e2cn",
                          "name": "second_checklist_answer",
                          "value": "second_checklist_answer",
                          "classifications": []
                        },
                        {
                          "feature_id": "c8c5c1d3-c805-4cde-9522-254d390ca880",
                          "feature_schema_id": "clhs21cpz0c7i07y1hytkftpk",
                          "name": "third_checklist_answer",
                          "value": "third_checklist_answer",
                          "classifications": []
                        }
                      ]
                    },
                    {
                      "feature_id": "dd36f2ec-91be-4bf6-9f6e-7531598628c0",
                      "feature_schema_id": "clhs21cpz0c7l07y160gh5xdw",
                      "name": "radio_question_geo",
                      "value": "radio_question_geo",
                      "radio_answer": {
                        "feature_id": "06a746a8-78c0-4bb5-9f71-65e8303155b5",
                        "feature_schema_id": "clhs21cpz0c7m07y188gjbqbj",
                        "name": "first_radio_answer",
                        "value": "first_radio_answer",
                        "classifications": []
                      }
                    }
                  ],
                  "relationships": []
                }
              }
            ]
          }
        }
      }
    },
    "models": {}
  }
  ```
</CodeGroup>


# Export glossary
Source: https://docs.labelbox.com/horizon/developer-guides/export-glossary

Complete list of definitions for the fields included in exports.

Labelbox exports give you more flexibility and control to retrieve the most valuable information from your projects. You can now select and export a subselection of the data rows of most interest based on your predefined or new parameters in the Data Rows tab. You can also export more detailed information from these data rows, and include or exclude relevant attributes in your export. We also simplified and standardized the annotation formats.

For more details on how to create exports, along with complete samples, please view the following pages:

<CardGroup>
  <Card title="Export labels from project" icon="angle-right" href="/horizon/guides/export-labels" />

  <Card title="Export model run data" icon="angle-right" href="/horizon/guides/export-data-for-model-training" />
</CardGroup>

Below is a glossary of the fields that can appear in an export.

## Export fields

Labelbox exports aligns with the data-row-centric paradigm, thus every line, regardless of the export time, will include information on each included data row.

| Field | Included | Description |
| - | - | - |
| `data_row` | Always | A dictionary containing the fields explained the `data_row` table below. |
| `media_attributes` | Optional | See [Media attributes](/horizon/developer-guides/media-attributes) |
| `attachments` | Optional | See [Attachments](/horizon/developer-guides/attachments) |
| `metadata_fields` | Optional | See [Metadata](/horizon/developer-guides/metadata) |
| `embeddings` | Optional | A list of dictionaries containing precomputed and custom embeddings |

# `data_row`

| Field | Included | Description |
| - | - | - |
| `id` | Always | The ID of the data row. |
| `global_key` | If utilized | The global key of the data row. |
| `row_data` | Always | The URL to your cloud-hosted file |
| `details` | Optional | A dictionary containing the fields explained in the `details` table below. When exporting through the SDK, these fields are included by setting the `data_row_details` parameter to `True`. |

## `details`

| Field | Included | Description |
| - | - | - |
| `dataset_id` | Always | The ID of the dataset to which the data row belongs. |
| `dataset_name` | Always | The name given to the dataset. |
| `created_at` | Always | A timestamp that indicates when the data row was created. |
| `updated_at` | Always | A timestamp indicating when the data row was most recently updated. |
| `last_activity_at` | Always\* | A timestamp indicating the last time *any* activity was performed on the data row. |
| `created_by` | Always | The email address of the user that created the data row. |

\* The `last_activity_at` field is *not* included in exports from a model run.

# `projects`

The `projects` field contains a dictionary in which the keys are project IDs and the values consist of the fields explained below.

In a project-based export, there will only be a singular project ID in this dictionary. Yet, when exporting from the Catalog, consider that a data row may have been labeled in multiple projects, and thus the dictionary will have multiple keys.

Project IDs are used in favor of project names in order to enforce uniqueness.

| Field | Included | Description |
| - | - | - |
| `name` | Always | The name of the project in which the data row was labeled. |
| `labels` | Always | Contains a list of dictionaries comprised of the fields explained in the `labels` section below. |
| `project_details` | Optional | A dictionary containing the fields explained in the `project_details` table below. |

## `labels`

| Field | Included | Description |
| - | - | - |
| `label_kind` | Always | For labels made on assets of most media types, the value is `Default`. For frame-based assets, the value is `Video`. |
| `version` | Always | Used to track updates made to export formats. At present, the value will always be `1.0.0`. |
| `id` | Always | The ID of the label. |
| `label_details` | Optional | A dictionary containing the fields explained in the `label_details` table below. |
| `performance_details` | Optional | A dictionary containing the fields explained in the `performance_details` table below. |
| `annotations` | Always | See the annotation export formats broken down by asset type, beginning [here](/horizon/developer-guides/export-image-annotations#annotation-export-formats) with images. |

### `label_details`

| Field | Included | Description |
| - | - | - |
| `created_at` | Always | A timestamp indicating when the label was created. |
| `updated_at` | Always | A timestamp indicating when the label was most recently updated. |
| `created_by` | Always | The email address of the user that created the label. |
| `reviews` | Always | \[Legacy] For projects using [Workflows](/horizon/guides/workflows) for review, please use `workflow_history` [here](/horizon/developer-guides/export-glossary#workflow_history). Information on the thumbs up/down reviews created on this label. Contains a list of dictionaries comprised of the fields explained in the `reviews` table below. |

#### `reviews`(for legacy only)

| Field | Included | Description |
| - | - | - |
| `reviewed_at` | Situational | A timestamp that indicates when the review was created. |
| `reviewed_by` | Situational | The email address of the user that created the review. |
| `review_action` | Situational | The type of review created; either `Approve` or `Reject`. |

### `performance_details`

| Field | Included | Description |
| - | - | - |
| `seconds_to_create` | Always | The total time in seconds spent creating the label, including initial labeling, review, and rework. |
| `seconds_to_review` | Always | The number of seconds spent reviewing the label. |
| `skipped` | Always | A boolean value expressing whether or not the asset was skipped. A value of `true` indicates the asset was skipped. |

<Info>
  ### Delay on value update

  Each `performance_details`field may take up to 30 minutes to update after you labeling, reviewing, reworking, or submitting the label.
</Info>

#### `performance_details_v2`

The breakdown of time spent on each workflow task (initial labeling, review, and rework).

| Field | Included | Description |
| - | - | - |
| `seconds_to_label` | Always | The total number of seconds spent on the initial labeling task. |
| `seconds_to_review` | Always | The number of seconds spent reviewing the label. |
| `seconds_to_rework` | Always | The number of seconds spent reworking the label. |
| `seconds_total` | Always | The total time in seconds spent creating the label, including initial labeling, review, and rework. |
| `skipped` | Always | A boolean value indicating whether the asset was skipped. A value of `true` indicates the data row was skipped. |

### `annotations`

| Field | Included | Description |
| - | - | - |
| `objects` | Always | Tool annotations (e.g., bounding box, masks, polygon, etc ..) |
| `classifications` | Always | Classification annotations (radio, checklist) |
| `frames` | Videos only | Dictionary with per-frame annotations (objects and classifications) |
| `segments` | Videos only | A dictionary where each key is a `feature_id`, and the corresponding value is a list of frame numbers representing the range of frames where that feature exists. |
| `key_frame_feature_map` | Videos only | A dictionary where each key is a `feature_id`, and the corresponding value is a list of frame numbers representing the frames where that feature exists. |
| `feature_id` | Always | Unique identifier for an annotation in a label; this id is also present on all classification answers. |
| `feature_schema_id` | Always | Unique identifier for an ontology tool or classification; this id is also present on all classification answers. |
| `annotation_kind` | Situational | The kind of tool utilized by the annotation (e.g. ImageBoundingBox, ImagePolyline) |
| `name` | Always | Name given to the tool or classification |
| `value` | Always | Normalized name of the tool or classification: if a user creates a tool named "Apple Pear," this field will show the normalized version "apple\_pear." |

## `project_details`

| Field | Included | Description |
| - | - | - |
| `ontology_id` | Always | The ID of the ontology connected to the project. |
| `task_id` | Always | The ID of the task the data row currently is in in the project. |
| `task_name` | Always | The name of the task the data row currently is in in the project. |
| `batch_id` | Always | The ID of the batch in which the data row was sent to the project. |
| `batch_name` | Always | The name of the batch in which the data row was sent to the project. |
| `workflow_status` | Always | The status of the data row in the project (either `TO_LABEL`, `IN_REWORK`, `IN_REVIEW`, or `DONE`) |
| `priority` | Always | The priority assigned to the batch. |
| `selected_label_id` | If utilized | The ID of the label that was selected as the "winner" amongst the labels made on the data row. |
| `consensus_expected_label_count` | Always | The number of labels that were expected to be created on this data row according to the [consensus](/horizon/guides/consensus) settings. |
| `workflow_history` | Always | Information on the progression of the labeled data row through the project's workflow. Contains a list of dictionaries comprised of the fields explained in the `workflow_history` table below. |

### `workflow_history`

| Field | Included | Description |
| - | - | - |
| `action` | Always | The action that was performed on the data row in this specific step. Typical actions are: `Move` : change of task queue `Accept`or `Reject`: selection of a review score `Rework`: follows `Reject` |
| `created_at` | Always | A timestamp that indicates when this action on the data row occurred. |
| `created_by` | Always | The email address of the user that performed this action. |
| `previous_task_name` | Situational | The name of the workflow task in which the data row began this action. |
| `previous_task_id` | Situational | The ID of the workflow task in which the data row began this action. |
| `next_task_name` | Situational | The name of the workflow task in which the data row concluded this action. |
| `next_task_id` | Situational | The ID of the workflow task in which the data row concluded this action. |

# `experiments`

The `experiments` field contains a dictionary in which the keys are model experiment IDs, and the values consist of the `name` and a dictionary of model `runs`.

In a model-based export, only a singular model ID will be in this dictionary. Yet, when exporting from the Catalog, consider that a data row may be included in multiple models, and thus the dictionary will have multiple keys.

Model experiment IDs are used in favor of model names to enforce uniqueness.

| Field | Included | Description |
| - | - | - |
| `name` | Always | The name of the model in which the data row appears. |
| `runs` | Always | A dictionary where the keys are the IDs of the model runs in which the data row appears. |

## `runs`

The model `runs` field contains a dictionary in which the keys are model run IDs, and the values consist of the fields explained below.

Model run IDs are used in favor of model run names to enforce uniqueness.

| Field | Included | Description |
| - | - | - |
| `name` | Always | The name of the model run. |
| `run_data_row_id` | Always | A unique ID for the data row in the context of the model run. |
| `labels` | Situational | Ground truth annotations can optionally be sent to a model run as labels. If present, the `labels` will appear in the same format as in a project-based export, which is detailed in the `labels` table above, though the optional `label_details` and `performance_details` sections are always excluded in the context of a model run. |
| `predictions` | Optional | A dictionary containing the fields explained in the `predictions` table below. |
| `split` | Situational | The value will be included if the data row is assigned to a split. The potential outputs are `Training`, `Validation`, or`Test`. When exporting through the SDK, these fields are included by setting the `model_run_details` parameter to `True`. |

### `predictions`

| Field | Included | Description |
| - | - | - |
| `label_kind` | Always | For labels made on assets of most media types, the value is `Default`. For frame-based assets, the value is `Video`. |
| `version` | Always | Used to track updates made to export formats. At present, the value will always be `1.0.0`. |
| `id` | Always | The ID of the set of predictions. |
| `annotations` | Always | See the annotation export formats broken down by asset type, beginning [here](/horizon/developer-guides/export-image-annotations#annotation-export-formats) with images. |


# Export HTML annotations
Source: https://docs.labelbox.com/horizon/developer-guides/export-html-annotations

How to export HTML data and sample export formats.

<CardGroup>
  <Card title="Open In Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/exports/export_data.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/exports/export_data.ipynb" />
</CardGroup>

## Export JSON annotations

<CodeGroup>
  ```python Export theme={"theme":"css-variables"}
  # Set the export params to include/exclude certain fields.
  export_params= {
    "attachments": True,
    "metadata_fields": True,
    "data_row_details": True,
    "project_details": True,
    "label_details": True,
    "performance_details": True,
    "interpolated_frames": True
  }

  # Note: Filters follow AND logic, so typically using one filter is sufficient.

  filters= {
    "last_activity_at": ["2000-01-01 00:00:00", "2050-01-01 00:00:00"],
    "workflow_status": "<wkf-status>"
  }

  export_task = project.export(params=export_params, filters=filters)
  export_task.wait_till_done()

  # Stream the export using a callback function

  def json_stream_handler(output: labelbox.BufferedJsonConverterOutput):
  print(output.json)

  export_task.get_buffered_stream(stream_type=labelbox.StreamType.RESULT).start(stream_handler=json_stream_handler)

  # Collect all exported data into a list

  export_json = [data_row.json for data_row in export_task.get_buffered_stream()]

  print("file size: ", export_task.get_total_file_size(stream_type=lb.StreamType.RESULT))
  print("line count: ", export_task.get_total_lines(stream_type=lb.StreamType.RESULT))

  ```
</CodeGroup>

## Annotation export formats

### Classification - Radio

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "clgva8bo5hmfs1401auduvxht",
    "feature_schema_id": "clgva7wwb0l9r07x54f9h826z",
    "name": "radio_html",
    "value": "radio_html",
    "radio_answer": {
      "feature_id": "clgva8bo5hmft1401nhfbg7ja",
      "feature_schema_id": "clgva7wwb0l9u07x55fyxa5g3",
      "name": "second_radio_answer",
      "value": "second_radio_answer",
      "classifications": []
    }
  }
  ```
</CodeGroup>

### Classification - Checklist

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "clgva8bo5hmfp14016xok5j73",
    "feature_schema_id": "clgva7wwa0l9l07x558vydexo",
    "name": "checklist_html",
    "value": "checklist_html",
    "checklist_answers": [
      {
        "feature_id": "clgva8bo5hmfq1401xnngji4s",
        "feature_schema_id": "clgva7wwb0l9m07x5671qe9ej",
        "name": "first_checklist_answer",
        "value": "first_checklist_answer",
        "classifications": []
      },
      {
        "feature_id": "clgva8bo5hmfr1401w191mdo5",
        "feature_schema_id": "clgva7wwb0l9o07x5h08p9572",
        "name": "second_checklist_answer",
        "value": "second_checklist_answer",
        "classifications": []
      }
    ]
  }
  ```
</CodeGroup>

### Classification - Free-form text

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "clgva8bo5hmfo140186f07jci",
    "feature_schema_id": "clgva7wwa0l9j07x51e4t0bq8",
    "name": "text_html",
    "value": "text_html",
    "text_answer": {
      "content": "sample text"
    }
  }
  ```
</CodeGroup>

## Sample project export

<CodeGroup>
  ```json JSON expandable theme={"theme":"css-variables"}
  {
    "data_row": {
      "id": "clfskiyrx0e0q078k40evbanc",
      "global_key": "sample_html_1.html",
      "row_data": "https://storage.googleapis.com/labelbox-datasets/html_sample_data/sample_html_1.html",
      "details": {
        "dataset_id": "clfskixq70ym7071scn9l16z0",
        "dataset_name": "html_annotation_import_demo_dataset",
        "created_at": "2023-03-28T18:05:11.000+00:00",
        "updated_at": "2024-01-19T23:05:36.279+00:00",
        "last_activity_at": "2023-04-24T20:19:59.000+00:00",
        "created_by": "[email protected]"
      }
    },
    "media_attributes": {
      "mime_type": "text/html"
    },
    "attachments": [],
    "metadata_fields": [],
    "projects": {
      "clgva7x400opk07x73jgs6p8o": {
        "name": "HTML Import Annotation Demo",
        "labels": [
          {
            "label_kind": "Default",
            "version": "1.0.0",
            "id": "clgva8bonhmg21401hcpq6tee",
            "label_details": {
              "created_at": "2023-04-24T20:19:59.000+00:00",
              "updated_at": "2023-04-24T20:19:59.000+00:00",
              "created_by": "[email protected]",
              "content_last_updated_at": "2023-04-24T20:19:59.466+00:00",
              "reviews": []
            },
            "performance_details": {
              "seconds_to_create": 0,
              "seconds_to_review": 0,
              "skipped": false
            },
            "annotations": {
              "objects": [],
              "classifications": [
                {
                  "feature_id": "clgva8bo5hmfo140186f07jci",
                  "feature_schema_id": "clgva7wwa0l9j07x51e4t0bq8",
                  "name": "text_html",
                  "value": "text_html",
                  "text_answer": {
                    "content": "sample text"
                  }
                },
                {
                  "feature_id": "clgva8bo5hmfp14016xok5j73",
                  "feature_schema_id": "clgva7wwa0l9l07x558vydexo",
                  "name": "checklist_html",
                  "value": "checklist_html",
                  "checklist_answers": [
                    {
                      "feature_id": "clgva8bo5hmfq1401xnngji4s",
                      "feature_schema_id": "clgva7wwb0l9m07x5671qe9ej",
                      "name": "first_checklist_answer",
                      "value": "first_checklist_answer",
                      "classifications": []
                    },
                    {
                      "feature_id": "clgva8bo5hmfr1401w191mdo5",
                      "feature_schema_id": "clgva7wwb0l9o07x5h08p9572",
                      "name": "second_checklist_answer",
                      "value": "second_checklist_answer",
                      "classifications": []
                    }
                  ]
                },
                {
                  "feature_id": "clgva8bo5hmfs1401auduvxht",
                  "feature_schema_id": "clgva7wwb0l9r07x54f9h826z",
                  "name": "radio_html",
                  "value": "radio_html",
                  "radio_answer": {
                    "feature_id": "clgva8bo5hmft1401nhfbg7ja",
                    "feature_schema_id": "clgva7wwb0l9u07x55fyxa5g3",
                    "name": "second_radio_answer",
                    "value": "second_radio_answer",
                    "classifications": []
                  }
                },
                {
                  "feature_id": "clgva8bo5hmfu1401yuz7wvuy",
                  "feature_schema_id": "clgva7wwb0l9x07x57si1gihw",
                  "name": "nested_checklist_question",
                  "value": "nested_checklist_question",
                  "checklist_answers": [
                    {
                      "feature_id": "clgva8bo6hmfv14015d33m625",
                      "feature_schema_id": "clgva7wwb0l9y07x5dt6l44qv",
                      "name": "first_checklist_answer",
                      "value": "first_checklist_answer",
                      "classifications": [
                        {
                          "feature_id": "clgva8bo6hmfw1401u59hfep5",
                          "feature_schema_id": "clgva7wwb0l9z07x5bvtx0agv",
                          "name": "sub_checklist_question",
                          "value": "sub_checklist_question",
                          "checklist_answers": [
                            {
                              "feature_id": "clgva8bo6hmfx14018asstw29",
                              "feature_schema_id": "clgva7wwb0la007x515qu9lje",
                              "name": "first_sub_checklist_answer",
                              "value": "first_sub_checklist_answer",
                              "classifications": []
                            }
                          ]
                        }
                      ]
                    }
                  ]
                },
                {
                  "feature_id": "clgva8bo6hmfy1401bivtvjjt",
                  "feature_schema_id": "clgva7wwb0la507x583lg7mvf",
                  "name": "nested_radio_question",
                  "value": "nested_radio_question",
                  "radio_answer": {
                    "feature_id": "clgva8bo6hmfz1401y555xn92",
                    "feature_schema_id": "clgva7wwb0la607x5h39u2z31",
                    "name": "first_radio_answer",
                    "value": "first_radio_answer",
                    "classifications": [
                      {
                        "feature_id": "clgva8bo6hmg01401agmna01o",
                        "feature_schema_id": "clgva7wwb0la707x5hxwe16bo",
                        "name": "sub_radio_question",
                        "value": "sub_radio_question",
                        "radio_answer": {
                          "feature_id": "clgva8bo6hmg11401dzqw0od8",
                          "feature_schema_id": "clgva7wwb0la807x51792e6il",
                          "name": "first_sub_radio_answer",
                          "value": "first_sub_radio_answer",
                          "classifications": []
                        }
                      }
                    ]
                  }
                }
              ],
              "relationships": []
            }
          }
        ],
        "project_details": {
          "ontology_id": "clgva7wto0l9i07x5emb8b4uo",
          "task_name": "Done",
          "batch_id": "5ba71600-e2dd-11ed-8ee3-e3123dc756e1",
          "batch_name": "first-batch-html-demo",
          "workflow_status": "DONE",
          "priority": 5,
          "consensus_expected_label_count": 1,
          "workflow_history": []
        },
        "project_tags": []
      }
    }
  }
  ```
</CodeGroup>

## Sample model run export

<CodeGroup>
  ```json JSON expandable theme={"theme":"css-variables"}
  {
    "data_row": {
      "id": "clfcvkqx708ba079n4nfo1drf",
      "global_key": "sample_html_2.html",
      "row_data": "https://storage.googleapis.com/labelbox-datasets/html_sample_data/sample_html_2.html",
      "details": {
        "dataset_id": "clfcvkpzc0v8l07yt9vhc16sv",
        "dataset_name": "html demo dataset",
        "created_at": "2023-03-17T18:30:11.000+00:00",
        "updated_at": "2024-01-19T22:55:40.282+00:00",
        "created_by": "[email protected]"
      }
    },
    "media_attributes": {
      "mime_type": "text/html"
    },
    "attachments": [],
    "metadata_fields": [],
    "experiments": {
      "a414cfe7-f6ab-094b-8307-68d5a7272397": {
        "name": "HTML_model_run_1d21d941-f392-41e9-85d4-a52b3a3b5bfd",
        "runs": {
          "a414cfe8-690d-0b10-6b9f-7e9bf6a7dd1c": {
            "name": "iteration 1",
            "run_data_row_id": "d2bdab47-af16-4052-a310-74fa324cda5f",
            "labels": [
              {
                "label_kind": "Default",
                "version": "1.0.0",
                "id": "cluu1x19z0out148p9vadgmcu",
                "annotations": {
                  "objects": [],
                  "classifications": [
                    {
                      "feature_id": "3625d4eb-ca0f-0385-9c74-c3538b40621c",
                      "feature_schema_id": "cluu1wl30002007wo1gnn8htz",
                      "name": "free_text",
                      "value": "free_text",
                      "text_answer": {
                        "content": "sample text"
                      }
                    },
                    {
                      "feature_id": "47d39827-416f-042f-a9f5-7d5bf45ba884",
                      "feature_schema_id": "cluu1wl2z001k07wo17r02qrr",
                      "name": "nested_radio_question",
                      "value": "nested_radio_question",
                      "radio_answer": {
                        "feature_id": "89c36638-ec65-095c-893a-49417a1611fc",
                        "feature_schema_id": "cluu1wl2z001l07wo6kll5fc0",
                        "name": "first_radio_answer",
                        "value": "first_radio_answer",
                        "classifications": [
                          {
                            "feature_id": "5de3b3eb-78a8-06e8-b420-491234d05743",
                            "feature_schema_id": "cluu1wl2z001m07wodiwh0hjp",
                            "name": "sub_radio_question",
                            "value": "sub_radio_question",
                            "radio_answer": {
                              "feature_id": "684b6ca5-e5d4-0684-991d-597e5ea76700",
                              "feature_schema_id": "cluu1wl2z001n07wo8ey5f1yw",
                              "name": "first_sub_radio_answer",
                              "value": "first_sub_radio_answer",
                              "classifications": []
                            }
                          }
                        ]
                      }
                    },
                    {
                      "feature_id": "9297d521-2b0a-04bd-b88b-b5512b85ec20",
                      "feature_schema_id": "cluu1wl2z001s07woc5xmcz68",
                      "name": "checklist_question",
                      "value": "checklist_question",
                      "checklist_answers": [
                        {
                          "feature_id": "add68a9d-01b1-039a-bbdf-c96f63d408e6",
                          "feature_schema_id": "cluu1wl30001v07wocp3hg8a0",
                          "name": "second_checklist_answer",
                          "value": "second_checklist_answer",
                          "classifications": []
                        },
                        {
                          "feature_id": "ce4eca3a-b2f5-0861-954f-ad7ff07c1a7f",
                          "feature_schema_id": "cluu1wl30001x07wobpdwgf4w",
                          "name": "third_checklist_answer",
                          "value": "third_checklist_answer",
                          "classifications": []
                        },
                        {
                          "feature_id": "fab6d119-206f-0b6d-a5a7-8a08e17ed8cf",
                          "feature_schema_id": "cluu1wl2z001t07wobsowcibp",
                          "name": "first_checklist_answer",
                          "value": "first_checklist_answer",
                          "classifications": []
                        }
                      ]
                    },
                    {
                      "feature_id": "9c1c6bad-ca20-08c9-8d1a-78907e65ebe4",
                      "feature_schema_id": "cluu1wl2z001g07wo5elxfwut",
                      "name": "radio_question",
                      "value": "radio_question",
                      "radio_answer": {
                        "feature_id": "28ea7b46-b891-0285-a8a3-3ba7a30cdeb4",
                        "feature_schema_id": "cluu1wl2z001h07wo760uc0sb",
                        "name": "first_radio_answer",
                        "value": "first_radio_answer",
                        "classifications": []
                      }
                    },
                    {
                      "feature_id": "d48dce83-f514-0728-9b5a-deb045493dc3",
                      "feature_schema_id": "cluu1wl30002207woa3u870v7",
                      "name": "nested_checklist_question",
                      "value": "nested_checklist_question",
                      "checklist_answers": [
                        {
                          "feature_id": "ca4f111b-c6f6-013a-a38e-0ce1ee77b370",
                          "feature_schema_id": "cluu1wl30002307wogixwdc5z",
                          "name": "first_checklist_answer",
                          "value": "first_checklist_answer",
                          "classifications": [
                            {
                              "feature_id": "6d08884d-ff25-0a41-a5aa-875e6385ea1e",
                              "feature_schema_id": "cluu1wl30002407woea6e102z",
                              "name": "sub_checklist_question",
                              "value": "sub_checklist_question",
                              "checklist_answers": [
                                {
                                  "feature_id": "02db9eb9-6763-0a2c-92ee-cdf3b11b8de4",
                                  "feature_schema_id": "cluu1wl30002507wo07av3u9e",
                                  "name": "first_sub_checklist_answer",
                                  "value": "first_sub_checklist_answer",
                                  "classifications": []
                                }
                              ]
                            }
                          ]
                        }
                      ]
                    }
                  ],
                  "relationships": []
                }
              }
            ],
            "predictions": [
              {
                "label_kind": "Default",
                "version": "1.0.0",
                "id": "cluu1x19z0out148p9vadgmcu",
                "annotations": {
                  "objects": [],
                  "classifications": [
                    {
                      "feature_id": "162ea9d9-c6e8-48e9-b4fe-1beb5ec35e71",
                      "feature_schema_id": "cluu1wl2z001k07wo17r02qrr",
                      "name": "nested_radio_question",
                      "value": "nested_radio_question",
                      "radio_answer": {
                        "feature_id": "ecdb63be-bd8b-4768-ab51-6399e38c3fd8",
                        "feature_schema_id": "cluu1wl2z001l07wo6kll5fc0",
                        "name": "first_radio_answer",
                        "value": "first_radio_answer",
                        "classifications": [
                          {
                            "feature_id": "9bdb8546-6ee7-4993-bea7-9de7d1d4f0d5",
                            "feature_schema_id": "cluu1wl2z001m07wodiwh0hjp",
                            "name": "sub_radio_question",
                            "value": "sub_radio_question",
                            "radio_answer": {
                              "feature_id": "a0466f11-c940-4217-9589-56f261425e43",
                              "feature_schema_id": "cluu1wl2z001n07wo8ey5f1yw",
                              "name": "first_sub_radio_answer",
                              "value": "first_sub_radio_answer",
                              "classifications": []
                            }
                          }
                        ]
                      }
                    },
                    {
                      "feature_id": "2b7e8e1b-a55d-4ac6-80e2-629bc64c5b97",
                      "feature_schema_id": "cluu1wl30002007wo1gnn8htz",
                      "name": "free_text",
                      "value": "free_text",
                      "text_answer": {
                        "content": "sample text"
                      }
                    },
                    {
                      "feature_id": "347dd2d9-7729-4ebd-970a-c4eedb1d41cc",
                      "feature_schema_id": "cluu1wl30002207woa3u870v7",
                      "name": "nested_checklist_question",
                      "value": "nested_checklist_question",
                      "checklist_answers": [
                        {
                          "feature_id": "1a17cdf4-2159-4c21-b1d8-7b9490d8bb14",
                          "feature_schema_id": "cluu1wl30002307wogixwdc5z",
                          "name": "first_checklist_answer",
                          "value": "first_checklist_answer",
                          "classifications": [
                            {
                              "feature_id": "ab20eb98-3002-4108-a541-37741a9a1531",
                              "feature_schema_id": "cluu1wl30002407woea6e102z",
                              "name": "sub_checklist_question",
                              "value": "sub_checklist_question",
                              "checklist_answers": [
                                {
                                  "feature_id": "31e09418-2b8e-4759-bd71-4f88302398ce",
                                  "feature_schema_id": "cluu1wl30002507wo07av3u9e",
                                  "name": "first_sub_checklist_answer",
                                  "value": "first_sub_checklist_answer",
                                  "classifications": []
                                }
                              ]
                            }
                          ]
                        }
                      ]
                    },
                    {
                      "feature_id": "70f54814-79c0-4d66-84c5-531e8b53e657",
                      "feature_schema_id": "cluu1wl2z001g07wo5elxfwut",
                      "name": "radio_question",
                      "value": "radio_question",
                      "radio_answer": {
                        "feature_id": "214cb3db-da72-4692-9fa4-c86e2933b11e",
                        "feature_schema_id": "cluu1wl2z001h07wo760uc0sb",
                        "name": "first_radio_answer",
                        "value": "first_radio_answer",
                        "classifications": []
                      }
                    },
                    {
                      "feature_id": "baa18a09-6626-48cb-9b4c-7d75aa7d8c57",
                      "feature_schema_id": "cluu1wl2z001s07woc5xmcz68",
                      "name": "checklist_question",
                      "value": "checklist_question",
                      "checklist_answers": [
                        {
                          "feature_id": "c1cf3d18-238f-4817-9fed-93c14954c153",
                          "feature_schema_id": "cluu1wl30001v07wocp3hg8a0",
                          "name": "second_checklist_answer",
                          "value": "second_checklist_answer",
                          "classifications": []
                        },
                        {
                          "feature_id": "cf4606ab-1a5b-4c50-87a3-1f2b5dfc9547",
                          "feature_schema_id": "cluu1wl30001x07wobpdwgf4w",
                          "name": "third_checklist_answer",
                          "value": "third_checklist_answer",
                          "classifications": []
                        },
                        {
                          "feature_id": "f4f28750-5a7f-4da0-a081-b4da8eacb667",
                          "feature_schema_id": "cluu1wl2z001t07wobsowcibp",
                          "name": "first_checklist_answer",
                          "value": "first_checklist_answer",
                          "classifications": []
                        }
                      ]
                    }
                  ],
                  "relationships": []
                }
              }
            ]
          }
        }
      }
    },
    "models": {}
  }
  ```
</CodeGroup>


# Export image annotations
Source: https://docs.labelbox.com/horizon/developer-guides/export-image-annotations

How to export image annotations and sample export formats.

<CardGroup>
  <Card title="Open In Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/exports/export_data.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/exports/export_data.ipynb" />
</CardGroup>

## Export as JSON

<CodeGroup>
  ```python Export theme={"theme":"css-variables"}
  # Set the export params to include/exclude certain fields.
  export_params= {
    "attachments": True,
    "metadata_fields": True,
    "data_row_details": True,
    "project_details": True,
    "label_details": True,
    "performance_details": True,
    "interpolated_frames": True
  }

  # Note: Filters follow AND logic, so typically using one filter is sufficient.

  filters= {
    "last_activity_at": ["2000-01-01 00:00:00", "2050-01-01 00:00:00"],
    "workflow_status": "<wkf-status>"
  }

  export_task = project.export(params=export_params, filters=filters)
  export_task.wait_till_done()

  # Stream the export using a callback function

  def json_stream_handler(output: labelbox.BufferedJsonConverterOutput):
    print(output.json)

  export_task.get_buffered_stream(stream_type=labelbox.StreamType.RESULT).start(stream_handler=json_stream_handler)

  # Collect all exported data into a list

  export_json = [data_row.json for data_row in export_task.get_buffered_stream()]

  print("file size: ", export_task.get_total_file_size(stream_type=lb.StreamType.RESULT))
  print("line count: ", export_task.get_total_lines(stream_type=lb.StreamType.RESULT))

  ```
</CodeGroup>

## Annotation export formats

### Bounding box

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "clrf5csvi6ofm07lsf9pygwvi",
    "feature_schema_id": "clrf5ck4a0b9b071paa9ncu15",
    "name": "bounding_box",
    "annotation_kind": "ImageBoundingBox",
    "classifications": [],
    "bounding_box": {
      "top": 1264.0,
      "left": 2096.0,
      "height": 425.0,
      "width": 144.0
    }
  }
  ```
</CodeGroup>

### Polygon

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "clr0x0ol83tvt07ea4has2a4l",
    "feature_schema_id": "clpk2me9t02v707xn08cv4ve5",
    "name": "polygon",
    "annotation_kind": "ImagePolygon",
    "classifications": [],
    "polygon": [
      {
        "x": 1489.581,
        "y": 183.934
      },
      {
        "x": 1489.581,
        "y": 183.934
      }
    ]
  }
  ```
</CodeGroup>

### Polyline

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
   {
     "feature_id": "clr0x0ol83tvk07eagehegjde",
     "feature_schema_id": "clpk2me9u02vh07xnfcmtbqvj",
     "name": "polyline",
     "annotation_kind": "ImagePolyline",
     "classifications": [],
     "line": [
       {
         "x": 2534.353,
         "y": 249.471
       },
       {
         "x": 90.929,
         "y": 326.412
       }
     ]
   }
  ```
</CodeGroup>

### Point

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "clr0x0ol83tvr07ea2fyl69g9",
    "feature_schema_id": "clpk2me9u02vf07xnbpk45wxa",
    "name": "point",
    "annotation_kind": "ImagePoint",
    "classifications": [],
    "point": {
      "x": 1166.606,
      "y": 1441.768
    }
  }
  ```
</CodeGroup>

### Mask

Regardless of which [tool](/horizon/guides/image-editor#segmentation-mask) is used to create a segmentation mask, the annotation will appear in the following form in the export. Whether you use AutoSegment, the brush tool, the pen tool, or the fill tool, all masks take the same format in exports.

#### Composite masks

[Open this Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/exports/composite_mask_export.ipynb) for an interactive tutorial on exporting composite masks.

A composite mask is a cloud-hosted PNG file that contains all raster segmentation mask instances on a labeled data row (one composite mask per label).

When you export data rows containing segmentation masks, you can use `composite_mask.url` to access all segmentation masks on that data row. To access a single instance of a segmentation mask, use the `color_rgb` values. The `color_rgb` value represents the color of a single annotation or instance mask in a label.

##### Sample composite masks export

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "clsivokxg00043b70xuf7x5tz",
    "feature_schema_id": "clsgbcaqw000j071t2ga23jnc",
    "name": "car",
    "value": "car",
    "annotation_kind": "ImageSegmentationMask",
    "classifications": [],
    "mask": {
      "url": "https://api.labelbox.com/api/v1/projects/clsivnieg050k074i2jqaa0c3/annotations/clsivokxg00043b70xuf7x5tz/index/1/mask"
    },
    "composite_mask": {
      // This URL contains all mask instances on the data row
      "url": "https://api.labelbox.com/api/v1/tasks/clsiw88n205mw072ke4n78y5o/masks/clsivok5f00023b70r1a4fgzy",
      // This RGB reflects a single mask instance
      "color_rgb": [
        31,
        185,
        101
      ]
    }
  }
  ```
</CodeGroup>

#### Access mask data

To access the mask data, you must pass your Labelbox API key stored in `client.headers` in an API request.

When you grab a URL from the mask annotation in the export, the `project_id` and `feature_id` will already be in place, as shown above. Here, we provide the framework for structuring a URL with any project ID and feature ID.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  import urllib.request
  from PIL import Image

  # Provide a project ID and feature ID if using instance masks. Alternatively, replace the entire mask_url with a URL grabbed from your export.
  project_id = ""
  feature_id = ""

  mask_url = f"https://api.labelbox.com/api/v1/projects/{project_id}/annotations/{feature_id}/index/1/mask"

  # Instead of providing the individual variables, we recommend utilizing the URL from export for composite mask
  composite_mask_url = f"https://api.labelbox.com/api/v1/tasks/{task_id}/masks/{composite_mask_id}/index/1"

  # Make the API request
  req = urllib.request.Request(mask_url, headers=client.headers)

  # Optionally, print the image of the mask
  image = Image.open(urllib.request.urlopen(req))
  image.show()
  ```
</CodeGroup>

### Relationship

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "clur69pfa00023b6rkftlkucz",
    "feature_schema_id": "cltd3vkc201ab07ts1f9dhpkt",
    "name": "relationship",
    "value": "relationship",
    "annotation_kind": "ImageUnidirectionalRelationship",
    "classifications": [],
    "unidirectional_relationship": {
      "source": "clur5tlyq1n2d07dcg9xldg90",
      "target": "clur5tlyr1n2l07dc1jsf15qo"
    }
  }
  ```
</CodeGroup>

### Classification - radio

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "clur5tlyr1n2k07dcezk08npu",
    "feature_schema_id": "cltd3vkc201ad07ts3t78873p",
    "name": "radio_question",
    "value": "radio_question",
    "radio_answer": {
      "feature_id": "clur5tlyr1n2y07dcajpyg0d4",
      "feature_schema_id": "cltd3vkc201ag07tsfoc2bdpi",
      "name": "second_radio_answer",
      "value": "second_radio_answer",
      "classifications": []
    }
  }
  ```
</CodeGroup>

### Classification - checklist

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "clur5tlyr1n2p07dchs791672",
    "feature_schema_id": "cltd3vkc201aj07tsait4cbzh",
    "name": "checklist_question",
    "value": "checklist_question",
    "checklist_answers": [
      {
        "feature_id": "clur5tlyr1n2z07dc3abv19e8",
        "feature_schema_id": "cltd3vkc201ak07tsfqxc740z",
        "name": "first_checklist_answer",
        "value": "first_checklist_answer",
        "classifications": []
      },
      {
        "feature_id": "clur5tlyr1n3007dc4vbr0628",
        "feature_schema_id": "cltd3vkc201am07ts7s9e7x7k",
        "name": "second_checklist_answer",
        "value": "second_checklist_answer",
        "classifications": []
      }
    ]
  }
  ```
</CodeGroup>

### Classification - free-form text

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "clur5tlyr1n2g07dc7wdv7p5z",
    "feature_schema_id": "cltd3vkc201ap07ts58gma0bb",
    "name": "free_text",
    "value": "free_text",
    "text_answer": {
      "content": "sample text"
    }
  }
  ```
</CodeGroup>

## Sample project export

<CodeGroup>
  ```json JSON expandable theme={"theme":"css-variables"}
  {
    "data_row": {
      "id": "clf9spzxq029m071nehy60a6v",
      "global_key": "2560px-Kitano_Street_Kobe01s5s4110.jpeg",
      "row_data": "https://storage.googleapis.com/labelbox-datasets/image_sample_data/2560px-Kitano_Street_Kobe01s5s4110.jpeg",
      "details": {
        "dataset_id": "clf9spwgw04w207ymfwy826qv",
        "dataset_name": "demo_dataset_img",
        "created_at": "2023-03-15T14:46:58.000+00:00",
        "updated_at": "2023-03-15T14:46:58.000+00:00",
        "last_activity_at": "2024-04-08T16:34:01.000+00:00",
        "created_by": "[email protected]"
      }
    },
    "media_attributes": {
      "height": 1707,
      "width": 2560,
      "mime_type": "image/jpeg"
    },
    "attachments": [],
    "metadata_fields": [],
    "embeddings": [
      {
        "id": "cn1cywy1r077807zblvo0m884",
        "name": "my_custom_embedding_2048_dimensions",
        "dimensions": 2048,
        "is_custom": true,
        "values": [
          {
            "value": [
              1.1265901290930636,
              1.0413733599834287,
              1.3041214120015427,
              1.893491997791636 ...
            ]
          }
        ]
      },
      {
        "id": "c200000000000000000000000",
        "name": "Image Embedding V2 (CLIP ViT-B/32)",
        "dimensions": 512,
        "is_custom": false,
        "values": [
          {
            "value": [
              -0.4192526042461395,
              -0.048827216029167175 ...
            ]
          }
        ]
      }
    ],
    "projects": {
      "cltd3vkju021907u63w2odj69": {
        "name": "Image Annotation Import Demo",
        "labels": [
          {
            "label_kind": "Default",
            "version": "1.0.0",
            "id": "clur5tlda1n2a07dc0i0zdd4k",
            "label_details": {
              "created_at": "2024-04-08T16:21:26.000+00:00",
              "updated_at": "2024-04-08T16:34:02.000+00:00",
              "created_by": "[email protected]",
              "content_last_updated_at": "2024-04-08T16:34:00.513+00:00",
              "reviews": []
            },
            "performance_details": {
              "seconds_to_create": 18,
              "seconds_to_review": 6,
              "skipped": false
            },
            "annotations": {
              "objects": [
                {
                  "feature_id": "clur5tlyq1n2d07dcg9xldg90",
                  "feature_schema_id": "cltd3vkc1019r07ts2lru4hi4",
                  "name": "bounding_box",
                  "value": "bounding_box",
                  "annotation_kind": "ImageBoundingBox",
                  "classifications": [],
                  "bounding_box": {
                    "top": 1264.0,
                    "left": 2096.0,
                    "height": 425.0,
                    "width": 144.0
                  }
                },
                {
                  "feature_id": "clur5tlyr1n2f07dcf78sa4y5",
                  "feature_schema_id": "cltd3vkc201a707ts9ullbhfo",
                  "name": "point",
                  "value": "point",
                  "annotation_kind": "ImagePoint",
                  "classifications": [],
                  "point": {
                    "x": 1166.606,
                    "y": 1441.768
                  }
                },
                {
                  "feature_id": "clur5tlyr1n2h07dc1pxnhoj3",
                  "feature_schema_id": "cltd3vkc1019t07ts2bk6hjjj",
                  "name": "bbox_with_radio_subclass",
                  "value": "bbox_with_radio_subclass",
                  "annotation_kind": "ImageBoundingBox",
                  "classifications": [
                    {
                      "feature_id": "clur5tlyr1n2v07dcbnnfh1ah",
                      "feature_schema_id": "cltd3vkc1019u07tsaari5i2k",
                      "name": "sub_radio_question",
                      "value": "sub_radio_question",
                      "radio_answer": {
                        "feature_id": "clur5tlyr1n3607dc98qpagxb",
                        "feature_schema_id": "cltd3vkc1019v07tsbow08f9u",
                        "name": "first_sub_radio_answer",
                        "value": "first_sub_radio_answer",
                        "classifications": []
                      }
                    }
                  ],
                  "bounding_box": {
                    "top": 933.0,
                    "left": 541.0,
                    "height": 191.0,
                    "width": 330.0
                  }
                },
                {
                  "feature_id": "clur5tlyr1n2j07dc50cs4e11",
                  "feature_schema_id": "cltd3vkc201a907ts69dr5m86",
                  "name": "polyline",
                  "value": "polyline",
                  "annotation_kind": "ImagePolyline",
                  "classifications": [],
                  "line": [
                    {
                      "x": 2534.353,
                      "y": 249.471
                    },
                    {
                      "x": 2429.492,
                      "y": 182.092
                    },
                    {
                      "x": 2294.322,
                      "y": 221.962
                    },
                    {
                      "x": 2224.491,
                      "y": 180.463
                    },
                    {
                      "x": 2136.123,
                      "y": 204.716
                    },
                    {
                      "x": 1712.247,
                      "y": 173.949
                    },
                    {
                      "x": 1703.838,
                      "y": 84.438
                    },
                    {
                      "x": 1579.772,
                      "y": 82.61
                    },
                    {
                      "x": 1583.442,
                      "y": 167.552
                    },
                    {
                      "x": 1478.869,
                      "y": 164.903
                    },
                    {
                      "x": 1418.941,
                      "y": 318.149
                    },
                    {
                      "x": 1243.128,
                      "y": 400.815
                    },
                    {
                      "x": 1022.067,
                      "y": 319.007
                    },
                    {
                      "x": 892.367,
                      "y": 379.216
                    },
                    {
                      "x": 670.273,
                      "y": 364.408
                    },
                    {
                      "x": 613.114,
                      "y": 288.16
                    },
                    {
                      "x": 377.559,
                      "y": 238.251
                    },
                    {
                      "x": 368.087,
                      "y": 185.064
                    },
                    {
                      "x": 246.557,
                      "y": 167.286
                    },
                    {
                      "x": 236.648,
                      "y": 285.61
                    },
                    {
                      "x": 90.929,
                      "y": 326.412
                    }
                  ]
                },
                {
                  "feature_id": "clur5tlyr1n2l07dc1jsf15qo",
                  "feature_schema_id": "cltd3vkc1019r07ts2lru4hi4",
                  "name": "bounding_box",
                  "value": "bounding_box",
                  "annotation_kind": "ImageBoundingBox",
                  "classifications": [],
                  "bounding_box": {
                    "top": 1346.0,
                    "left": 2272.0,
                    "height": 358.0,
                    "width": 144.0
                  }
                },
                {
                  "feature_id": "clur5tlyr1n2m07dc4x6xhzep",
                  "feature_schema_id": "cltd3vkc1019z07ts7g1n3o5m",
                  "name": "polygon",
                  "value": "polygon",
                  "annotation_kind": "ImagePolygon",
                  "classifications": [],
                  "polygon": [
                    {
                      "x": 1489.581,
                      "y": 183.934
                    },
                    {
                      "x": 2278.306,
                      "y": 256.885
                    },
                    {
                      "x": 2428.197,
                      "y": 200.437
                    },
                    {
                      "x": 2560.0,
                      "y": 335.419
                    },
                    {
                      "x": 2557.386,
                      "y": 503.165
                    },
                    {
                      "x": 2320.596,
                      "y": 503.103
                    },
                    {
                      "x": 2156.083,
                      "y": 628.943
                    },
                    {
                      "x": 2161.111,
                      "y": 785.519
                    },
                    {
                      "x": 2002.115,
                      "y": 894.647
                    },
                    {
                      "x": 1838.456,
                      "y": 877.874
                    },
                    {
                      "x": 1436.53,
                      "y": 874.636
                    },
                    {
                      "x": 1411.403,
                      "y": 758.579
                    },
                    {
                      "x": 1353.853,
                      "y": 751.74
                    },
                    {
                      "x": 1345.264,
                      "y": 453.461
                    },
                    {
                      "x": 1426.011,
                      "y": 421.129
                    },
                    {
                      "x": 1489.581,
                      "y": 183.934
                    }
                  ]
                },
                {
                  "feature_id": "clur5tlyr1n2n07dc4qot5gbx",
                  "feature_schema_id": "cltd3vkc1019r07ts2lru4hi4",
                  "name": "bounding_box",
                  "value": "bounding_box",
                  "annotation_kind": "ImageBoundingBox",
                  "classifications": [],
                  "bounding_box": {
                    "top": 977.0,
                    "left": 1690.0,
                    "height": 330.0,
                    "width": 225.0
                  }
                },
                {
                  "feature_id": "clur5tlyr1n2q07dc5joc4k7u",
                  "feature_schema_id": "cltd3vkc101a107ts95sk63u0",
                  "name": "mask",
                  "value": "mask",
                  "annotation_kind": "ImageSegmentationMask",
                  "classifications": [],
                  "mask": {
                    "url": "https://api.labelbox.com/api/v1/projects/cltd3vkju021907u63w2odj69/annotations/clur5tlyr1n2q07dc5joc4k7u/index/1/mask"
                  },
                  "composite_mask": {
                    "url": "https://api.labelbox.com/api/v1/tasks/clur6dp8r04ul07wrgp9q4pla/masks/clur5tlyr1n2o07dc99lw3r8p/index/1",
                    "color_rgb": [
                      10,
                      10,
                      125
                    ]
                  }
                },
                {
                  "feature_id": "clur5tlyr1n2r07dc6ym4d4mc",
                  "feature_schema_id": "cltd3vkc201a307tsforlev07",
                  "name": "mask_with_text_subclass",
                  "value": "mask_with_text_subclass",
                  "annotation_kind": "ImageSegmentationMask",
                  "classifications": [
                    {
                      "feature_id": "clur5tlyr1n2x07dc84a12l8i",
                      "feature_schema_id": "cltd3vkc201a407tsfo3agh7x",
                      "name": "sub_free_text",
                      "value": "sub_free_text",
                      "text_answer": {
                        "content": "free text answer sample"
                      }
                    }
                  ],
                  "mask": {
                    "url": "https://api.labelbox.com/api/v1/projects/cltd3vkju021907u63w2odj69/annotations/clur5tlyr1n2r07dc6ym4d4mc/index/1/mask"
                  },
                  "composite_mask": {
                    "url": "https://api.labelbox.com/api/v1/tasks/clur6dp8r04ul07wrgp9q4pla/masks/clur5tlyr1n2o07dc99lw3r8p/index/1",
                    "color_rgb": [
                      105,
                      136,
                      97
                    ]
                  }
                },
                {
                  "feature_id": "clur5tlyr1n2s07dc739a6kqs",
                  "feature_schema_id": "cltd3vkc201a307tsforlev07",
                  "name": "mask_with_text_subclass",
                  "value": "mask_with_text_subclass",
                  "annotation_kind": "ImageSegmentationMask",
                  "classifications": [
                    {
                      "feature_id": "clur5tlyr1n2w07dccdv0g7gg",
                      "feature_schema_id": "cltd3vkc201a407tsfo3agh7x",
                      "name": "sub_free_text",
                      "value": "sub_free_text",
                      "text_answer": {
                        "content": "free text answer sample"
                      }
                    }
                  ],
                  "mask": {
                    "url": "https://api.labelbox.com/api/v1/projects/cltd3vkju021907u63w2odj69/annotations/clur5tlyr1n2s07dc739a6kqs/index/1/mask"
                  },
                  "composite_mask": {
                    "url": "https://api.labelbox.com/api/v1/tasks/clur6dp8r04ul07wrgp9q4pla/masks/clur5tlyr1n2o07dc99lw3r8p/index/1",
                    "color_rgb": [
                      7,
                      85,
                      11
                    ]
                  }
                },
                {
                  "feature_id": "clur5tlyr1n2t07dc0oxlgoc8",
                  "feature_schema_id": "cltd3vkc201a307tsforlev07",
                  "name": "mask_with_text_subclass",
                  "value": "mask_with_text_subclass",
                  "annotation_kind": "ImageSegmentationMask",
                  "classifications": [
                    {
                      "feature_id": "clur5tlyr1n3107dccy5pbudr",
                      "feature_schema_id": "cltd3vkc201a407tsfo3agh7x",
                      "name": "sub_free_text",
                      "value": "sub_free_text",
                      "text_answer": {
                        "content": "free text answer sample"
                      }
                    }
                  ],
                  "mask": {
                    "url": "https://api.labelbox.com/api/v1/projects/cltd3vkju021907u63w2odj69/annotations/clur5tlyr1n2t07dc0oxlgoc8/index/1/mask"
                  },
                  "composite_mask": {
                    "url": "https://api.labelbox.com/api/v1/tasks/clur6dp8r04ul07wrgp9q4pla/masks/clur5tlyr1n2o07dc99lw3r8p/index/1",
                    "color_rgb": [
                      35,
                      34,
                      212
                    ]
                  }
                },
                {
                  "feature_id": "clur5tlyr1n2u07dc5wjmdm3v",
                  "feature_schema_id": "cltd3vkc101a107ts95sk63u0",
                  "name": "mask",
                  "value": "mask",
                  "annotation_kind": "ImageSegmentationMask",
                  "classifications": [],
                  "mask": {
                    "url": "https://api.labelbox.com/api/v1/projects/cltd3vkju021907u63w2odj69/annotations/clur5tlyr1n2u07dc5wjmdm3v/index/1/mask"
                  },
                  "composite_mask": {
                    "url": "https://api.labelbox.com/api/v1/tasks/clur6dp8r04ul07wrgp9q4pla/masks/clur5tlyr1n2o07dc99lw3r8p/index/1",
                    "color_rgb": [
                      232,
                      194,
                      86
                    ]
                  }
                }
              ],
              "classifications": [
                {
                  "feature_id": "clur5tlyr1n2e07dc2bez8p1u",
                  "feature_schema_id": "cltd3vkc201ar07ts3gq6a6sn",
                  "name": "nested_radio_question",
                  "value": "nested_radio_question",
                  "radio_answer": {
                    "feature_id": "clur5tlyr1n3307dcexth5ff7",
                    "feature_schema_id": "cltd3vkc201as07tsh2prgosu",
                    "name": "first_radio_answer",
                    "value": "first_radio_answer",
                    "classifications": [
                      {
                        "feature_id": "clur5tlyr1n3407dc306b5xib",
                        "feature_schema_id": "cltd3vkc201at07tsdjx9dlbw",
                        "name": "sub_radio_question",
                        "value": "sub_radio_question",
                        "radio_answer": {
                          "feature_id": "clur5tlyr1n3507dcef304jmb",
                          "feature_schema_id": "cltd3vkc201au07ts7nk47qny",
                          "name": "first_sub_radio_answer",
                          "value": "first_sub_radio_answer",
                          "classifications": []
                        }
                      }
                    ]
                  }
                },
                {
                  "feature_id": "clur5tlyr1n2g07dc7wdv7p5z",
                  "feature_schema_id": "cltd3vkc201ap07ts58gma0bb",
                  "name": "free_text",
                  "value": "free_text",
                  "text_answer": {
                    "content": "sample text"
                  }
                },
                {
                  "feature_id": "clur5tlyr1n2i07dcd8lx87be",
                  "feature_schema_id": "cltd3vkc201az07ts7st3fn79",
                  "name": "nested_checklist_question",
                  "value": "nested_checklist_question",
                  "checklist_answers": [
                    {
                      "feature_id": "clur5tlyr1n3207dc4u4nfg08",
                      "feature_schema_id": "cltd3vkc201b007tsdhs07baf",
                      "name": "first_checklist_answer",
                      "value": "first_checklist_answer",
                      "classifications": [
                        {
                          "feature_id": "clur5tlyr1n3707dc2cy805ac",
                          "feature_schema_id": "cltd3vkc201b107ts12b1al1j",
                          "name": "sub_checklist_question",
                          "value": "sub_checklist_question",
                          "checklist_answers": [
                            {
                              "feature_id": "clur5tlyr1n3807dcgpgih5qj",
                              "feature_schema_id": "cltd3vkc201b207ts434b9iyx",
                              "name": "first_sub_checklist_answer",
                              "value": "first_sub_checklist_answer",
                              "classifications": []
                            }
                          ]
                        }
                      ]
                    }
                  ]
                },
                {
                  "feature_id": "clur5tlyr1n2k07dcezk08npu",
                  "feature_schema_id": "cltd3vkc201ad07ts3t78873p",
                  "name": "radio_question",
                  "value": "radio_question",
                  "radio_answer": {
                    "feature_id": "clur5tlyr1n2y07dcajpyg0d4",
                    "feature_schema_id": "cltd3vkc201ag07tsfoc2bdpi",
                    "name": "second_radio_answer",
                    "value": "second_radio_answer",
                    "classifications": []
                  }
                },
                {
                  "feature_id": "clur5tlyr1n2p07dchs791672",
                  "feature_schema_id": "cltd3vkc201aj07tsait4cbzh",
                  "name": "checklist_question",
                  "value": "checklist_question",
                  "checklist_answers": [
                    {
                      "feature_id": "clur5tlyr1n2z07dc3abv19e8",
                      "feature_schema_id": "cltd3vkc201ak07tsfqxc740z",
                      "name": "first_checklist_answer",
                      "value": "first_checklist_answer",
                      "classifications": []
                    },
                    {
                      "feature_id": "clur5tlyr1n3007dc4vbr0628",
                      "feature_schema_id": "cltd3vkc201am07ts7s9e7x7k",
                      "name": "second_checklist_answer",
                      "value": "second_checklist_answer",
                      "classifications": []
                    }
                  ]
                }
              ],
              "relationships": [
                {
                  "feature_id": "clur69pfa00023b6rkftlkucz",
                  "feature_schema_id": "cltd3vkc201ab07ts1f9dhpkt",
                  "name": "relationship",
                  "value": "relationship",
                  "annotation_kind": "ImageUnidirectionalRelationship",
                  "classifications": [],
                  "unidirectional_relationship": {
                    "source": "clur5tlyq1n2d07dcg9xldg90",
                    "target": "clur5tlyr1n2l07dc1jsf15qo"
                  }
                }
              ]
            },
            "z_order": [
                "clur5tlyq1n2d07dcg9xldg90",
                "clur5tlyr1n2f07dcf78sa4y5",
                "clur5tlyr1n2j07dc50cs4e11",
                "clur5tlyr1n2l07dc1jsf15qo",
                "clur5tlyr1n2m07dc4x6xhzep",
                "clur5tlyr1n2n07dc4qot5gbx",
                "clur5tlyr1n2q07dc5joc4k7u",
                "clur5tlyr1n2r07dc6ym4d4mc",
                "clur5tlyr1n2s07dc739a6kqs",
                "clur5tlyr1n2u07dc5wjmdm3v"
            ]
          }
        ],
        "project_details": {
          "ontology_id": "cltd3vkbb019q07ts23eedy4o",
          "task_id": "a384279a-45c0-4200-b85a-3c18469b3809",
          "task_name": "Initial review task",
          "batch_id": "3f724e20-da3d-11ee-86a4-0145011952fd",
          "batch_name": "image-demo-batch",
          "workflow_status": "IN_REVIEW",
          "priority": 1,
          "consensus_expected_label_count": 1,
          "workflow_history": [
            {
              "action": "Move",
              "created_at": "2024-04-08T16:21:26.492+00:00",
              "created_by": "[email protected]",
              "previous_task_name": "Initial labeling task",
              "previous_task_id": "22ee463b-fbe3-0791-8159-94083be9dee3",
              "next_task_name": "Initial review task",
              "next_task_id": "a384279a-45c0-4200-b85a-3c18469b3809"
            },
            {
              "action": "Move",
              "created_at": "2024-04-08T16:21:26.481+00:00",
              "created_by": "[email protected]",
              "next_task_name": "Initial labeling task",
              "next_task_id": "22ee463b-fbe3-0791-8159-94083be9dee3"
            }
          ]
        },
        "project_tags": []
      }
    }
  }
  ```
</CodeGroup>

## Sample model run export

<CodeGroup>
  ```json JSON expandable theme={"theme":"css-variables"}
  {
    "data_row": {
      "id": "clo78lhhn3dxb07322mnn8plu",
      "global_key": "2560px-Kitano_Street_Kobe01s.jpeg",
      "row_data": "https://storage.googleapis.com/labelbox-datasets/image_sample_data/2560px-Kitano_Street_Kobe01s5s4110.jpeg",
      "details": {
        "dataset_id": "clo78lg8f003007321fe52lje",
        "dataset_name": "image_prediction_demo",
        "created_at": "2023-10-26T13:45:24.929+00:00",
        "updated_at": "2023-10-26T13:45:25.965+00:00",
        "created_by": "[email protected]"
      }
    },
    "media_attributes": {
      "height": 1707,
      "width": 2560,
      "mime_type": "image/jpeg",
      "exif_rotation": "1"
    },
    "attachments": [],
    "metadata_fields": [],
     "embeddings": [
      {
        "id": "cn1cywy1r077807zblvo0m884",
        "name": "my_custom_embedding_2048_dimensions",
        "dimensions": 2048,
        "is_custom": true,
        "values": [
          {
            "value": [
              1.1265901290930636,
              1.0413733599834287,
              1.3041214120015427,
              1.893491997791636 ...
            ]
          }
        ]
      },
      {
        "id": "c200000000000000000000000",
        "name": "Image Embedding V2 (CLIP ViT-B/32)",
        "dimensions": 512,
        "is_custom": false,
        "values": [
          {
            "value": [
              -0.4192526042461395,
              -0.048827216029167175 ...
            ]
          }
        ]
      }
    ],
    "experiments": {
      "a2fb1fec-83fb-0858-2cfb-98f4757eae95": {
        "name": "image_model_run_3e5bf56c-31a4-4eed-a731-9ca85e783225",
        "runs": {
          "a2fb1fec-b8a5-0f96-b61e-0d5643fb8dec": {
            "name": "iteration 1",
            "run_data_row_id": "42630da9-0dc4-462b-9aa0-5a17e85ff46a",
            "labels": [
              {
                "label_kind": "Default",
                "version": "1.0.0",
                "id": "clpk3h87e005r14vsp4c33sfq",
                "annotations": {
                  "objects": [
                    {
                      "feature_id": "164937d0-7bb2-0f03-8031-34b57799736b",
                      "feature_schema_id": "clpk3gs0t054y07vig605dopr",
                      "name": "polygon",
                      "value": "polygon",
                      "annotation_kind": "ImagePolygon",
                      "classifications": [],
                      "polygon": [
                        {
                          "x": 1489.581,
                          "y": 183.934
                        },
                        {
                          "x": 2278.306,
                          "y": 256.885
                        },
                        {
                          "x": 2428.197,
                          "y": 200.437
                        },
                        {
                          "x": 2560.0,
                          "y": 335.419
                        },
                        {
                          "x": 2557.386,
                          "y": 503.165
                        },
                        {
                          "x": 2320.596,
                          "y": 503.103
                        },
                        {
                          "x": 2156.083,
                          "y": 628.943
                        },
                        {
                          "x": 2161.111,
                          "y": 785.519
                        },
                        {
                          "x": 2002.115,
                          "y": 894.647
                        },
                        {
                          "x": 1838.456,
                          "y": 877.874
                        },
                        {
                          "x": 1436.53,
                          "y": 874.636
                        },
                        {
                          "x": 1411.403,
                          "y": 758.579
                        },
                        {
                          "x": 1353.853,
                          "y": 751.74
                        },
                        {
                          "x": 1345.264,
                          "y": 453.461
                        },
                        {
                          "x": 1426.011,
                          "y": 421.129
                        },
                        {
                          "x": 1489.581,
                          "y": 183.934
                        }
                      ]
                    },
                    {
                      "feature_id": "2738b5f8-e586-0090-974a-59b6c9b46ee2",
                      "feature_schema_id": "clpk3gs0t055207vi1baq8bke",
                      "name": "mask_with_text_subclass",
                      "value": "mask_with_text_subclass",
                      "annotation_kind": "ImageSegmentationMask",
                      "classifications": [
                        {
                          "feature_id": "9001b75c-bb70-0b40-afbf-9479ecfa9f99",
                          "feature_schema_id": "clpk3gs0t055307vi90s32rhi",
                          "name": "sub_free_text",
                          "value": "sub_free_text",
                          "text_answer": {
                            "content": "free text answer"
                          }
                        }
                      ],
                      "mask": {
                        "url": "https://api.labelbox.com/api/v1/models/a2fb1fec-83fb-0858-2cfb-98f4757eae95/annotations/2738b5f8-e586-0090-974a-59b6c9b46ee2/index/1/mask"
                      },
                      "composite_mask": {
                        "url": "https://api.labelbox.com/api/v1/tasks/clur6mb7a0dz107xod5w53p61/masks/clpk3h7uk005114vsgru34yw7/index/1",
                        "color_rgb": [
                          41,
                          24,
                          175
                        ]
                      }
                    },
                    {
                      "feature_id": "4defcaa5-623f-0589-932a-b703e1116fa4",
                      "feature_schema_id": "clpk3gs0t055807vi49uy1rcm",
                      "name": "polyline",
                      "value": "polyline",
                      "annotation_kind": "ImagePolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 2534.353,
                          "y": 249.471
                        },
                        {
                          "x": 2429.492,
                          "y": 182.092
                        },
                        {
                          "x": 2294.322,
                          "y": 221.962
                        },
                        {
                          "x": 2224.491,
                          "y": 180.463
                        },
                        {
                          "x": 2136.123,
                          "y": 204.716
                        },
                        {
                          "x": 1712.247,
                          "y": 173.949
                        },
                        {
                          "x": 1703.838,
                          "y": 84.438
                        },
                        {
                          "x": 1579.772,
                          "y": 82.61
                        },
                        {
                          "x": 1583.442,
                          "y": 167.552
                        },
                        {
                          "x": 1478.869,
                          "y": 164.903
                        },
                        {
                          "x": 1418.941,
                          "y": 318.149
                        },
                        {
                          "x": 1243.128,
                          "y": 400.815
                        },
                        {
                          "x": 1022.067,
                          "y": 319.007
                        },
                        {
                          "x": 892.367,
                          "y": 379.216
                        },
                        {
                          "x": 670.273,
                          "y": 364.408
                        },
                        {
                          "x": 613.114,
                          "y": 288.16
                        },
                        {
                          "x": 377.559,
                          "y": 238.251
                        },
                        {
                          "x": 368.087,
                          "y": 185.064
                        },
                        {
                          "x": 246.557,
                          "y": 167.286
                        },
                        {
                          "x": 236.648,
                          "y": 285.61
                        },
                        {
                          "x": 90.929,
                          "y": 326.412
                        }
                      ]
                    },
                    {
                      "feature_id": "7d129546-45a6-060b-918e-35caec234d31",
                      "feature_schema_id": "clpk3gs0s054s07vi3ddo8had",
                      "name": "bbox_with_radio_subclass",
                      "value": "bbox_with_radio_subclass",
                      "annotation_kind": "ImageBoundingBox",
                      "classifications": [
                        {
                          "feature_id": "cccba37c-b7a8-095b-8679-dd20aece6f90",
                          "feature_schema_id": "clpk3gs0s054t07vi17aih47e",
                          "name": "sub_radio_question",
                          "value": "sub_radio_question",
                          "radio_answer": {
                            "feature_id": "ea336ea5-041c-02b6-9e8b-6ac1098351b1",
                            "feature_schema_id": "clpk3gs0s054u07vicrttdlgj",
                            "name": "first_sub_radio_answer",
                            "value": "first_sub_radio_answer",
                            "classifications": []
                          }
                        }
                      ],
                      "bounding_box": {
                        "top": 933.0,
                        "left": 541.0,
                        "height": 191.0,
                        "width": 330.0
                      }
                    },
                    {
                      "feature_id": "a3e0496d-3963-0d40-af89-48a907fbb48d",
                      "feature_schema_id": "clpk3gs0s054q07vi3c30d1be",
                      "name": "bounding_box",
                      "value": "bounding_box",
                      "annotation_kind": "ImageBoundingBox",
                      "classifications": [],
                      "bounding_box": {
                        "top": 977.0,
                        "left": 1690.0,
                        "height": 330.0,
                        "width": 225.0
                      }
                    },
                    {
                      "feature_id": "f2fc7c1f-7f10-0fc2-b676-fa2591f51a27",
                      "feature_schema_id": "clpk3gs0t055607vi59sshqn0",
                      "name": "point",
                      "value": "point",
                      "annotation_kind": "ImagePoint",
                      "classifications": [],
                      "point": {
                        "x": 1166.606,
                        "y": 1441.768
                      }
                    },
                    {
                      "feature_id": "fb60f23f-8ae1-05b3-a4d8-353839975d59",
                      "feature_schema_id": "clpk3gs0t055007vi4h7fgm59",
                      "name": "mask",
                      "value": "mask",
                      "annotation_kind": "ImageSegmentationMask",
                      "classifications": [],
                      "mask": {
                        "url": "https://api.labelbox.com/api/v1/models/a2fb1fec-83fb-0858-2cfb-98f4757eae95/annotations/fb60f23f-8ae1-05b3-a4d8-353839975d59/index/1/mask"
                      },
                      "composite_mask": {
                        "url": "https://api.labelbox.com/api/v1/tasks/clur6mb7a0dz107xod5w53p61/masks/clpk3h7uk005114vsgru34yw7/index/1",
                        "color_rgb": [
                          186,
                          26,
                          40
                        ]
                      }
                    }
                  ],
                  "classifications": [
                    {
                      "feature_id": "1be71b85-08da-0ade-bbe4-cfdd431d3e8a",
                      "feature_schema_id": "clpk3gs0t055g07viaine2doh",
                      "name": "checklist_question",
                      "value": "checklist_question",
                      "checklist_answers": [
                        {
                          "feature_id": "0104948e-7be5-032d-80ca-87d052346618",
                          "feature_schema_id": "clpk3gs0t055j07vi4h3y9wso",
                          "name": "second_checklist_answer",
                          "value": "second_checklist_answer",
                          "classifications": []
                        },
                        {
                          "feature_id": "cb587406-5db3-0de2-99f0-cf6e6c344f43",
                          "feature_schema_id": "clpk3gs0t055h07vi1hp0acnb",
                          "name": "first_checklist_answer",
                          "value": "first_checklist_answer",
                          "classifications": []
                        }
                      ]
                    },
                    {
                      "feature_id": "5b1f4994-1bad-03a1-bf0f-1a0b4682ea25",
                      "feature_schema_id": "clpk3gs0t055m07vi53ow6auc",
                      "name": "free_text",
                      "value": "free_text",
                      "text_answer": {
                        "content": "sample text"
                      }
                    },
                    {
                      "feature_id": "88f8cd4d-e1ca-0add-a7e8-a7349def31bc",
                      "feature_schema_id": "clpk3gs0t055a07vifk5701ua",
                      "name": "radio_question",
                      "value": "radio_question",
                      "radio_answer": {
                        "feature_id": "ee141c0d-064f-0de8-b746-e61ac14fbe62",
                        "feature_schema_id": "clpk3gs0t055d07vibryq8f4l",
                        "name": "second_radio_answer",
                        "value": "second_radio_answer",
                        "classifications": []
                      }
                    },
                    {
                      "feature_id": "cc8e57d8-5080-0b43-bab1-6e4d78b684de",
                      "feature_schema_id": "clpk3gs0t055o07vifjxs63xi",
                      "name": "nested_radio_question",
                      "value": "nested_radio_question",
                      "radio_answer": {
                        "feature_id": "81690da4-5cb5-037f-9a7b-3dd7fefc5481",
                        "feature_schema_id": "clpk3gs0t055p07vigvkkeii3",
                        "name": "first_radio_answer",
                        "value": "first_radio_answer",
                        "classifications": [
                          {
                            "feature_id": "37453082-23dd-0c38-bdcb-90705102bc6e",
                            "feature_schema_id": "clpk3gs0t055q07vifyau26jb",
                            "name": "sub_radio_question",
                            "value": "sub_radio_question",
                            "radio_answer": {
                              "feature_id": "4b014520-7193-04d9-8d69-f800316b60f2",
                              "feature_schema_id": "clpk3gs0t055r07vi406q70ga",
                              "name": "first_sub_radio_answer",
                              "value": "first_sub_radio_answer",
                              "classifications": []
                            }
                          }
                        ]
                      }
                    },
                    {
                      "feature_id": "f1d18441-41cd-0ff7-9f4f-75384e5a7b51",
                      "feature_schema_id": "clpk3gs0u055w07vifjngez8o",
                      "name": "nested_checklist_question",
                      "value": "nested_checklist_question",
                      "checklist_answers": [
                        {
                          "feature_id": "a3c8745f-8151-0256-8b6d-8ac449797fd7",
                          "feature_schema_id": "clpk3gs0u055x07vibwdt1pk3",
                          "name": "first_checklist_answer",
                          "value": "first_checklist_answer",
                          "classifications": [
                            {
                              "feature_id": "aa00a3ff-1af4-038a-9be5-97a468656bc5",
                              "feature_schema_id": "clpk3gs0u055y07vi3srh1g0l",
                              "name": "sub_checklist_question",
                              "value": "sub_checklist_question",
                              "checklist_answers": [
                                {
                                  "feature_id": "0608a0d7-d9d9-0347-843b-3297e58a25dc",
                                  "feature_schema_id": "clpk3gs0u055z07vifkpvgap4",
                                  "name": "first_sub_checklist_answer",
                                  "value": "first_sub_checklist_answer",
                                  "classifications": []
                                }
                              ]
                            }
                          ]
                        }
                      ]
                    }
                  ],
                  "relationships": []
                }
              }
            ],
            "predictions": [
              {
                "label_kind": "Default",
                "version": "1.0.0",
                "id": "clpk3h87e005r14vsp4c33sfq",
                "annotations": {
                  "objects": [
                    {
                      "feature_id": "028389ae-93ff-4614-a37e-e867a310371c",
                      "feature_schema_id": "clpk3gs0s054q07vi3c30d1be",
                      "name": "bounding_box",
                      "value": "bounding_box",
                      "annotation_kind": "ImageBoundingBox",
                      "classifications": [],
                      "bounding_box": {
                        "top": 977.0,
                        "left": 1690.0,
                        "height": 330.0,
                        "width": 225.0
                      }
                    },
                    {
                      "feature_id": "37415b2f-f376-47b9-8f6a-6429c3efa6af",
                      "feature_schema_id": "clpk3gs0t054y07vig605dopr",
                      "name": "polygon",
                      "value": "polygon",
                      "annotation_kind": "ImagePolygon",
                      "classifications": [],
                      "polygon": [
                        {
                          "x": 1489.581,
                          "y": 183.934
                        },
                        {
                          "x": 2278.306,
                          "y": 256.885
                        },
                        {
                          "x": 2428.197,
                          "y": 200.437
                        },
                        {
                          "x": 2560.0,
                          "y": 335.419
                        },
                        {
                          "x": 2557.386,
                          "y": 503.165
                        },
                        {
                          "x": 2320.596,
                          "y": 503.103
                        },
                        {
                          "x": 2156.083,
                          "y": 628.943
                        },
                        {
                          "x": 2161.111,
                          "y": 785.519
                        },
                        {
                          "x": 2002.115,
                          "y": 894.647
                        },
                        {
                          "x": 1838.456,
                          "y": 877.874
                        },
                        {
                          "x": 1436.53,
                          "y": 874.636
                        },
                        {
                          "x": 1411.403,
                          "y": 758.579
                        },
                        {
                          "x": 1353.853,
                          "y": 751.74
                        },
                        {
                          "x": 1345.264,
                          "y": 453.461
                        },
                        {
                          "x": 1426.011,
                          "y": 421.129
                        },
                        {
                          "x": 1489.581,
                          "y": 183.934
                        }
                      ]
                    },
                    {
                      "feature_id": "55bee3a4-59c2-4d85-95b3-4584c315037e",
                      "feature_schema_id": "clpk3gs0t055607vi59sshqn0",
                      "name": "point",
                      "value": "point",
                      "annotation_kind": "ImagePoint",
                      "classifications": [],
                      "point": {
                        "x": 1166.606,
                        "y": 1441.768
                      }
                    },
                    {
                      "feature_id": "671736d6-7c30-4399-bec4-a8a1779abf4d",
                      "feature_schema_id": "clpk3gs0t055007vi4h7fgm59",
                      "name": "mask",
                      "value": "mask",
                      "annotation_kind": "ImageSegmentationMask",
                      "classifications": [],
                      "mask": {
                        "url": "https://api.labelbox.com/api/v1/models/a2fb1fec-83fb-0858-2cfb-98f4757eae95/annotations/671736d6-7c30-4399-bec4-a8a1779abf4d/index/1/mask"
                      },
                      "composite_mask": {
                        "url": "https://api.labelbox.com/api/v1/tasks/clur6mb7a0dz107xod5w53p61/masks/clpk3gxbq00oi14ok1yykgp85/index/1",
                        "color_rgb": [
                          36,
                          67,
                          66
                        ]
                      }
                    },
                    {
                      "feature_id": "7e103ed1-f66f-4cf9-9a36-3e08be3a9648",
                      "feature_schema_id": "clpk3gs0t055807vi49uy1rcm",
                      "name": "polyline",
                      "value": "polyline",
                      "annotation_kind": "ImagePolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 2534.353,
                          "y": 249.471
                        },
                        {
                          "x": 2429.492,
                          "y": 182.092
                        },
                        {
                          "x": 2294.322,
                          "y": 221.962
                        },
                        {
                          "x": 2224.491,
                          "y": 180.463
                        },
                        {
                          "x": 2136.123,
                          "y": 204.716
                        },
                        {
                          "x": 1712.247,
                          "y": 173.949
                        },
                        {
                          "x": 1703.838,
                          "y": 84.438
                        },
                        {
                          "x": 1579.772,
                          "y": 82.61
                        },
                        {
                          "x": 1583.442,
                          "y": 167.552
                        },
                        {
                          "x": 1478.869,
                          "y": 164.903
                        },
                        {
                          "x": 1418.941,
                          "y": 318.149
                        },
                        {
                          "x": 1243.128,
                          "y": 400.815
                        },
                        {
                          "x": 1022.067,
                          "y": 319.007
                        },
                        {
                          "x": 892.367,
                          "y": 379.216
                        },
                        {
                          "x": 670.273,
                          "y": 364.408
                        },
                        {
                          "x": 613.114,
                          "y": 288.16
                        },
                        {
                          "x": 377.559,
                          "y": 238.251
                        },
                        {
                          "x": 368.087,
                          "y": 185.064
                        },
                        {
                          "x": 246.557,
                          "y": 167.286
                        },
                        {
                          "x": 236.648,
                          "y": 285.61
                        },
                        {
                          "x": 90.929,
                          "y": 326.412
                        }
                      ]
                    },
                    {
                      "feature_id": "f3a445af-0475-4ad1-892b-fbbcce26073f",
                      "feature_schema_id": "clpk3gs0s054s07vi3ddo8had",
                      "name": "bbox_with_radio_subclass",
                      "value": "bbox_with_radio_subclass",
                      "annotation_kind": "ImageBoundingBox",
                      "classifications": [
                        {
                          "feature_id": "d8427b71-d516-4686-bc52-660d1133c18d",
                          "feature_schema_id": "clpk3gs0s054t07vi17aih47e",
                          "name": "sub_radio_question",
                          "value": "sub_radio_question",
                          "radio_answer": {
                            "feature_id": "494d3af6-9bf8-477f-8827-dac0dd4c42a3",
                            "feature_schema_id": "clpk3gs0s054u07vicrttdlgj",
                            "name": "first_sub_radio_answer",
                            "value": "first_sub_radio_answer",
                            "classifications": []
                          }
                        }
                      ],
                      "bounding_box": {
                        "top": 933.0,
                        "left": 541.0,
                        "height": 191.0,
                        "width": 330.0
                      }
                    }
                  ],
                  "classifications": [
                    {
                      "feature_id": "22778fd4-3fd3-48ad-9007-a8f2fc22d664",
                      "feature_schema_id": "clpk3gs0t055o07vifjxs63xi",
                      "name": "nested_radio_question",
                      "value": "nested_radio_question",
                      "radio_answer": {
                        "feature_id": "b992d812-7d0b-49b1-82da-14bcff5a76ef",
                        "feature_schema_id": "clpk3gs0t055p07vigvkkeii3",
                        "name": "first_radio_answer",
                        "value": "first_radio_answer",
                        "classifications": [
                          {
                            "feature_id": "877d229e-c877-4aa7-a034-ddde21fa8417",
                            "feature_schema_id": "clpk3gs0t055q07vifyau26jb",
                            "name": "sub_radio_question",
                            "value": "sub_radio_question",
                            "radio_answer": {
                              "feature_id": "803cbe24-7c2f-400c-8d73-fb19bdc5302c",
                              "feature_schema_id": "clpk3gs0t055r07vi406q70ga",
                              "name": "first_sub_radio_answer",
                              "value": "first_sub_radio_answer",
                              "classifications": []
                            }
                          }
                        ]
                      }
                    },
                    {
                      "feature_id": "2555cb5f-5a90-40f9-987f-7a2899688ad6",
                      "feature_schema_id": "clpk3gs0t055a07vifk5701ua",
                      "name": "radio_question",
                      "value": "radio_question",
                      "radio_answer": {
                        "feature_id": "246406e7-162c-43c5-a1ff-ca555be51e55",
                        "feature_schema_id": "clpk3gs0t055d07vibryq8f4l",
                        "name": "second_radio_answer",
                        "value": "second_radio_answer",
                        "classifications": []
                      }
                    },
                    {
                      "feature_id": "e4f143b7-5e3d-4f17-8202-efc78cd73f6e",
                      "feature_schema_id": "clpk3gs0t055m07vi53ow6auc",
                      "name": "free_text",
                      "value": "free_text",
                      "text_answer": {
                        "content": "sample text"
                      }
                    },
                    {
                      "feature_id": "ec7d6588-3c15-4fb6-baa1-3508c476bc32",
                      "feature_schema_id": "clpk3gs0u055w07vifjngez8o",
                      "name": "nested_checklist_question",
                      "value": "nested_checklist_question",
                      "checklist_answers": [
                        {
                          "feature_id": "71fd6407-5fc6-43e4-8196-78487f0f9ec8",
                          "feature_schema_id": "clpk3gs0u055x07vibwdt1pk3",
                          "name": "first_checklist_answer",
                          "value": "first_checklist_answer",
                          "classifications": [
                            {
                              "feature_id": "ab8bf663-3fba-4dd6-9e09-84e0bbd713d0",
                              "feature_schema_id": "clpk3gs0u055y07vi3srh1g0l",
                              "name": "sub_checklist_question",
                              "value": "sub_checklist_question",
                              "checklist_answers": [
                                {
                                  "feature_id": "58c98b94-b360-4ceb-b5ea-7765e096c40a",
                                  "feature_schema_id": "clpk3gs0u055z07vifkpvgap4",
                                  "name": "first_sub_checklist_answer",
                                  "value": "first_sub_checklist_answer",
                                  "classifications": []
                                }
                              ]
                            }
                          ]
                        }
                      ]
                    },
                    {
                      "feature_id": "f9107ba1-8020-4b7a-8066-b80167a9cd6a",
                      "feature_schema_id": "clpk3gs0t055g07viaine2doh",
                      "name": "checklist_question",
                      "value": "checklist_question",
                      "checklist_answers": [
                        {
                          "feature_id": "88cc2ed7-8b19-4396-b9ee-36949d23fac7",
                          "feature_schema_id": "clpk3gs0t055h07vi1hp0acnb",
                          "name": "first_checklist_answer",
                          "value": "first_checklist_answer",
                          "classifications": []
                        },
                        {
                          "feature_id": "b70e6eb1-2ee9-42f3-b47a-a56e764ea4ac",
                          "feature_schema_id": "clpk3gs0t055j07vi4h3y9wso",
                          "name": "second_checklist_answer",
                          "value": "second_checklist_answer",
                          "classifications": []
                        }
                      ]
                    }
                  ],
                  "relationships": []
                }
              }
            ]
          }
        }
      }
    },
    "models": {}
  }
  ```
</CodeGroup>


# Export LLM response evaluation annotations
Source: https://docs.labelbox.com/horizon/developer-guides/export-llm-response-evaluation-annotations

How to export LLM response evaluation annotations and sample export formats.

<CardGroup>
  <Card title="Open In Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/exports/export_data.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/exports/export_data.ipynb" />
</CardGroup>

## Export JSON annotations

<CodeGroup>
  ```python Export theme={"theme":"css-variables"}
  # Set the export params to include/exclude certain fields.
  export_params= {
    "attachments": True,
    "metadata_fields": True,
    "data_row_details": True,
    "project_details": True,
    "label_details": True,
    "performance_details": True,
    "interpolated_frames": True
  }

  # Note: Filters follow AND logic, so typically using one filter is sufficient.
  filters= {
    "last_activity_at": ["2000-01-01 00:00:00", "2050-01-01 00:00:00"],
    "workflow_status": "<wkf-status>"
  }

  export_task = project.export(params=export_params, filters=filters)
  export_task.wait_till_done()

  # Stream the export using a callback function
  def json_stream_handler(output: labelbox.BufferedJsonConverterOutput):
    print(output.json)

  export_task.get_buffered_stream(stream_type=labelbox.StreamType.RESULT).start(stream_handler=json_stream_handler)

  # Collect all exported data into a list
  export_json = [data_row.json for data_row in export_task.get_buffered_stream()]

  print("file size: ", export_task.get_total_file_size(stream_type=lb.StreamType.RESULT))
  print("line count: ", export_task.get_total_lines(stream_type=lb.StreamType.RESULT))

  ```
</CodeGroup>

## Annotation export formats

### Classification - Radio

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
  		"feature_id": "clp8gzc7a000h07iu2uqcfziq",
  		"feature_schema_id": "clp8fepqc02nd071phl4tck59",
  		"name": "Choose the best response",
  		"radio_answer": {
  				"feature_id": "clp8gzc7a000i07iu46z39uwl",
  				"feature_schema_id": "clp8fepqc02ng071p9ufm76xj",
  				"name": "Response B",
  				"classifications": []
  		}
  }
  ```
</CodeGroup>

### Classification - Free-form text

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
      "feature_id": "clph8rvw700013b6pbsq1lwcg",
      "feature_schema_id": "clp8fepqc02nl071p8z948pd4",
      "name": "Provide a reason for your choice",
      "text_answer": {
          "content": "Clear recommendation"
      }
  }
  ```
</CodeGroup>

### Classification - Checklist

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
      "feature_id": "clpiyav9m00053b6mpe2qk0im",
      "feature_schema_id": "clpiya8h507dx072efo9kb97j",
      "name": "Evaluations done",
      "checklist_answers": [
          {
              "feature_id": "clpiyav9m00043b6mvhur1p19",
              "name": "Usefulness",
              "classifications": []
          },
          {
              "feature_id": "clpiyax4d00073b6mqyyt0504",
              "name": "Toxicity",
              "classifications": []
          }
      ]
  }
  ```
</CodeGroup>

## Sample project export

<CodeGroup>
  ```json JSON expandable theme={"theme":"css-variables"}
  {
      "data_row": {
          "id": "clpiqgsvn4lxr079096g5mt9v",
          "global_key": "0914c7da-5999-464b-b621-054a5247cf48",
          "row_data": "https://storage.googleapis.com/labelbox-developer-testing-assets/Pranoy/shopping_3.json",
          "details": {
              "dataset_id": "clpiqgrsr00f60737sqeiaf4v",
              "dataset_name": "pairwise_demo_42454af2-b7b5-45c0-a25a-90f97efd49be",
              "created_at": "2023-11-28T19:30:49.729+00:00",
              "updated_at": "2023-11-28T19:30:50.609+00:00",
              "last_activity_at": "2023-11-28T23:11:08.000+00:00",
              "created_by": "[email protected]"
          }
      },
      "media_attributes": {
          "mime_type": "application/vnd.labelbox.conversational",
          "labelable_ids": [
              "message-1",
              "message-3"
          ],
          "message_count": 4
      },
      "attachments": [],
      "metadata_fields": [],
      "projects": {
          "clpiy2itz08vg0705akh60zc1": {
              "name": "Test LLM Human Preference",
              "labels": [
                  {
                      "label_kind": "Default",
                      "version": "1.0.0",
                      "id": "clpiyah63071f07e4756yer2d",
                      "label_details": {
                          "created_at": "2023-11-28T23:11:08.000+00:00",
                          "updated_at": "2023-11-28T23:11:07.000+00:00",
                          "created_by": "[email protected]",
                          "content_last_updated_at": "2023-11-28T23:11:07.568+00:00",
                          "reviews": []
                      },
                      "annotations": {
                          "objects": [],
                          "classifications": [
                              {
                                  "feature_id": "clpiybl7h000g3b6mil1amjme",
                                  "feature_schema_id": "clpiqh0v104mq0729homqffup",
                                  "name": "Choose the best response",
                                  "radio_answer": {
                                      "feature_id": "clpiybl7h000f3b6m03fn8ry3",
                                      "feature_schema_id": "clpiqh0v104mt0729atflgvvw",
                                      "name": "Response B",
                                      "classifications": []
                                  }
                              },
                              {
                                  "feature_id": "clpiybzp3000h3b6m9uowaci4",
                                  "feature_schema_id": "clpiqh0v104my072907gt7id0",
                                  "name": "Provide a reason for your choice",
                                  "text_answer": {
                                      "content": "Direct approach that is convenient"
                                  }
                              },
                              {
                                  "feature_id": "clpiyc10q000j3b6m01i2d1pv",
                                  "feature_schema_id": "clpiya8h507dx072efo9kb97j",
                                  "name": "Evaluations done",
                                  "checklist_answers": [
                                      {
                                          "feature_id": "clpiyc10q000i3b6mhhvhdzqs",
                                          "name": "Relevance",
                                          "classifications": []
                                      },
                                      {
                                          "feature_id": "clpiyc1ie000l3b6m3kpff0lp",
                                          "name": "Usefulness",
                                          "classifications": []
                                      }
                                  ]
                              }
                          ],
                          "relationships": []
                      }
                  }
              ],
              "project_details": {
                  "ontology_id": "clpiqh0ud04mp0729adeoccsl",
                  "task_id": "33e77b4d-424f-49ba-8792-628c1cb746b7",
                  "task_name": "Initial review task",
                  "batch_id": "66882c10-8e42-11ee-8f8d-71857f06db2f",
                  "batch_name": "test_llm_human_preference_batch_1",
                  "workflow_status": "IN_REVIEW",
                  "priority": 5,
                  "consensus_expected_label_count": 1,
                  "workflow_history": [
                      {
                          "action": "Move",
                          "created_at": "2023-11-28T23:11:07.654+00:00",
                          "created_by": "[email protected]",
                          "previous_task_name": "Initial labeling task",
                          "previous_task_id": "8f9917a7-6030-0914-befc-ac06f59a7ec9",
                          "next_task_name": "Initial review task",
                          "next_task_id": "33e77b4d-424f-49ba-8792-628c1cb746b7"
                      },
                      {
                          "action": "Move",
                          "created_at": "2023-11-28T23:11:07.631+00:00",
                          "created_by": "[email protected]",
                          "next_task_name": "Initial labeling task",
                          "next_task_id": "8f9917a7-6030-0914-befc-ac06f59a7ec9"
                      }
                  ]
              },
              "project_tags": []
          }
      }
  }
  ```
</CodeGroup>


# Export multimodal chat annotations
Source: https://docs.labelbox.com/horizon/developer-guides/export-multimodal-chat-annotations

How to export multimodal chat annotations and sample export formats.

<CardGroup>
  <Card title="Open In Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/exports/export_data.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/exports/export_data.ipynb" />
</CardGroup>

## Export JSON annotations

<CodeGroup>
  ```python Export theme={"theme":"css-variables"}
  # Set the export params to include/exclude certain fields.
  export_params= {
    "attachments": True,
    "metadata_fields": True,
    "data_row_details": True,
    "project_details": True,
    "label_details": True,
    "performance_details": True,
    "interpolated_frames": True
  }

  # Note: Filters follow AND logic, so typically using one filter is sufficient.

  filters= {
    "last_activity_at": ["2000-01-01 00:00:00", "2050-01-01 00:00:00"],
    "workflow_status": "<wkf-status>"
  }

  export_task = project.export(params=export_params, filters=filters)
  export_task.wait_till_done()

  # Stream the export using a callback function

  def json_stream_handler(output: labelbox.BufferedJsonConverterOutput):
  print(output.json)

  export_task.get_buffered_stream(stream_type=labelbox.StreamType.RESULT).start(stream_handler=json_stream_handler)

  # Collect all exported data into a list

  export_json = [data_row.json for data_row in export_task.get_buffered_stream()]

  print("file size: ", export_task.get_total_file_size(stream_type=lb.StreamType.RESULT))
  print("line count: ", export_task.get_total_lines(stream_type=lb.StreamType.RESULT))

  ```
</CodeGroup>

## Annotation export formats

### Classification - Radio

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "clws3nkhh00043b6p7qktjffs",
    "feature_schema_id": "clws3n1yu055907wlg1uh1i8g",
    "name": "Context subject",
    "value": "context_subject",
    "message_id": "clws2arr200073b6p7ehwkoh4",
    "conversational_radio_answer": {
      "feature_id": "clws3nkhh00033b6ps99i1d56",
      "feature_schema_id": "clws3n1yu055a07wl2axp39nq",
      "name": "Car",
      "value": "car",
      "classifications": []
    }
  }
  ```
</CodeGroup>

### Classification - Free-form text

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "clws3or5q00093b6p00oxtkve",
    "feature_schema_id": "clws3n1yv055v07wlbnd57c9h",
    "name": "Additional comment",
    "value": "additional_comment",
    "text_answer": {
      "content": "Color: white"
    }
  }
  ```
</CodeGroup>

### Classification - Checklist

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "clpiyav9m00053b6mpe2qk0im",
    "feature_schema_id": "clpiya8h507dx072efo9kb97j",
    "name": "Evaluations done",
    "checklist_answers": [
      {
        "feature_id": "clpiyav9m00043b6mvhur1p19",
        "name": "Usefulness",
        "classifications": []
      },
      {
        "feature_id": "clpiyax4d00073b6mqyyt0504",
        "name": "Toxicity",
        "classifications": []
      }
    ]
  }
  ```
</CodeGroup>

### Message Ranking

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "clwrt25ar00093b6po9g7fiu3",
    "feature_schema_id": "clwkfbcnn01kl07z68er64ffi",
    "name": "Rank given prompt",
    "annotation_kind": "MessageRanking",
    "classifications": [],
    "ranked_messages": {
      "parent_message_id": "clwrt16zz00033b6p6ud7w233",
      "ranked_messages": [
        {
          "message_id": "clwrt1fw700063b6pe8zpo1xp",
          "order": 1,
          "model_config_id": "e606593e-def3-4662-9737-3fc4ac412dd3"
        },
        {
          "message_id": "clwrt1fw700073b6p8k3vkh1x",
          "order": 2,
          "model_config_id": "8fcda464-a0c7-4300-a9fa-c9a5cf82cab2"
        }
      ]
    }
  }
  ```
</CodeGroup>

### Message selection

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "clwrt2dru000b3b6pkj3i5zxk",
    "feature_schema_id": "clwknckmp030v07xp6kow09xd",
    "name": "Subject relevance",
    "annotation_kind": "MessageMultiSelection",
    "classifications": [],
    "selected_messages": {
      "parent_message_id": "clwrt16zz00033b6p6ud7w233",
      "messages": [
        {
          "message_id": "clwrt1fw700063b6pe8zpo1xp",
          "model_config_id": "e606593e-def3-4662-9737-3fc4ac412dd3"
        }
      ]
    }
  }
  ```
</CodeGroup>

### Message step reasoning

<CodeGroup>
  ```json JSON expandable theme={"theme":"css-variables"}
  {
      "feature_id": "cm3gp5d39000k3b6tpet38ui9",
      "feature_schema_id": "cm3ffq1mm04t407we109o1hyi",
      "name": "my message step task",
      "annotation_kind": "StepReasoning",
      "classifications": [],
      "steps": {
          "parent_message_id": "cm3gp54f500073b6t26gyznra",
          "steps": [
              {
                  "step_id": "cm3gp54q6000a3b6t4563k4ba",
                  "content": "Here's a simple way to make a cup of tea:",
                  "variant_id": 0,
                  "variant": "correct"
              },
              {
                  "step_id": "cm3gp54q6000b3b6tsuiaudjb",
                  "content": "1. Boil water: Fill a kettle with water and bring it to a boil.",
                  "variant_id": 0,
                  "variant": "correct"
              },
              {
                  "step_id": "cm3gp54q6000d3b6tozd9neiq",
                  "content": "2. Add tea: Once the water is boiling, empty your now preheated teacup and add your tea. This can be a tea bag or loose tea leaves in a tea infuser. The amount of tea you add will depend on how strong you like your tea.",
                  "variant_id": 0,
                  "variant": "correct"
              },
              {
                  "step_id": "cm3gp54q6000e3b6t8y4k7grw",
                  "content": "3. Pour hot water: Pour the boiling water into the cup, over the tea.",
                  "variant_id": 0,
                  "variant": "correct"
              },
              {
                  "step_id": "cm3gp54q6000f3b6tddynpg1y",
                  "content": "4. Steep: Let the tea steep for a few minutes. The exact time will depend on the type of tea and how strong you like it. A general guide is 3-5 minutes. After the tea has steeped, remove the tea bag or infuser from the cup.",
                  "variant_id": 0,
                  "variant": "correct"
              },
              {
                  "step_id": "cm3gp54q6000h3b6t7mopupfy",
                  "content": "5. Add extras: If you like, you can now add extras such as milk, sugar, honey, or lemon. ",
                  "variant_id": 0,
                  "variant": "correct"
              }
          ],
          "root_step_id": "cm3gp54q6000a3b6t4563k4ba"
      }
  },
  {
      "feature_id": "cm3gp8byj00103b6t9904n65h",
      "feature_schema_id": "cm3ffq1mm04t407we109o1hyi",
      "name": "my message step task",
      "annotation_kind": "StepReasoning",
      "classifications": [],
      "steps": {
          "parent_message_id": "cm3gp82di000o3b6tr13dwupc",
          "steps": [
              {
                  "step_id": "cm3gp82mx000p3b6tlkd0c8a6",
                  "content": "Sure, here's more detail on step 5:",
                  "variant_id": 0,
                  "variant": "correct"
              },
              {
                  "step_id": "cm3gp82mx000q3b6tptjgz0ol",
                  "content": "5. Add extras: ",
                  "variant_id": 0,
                  "variant": "correct"
              },
              {
                  "step_id": "cm3gp82mx000r3b6t7mqcilx6",
                  "content": "   - Milk: Some people prefer to add a splash of milk to their tea to make it creamier and to soften the flavor. The amount of milk can vary based on personal preference. Some teas, like black teas, often pair well with milk. ",
                  "variant_id": 0,
                  "variant": "correct"
              },
              {
                  "step_id": "cm3gp82mx000s3b6tntnvhpfc",
                  "content": "   - Sugar: If you prefer your tea sweet, you can add sugar. Start with a teaspoon and then adjust to taste. You can also use sweeteners like stevia or sucralose if you're trying to avoid sugar.",
                  "variant_id": 2,
                  "variant": "corrected",
                  "alternative_steps": [
                      {
                          "content": "   - Agave: If you prefer your tea sweet, you can add agave syrup. Start with a teaspoon and then adjust to taste. Agave is a natural sweetener that is sweeter than sugar, so you might need less of it.",
                          "variant_id": 0,
                          "variant": "neutral"
                      },
                      {
                          "content": "   - Maple Syrup: If you prefer your tea sweet, you can add maple syrup. Start with a teaspoon and then adjust to taste. Maple syrup can add a unique, slightly smoky sweetness to your tea.",
                          "variant_id": 0,
                          "variant": "correct"
                      },
                      {
                          "content": "   - Dates: If you prefer your tea sweet, you can blend dates and add the paste to your tea. Dates are a natural sweetener and can also add a bit of texture to your tea.",
                          "variant_id": 1,
                          "variant": "neutral"
                      }
                  ],
                  "rewritten_content": "Sugar: If you prefer sweet tea, add sugar or other sweeteners.",
                  "justification": "The response provided recommendations that pose health risks we should avoid."
              },
              {
                  "step_id": "cm3gpcrij00133b6tz9766jfw",
                  "content": "- Sugar: Start with a teaspoon of sugar, stir it in and then taste. Add more if needed. Remember, you can always add more sugar but you can't take it away once it's added.",
                  "variant_id": 0,
                  "variant": "correct"
              },
              {
                  "step_id": "cm3gpcrij00153b6t0woknwgo",
                  "content": "- Lemon: A slice of lemon or a splash of lemon juice can add a refreshing citrus note to your tea. This is often paired with black tea.",
                  "variant_id": 0,
                  "variant": "correct"
              },
              {
                  "step_id": "cm3gpcrij00163b6tcetum5w1",
                  "content": "- Herbs and Spices: Some people like to add herbs and spices to their tea. For example, mint leaves, a cinnamon stick, or a few cardamom pods can add a unique flavor to your tea.",
                  "variant_id": 0,
                  "variant": "correct"
              }
          ],
          "root_step_id": "cm3gp82mx000p3b6tlkd0c8a6"
      }
  }
  ```
</CodeGroup>

### Prompt rating

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
      "feature_id": "cm46ljjhs000i3b6tan025id0",
      "feature_schema_id": "cm46l1i4n02yp07s99isra6v9",
      "name": "Rate",
      "annotation_kind": "PromptIssue",
      "classifications": [
          {
              "feature_id": "cm46ljm67000k3b6t34goxv5r",
              "feature_schema_id": "cm46l1i4n02yq07s9g32o677i",
              "name": "Rate the prompt",
              "value": "rate_the_prompt",
              "checklist_answers": [
                  {
                      "feature_id": "cm46ljm67000j3b6tpr2rvaip",
                      "feature_schema_id": "cm46l1i4n02yt07s9haoj7aek",
                      "name": "This prompt contains a false, offensive, or controversial premise (eg. \u201cwhy does 1+1=3\u201d?)",
                      "value": "this_prompt_contains_a_false_offensive_or_controversial_premise_eg_why_does_1_1_3",
                      "classifications": []
                  }
              ]
          }
      ],
      "prompt_message_id": "cm46li41v000d3b6tdqzc9nv2"
  },
  {
      "feature_id": "cm46lk2pr000q3b6ti5c02fbb",
      "feature_schema_id": "cm46l1i4n02yp07s99isra6v9",
      "name": "Rate",
      "annotation_kind": "PromptIssue",
      "classifications": [
          {
              "feature_id": "cm46lk3pv000s3b6tzlpvifq5",
              "feature_schema_id": "cm46l1i4n02yq07s9g32o677i",
              "name": "Rate the prompt",
              "value": "rate_the_prompt",
              "checklist_answers": [
                  {
                      "feature_id": "cm46lk3pv000r3b6t0vueiz56",
                      "feature_schema_id": "cm46l1i4n02yv07s9ccti4qv9",
                      "name": "This prompt is not self-contained, i.e. the prompt cannot be understood without additional context about previous turns, account information or images.",
                      "value": "this_prompt_is_not_self_contained_i_e_the_prompt_cannot_be_understood_without_additional_context_about_previous_turns_account_information_or_images",
                      "classifications": []
                  }
              ]
          }
      ],
      "prompt_message_id": "cm46ljwcy000l3b6tmk9oua72"
  }
  ```
</CodeGroup>

### Fact-checking

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
      "feature_id": "cm46myu0200193b6ttjhe11n5",
      "feature_schema_id": "cm46m7j0k02t707uaesr2fhjq",
      "name": "Fact-checking",
      "annotation_kind": "FactChecking",
      "classifications": [],
      "steps": {
          "parent_message_id": "cm46mvig0000z3b6t4c1qzqbr",
          "steps": [
              {
                  "content": "The Triassic period, which occurred approximately 252 to 201 million years ago, was a time of significant evolutionary change. While many species from that era have gone extinct, some lineages have persisted to the present day. Here are a few examples of animals and plants from the Triassic period that have descendants or close relatives still existing today:",
                  "variant": ""
              },
              {
                  "content": "1. **Crocodilians**: The ancestors of modern crocodiles and alligators first appeared in the Triassic period. While the specific species from that time are extinct, their descendants have survived and evolved into the crocodilians we know today.",
                  "variant_id": 4,
                  "variant": "cant_confidently_assess"
              },
              {
                  "content": "2. **Turtles**: Turtles also have ancient origins, with some of their earliest ancestors appearing in the late Triassic period. Modern turtles are part of this long lineage.",
                  "variant_id": 1,
                  "variant": "inaccurate",
                  "justification": "Modern turtles are different from Eunotosaurus in the Triassic period."
              },
              {
                  "content": "3. **Lepidosaurs**: This group includes modern lizards and snakes. Their ancestors were present during the Triassic period, and they have since diversified into the many species we see today.",
                  "variant_id": 4,
                  "variant": "cant_confidently_assess"
              },
              {
                  "content": "4. **Horsetails and Ferns**: These plants were common during the Triassic period and have survived to the present day. While the specific species from that time are not the same, the groups themselves have persisted.",
                  "variant_id": 4,
                  "variant": "cant_confidently_assess"
              },
              {
                  "content": "5. **Ginkgo**: The Ginkgo tree, often referred to as a \"living fossil,\" has ancient origins dating back to the Triassic period. The modern Ginkgo biloba is the only surviving species of this once-diverse group.",
                  "variant_id": 0,
                  "variant": "accurate",
                  "justification": "Ginkgo first appeared over 290 million years ago, which is the Triassic period."
              },
              {
                  "content": "6. **Conifers**: These gymnosperms were already present during the Triassic period and have continued to thrive, with many species existing today.",
                  "variant_id": 4,
                  "variant": "cant_confidently_assess"
              },
              {
                  "content": "These examples illustrate the resilience and adaptability of certain lineages that have managed to survive through multiple mass extinctions and environmental changes over millions of years.",
                  "variant_id": 3,
                  "variant": "unsupported"
              }
          ]
      }
  }
  ```
</CodeGroup>

## Sample project export

<CodeGroup>
  ```json JSON expandable theme={"theme":"css-variables"}
  {
    "data_row": {
      "id": "clwrm93488cpd0734f740r9lr",
      "global_key": "clwrm92sb0grg07wncr6dhihw",
      "row_data": "{\"type\":\"application/vnd.labelbox.conversational.model-chat-evaluation\",\"version\":2,\"actors\":{\"clwrt16zz00023b6pg6hlhtub\":{\"role\":\"human\",\"metadata\":{\"name\":\"User\"}},\"clwrt17a900043b6py2tepemd\":{\"role\":\"model\",\"metadata\":{\"modelConfigId\":\"e606593e-def3-4662-9737-3fc4ac412dd3\"}},\"clwrt17a900053b6p0xtyv1im\":{\"role\":\"model\",\"metadata\":{\"modelConfigId\":\"8fcda464-a0c7-4300-a9fa-c9a5cf82cab2\"}}},\"messages\":{\"clwrt16zz00033b6p6ud7w233\":{\"actorId\":\"clwrt16zz00023b6pg6hlhtub\",\"content\":[{\"type\":\"text\",\"content\":\"Is there a car in this image?\"},{\"type\":\"fileData\",\"fileUri\":\"https://c7.staticflickr.com/3/2447/3799589580_4b73283f92_o.jpg\",\"mimeType\":\"image/jpeg\"}],\"childMessageIds\":[\"clwrt1fw700063b6pe8zpo1xp\",\"clwrt1fw700073b6p8k3vkh1x\"]},\"clwrt1fw700063b6pe8zpo1xp\":{\"actorId\":\"clwrt17a900043b6py2tepemd\",\"content\":[{\"type\":\"text\",\"content\":\"Yes, there is a car in this image. It appears to be a white hatchback parked in a parking lot.\"}],\"childMessageIds\":[]},\"clwrt1fw700073b6p8k3vkh1x\":{\"actorId\":\"clwrt17a900053b6p0xtyv1im\",\"content\":[{\"type\":\"text\",\"content\":\" Yes, there is a white Volkswagen Golf in the image.\"}],\"childMessageIds\":[]}},\"rootMessageIds\":[\"clwrt16zz00033b6p6ud7w233\"]}",
      "details": {
        "dataset_id": "clwkf82ul001l0755mhckcmuy",
        "dataset_name": "Model evaluation 101",
        "created_at": "2024-05-29T09:20:43.342+00:00",
        "updated_at": "2024-05-29T12:31:52.027+00:00",
        "last_activity_at": "2024-05-29T12:33:21.203+00:00",
        "created_by": "[email protected]"
      }
    },
    "media_attributes": {
      "asset_type": "conversational",
      "mime_type": "application/vnd.labelbox.conversational.model-chat-evaluation",
      "labelable_ids": [
        "clwrt16zz00033b6p6ud7w233",
        "clwrt1fw700063b6pe8zpo1xp",
        "clwrt1fw700073b6p8k3vkh1x"
      ],
      "message_count": 3
    },
    "attachments": [],
    "metadata_fields": [],
    "projects": {
      "clwrm92jc0grc07wncq654q5s": {
        "name": "Multi modal chat evalution",
        "labels": [
          {
            "label_kind": "Default",
            "version": "1.0.0",
            "id": "clwrsyrdp02mr07x5378j3ysr",
            "label_details": {
              "created_at": "2024-05-29T12:33:21.000+00:00",
              "updated_at": "2024-05-29T12:33:21.000+00:00",
              "created_by": "[email protected]",
              "content_last_updated_at": "2024-05-29T12:33:20.787+00:00",
              "reviews": []
            },
            "performance_details": {
              "seconds_to_create": 250,
              "seconds_to_review": 0,
              "skipped": false
            },
            "annotations": {
              "objects": [
                {
                  "feature_id": "clwrt25ar00093b6po9g7fiu3",
                  "feature_schema_id": "clwkfbcnn01kl07z68er64ffi",
                  "name": "Rank given prompt",
                  "annotation_kind": "MessageRanking",
                  "classifications": [],
                  "ranked_messages": {
                    "parent_message_id": "clwrt16zz00033b6p6ud7w233",
                    "ranked_messages": [
                      {
                        "message_id": "clwrt1fw700063b6pe8zpo1xp",
                        "order": 1,
                        "model_config_id": "e606593e-def3-4662-9737-3fc4ac412dd3"
                      },
                      {
                        "message_id": "clwrt1fw700073b6p8k3vkh1x",
                        "order": 2,
                        "model_config_id": "8fcda464-a0c7-4300-a9fa-c9a5cf82cab2"
                      }
                    ]
                  }
                },
                {
                  "feature_id": "clwrt2dru000b3b6pkj3i5zxk",
                  "feature_schema_id": "clwknckmp030v07xp6kow09xd",
                  "name": "Subject relevance",
                  "annotation_kind": "MessageMultiSelection",
                  "classifications": [],
                  "selected_messages": {
                    "parent_message_id": "clwrt16zz00033b6p6ud7w233",
                    "messages": [
                      {
                        "message_id": "clwrt1fw700063b6pe8zpo1xp",
                        "model_config_id": "e606593e-def3-4662-9737-3fc4ac412dd3"
                      }
                    ]
                  }
                }
              ],
              "classifications": [],
              "relationships": []
            }
          }
        ],
        "project_details": {
          "ontology_id": "clwkn6leb034b07waeolfb7tl",
          "task_id": "0c392f64-bc65-44e6-8848-ae5a12000328",
          "task_name": "Initial review task",
          "batch_id": "c2267a60-1d9c-11ef-9edf-ed6f8eede535",
          "batch_name": "batch_clwrm92jc0grc07wncq654q5s_clwrm9eph006d0750b3s5b2cr",
          "workflow_status": "IN_REVIEW",
          "priority": 5,
          "consensus_expected_label_count": 2,
          "workflow_history": [
            {
              "action": "Move",
              "created_at": "2024-05-29T12:33:21.160+00:00",
              "created_by": "[email protected]",
              "previous_task_name": "Initial labeling task",
              "previous_task_id": "ebf41e2b-e231-0e43-adee-92252c1858dd",
              "next_task_name": "Initial review task",
              "next_task_id": "0c392f64-bc65-44e6-8848-ae5a12000328"
            },
            {
              "action": "Move",
              "created_at": "2024-05-29T12:33:21.151+00:00",
              "created_by": "[email protected]",
              "next_task_name": "Initial labeling task",
              "next_task_id": "ebf41e2b-e231-0e43-adee-92252c1858dd"
            }
          ]
        },
        "project_model_configs": {
          "models": {
            "5d9647e7-aafe-4275-944d-c2e9b122105a": {
              "name": "Google Gemini Pro Vision",
              "created_at": "2023-12-13T21:51:42.657+00:00"
            },
            "036aeb03-b0b4-442e-a6e1-161382bc1228": {
              "name": "OpenAI GPT-4o",
              "created_at": "2024-05-23T23:56:03.078+00:00"
            }
          },
          "model_configs": {
            "e606593e-def3-4662-9737-3fc4ac412dd3": {
              "name": "GPT_4o_multi_modal",
              "model_id": "036aeb03-b0b4-442e-a6e1-161382bc1228",
              "inference_params": {
                "top_p": 1,
                "detail": "high",
                "temperature": 0,
                "use_text_only": false,
                "max_new_tokens": 1024,
                "use_image_only": false,
                "use_attachments": false
              }
            },
            "8fcda464-a0c7-4300-a9fa-c9a5cf82cab2": {
              "name": "Gemini_pro_vison_multi_modal",
              "model_id": "5d9647e7-aafe-4275-944d-c2e9b122105a",
              "inference_params": {
                "use_attachments": false
              }
            }
          },
          "project_model_configs": [
            {
              "id": "clwrsxr1y009i07v93w2h45qm",
              "model_id": "036aeb03-b0b4-442e-a6e1-161382bc1228",
              "model_config_id": "e606593e-def3-4662-9737-3fc4ac412dd3"
            },
            {
              "id": "clwrsyext00ar07v5czpo7w2a",
              "model_id": "5d9647e7-aafe-4275-944d-c2e9b122105a",
              "model_config_id": "8fcda464-a0c7-4300-a9fa-c9a5cf82cab2"
            }
          ]
        },
        "project_tags": []
      }
    }
  ```
</CodeGroup>


# Export overview
Source: https://docs.labelbox.com/horizon/developer-guides/export-overview

How to export data, with examples for each type of export along with details on optional parameters and filters.

<CardGroup>
  <Card title="Open in Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/exports/export_data.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/exports/export_data.ipynb" />
</CardGroup>

To export data via the SDK, use `export()`, a scalable and efficient method that allows streaming of unlimited data rows while providing a unique task object for tracking progress.

<Warning>
  ### export\_v2() is deprecated

  The previously available `export_v2()` method has been deprecated and will be removed in version 7.0 of the SDK. If you're currently using `export_v2()`, we strongly encourage you to switch to `export()` for its enhanced streamable implementation.
</Warning>

To learn how to export data rows from a project via the app UI, visit [Export labels from project](/horizon/guides/export-labels).

## Export specifications

| Data type | Annotation export formats | Project export | Model run export |
| - | - | - | - |
| Image | [See export formats](/horizon/developer-guides/export-image-annotations#annotation-export-formats) | [See sample](/horizon/developer-guides/export-image-annotations#sample-project-export) | [See sample](/horizon/developer-guides/export-image-annotations#sample-model-run-export) |
| Video | [See export formats](/horizon/developer-guides/export-video-annotations#annotation-export-formats) | [See sample](/horizon/developer-guides/export-video-annotations#sample-project-export) | [See sample](/horizon/developer-guides/export-video-annotations#sample-model-run-export) |
| Text | [See export formats](/horizon/developer-guides/export-text-annotations#annotation-export-formats) | [See sample](/horizon/developer-guides/export-text-annotations#sample-project-export) | [See sample](/horizon/developer-guides/export-text-annotations#sample-model-run-export) |
| Geospatial | [See export formats](/horizon/developer-guides/export-geospatial-annotations#annotation-export-formats) | [See sample](/horizon/developer-guides/export-geospatial-annotations#sample-project-export) | [See sample](/horizon/developer-guides/export-geospatial-annotations#sample-model-run-export) |
| Documents | [See export formats](/horizon/developer-guides/export-document-annotations#annotation-export-formats) | [See sample](/horizon/developer-guides/export-document-annotations#sample-project-export) | Not supported yet |
| Audio | [See export formats](/horizon/developer-guides/export-audio-annotations#annotation-export-formats) | [See sample](/horizon/developer-guides/export-audio-annotations#sample-project-export) | Not supported yet |
| Conversational text | [See export formats](/horizon/developer-guides/export-conversational-text-annotations#annotation-export-formats) | [See sample](/horizon/developer-guides/export-conversational-text-annotations#sample-project-export) | Not supported yet |
| HTML | [See export formats](/horizon/developer-guides/export-html-annotations#annotation-export-formats) | [See sample](/horizon/developer-guides/export-html-annotations#sample-project-export) | Not supported yet |

There are three ways to export data from Labelbox: export from Catalog, export from a labeling project, and export from a model run.

## Required & optional fields

Below is the complete list of required and optional fields supported for exports.\
To include the `location.token` field in your export of text data, set `data_row_details` to `True`.

| Field | Description | Project export | Model run export | Catalog export |
| - | - | - | - | - |
| `data_row` | Contains the basic information of the data row: - `id` - `row_data` - `global_key` - `data_row_details` (optional, see below) | Always | Always | Always |
| `data_row_details` | Contains additional details of the data row: - `dataset_id` - `dataset_name` - `created_at` - `updated_at` - `last_activity_at` - `created_by` | Optional | Optional | Optional |
| `media_attributes` | See [Media attributes](/horizon/developer-guides/media-attributes) | Optional | Optional | Optional |
| `attachments` | See [Attachments](/horizon/guides/label-data#attachments) | Optional | Optional | Optional |
| `metadata_fields` | See [Metadata](/horizon/developer-guides/metadata) | Optional | Optional | Optional |
| `embeddings` | Contains a list of dictionaries with precomputed and custom embeddings | Optional | Optional | Optional |
| `projects` | Contains the ID of the project in which the data row was labeled. | Always | n/a | Optional |
| `<project_id>` | Contains the following sections, which are expanded on below: - `labels` - `project_details` | Always | n/a | Optional |
| `labels` | Contains a list of labels attached to this data row: - `label_kind` - `version` - `id` - `annotations` | Always | Always | Optional |
| `label_details` | Contains details of each specific label: - `created_at` - `updated_at` - `created_by` - `reviews` | Optional | n/a | Optional |
| `performance_details` | Contains label-specific [performance details](/horizon/developer-guides/export-glossary#performance_details): - `seconds_to_create` - `seconds_to_review` - `skipped` - `performance_details_v2`, which contains: - `seconds_to_create` - `seconds_to_review` - `seconds_to_rework` - `seconds_total` | Optional | n/a | Optional |
| `project_details` | Contains project-specific information about this data row: - `ontology_id` - `task_id` - `task_name` - `batch_id` - `batch_name` - `workflow_status` - `priority` - `selected_label_id` - `consensus_expected_label_count` - `workflow_history` | Optional | n/a | Optional |
| `project_tags` | See [Project tags](/horizon/guides/project-tagging) | Always | n/a | n/a |
| `experiments` | Contains the ID of the model experiment(s) in which the data row was stored. | n/a | Always | Optional |
| `<model_experiment_id>` | Contains the following sections, which are expanded on below: - `name` - `runs` | n/a | Always | Optional |
| `name` | Name of the model. | n/a | Always | Optional |
| `runs` | Contains the ID of the model run(s) in which the data row was stored. | n/a | Always | Optional |
| `<model_run_id>` | Contains the following sections, which are expanded on below: - `name` - `annotation_group_id` - `labels` - `predictions` - `split` | n/a | Always | Optional |
| `name` | Name of the model run. | n/a | Always | Optional |
| `run_data_row_id` | Model run data row ID, similar to `data_row_id` but in a model run's context. | n/a | Always | Optional |
| `labels` | Contains a list of the ground truth labels attached to this data row and included in this model run: - `label_kind` - `version` - `id` - `annotations` | n/a | Always | Always |
| `predictions` | Contains a list of predictions attached to this data row and included in this model run: - `label_kind` - `version` - `id` - `annotations` | n/a | Optional | Optional |
| `split` | Contains the split the data row belongs to (either `Training`, `Validation`, or `Test`). | n/a | Optional | Optional |

## Optional parameters and filters

### Parameters

When you export data rows from a project, a model run, or Catalog, you can set parameters to include optional fields in the exports. The table below expresses the parameters available for each type of export.

| Parameter | Project export | Model run export | Dataset export (Catalog) | Slice export (Catalog) |
| - | - | - | - | - |
| `attachments` | ✔ | ✔ | ✔ | ✔ |
| `metadata_fields` | ✔ | ✔ | ✔ | ✔ |
| `embeddings` | ✔ | ✔ | ✔ | ✔ |
| `data_row_details` | ✔ | ✔ | ✔ | ✔ |
| `project_details` | ✔ | - | ✔ | ✔ |
| `label_details` | ✔ | - | ✔ | ✔ |
| `performance_details` | ✔ | - | ✔ | ✔ |
| `interpolated_frames` | ✔ | ✔ | ✔ | ✔ |
| `predictions` | - | ✔ | - | - |
| `model_run_details` | - | ✔ | - | - |
| `model_run_ids` | - | - | ✔ | ✔ |
| `project_ids` | - | - | ✔ | ✔ |
| `all_projects` | - | - | ✔ | ✔ |
| `all_model_runs` | - | - | ✔ | ✔ |

For explanations of each field and subfield, see [Export glossary](/horizon/developer-guides/export-glossary). For a detailed explanation of the `project_ids`, `model_run_ids`, `all_projects`, and `all_model_runs` parameters, see [Export data rows from Catalog](/horizon/developer-guides/export-overview#export-data-rows-from-catalog) below.

To learn how to apply these filters, see the below sections specific to each export type.

### Filters

You can use filters to select a subset of data rows to export. The table below contains the filters supported for each export type. You can apply multiple supported filters to the same export. Combinations of filters apply `AND` operator logic.

| Filter | Project export | Model run export | Dataset export (Catalog) | Slice export (Catalog) |
| - | - | - | - | - |
| `last_activity_at` | ✔ | - | ✔ | - |
| `label_created_at` | ✔ | - | ✔ | - |
| `workflow_status` | ✔ | - | - | - |
| `batch_ids` | ✔ | - | - | - |
| `global_keys` | ✔ | - | ✔ | - |
| `data_row_ids` | ✔ | - | ✔ | - |

The `last_activity_at` and `label_created_at` filters take the structure of `[<start_date>, <end_date>]` and can have the following formats:

* `YYYY-MM-DD`(this is an alias of `YYYY-MM-DD 00:00:00`)
* `YYYY-MM-DD hh:mm:ss`
* `YYYY-MM-DDThh:mm:ss±hhmm` (ISO 8601)
* `None`

The ISO 8601 format allows you to specify the timezone, while the other two formats assume the timezone from the user's workspace settings.

#### Last activity at

The `last_activity_at` filter captures only the data rows where the following changes have been made in the specified time frame:

* Changes update a data row's data (`rowData`), external ID (`externalId`), or global key (`globalKey`)
* Changes are made to annotations, attachments, embeddings, or metadata
* Data rows are added to batches
* Data row labeling tasks change
* Labels, reviews, comments, or issues are added to a project containing the data row

Data rows in multiple projects update `last_activity_at` when such changes occur in any project containing the data rows.

#### Label created at

The `label_created_at` filter captures only the data rows where labels have been created in the specified time frame.

#### Workflow status

The `workflow_status` filter allows you to export only the data rows in a specific status of a project's workflow. The filter accepts the following values:

* `ToLabel`
* `InReview`
* `InRework`
* `Done`

This filter only accepts one value. For example, `filters = {"workflow_status": "InReview"}`.

#### Batch IDs

The `batch_ids` filter allows you to export only the data rows in a specific batch or batches. This filter accepts a list of batch IDs. For example, `filters = {"batch_ids": ["batch_id_1", "batch_id_2"]}`.

To get the batches sent to a project and their associated information, you can use the `project.batches()` method. For more information, see [Get the batches](/horizon/developer-guides/project#get-the-batches).

#### Global keys

The `global_keys` filter allows you to export only the data rows with the specified global keys within a project or dataset. This filter accepts a list containing up to 2,000 values. For example, `filters = {"global_keys": ["global_key_1", "global_key_2"]}`.

#### Data row IDs

The `data_row_ids` filter allows you to export only the data rows with the specified IDs within a project or dataset. This filter accepts a list containing up to 2,000 values. For example, `filters = {"data_row_ids": ["data_row_id_1", "data_row_id_2"]}`.

## Streamable exports

Streamable exports (compatible with SDK versions 3.56 and above) allow you to get real-time data flow and updates using any of the following streamable export methods:

1. `data_row.export()`
2. `dataset.export()`
3. `model_run.export()`
4. `project.export()`
5. `slice.export()`

The return type of these methods is an object of the class `ExportTask`. This class serves as a wrapper around a [Task Objects](/horizon/developer-guides/tasks). Because of this relationship, most of the features present in the `Task` class are also available in the `ExportTask` class.

`ExportTask` supports the following methods and properties from `Task`:

* uid
* deleted
* wait\_till\_done
* completion\_percentage
* created\_at
* name
* status
* type
* updated\_at
* get\_task
* organization
* created\_by

### Creating an ExportTask instance

An instance of an `ExportTask` can be obtained via the `export()` method on the classes mentioned above, or by executing the following:

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  export_task = lb.ExportTask.get_task(client, task_id)
  # where `task_id` has to be of type `ExportTask`
  export_task.wait_till_done()
  ```
</CodeGroup>

### Checking for results and errors

To check if a task has a result/errors, the following methods can be executed:

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  if not export_task.has_result():  
    print("no results")

  if export_task.has_errors():  
    print("there are errors")  
  # These method will raise an ExportTask.TaskNotReadyExceptionexception if the task is neither in a COMPLETE or FAILED state.
  ```
</CodeGroup>

### Streaming results

To stream the results of exported data rows:

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Start export from project
  export_task = project.export()
  export_task.wait_till_done()

  # Stream the export using a callback function
  def json_stream_handler(output: labelbox.BufferedJsonConverterOutput):
    print(output.json)

  export_task.get_buffered_stream(stream_type=labelbox.StreamType.RESULT).start(stream_handler=json_stream_handler)

  # Collect all exported data into a list
  export_json = [data_row.json for data_row in export_task.get_buffered_stream()]
  ```
</CodeGroup>

### Simplified usage

For fine-grained control over the streaming process, you can use a for loop to iterate through the converted items in the stream. This allows you to implement custom streaming logic, process partial results, or apply additional filtering.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  export_task = project.export()
  export_task.wait_till_done()

  for data_row in export_task.get_buffered_stream():
    print(data_row.json)
  ```
</CodeGroup>

#### Start streaming at an offset or line

You can define a particular offset to initiate streaming. In the given example, the stream will start from offset 25,548.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  export_task.get_buffered_stream().with_offset(25548).start(stream_handler=json_stream_handler)
  ```
</CodeGroup>

<Info>
  ### Note:

  Selecting a random offset might result in positioning within the middle of a JSON string, and this behavior is entirely acceptable. The impact of this choice will become apparent in the output as soon as the streaming starts.
</Info>

Likewise, a specific line can be specified. In the following example, the stream will skip the first 348 lines and start with the 349th line, where a single JSON string is considered a line.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  export_task.get_buffered_stream().with_line(348).start(stream_handler=json_stream_handler)
  ```
</CodeGroup>

<Info>
  ### Note:

  offsets and lines are indexed starting from 0, thus `with_line(3)` will start streaming from the 4th line.
</Info>

The offset within `with_offset()` cannot exceed the total size, and line in `with_line()` cannot exceed the total number of lines returned by these methods; otherwise, a `ValueError exception` will be raised.

#### Print output size

`ExportTask` has two methods to output the total size of the exported file and the total number of lines it contains:

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  total_size = export_task.get_total_file_size(lb.StreamType.RESULT)
  total_lines = export_task.get_total_lines(lb.StreamType.ERRORS)
  ```
</CodeGroup>

#### Save export results and log errors

You can store export results in a JSON file and log any errors for monitoring or further processing, like the following example:

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  try:
      export_task = project.export()
      export_task.wait_till_done()

      # Retrieve and save the export data
      export_data = list(export_task.result)
      with open(EXPORT_FILE_PATH, "w") as file:
          json.dump(export_data, file, indent=4)
      print(f"Exported data saved to {EXPORT_FILE_PATH}")

  except Exception as e:
      # Log the error with a timestamp for monitoring
      error_message = f"Error during export: {str(e)}"
      logging.error(error_message)
      print("An error occurred. Check the log file for details.")
  ```
</CodeGroup>

### Cancel export tasks

You can cancel an ongoing export task before it completes, like the following example:

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Cancel the task before it completes
  success = client.cancel_task(export_task.uid)
  assert success is True

  # Verify the task was cancelled
  cancelled_task = client.get_task_by_id(export_task.uid)
  assert cancelled_task.status in ["CANCELING", "CANCELED"]
  ```
</CodeGroup>

## Export data rows from a project

When you export data rows from a project, you can narrow down your data rows by label status, metadata, batch, annotations, and workflow history. Then, when you export from a project, you may choose to include or exclude certain attributes in your export.

See the table at the top of this page to find the JSON export formats for each data type.

### Export from a project

<CodeGroup>
  ```python Export theme={"theme":"css-variables"}
  # The return type of this method is an `ExportTask`, which is a wrapper of a`Task`
  # Most of `Task` features are also present in `ExportTask`.
  export_params= {
    "attachments": True,
    "metadata_fields": True,
    "data_row_details": True,
    "project_details": True,
    "label_details": True,
    "performance_details": True,
    "interpolated_frames": True
  }

  # Note: Filters follow AND logic, so typically using one filter is sufficient.
  filters= {
    "last_activity_at": ["2000-01-01 00:00:00", "2050-01-01 00:00:00"],
    "label_created_at": ["2000-01-01 00:00:00", "2050-01-01 00:00:00"],
    "workflow_status": "InReview",
    "batch_ids": ["batch_id_1", "batch_id_2"],
    "data_row_ids": ["data_row_id_1", "data_row_id_2"],
    "global_keys": ["global_key_1", "global_key_2"]
  }

  export_task = project.export(params=export_params, filters=filters)
  export_task.wait_till_done()

  # Stream the export using a callback function
  def json_stream_handler(output: labelbox.BufferedJsonConverterOutput):
    print(output.json)

  export_task.get_buffered_stream(stream_type=labelbox.StreamType.RESULT).start(stream_handler=json_stream_handler)

  # Collect all exported data into a list
  export_json = [data_row.json for data_row in export_task.get_buffered_stream()]

  print("file size: ", export_task.get_total_file_size(stream_type=lb.StreamType.RESULT))
  print("line count: ", export_task.get_total_lines(stream_type=lb.StreamType.RESULT))
  ```
</CodeGroup>

## Export data rows from Catalog

You can export data rows and all their information from a [Dataset](/horizon/developer-guides/dataset) or a [Catalog Slice](/horizon/guides/slices).

When exporting from Catalog, you can include information about a data row from all projects and model runs to which it belongs. Specifically, you can export the labels from multiple projects and/or the predictions from multiple model runs for the selected data rows. You can use the `all_projects` and `all_model_runs` parameters to get information from all projects and model runs attached to your data row.

As shown below, the `project_ids` and `model_run_ids` parameters accept a list of IDs.

See the table at the top of this page to find the JSON export formats for each data type.

### Export from a dataset

<CodeGroup>
  ```python Export theme={"theme":"css-variables"}
  export_params= {
    "attachments": True,
    "metadata_fields": True,
    "data_row_details": True,
    "project_details": True,
    "label_details": True,
    "performance_details": True,
    "interpolated_frames": True,
    "all_projects": True,
    "all_model_runs": True,
    "project_ids": ["<project_id_1>", "<project_id_2>"],
    "model_run_ids": ["<model_run_id_1>", "<model_run_id_2>"] 
  }

  # Note: Filters follow AND logic, so typically using one filter is sufficient.
  filters= {
    "last_activity_at": ["2000-01-01 00:00:00", "2050-01-01 00:00:00"]
  }

  dataset = client.get_dataset("<dataset_id>")
  export_task = dataset.export(params=export_params, filters=filters)
  export_task.wait_till_done()

  # Stream the export using a callback function
  def json_stream_handler(output: labelbox.BufferedJsonConverterOutput):
    print(output.json)

  export_task.get_buffered_stream(stream_type=labelbox.StreamType.RESULT).start(stream_handler=json_stream_handler)

  # Collect all exported data into a list
  export_json = [data_row.json for data_row in export_task.get_buffered_stream()]

  print("file size: ", export_task.get_total_file_size(stream_type=lb.StreamType.RESULT))
  print("line count: ", export_task.get_total_lines(stream_type=lb.StreamType.RESULT))
  ```
</CodeGroup>

### Export a list of selected data rows from a dataset

<CodeGroup>
  ```python Export theme={"theme":"css-variables"}
  export_params= {
    "attachments": True,
    "metadata_fields": True,
    "data_row_details": True,
    "project_details": True,
    "label_details": True,
    "performance_details": True,
    "interpolated_frames": True,
    "all_projects": True,
    "all_model_runs": True,
    "project_ids": ["<project_id_1>", "<project_id_2>"],
    "model_run_ids": ["<model_run_id_1>", "<model_run_id_2>"] 
  }

  # Note: Filters follow AND logic, so typically using one filter is sufficient.
  filters= {
    "data_row_ids": ["data_row_id_1", "data_row_id_2"],
    "global_keys": ["global_key_1", "global_key_2"]
  }

  export_task = dataset.export(params=export_params, filters=filters)
  export_task.wait_till_done()

  # Stream the export using a callback function
  def json_stream_handler(output: labelbox.BufferedJsonConverterOutput):
    print(output.json)

  export_task.get_buffered_stream(stream_type=labelbox.StreamType.RESULT).start(stream_handler=json_stream_handler)

  # Collect all exported data into a list
  export_json = [data_row.json for data_row in export_task.get_buffered_stream()]

  print("file size: ", export_task.get_total_file_size(stream_type=lb.StreamType.RESULT))
  print("line count: ", export_task.get_total_lines(stream_type=lb.StreamType.RESULT))
  ```
</CodeGroup>

### Export from a slice

<CodeGroup>
  ```python Export theme={"theme":"css-variables"}
  # Set the export params to include/exclude certain fields.
  export_params = {
    "attachments": True,
    "metadata_fields": True,
    "data_row_details": True,
    "project_details": True,
    "label_details": True,
    "performance_details": True,
    "interpolated_frames": True,
    "all_projects": True,
    "all_model_runs": True,
    "project_ids": ["<project_id_1>", "<project_id_2>"],
    "model_run_ids": ["<model_run_id_1>", "<model_run_id_2>"]
  }

  catalog_slice = client.get_catalog_slice("<slice_id>")
  export_task = catalog_slice.export(params=export_params)
  export_task.wait_till_done()

  # Stream the export using a callback function
  def json_stream_handler(output: labelbox.BufferedJsonConverterOutput):
    print(output.json)

  export_task.get_buffered_stream(stream_type=labelbox.StreamType.RESULT).start(stream_handler=json_stream_handler)

  # Collect all exported data into a list
  export_json = [data_row.json for data_row in export_task.get_buffered_stream()]

  print("file size: ", export_task.get_total_file_size(stream_type=lb.StreamType.RESULT))
  print("line count: ", export_task.get_total_lines(stream_type=lb.StreamType.RESULT))
  ```
</CodeGroup>

### Export your entire Catalog

<CodeGroup>
  ```python Export theme={"theme":"css-variables"}
  # Set the export params to include/exclude certain fields.
  export_params = {
    "attachments": True,
    "embeddings": True,
    "metadata_fields": True,
    "data_row_details": True,
    "project_details": True,
    "label_details": True,
    "performance_details": True,
    "interpolated_frames": True,
    "all_projects": True,
    "all_model_runs": True,
    "project_ids": ["<project_id_1>", "<project_id_2>"],
    "model_run_ids": ["<model_run_id_1>", "<model_run_id_2>"]
  }

  catalog = client.get_catalog()
  export_task = catalog.export(params=export_params)
  export_task.wait_till_done()

  # Stream the export using a callback function
  def json_stream_handler(output: labelbox.BufferedJsonConverterOutput):
    print(output.json)

  export_task.get_buffered_stream(stream_type=labelbox.StreamType.RESULT).start(stream_handler=json_stream_handler)

  # Collect all exported data into a list
  export_json = [data_row.json for data_row in export_task.get_buffered_stream()]

  print("file size: ", export_task.get_total_file_size(stream_type=lb.StreamType.RESULT))
  print("line count: ", export_task.get_total_lines(stream_type=lb.StreamType.RESULT))
  ```
</CodeGroup>

## Export data rows from a model run

See the table at the top of this page to find the JSON export formats for each data type.

<CodeGroup>
  ```python Export theme={"theme":"css-variables"}
  # Set the export params to include/exclude certain fields.
  export_params = {
    "attachments": True,
    "metadata_fields": True,
    "data_row_details": True,
    "interpolated_frames": True,
    "predictions": True
  }

  model_run = client.get_model_run("<model_run_id>")
  export_task = model_run.export(params=export_params)
  export_task.wait_till_done()

  # Stream the export using a callback function
  def json_stream_handler(output: labelbox.BufferedJsonConverterOutput):
    print(output.json)

  export_task.get_buffered_stream(stream_type=labelbox.StreamType.RESULT).start(stream_handler=json_stream_handler)

  # Collect all exported data into a list
  export_json = [data_row.json for data_row in export_task.get_buffered_stream()]

  print("file size: ", export_task.get_total_file_size(stream_type=lb.StreamType.RESULT))
  print("line count: ", export_task.get_total_lines(stream_type=lb.StreamType.RESULT))
  ```
</CodeGroup>


# Export prompt and response data
Source: https://docs.labelbox.com/horizon/developer-guides/export-prompt-response-data

How to export prompt and response data

<CardGroup>
  <Card title="Open In Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/exports/export_data.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/exports/export_data.ipynb" />
</CardGroup>

## Export JSON annotations

<CodeGroup>
  ```python Export theme={"theme":"css-variables"}
  # Set the export params to include/exclude certain fields.
  export_params= {
    "attachments": True,
    "metadata_fields": True,
    "data_row_details": True,
    "project_details": True,
    "label_details": True,
    "performance_details": True,
    "interpolated_frames": True
  }

  # Note: Filters follow AND logic, so typically using one filter is sufficient.

  filters= {
    "last_activity_at": ["2000-01-01 00:00:00", "2050-01-01 00:00:00"],
    "workflow_status": "<wkf-status>"
  }

  export_task = project.export(params=export_params, filters=filters)
  export_task.wait_till_done()

  # Stream the export using a callback function

  def json_stream_handler(output: labelbox.BufferedJsonConverterOutput):
    print(output.json)

  export_task.get_buffered_stream(stream_type=labelbox.StreamType.RESULT).start(stream_handler=json_stream_handler)

  # Collect all exported data into a list

  export_json = [data_row.json for data_row in export_task.get_buffered_stream()]

  print("file size: ", export_task.get_total_file_size(stream_type=lb.StreamType.RESULT))
  print("line count: ", export_task.get_total_lines(stream_type=lb.StreamType.RESULT))

  ```
</CodeGroup>

## Prompt export format

### Humans generate prompts

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "cldne96y301wy13yd0wp5z87y",
    "feature_schema_id": "cljg9my6h01000741aemmcln8",
    "name": "sample_name",
    "text_answer": {
      "content": "sample text"
    }
  }
  ```
</CodeGroup>

## Response export formats

### Response - Radio

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "cldne96y201wq13ydu0qcc2up",
    "feature_schema_id": "cljggfygv0chn070w1v131s3v",
    "name": "sample_radio_name",
    "radio_answer": {
      "feature_id": "cldne96y201wr13yd23kr1pcr",
      "feature_schema_id": "cljgggcs200083b6lqbpml10p",
      "name": "first_radio_answer",
      "classifications": []
    }
  }
  ```
</CodeGroup>

### Response - Checklist

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "cldne96y201wu13ydohrclpra",
    "feature_schema_id": "cljgegd1p07m4073cfmfy5xkx",
    "name": "checklist_question",
    "checklist_answers": [
      {
        "feature_id": "cldne96y301wv13ydatuxugbt",
        "name": "first_checklist_answer",
        "classifications": []
      },
      {
        "feature_id": "cldne96y301ww13yds4zkk49u",
        "name": "second_checklist_answer",
        "classifications": []
      },
      {
        "feature_id": "cldne96y301wx13ydvb5x2w6o",
        "name": "third_checklist_answer",
        "classifications": []
      }
    ]
  }
  ```
</CodeGroup>

### Response - Text

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "cldne96y301wy13yd0wp5z87y",
    "feature_schema_id": "cljg9my6h01000741aemmcln8",
    "name": "sample_name",
    "text_answer": {
      "content": "sample text"
    }
  }
  ```
</CodeGroup>

## Sample project export

<CodeGroup>
  ```json JSON expandable theme={"theme":"css-variables"}
  {
    "data_row": {
      "id": "clpvnouh04uyy0723mmru42qn",
      "global_key": "clpvnou2v03js07xsghfo2nzc",
      "row_data": "{\"type\":\"application/llm.prompt-response-creation\",\"version\":1}",
      "details": {
        "dataset_id": "clpvnou0z004c0724pd4cmw8g",
        "dataset_name": "test-humans-generate-prompts-reponses-dataset",
        "created_at": "2023-12-07T20:34:06.540+00:00",
        "updated_at": "2023-12-07T20:34:06.815+00:00",
        "last_activity_at": "2024-04-10T15:21:31.000+00:00",
        "created_by": "[email protected]"
      }
    },
    "media_attributes": {
      "mime_type": "application/llm.prompt-response-creation"
    },
    "attachments": [],
    "metadata_fields": [],
    "projects": {
      "clpvnotzb03jo07xs48r7ewka": {
        "name": "Andrea-test-humans-generate-prompts-responses",
        "labels": [
          {
            "label_kind": "Default",
            "version": "1.0.0",
            "id": "clutyida4041a07h4a8ojbu1g",
            "label_details": {
              "created_at": "2024-04-10T15:21:31.000+00:00",
              "updated_at": "2024-04-10T15:21:31.000+00:00",
              "created_by": "[email protected]",
              "content_last_updated_at": "2024-04-10T15:21:31.137+00:00",
              "reviews": []
            },
            "performance_details": {
              "seconds_to_create": 28,
              "seconds_to_review": 0,
              "skipped": false
            },
            "annotations": {
              "objects": [],
              "classifications": [
                {
                  "feature_id": "clutykado00013b6rw65zj51e",
                  "feature_schema_id": "clutyjef000xk07wfeurhc2qb",
                  "name": "Is this a shirt?",
                  "value": "Is this a shirt?",
                  "text_answer": {
                    "content": "Potentially this is a shirt, but keep in mind this is not a good quality prompt"
                  }
                },
                {
                  "feature_id": "clutykap700033b6rmmlkpo6u",
                  "feature_schema_id": "clutyjef000xm07wf5ym80ud6",
                  "name": "Yes",
                  "value": "yes",
                  "radio_answer": {
                    "feature_id": "clutykap700023b6rhktauyzc",
                    "feature_schema_id": "clutyjef000xn07wf6smvc54b",
                    "name": "Red shirt",
                    "value": "red_shirt",
                    "classifications": []
                  }
                }
              ],
              "relationships": []
            }
          }
        ],
        "project_details": {
          "ontology_id": "clpvnqsqz01kv07zydvbscxzq",
          "task_id": "14b02ec0-71f3-4d1f-b720-c6318e9a9346",
          "task_name": "Initial review task",
          "batch_id": "00870bc0-9540-11ee-a202-8d3b90bd1707",
          "batch_name": "batch_clpvnotzb03jo07xs48r7ewka",
          "workflow_status": "IN_REVIEW",
          "priority": 5,
          "consensus_expected_label_count": 2,
          "workflow_history": [
            {
              "action": "Move",
              "created_at": "2024-04-10T15:21:31.517+00:00",
              "created_by": "[email protected]",
              "previous_task_name": "Initial labeling task",
              "previous_task_id": "1d0062f9-dfd9-0f86-baed-823235868a8c",
              "next_task_name": "Initial review task",
              "next_task_id": "14b02ec0-71f3-4d1f-b720-c6318e9a9346"
            },
            {
              "action": "Move",
              "created_at": "2024-04-10T15:21:31.506+00:00",
              "created_by": "[email protected]",
              "next_task_name": "Initial labeling task",
              "next_task_id": "1d0062f9-dfd9-0f86-baed-823235868a8c"
            }
          ]
        },
        "project_tags": []
      }
    }
  }
  ```
</CodeGroup>


# Export quick reference
Source: https://docs.labelbox.com/horizon/developer-guides/export-quick-reference



## Image

### Sample Project Export

```json JSON expandable theme={"theme":"css-variables"}
{
  "data_row": {
    "id": "clf9spzxq029m071nehy60a6v",
    "global_key": "2560px-Kitano_Street_Kobe01s5s4110.jpeg",
    "row_data": "https://storage.googleapis.com/labelbox-datasets/image_sample_data/2560px-Kitano_Street_Kobe01s5s4110.jpeg",
    "details": {
      "dataset_id": "clf9spwgw04w207ymfwy826qv",
      "dataset_name": "demo_dataset_img",
      "created_at": "2023-03-15T14:46:58.000+00:00",
      "updated_at": "2023-03-15T14:46:58.000+00:00",
      "last_activity_at": "2024-04-08T16:34:01.000+00:00",
      "created_by": "labelbox.public.user@labelbox.com"
    }
  },
  "media_attributes": {
    "height": 1707,
    "width": 2560,
    "mime_type": "image/jpeg"
  },
  "attachments": [],
  "metadata_fields": [],
  "embeddings": [
    {
      "id": "cn1cywy1r077807zblvo0m884",
      "name": "my_custom_embedding_2048_dimensions",
      "dimensions": 2048,
      "is_custom": true,
      "values": [
        {
          "value": [
            1.1265901290930636,
            1.0413733599834287,
            1.3041214120015427,
            1.893491997791636 ...
          ]
        }
      ]
    },
    {
      "id": "c200000000000000000000000",
      "name": "Image Embedding V2 (CLIP ViT-B/32)",
      "dimensions": 512,
      "is_custom": false,
      "values": [
        {
          "value": [
            -0.4192526042461395,
            -0.048827216029167175 ...
          ]
        }
      ]
    }
  ],
  "projects": {
    "cltd3vkju021907u63w2odj69": {
      "name": "Image Annotation Import Demo",
      "labels": [
        {
          "label_kind": "Default",
          "version": "1.0.0",
          "id": "clur5tlda1n2a07dc0i0zdd4k",
          "label_details": {
            "created_at": "2024-04-08T16:21:26.000+00:00",
            "updated_at": "2024-04-08T16:34:02.000+00:00",
            "created_by": "user@labelbox.com",
            "content_last_updated_at": "2024-04-08T16:34:00.513+00:00",
            "reviews": []
          },
          "performance_details": {
            "seconds_to_create": 18,
            "seconds_to_review": 6,
            "skipped": false
          },
          "annotations": {
            "objects": [
              {
                "feature_id": "clur5tlyq1n2d07dcg9xldg90",
                "feature_schema_id": "cltd3vkc1019r07ts2lru4hi4",
                "name": "bounding_box",
                "value": "bounding_box",
                "annotation_kind": "ImageBoundingBox",
                "classifications": [],
                "bounding_box": {
                  "top": 1264.0,
                  "left": 2096.0,
                  "height": 425.0,
                  "width": 144.0
                }
              },
              {
                "feature_id": "clur5tlyr1n2f07dcf78sa4y5",
                "feature_schema_id": "cltd3vkc201a707ts9ullbhfo",
                "name": "point",
                "value": "point",
                "annotation_kind": "ImagePoint",
                "classifications": [],
                "point": {
                  "x": 1166.606,
                  "y": 1441.768
                }
              },
              {
                "feature_id": "clur5tlyr1n2h07dc1pxnhoj3",
                "feature_schema_id": "cltd3vkc1019t07ts2bk6hjjj",
                "name": "bbox_with_radio_subclass",
                "value": "bbox_with_radio_subclass",
                "annotation_kind": "ImageBoundingBox",
                "classifications": [
                  {
                    "feature_id": "clur5tlyr1n2v07dcbnnfh1ah",
                    "feature_schema_id": "cltd3vkc1019u07tsaari5i2k",
                    "name": "sub_radio_question",
                    "value": "sub_radio_question",
                    "radio_answer": {
                      "feature_id": "clur5tlyr1n3607dc98qpagxb",
                      "feature_schema_id": "cltd3vkc1019v07tsbow08f9u",
                      "name": "first_sub_radio_answer",
                      "value": "first_sub_radio_answer",
                      "classifications": []
                    }
                  }
                ],
                "bounding_box": {
                  "top": 933.0,
                  "left": 541.0,
                  "height": 191.0,
                  "width": 330.0
                }
              },
              {
                "feature_id": "clur5tlyr1n2j07dc50cs4e11",
                "feature_schema_id": "cltd3vkc201a907ts69dr5m86",
                "name": "polyline",
                "value": "polyline",
                "annotation_kind": "ImagePolyline",
                "classifications": [],
                "line": [
                  {
                    "x": 2534.353,
                    "y": 249.471
                  },
                  {
                    "x": 2429.492,
                    "y": 182.092
                  },
                  {
                    "x": 2294.322,
                    "y": 221.962
                  },
                  {
                    "x": 2224.491,
                    "y": 180.463
                  },
                  {
                    "x": 2136.123,
                    "y": 204.716
                  },
                  {
                    "x": 1712.247,
                    "y": 173.949
                  },
                  {
                    "x": 1703.838,
                    "y": 84.438
                  },
                  {
                    "x": 1579.772,
                    "y": 82.61
                  },
                  {
                    "x": 1583.442,
                    "y": 167.552
                  },
                  {
                    "x": 1478.869,
                    "y": 164.903
                  },
                  {
                    "x": 1418.941,
                    "y": 318.149
                  },
                  {
                    "x": 1243.128,
                    "y": 400.815
                  },
                  {
                    "x": 1022.067,
                    "y": 319.007
                  },
                  {
                    "x": 892.367,
                    "y": 379.216
                  },
                  {
                    "x": 670.273,
                    "y": 364.408
                  },
                  {
                    "x": 613.114,
                    "y": 288.16
                  },
                  {
                    "x": 377.559,
                    "y": 238.251
                  },
                  {
                    "x": 368.087,
                    "y": 185.064
                  },
                  {
                    "x": 246.557,
                    "y": 167.286
                  },
                  {
                    "x": 236.648,
                    "y": 285.61
                  },
                  {
                    "x": 90.929,
                    "y": 326.412
                  }
                ]
              },
              {
                "feature_id": "clur5tlyr1n2l07dc1jsf15qo",
                "feature_schema_id": "cltd3vkc1019r07ts2lru4hi4",
                "name": "bounding_box",
                "value": "bounding_box",
                "annotation_kind": "ImageBoundingBox",
                "classifications": [],
                "bounding_box": {
                  "top": 1346.0,
                  "left": 2272.0,
                  "height": 358.0,
                  "width": 144.0
                }
              },
              {
                "feature_id": "clur5tlyr1n2m07dc4x6xhzep",
                "feature_schema_id": "cltd3vkc1019z07ts7g1n3o5m",
                "name": "polygon",
                "value": "polygon",
                "annotation_kind": "ImagePolygon",
                "classifications": [],
                "polygon": [
                  {
                    "x": 1489.581,
                    "y": 183.934
                  },
                  {
                    "x": 2278.306,
                    "y": 256.885
                  },
                  {
                    "x": 2428.197,
                    "y": 200.437
                  },
                  {
                    "x": 2560.0,
                    "y": 335.419
                  },
                  {
                    "x": 2557.386,
                    "y": 503.165
                  },
                  {
                    "x": 2320.596,
                    "y": 503.103
                  },
                  {
                    "x": 2156.083,
                    "y": 628.943
                  },
                  {
                    "x": 2161.111,
                    "y": 785.519
                  },
                  {
                    "x": 2002.115,
                    "y": 894.647
                  },
                  {
                    "x": 1838.456,
                    "y": 877.874
                  },
                  {
                    "x": 1436.53,
                    "y": 874.636
                  },
                  {
                    "x": 1411.403,
                    "y": 758.579
                  },
                  {
                    "x": 1353.853,
                    "y": 751.74
                  },
                  {
                    "x": 1345.264,
                    "y": 453.461
                  },
                  {
                    "x": 1426.011,
                    "y": 421.129
                  },
                  {
                    "x": 1489.581,
                    "y": 183.934
                  }
                ]
              },
              {
                "feature_id": "clur5tlyr1n2n07dc4qot5gbx",
                "feature_schema_id": "cltd3vkc1019r07ts2lru4hi4",
                "name": "bounding_box",
                "value": "bounding_box",
                "annotation_kind": "ImageBoundingBox",
                "classifications": [],
                "bounding_box": {
                  "top": 977.0,
                  "left": 1690.0,
                  "height": 330.0,
                  "width": 225.0
                }
              },
              {
                "feature_id": "clur5tlyr1n2q07dc5joc4k7u",
                "feature_schema_id": "cltd3vkc101a107ts95sk63u0",
                "name": "mask",
                "value": "mask",
                "annotation_kind": "ImageSegmentationMask",
                "classifications": [],
                "mask": {
                  "url": "https://api.labelbox.com/api/v1/projects/cltd3vkju021907u63w2odj69/annotations/clur5tlyr1n2q07dc5joc4k7u/index/1/mask"
                },
                "composite_mask": {
                  "url": "https://api.labelbox.com/api/v1/tasks/clur6dp8r04ul07wrgp9q4pla/masks/clur5tlyr1n2o07dc99lw3r8p/index/1",
                  "color_rgb": [
                    10,
                    10,
                    125
                  ]
                }
              },
              {
                "feature_id": "clur5tlyr1n2r07dc6ym4d4mc",
                "feature_schema_id": "cltd3vkc201a307tsforlev07",
                "name": "mask_with_text_subclass",
                "value": "mask_with_text_subclass",
                "annotation_kind": "ImageSegmentationMask",
                "classifications": [
                  {
                    "feature_id": "clur5tlyr1n2x07dc84a12l8i",
                    "feature_schema_id": "cltd3vkc201a407tsfo3agh7x",
                    "name": "sub_free_text",
                    "value": "sub_free_text",
                    "text_answer": {
                      "content": "free text answer sample"
                    }
                  }
                ],
                "mask": {
                  "url": "https://api.labelbox.com/api/v1/projects/cltd3vkju021907u63w2odj69/annotations/clur5tlyr1n2r07dc6ym4d4mc/index/1/mask"
                },
                "composite_mask": {
                  "url": "https://api.labelbox.com/api/v1/tasks/clur6dp8r04ul07wrgp9q4pla/masks/clur5tlyr1n2o07dc99lw3r8p/index/1",
                  "color_rgb": [
                    105,
                    136,
                    97
                  ]
                }
              },
              {
                "feature_id": "clur5tlyr1n2s07dc739a6kqs",
                "feature_schema_id": "cltd3vkc201a307tsforlev07",
                "name": "mask_with_text_subclass",
                "value": "mask_with_text_subclass",
                "annotation_kind": "ImageSegmentationMask",
                "classifications": [
                  {
                    "feature_id": "clur5tlyr1n2w07dccdv0g7gg",
                    "feature_schema_id": "cltd3vkc201a407tsfo3agh7x",
                    "name": "sub_free_text",
                    "value": "sub_free_text",
                    "text_answer": {
                      "content": "free text answer sample"
                    }
                  }
                ],
                "mask": {
                  "url": "https://api.labelbox.com/api/v1/projects/cltd3vkju021907u63w2odj69/annotations/clur5tlyr1n2s07dc739a6kqs/index/1/mask"
                },
                "composite_mask": {
                  "url": "https://api.labelbox.com/api/v1/tasks/clur6dp8r04ul07wrgp9q4pla/masks/clur5tlyr1n2o07dc99lw3r8p/index/1",
                  "color_rgb": [
                    7,
                    85,
                    11
                  ]
                }
              },
              {
                "feature_id": "clur5tlyr1n2t07dc0oxlgoc8",
                "feature_schema_id": "cltd3vkc201a307tsforlev07",
                "name": "mask_with_text_subclass",
                "value": "mask_with_text_subclass",
                "annotation_kind": "ImageSegmentationMask",
                "classifications": [
                  {
                    "feature_id": "clur5tlyr1n3107dccy5pbudr",
                    "feature_schema_id": "cltd3vkc201a407tsfo3agh7x",
                    "name": "sub_free_text",
                    "value": "sub_free_text",
                    "text_answer": {
                      "content": "free text answer sample"
                    }
                  }
                ],
                "mask": {
                  "url": "https://api.labelbox.com/api/v1/projects/cltd3vkju021907u63w2odj69/annotations/clur5tlyr1n2t07dc0oxlgoc8/index/1/mask"
                },
                "composite_mask": {
                  "url": "https://api.labelbox.com/api/v1/tasks/clur6dp8r04ul07wrgp9q4pla/masks/clur5tlyr1n2o07dc99lw3r8p/index/1",
                  "color_rgb": [
                    35,
                    34,
                    212
                  ]
                }
              },
              {
                "feature_id": "clur5tlyr1n2u07dc5wjmdm3v",
                "feature_schema_id": "cltd3vkc101a107ts95sk63u0",
                "name": "mask",
                "value": "mask",
                "annotation_kind": "ImageSegmentationMask",
                "classifications": [],
                "mask": {
                  "url": "https://api.labelbox.com/api/v1/projects/cltd3vkju021907u63w2odj69/annotations/clur5tlyr1n2u07dc5wjmdm3v/index/1/mask"
                },
                "composite_mask": {
                  "url": "https://api.labelbox.com/api/v1/tasks/clur6dp8r04ul07wrgp9q4pla/masks/clur5tlyr1n2o07dc99lw3r8p/index/1",
                  "color_rgb": [
                    232,
                    194,
                    86
                  ]
                }
              }
            ],
            "classifications": [
              {
                "feature_id": "clur5tlyr1n2e07dc2bez8p1u",
                "feature_schema_id": "cltd3vkc201ar07ts3gq6a6sn",
                "name": "nested_radio_question",
                "value": "nested_radio_question",
                "radio_answer": {
                  "feature_id": "clur5tlyr1n3307dcexth5ff7",
                  "feature_schema_id": "cltd3vkc201as07tsh2prgosu",
                  "name": "first_radio_answer",
                  "value": "first_radio_answer",
                  "classifications": [
                    {
                      "feature_id": "clur5tlyr1n3407dc306b5xib",
                      "feature_schema_id": "cltd3vkc201at07tsdjx9dlbw",
                      "name": "sub_radio_question",
                      "value": "sub_radio_question",
                      "radio_answer": {
                        "feature_id": "clur5tlyr1n3507dcef304jmb",
                        "feature_schema_id": "cltd3vkc201au07ts7nk47qny",
                        "name": "first_sub_radio_answer",
                        "value": "first_sub_radio_answer",
                        "classifications": []
                      }
                    }
                  ]
                }
              },
              {
                "feature_id": "clur5tlyr1n2g07dc7wdv7p5z",
                "feature_schema_id": "cltd3vkc201ap07ts58gma0bb",
                "name": "free_text",
                "value": "free_text",
                "text_answer": {
                  "content": "sample text"
                }
              },
              {
                "feature_id": "clur5tlyr1n2i07dcd8lx87be",
                "feature_schema_id": "cltd3vkc201az07ts7st3fn79",
                "name": "nested_checklist_question",
                "value": "nested_checklist_question",
                "checklist_answers": [
                  {
                    "feature_id": "clur5tlyr1n3207dc4u4nfg08",
                    "feature_schema_id": "cltd3vkc201b007tsdhs07baf",
                    "name": "first_checklist_answer",
                    "value": "first_checklist_answer",
                    "classifications": [
                      {
                        "feature_id": "clur5tlyr1n3707dc2cy805ac",
                        "feature_schema_id": "cltd3vkc201b107ts12b1al1j",
                        "name": "sub_checklist_question",
                        "value": "sub_checklist_question",
                        "checklist_answers": [
                          {
                            "feature_id": "clur5tlyr1n3807dcgpgih5qj",
                            "feature_schema_id": "cltd3vkc201b207ts434b9iyx",
                            "name": "first_sub_checklist_answer",
                            "value": "first_sub_checklist_answer",
                            "classifications": []
                          }
                        ]
                      }
                    ]
                  }
                ]
              },
              {
                "feature_id": "clur5tlyr1n2k07dcezk08npu",
                "feature_schema_id": "cltd3vkc201ad07ts3t78873p",
                "name": "radio_question",
                "value": "radio_question",
                "radio_answer": {
                  "feature_id": "clur5tlyr1n2y07dcajpyg0d4",
                  "feature_schema_id": "cltd3vkc201ag07tsfoc2bdpi",
                  "name": "second_radio_answer",
                  "value": "second_radio_answer",
                  "classifications": []
                }
              },
              {
                "feature_id": "clur5tlyr1n2p07dchs791672",
                "feature_schema_id": "cltd3vkc201aj07tsait4cbzh",
                "name": "checklist_question",
                "value": "checklist_question",
                "checklist_answers": [
                  {
                    "feature_id": "clur5tlyr1n2z07dc3abv19e8",
                    "feature_schema_id": "cltd3vkc201ak07tsfqxc740z",
                    "name": "first_checklist_answer",
                    "value": "first_checklist_answer",
                    "classifications": []
                  },
                  {
                    "feature_id": "clur5tlyr1n3007dc4vbr0628",
                    "feature_schema_id": "cltd3vkc201am07ts7s9e7x7k",
                    "name": "second_checklist_answer",
                    "value": "second_checklist_answer",
                    "classifications": []
                  }
                ]
              }
            ],
            "relationships": [
              {
                "feature_id": "clur69pfa00023b6rkftlkucz",
                "feature_schema_id": "cltd3vkc201ab07ts1f9dhpkt",
                "name": "relationship",
                "value": "relationship",
                "annotation_kind": "ImageUnidirectionalRelationship",
                "classifications": [],
                "unidirectional_relationship": {
                  "source": "clur5tlyq1n2d07dcg9xldg90",
                  "target": "clur5tlyr1n2l07dc1jsf15qo"
                }
              }
            ]
          }
        }
      ],
      "project_details": {
        "ontology_id": "cltd3vkbb019q07ts23eedy4o",
        "task_id": "a384279a-45c0-4200-b85a-3c18469b3809",
        "task_name": "Initial review task",
        "batch_id": "3f724e20-da3d-11ee-86a4-0145011952fd",
        "batch_name": "image-demo-batch",
        "workflow_status": "IN_REVIEW",
        "priority": 1,
        "consensus_expected_label_count": 1,
        "workflow_history": [
          {
            "action": "Move",
            "created_at": "2024-04-08T16:21:26.492+00:00",
            "created_by": "aovalle@labelbox.com",
            "previous_task_name": "Initial labeling task",
            "previous_task_id": "22ee463b-fbe3-0791-8159-94083be9dee3",
            "next_task_name": "Initial review task",
            "next_task_id": "a384279a-45c0-4200-b85a-3c18469b3809"
          },
          {
            "action": "Move",
            "created_at": "2024-04-08T16:21:26.481+00:00",
            "created_by": "aovalle@labelbox.com",
            "next_task_name": "Initial labeling task",
            "next_task_id": "22ee463b-fbe3-0791-8159-94083be9dee3"
          }
        ]
      }
    }
  }
}
```

### Sample Model Run Export

```json JSON expandable theme={"theme":"css-variables"}
{
  "data_row": {
    "id": "clo78lhhn3dxb07322mnn8plu",
    "global_key": "2560px-Kitano_Street_Kobe01s.jpeg",
    "row_data": "https://storage.googleapis.com/labelbox-datasets/image_sample_data/2560px-Kitano_Street_Kobe01s5s4110.jpeg",
    "details": {
      "dataset_id": "clo78lg8f003007321fe52lje",
      "dataset_name": "image_prediction_demo",
      "created_at": "2023-10-26T13:45:24.929+00:00",
      "updated_at": "2023-10-26T13:45:25.965+00:00",
      "created_by": "aovalle@labelbox.com"
    }
  },
  "media_attributes": {
    "height": 1707,
    "width": 2560,
    "mime_type": "image/jpeg",
    "exif_rotation": "1"
  },
  "attachments": [],
  "metadata_fields": [],
   "embeddings": [
    {
      "id": "cn1cywy1r077807zblvo0m884",
      "name": "my_custom_embedding_2048_dimensions",
      "dimensions": 2048,
      "is_custom": true,
      "values": [
        {
          "value": [
            1.1265901290930636,
            1.0413733599834287,
            1.3041214120015427,
            1.893491997791636 ...
          ]
        }
      ]
    },
    {
      "id": "c200000000000000000000000",
      "name": "Image Embedding V2 (CLIP ViT-B/32)",
      "dimensions": 512,
      "is_custom": false,
      "values": [
        {
          "value": [
            -0.4192526042461395,
            -0.048827216029167175 ...
          ]
        }
      ]
    }
  ],
  "experiments": {
    "a2fb1fec-83fb-0858-2cfb-98f4757eae95": {
      "name": "image_model_run_3e5bf56c-31a4-4eed-a731-9ca85e783225",
      "runs": {
        "a2fb1fec-b8a5-0f96-b61e-0d5643fb8dec": {
          "name": "iteration 1",
          "run_data_row_id": "42630da9-0dc4-462b-9aa0-5a17e85ff46a",
          "labels": [
            {
              "label_kind": "Default",
              "version": "1.0.0",
              "id": "clpk3h87e005r14vsp4c33sfq",
              "annotations": {
                "objects": [
                  {
                    "feature_id": "164937d0-7bb2-0f03-8031-34b57799736b",
                    "feature_schema_id": "clpk3gs0t054y07vig605dopr",
                    "name": "polygon",
                    "value": "polygon",
                    "annotation_kind": "ImagePolygon",
                    "classifications": [],
                    "polygon": [
                      {
                        "x": 1489.581,
                        "y": 183.934
                      },
                      {
                        "x": 2278.306,
                        "y": 256.885
                      },
                      {
                        "x": 2428.197,
                        "y": 200.437
                      },
                      {
                        "x": 2560.0,
                        "y": 335.419
                      },
                      {
                        "x": 2557.386,
                        "y": 503.165
                      },
                      {
                        "x": 2320.596,
                        "y": 503.103
                      },
                      {
                        "x": 2156.083,
                        "y": 628.943
                      },
                      {
                        "x": 2161.111,
                        "y": 785.519
                      },
                      {
                        "x": 2002.115,
                        "y": 894.647
                      },
                      {
                        "x": 1838.456,
                        "y": 877.874
                      },
                      {
                        "x": 1436.53,
                        "y": 874.636
                      },
                      {
                        "x": 1411.403,
                        "y": 758.579
                      },
                      {
                        "x": 1353.853,
                        "y": 751.74
                      },
                      {
                        "x": 1345.264,
                        "y": 453.461
                      },
                      {
                        "x": 1426.011,
                        "y": 421.129
                      },
                      {
                        "x": 1489.581,
                        "y": 183.934
                      }
                    ]
                  },
                  {
                    "feature_id": "2738b5f8-e586-0090-974a-59b6c9b46ee2",
                    "feature_schema_id": "clpk3gs0t055207vi1baq8bke",
                    "name": "mask_with_text_subclass",
                    "value": "mask_with_text_subclass",
                    "annotation_kind": "ImageSegmentationMask",
                    "classifications": [
                      {
                        "feature_id": "9001b75c-bb70-0b40-afbf-9479ecfa9f99",
                        "feature_schema_id": "clpk3gs0t055307vi90s32rhi",
                        "name": "sub_free_text",
                        "value": "sub_free_text",
                        "text_answer": {
                          "content": "free text answer"
                        }
                      }
                    ],
                    "mask": {
                      "url": "https://api.labelbox.com/api/v1/models/a2fb1fec-83fb-0858-2cfb-98f4757eae95/annotations/2738b5f8-e586-0090-974a-59b6c9b46ee2/index/1/mask"
                    },
                    "composite_mask": {
                      "url": "https://api.labelbox.com/api/v1/tasks/clur6mb7a0dz107xod5w53p61/masks/clpk3h7uk005114vsgru34yw7/index/1",
                      "color_rgb": [
                        41,
                        24,
                        175
                      ]
                    }
                  },
                  {
                    "feature_id": "4defcaa5-623f-0589-932a-b703e1116fa4",
                    "feature_schema_id": "clpk3gs0t055807vi49uy1rcm",
                    "name": "polyline",
                    "value": "polyline",
                    "annotation_kind": "ImagePolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 2534.353,
                        "y": 249.471
                      },
                      {
                        "x": 2429.492,
                        "y": 182.092
                      },
                      {
                        "x": 2294.322,
                        "y": 221.962
                      },
                      {
                        "x": 2224.491,
                        "y": 180.463
                      },
                      {
                        "x": 2136.123,
                        "y": 204.716
                      },
                      {
                        "x": 1712.247,
                        "y": 173.949
                      },
                      {
                        "x": 1703.838,
                        "y": 84.438
                      },
                      {
                        "x": 1579.772,
                        "y": 82.61
                      },
                      {
                        "x": 1583.442,
                        "y": 167.552
                      },
                      {
                        "x": 1478.869,
                        "y": 164.903
                      },
                      {
                        "x": 1418.941,
                        "y": 318.149
                      },
                      {
                        "x": 1243.128,
                        "y": 400.815
                      },
                      {
                        "x": 1022.067,
                        "y": 319.007
                      },
                      {
                        "x": 892.367,
                        "y": 379.216
                      },
                      {
                        "x": 670.273,
                        "y": 364.408
                      },
                      {
                        "x": 613.114,
                        "y": 288.16
                      },
                      {
                        "x": 377.559,
                        "y": 238.251
                      },
                      {
                        "x": 368.087,
                        "y": 185.064
                      },
                      {
                        "x": 246.557,
                        "y": 167.286
                      },
                      {
                        "x": 236.648,
                        "y": 285.61
                      },
                      {
                        "x": 90.929,
                        "y": 326.412
                      }
                    ]
                  },
                  {
                    "feature_id": "7d129546-45a6-060b-918e-35caec234d31",
                    "feature_schema_id": "clpk3gs0s054s07vi3ddo8had",
                    "name": "bbox_with_radio_subclass",
                    "value": "bbox_with_radio_subclass",
                    "annotation_kind": "ImageBoundingBox",
                    "classifications": [
                      {
                        "feature_id": "cccba37c-b7a8-095b-8679-dd20aece6f90",
                        "feature_schema_id": "clpk3gs0s054t07vi17aih47e",
                        "name": "sub_radio_question",
                        "value": "sub_radio_question",
                        "radio_answer": {
                          "feature_id": "ea336ea5-041c-02b6-9e8b-6ac1098351b1",
                          "feature_schema_id": "clpk3gs0s054u07vicrttdlgj",
                          "name": "first_sub_radio_answer",
                          "value": "first_sub_radio_answer",
                          "classifications": []
                        }
                      }
                    ],
                    "bounding_box": {
                      "top": 933.0,
                      "left": 541.0,
                      "height": 191.0,
                      "width": 330.0
                    }
                  },
                  {
                    "feature_id": "a3e0496d-3963-0d40-af89-48a907fbb48d",
                    "feature_schema_id": "clpk3gs0s054q07vi3c30d1be",
                    "name": "bounding_box",
                    "value": "bounding_box",
                    "annotation_kind": "ImageBoundingBox",
                    "classifications": [],
                    "bounding_box": {
                      "top": 977.0,
                      "left": 1690.0,
                      "height": 330.0,
                      "width": 225.0
                    }
                  },
                  {
                    "feature_id": "f2fc7c1f-7f10-0fc2-b676-fa2591f51a27",
                    "feature_schema_id": "clpk3gs0t055607vi59sshqn0",
                    "name": "point",
                    "value": "point",
                    "annotation_kind": "ImagePoint",
                    "classifications": [],
                    "point": {
                      "x": 1166.606,
                      "y": 1441.768
                    }
                  },
                  {
                    "feature_id": "fb60f23f-8ae1-05b3-a4d8-353839975d59",
                    "feature_schema_id": "clpk3gs0t055007vi4h7fgm59",
                    "name": "mask",
                    "value": "mask",
                    "annotation_kind": "ImageSegmentationMask",
                    "classifications": [],
                    "mask": {
                      "url": "https://api.labelbox.com/api/v1/models/a2fb1fec-83fb-0858-2cfb-98f4757eae95/annotations/fb60f23f-8ae1-05b3-a4d8-353839975d59/index/1/mask"
                    },
                    "composite_mask": {
                      "url": "https://api.labelbox.com/api/v1/tasks/clur6mb7a0dz107xod5w53p61/masks/clpk3h7uk005114vsgru34yw7/index/1",
                      "color_rgb": [
                        186,
                        26,
                        40
                      ]
                    }
                  }
                ],
                "classifications": [
                  {
                    "feature_id": "1be71b85-08da-0ade-bbe4-cfdd431d3e8a",
                    "feature_schema_id": "clpk3gs0t055g07viaine2doh",
                    "name": "checklist_question",
                    "value": "checklist_question",
                    "checklist_answers": [
                      {
                        "feature_id": "0104948e-7be5-032d-80ca-87d052346618",
                        "feature_schema_id": "clpk3gs0t055j07vi4h3y9wso",
                        "name": "second_checklist_answer",
                        "value": "second_checklist_answer",
                        "classifications": []
                      },
                      {
                        "feature_id": "cb587406-5db3-0de2-99f0-cf6e6c344f43",
                        "feature_schema_id": "clpk3gs0t055h07vi1hp0acnb",
                        "name": "first_checklist_answer",
                        "value": "first_checklist_answer",
                        "classifications": []
                      }
                    ]
                  },
                  {
                    "feature_id": "5b1f4994-1bad-03a1-bf0f-1a0b4682ea25",
                    "feature_schema_id": "clpk3gs0t055m07vi53ow6auc",
                    "name": "free_text",
                    "value": "free_text",
                    "text_answer": {
                      "content": "sample text"
                    }
                  },
                  {
                    "feature_id": "88f8cd4d-e1ca-0add-a7e8-a7349def31bc",
                    "feature_schema_id": "clpk3gs0t055a07vifk5701ua",
                    "name": "radio_question",
                    "value": "radio_question",
                    "radio_answer": {
                      "feature_id": "ee141c0d-064f-0de8-b746-e61ac14fbe62",
                      "feature_schema_id": "clpk3gs0t055d07vibryq8f4l",
                      "name": "second_radio_answer",
                      "value": "second_radio_answer",
                      "classifications": []
                    }
                  },
                  {
                    "feature_id": "cc8e57d8-5080-0b43-bab1-6e4d78b684de",
                    "feature_schema_id": "clpk3gs0t055o07vifjxs63xi",
                    "name": "nested_radio_question",
                    "value": "nested_radio_question",
                    "radio_answer": {
                      "feature_id": "81690da4-5cb5-037f-9a7b-3dd7fefc5481",
                      "feature_schema_id": "clpk3gs0t055p07vigvkkeii3",
                      "name": "first_radio_answer",
                      "value": "first_radio_answer",
                      "classifications": [
                        {
                          "feature_id": "37453082-23dd-0c38-bdcb-90705102bc6e",
                          "feature_schema_id": "clpk3gs0t055q07vifyau26jb",
                          "name": "sub_radio_question",
                          "value": "sub_radio_question",
                          "radio_answer": {
                            "feature_id": "4b014520-7193-04d9-8d69-f800316b60f2",
                            "feature_schema_id": "clpk3gs0t055r07vi406q70ga",
                            "name": "first_sub_radio_answer",
                            "value": "first_sub_radio_answer",
                            "classifications": []
                          }
                        }
                      ]
                    }
                  },
                  {
                    "feature_id": "f1d18441-41cd-0ff7-9f4f-75384e5a7b51",
                    "feature_schema_id": "clpk3gs0u055w07vifjngez8o",
                    "name": "nested_checklist_question",
                    "value": "nested_checklist_question",
                    "checklist_answers": [
                      {
                        "feature_id": "a3c8745f-8151-0256-8b6d-8ac449797fd7",
                        "feature_schema_id": "clpk3gs0u055x07vibwdt1pk3",
                        "name": "first_checklist_answer",
                        "value": "first_checklist_answer",
                        "classifications": [
                          {
                            "feature_id": "aa00a3ff-1af4-038a-9be5-97a468656bc5",
                            "feature_schema_id": "clpk3gs0u055y07vi3srh1g0l",
                            "name": "sub_checklist_question",
                            "value": "sub_checklist_question",
                            "checklist_answers": [
                              {
                                "feature_id": "0608a0d7-d9d9-0347-843b-3297e58a25dc",
                                "feature_schema_id": "clpk3gs0u055z07vifkpvgap4",
                                "name": "first_sub_checklist_answer",
                                "value": "first_sub_checklist_answer",
                                "classifications": []
                              }
                            ]
                          }
                        ]
                      }
                    ]
                  }
                ],
                "relationships": []
              }
            }
          ],
          "predictions": [
            {
              "label_kind": "Default",
              "version": "1.0.0",
              "id": "clpk3h87e005r14vsp4c33sfq",
              "annotations": {
                "objects": [
                  {
                    "feature_id": "028389ae-93ff-4614-a37e-e867a310371c",
                    "feature_schema_id": "clpk3gs0s054q07vi3c30d1be",
                    "name": "bounding_box",
                    "value": "bounding_box",
                    "annotation_kind": "ImageBoundingBox",
                    "classifications": [],
                    "bounding_box": {
                      "top": 977.0,
                      "left": 1690.0,
                      "height": 330.0,
                      "width": 225.0
                    }
                  },
                  {
                    "feature_id": "37415b2f-f376-47b9-8f6a-6429c3efa6af",
                    "feature_schema_id": "clpk3gs0t054y07vig605dopr",
                    "name": "polygon",
                    "value": "polygon",
                    "annotation_kind": "ImagePolygon",
                    "classifications": [],
                    "polygon": [
                      {
                        "x": 1489.581,
                        "y": 183.934
                      },
                      {
                        "x": 2278.306,
                        "y": 256.885
                      },
                      {
                        "x": 2428.197,
                        "y": 200.437
                      },
                      {
                        "x": 2560.0,
                        "y": 335.419
                      },
                      {
                        "x": 2557.386,
                        "y": 503.165
                      },
                      {
                        "x": 2320.596,
                        "y": 503.103
                      },
                      {
                        "x": 2156.083,
                        "y": 628.943
                      },
                      {
                        "x": 2161.111,
                        "y": 785.519
                      },
                      {
                        "x": 2002.115,
                        "y": 894.647
                      },
                      {
                        "x": 1838.456,
                        "y": 877.874
                      },
                      {
                        "x": 1436.53,
                        "y": 874.636
                      },
                      {
                        "x": 1411.403,
                        "y": 758.579
                      },
                      {
                        "x": 1353.853,
                        "y": 751.74
                      },
                      {
                        "x": 1345.264,
                        "y": 453.461
                      },
                      {
                        "x": 1426.011,
                        "y": 421.129
                      },
                      {
                        "x": 1489.581,
                        "y": 183.934
                      }
                    ]
                  },
                  {
                    "feature_id": "55bee3a4-59c2-4d85-95b3-4584c315037e",
                    "feature_schema_id": "clpk3gs0t055607vi59sshqn0",
                    "name": "point",
                    "value": "point",
                    "annotation_kind": "ImagePoint",
                    "classifications": [],
                    "point": {
                      "x": 1166.606,
                      "y": 1441.768
                    }
                  },
                  {
                    "feature_id": "671736d6-7c30-4399-bec4-a8a1779abf4d",
                    "feature_schema_id": "clpk3gs0t055007vi4h7fgm59",
                    "name": "mask",
                    "value": "mask",
                    "annotation_kind": "ImageSegmentationMask",
                    "classifications": [],
                    "mask": {
                      "url": "https://api.labelbox.com/api/v1/models/a2fb1fec-83fb-0858-2cfb-98f4757eae95/annotations/671736d6-7c30-4399-bec4-a8a1779abf4d/index/1/mask"
                    },
                    "composite_mask": {
                      "url": "https://api.labelbox.com/api/v1/tasks/clur6mb7a0dz107xod5w53p61/masks/clpk3gxbq00oi14ok1yykgp85/index/1",
                      "color_rgb": [
                        36,
                        67,
                        66
                      ]
                    }
                  },
                  {
                    "feature_id": "7e103ed1-f66f-4cf9-9a36-3e08be3a9648",
                    "feature_schema_id": "clpk3gs0t055807vi49uy1rcm",
                    "name": "polyline",
                    "value": "polyline",
                    "annotation_kind": "ImagePolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 2534.353,
                        "y": 249.471
                      },
                      {
                        "x": 2429.492,
                        "y": 182.092
                      },
                      {
                        "x": 2294.322,
                        "y": 221.962
                      },
                      {
                        "x": 2224.491,
                        "y": 180.463
                      },
                      {
                        "x": 2136.123,
                        "y": 204.716
                      },
                      {
                        "x": 1712.247,
                        "y": 173.949
                      },
                      {
                        "x": 1703.838,
                        "y": 84.438
                      },
                      {
                        "x": 1579.772,
                        "y": 82.61
                      },
                      {
                        "x": 1583.442,
                        "y": 167.552
                      },
                      {
                        "x": 1478.869,
                        "y": 164.903
                      },
                      {
                        "x": 1418.941,
                        "y": 318.149
                      },
                      {
                        "x": 1243.128,
                        "y": 400.815
                      },
                      {
                        "x": 1022.067,
                        "y": 319.007
                      },
                      {
                        "x": 892.367,
                        "y": 379.216
                      },
                      {
                        "x": 670.273,
                        "y": 364.408
                      },
                      {
                        "x": 613.114,
                        "y": 288.16
                      },
                      {
                        "x": 377.559,
                        "y": 238.251
                      },
                      {
                        "x": 368.087,
                        "y": 185.064
                      },
                      {
                        "x": 246.557,
                        "y": 167.286
                      },
                      {
                        "x": 236.648,
                        "y": 285.61
                      },
                      {
                        "x": 90.929,
                        "y": 326.412
                      }
                    ]
                  },
                  {
                    "feature_id": "f3a445af-0475-4ad1-892b-fbbcce26073f",
                    "feature_schema_id": "clpk3gs0s054s07vi3ddo8had",
                    "name": "bbox_with_radio_subclass",
                    "value": "bbox_with_radio_subclass",
                    "annotation_kind": "ImageBoundingBox",
                    "classifications": [
                      {
                        "feature_id": "d8427b71-d516-4686-bc52-660d1133c18d",
                        "feature_schema_id": "clpk3gs0s054t07vi17aih47e",
                        "name": "sub_radio_question",
                        "value": "sub_radio_question",
                        "radio_answer": {
                          "feature_id": "494d3af6-9bf8-477f-8827-dac0dd4c42a3",
                          "feature_schema_id": "clpk3gs0s054u07vicrttdlgj",
                          "name": "first_sub_radio_answer",
                          "value": "first_sub_radio_answer",
                          "classifications": []
                        }
                      }
                    ],
                    "bounding_box": {
                      "top": 933.0,
                      "left": 541.0,
                      "height": 191.0,
                      "width": 330.0
                    }
                  }
                ],
                "classifications": [
                  {
                    "feature_id": "22778fd4-3fd3-48ad-9007-a8f2fc22d664",
                    "feature_schema_id": "clpk3gs0t055o07vifjxs63xi",
                    "name": "nested_radio_question",
                    "value": "nested_radio_question",
                    "radio_answer": {
                      "feature_id": "b992d812-7d0b-49b1-82da-14bcff5a76ef",
                      "feature_schema_id": "clpk3gs0t055p07vigvkkeii3",
                      "name": "first_radio_answer",
                      "value": "first_radio_answer",
                      "classifications": [
                        {
                          "feature_id": "877d229e-c877-4aa7-a034-ddde21fa8417",
                          "feature_schema_id": "clpk3gs0t055q07vifyau26jb",
                          "name": "sub_radio_question",
                          "value": "sub_radio_question",
                          "radio_answer": {
                            "feature_id": "803cbe24-7c2f-400c-8d73-fb19bdc5302c",
                            "feature_schema_id": "clpk3gs0t055r07vi406q70ga",
                            "name": "first_sub_radio_answer",
                            "value": "first_sub_radio_answer",
                            "classifications": []
                          }
                        }
                      ]
                    }
                  },
                  {
                    "feature_id": "2555cb5f-5a90-40f9-987f-7a2899688ad6",
                    "feature_schema_id": "clpk3gs0t055a07vifk5701ua",
                    "name": "radio_question",
                    "value": "radio_question",
                    "radio_answer": {
                      "feature_id": "246406e7-162c-43c5-a1ff-ca555be51e55",
                      "feature_schema_id": "clpk3gs0t055d07vibryq8f4l",
                      "name": "second_radio_answer",
                      "value": "second_radio_answer",
                      "classifications": []
                    }
                  },
                  {
                    "feature_id": "e4f143b7-5e3d-4f17-8202-efc78cd73f6e",
                    "feature_schema_id": "clpk3gs0t055m07vi53ow6auc",
                    "name": "free_text",
                    "value": "free_text",
                    "text_answer": {
                      "content": "sample text"
                    }
                  },
                  {
                    "feature_id": "ec7d6588-3c15-4fb6-baa1-3508c476bc32",
                    "feature_schema_id": "clpk3gs0u055w07vifjngez8o",
                    "name": "nested_checklist_question",
                    "value": "nested_checklist_question",
                    "checklist_answers": [
                      {
                        "feature_id": "71fd6407-5fc6-43e4-8196-78487f0f9ec8",
                        "feature_schema_id": "clpk3gs0u055x07vibwdt1pk3",
                        "name": "first_checklist_answer",
                        "value": "first_checklist_answer",
                        "classifications": [
                          {
                            "feature_id": "ab8bf663-3fba-4dd6-9e09-84e0bbd713d0",
                            "feature_schema_id": "clpk3gs0u055y07vi3srh1g0l",
                            "name": "sub_checklist_question",
                            "value": "sub_checklist_question",
                            "checklist_answers": [
                              {
                                "feature_id": "58c98b94-b360-4ceb-b5ea-7765e096c40a",
                                "feature_schema_id": "clpk3gs0u055z07vifkpvgap4",
                                "name": "first_sub_checklist_answer",
                                "value": "first_sub_checklist_answer",
                                "classifications": []
                              }
                            ]
                          }
                        ]
                      }
                    ]
                  },
                  {
                    "feature_id": "f9107ba1-8020-4b7a-8066-b80167a9cd6a",
                    "feature_schema_id": "clpk3gs0t055g07viaine2doh",
                    "name": "checklist_question",
                    "value": "checklist_question",
                    "checklist_answers": [
                      {
                        "feature_id": "88cc2ed7-8b19-4396-b9ee-36949d23fac7",
                        "feature_schema_id": "clpk3gs0t055h07vi1hp0acnb",
                        "name": "first_checklist_answer",
                        "value": "first_checklist_answer",
                        "classifications": []
                      },
                      {
                        "feature_id": "b70e6eb1-2ee9-42f3-b47a-a56e764ea4ac",
                        "feature_schema_id": "clpk3gs0t055j07vi4h3y9wso",
                        "name": "second_checklist_answer",
                        "value": "second_checklist_answer",
                        "classifications": []
                      }
                    ]
                  }
                ],
                "relationships": []
              }
            }
          ]
        }
      }
    }
  },
  "models": {}
}
```

## Video

### Sample Project Export

```json JSON expandable theme={"theme":"css-variables"}
{
  "data_row": {
    "id": "clp1jolki07uy0725flku4bmv",
    "global_key": "sample-video-jellyfish.mp4",
    "row_data": "https://storage.googleapis.com/labelbox-datasets/video-sample-data/sample-video-2.mp4",
    "details": {
      "dataset_id": "clp1jokqp000p07307gtfy1w3",
      "dataset_name": "video_demo_dataset",
      "created_at": "2023-11-16T18:48:51.208+00:00",
      "updated_at": "2023-11-16T18:48:51.801+00:00",
      "last_activity_at": "2024-04-08T17:09:54.000+00:00",
      "created_by": "aovalle@labelbox.com"
    }
  },
  "media_attributes": {
    "height": 1080,
    "width": 1920,
    "mime_type": "video/mp4",
    "frame_rate": 10,
    "frame_count": 100
  },
  "attachments": [],
  "metadata_fields": [],
   "embeddings": [
    {
      "id": "c400000000000000000000000",
      "name": "Video Embedding V2 (Google MultiModal)",
      "dimensions": 1408,
      "is_custom": false,
      "values": [
        {
          "value": [
            -0.00489852577,
            0.0287129097 ...
          ]
        }
      ]
    }
  ],
  "projects": {
    "cltd977uu016y08xyggep8aiv": {
      "name": "Video Annotation Import Demo",
      "labels": [
        {
          "label_kind": "Video",
          "version": "1.0.0",
          "id": "cltd97caz01ub14ysq8rf4x8w",
          "label_details": {
            "created_at": "2024-03-04T18:07:33.000+00:00",
            "updated_at": "2024-03-04T18:07:33.000+00:00",
            "created_by": "aovalle@labelbox.com",
            "content_last_updated_at": "2024-03-04T18:07:33.369+00:00",
            "reviews": []
          },
          "performance_details": {
            "seconds_to_create": 0,
            "seconds_to_review": 0,
            "skipped": false
          },
          "annotations": {
            "frames": {
              "29": {
                "objects": {
                  "cltd97cag01tp14ysz3my848t": {
                    "feature_id": "cltd97cag01tp14ysz3my848t",
                    "feature_schema_id": "cltd977np037n07x5d1buckz9",
                    "name": "line_video_frame",
                    "value": "line_video_frame",
                    "annotation_kind": "VideoPolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 680.0,
                        "y": 100.0
                      },
                      {
                        "x": 100.0,
                        "y": 190.0
                      }
                    ]
                  }
                },
                "classifications": [
                  {
                    "feature_id": "cltd97cag01tc14yspm0lev2d",
                    "feature_schema_id": "cltd977np038307x55dlf3rgs",
                    "name": "checklist_class",
                    "value": "checklist_class",
                    "checklist_answers": [
                      {
                        "feature_id": "cltd97cag01td14ysxw2n8xkz",
                        "feature_schema_id": "cltd977np038407x51hcvdad7",
                        "name": "first_checklist_answer",
                        "value": "first_checklist_answer",
                        "classifications": []
                      },
                      {
                        "feature_id": "cltd97cag01te14ys1jdf1fku",
                        "feature_schema_id": "cltd977np038607x58epu4nb5",
                        "name": "second_checklist_answer",
                        "value": "second_checklist_answer",
                        "classifications": []
                      }
                    ]
                  }
                ]
              },
              "35": {
                "objects": {
                  "cltd97cag01tp14ysz3my848t": {
                    "feature_id": "cltd97cag01tp14ysz3my848t",
                    "feature_schema_id": "cltd977np037n07x5d1buckz9",
                    "name": "line_video_frame",
                    "value": "line_video_frame",
                    "annotation_kind": "VideoPolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 680.0,
                        "y": 100.0
                      },
                      {
                        "x": 100.0,
                        "y": 190.0
                      }
                    ]
                  }
                },
                "classifications": [
                  {
                    "feature_id": "cltd97cag01tc14yspm0lev2d",
                    "feature_schema_id": "cltd977np038307x55dlf3rgs",
                    "name": "checklist_class",
                    "value": "checklist_class",
                    "checklist_answers": [
                      {
                        "feature_id": "cltd97cag01td14ysxw2n8xkz",
                        "feature_schema_id": "cltd977np038407x51hcvdad7",
                        "name": "first_checklist_answer",
                        "value": "first_checklist_answer",
                        "classifications": []
                      },
                      {
                        "feature_id": "cltd97cag01te14ys1jdf1fku",
                        "feature_schema_id": "cltd977np038607x58epu4nb5",
                        "name": "second_checklist_answer",
                        "value": "second_checklist_answer",
                        "classifications": []
                      }
                    ]
                  }
                ]
              },
              "9": {
                "objects": {
                  "cltd97cag01tp14ysz3my848t": {
                    "feature_id": "cltd97cag01tp14ysz3my848t",
                    "feature_schema_id": "cltd977np037n07x5d1buckz9",
                    "name": "line_video_frame",
                    "value": "line_video_frame",
                    "annotation_kind": "VideoPolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 680.0,
                        "y": 100.00000000000001
                      },
                      {
                        "x": 100.00000000000001,
                        "y": 190.0
                      }
                    ]
                  }
                },
                "classifications": [
                  {
                    "feature_id": "cltd97cag01tf14ysho6tfl7j",
                    "feature_schema_id": "cltd977np038907x5ah8l9ha7",
                    "name": "radio_class",
                    "value": "radio_class",
                    "radio_answer": {
                      "feature_id": "cltd97cag01tg14ys171a3qqr",
                      "feature_schema_id": "cltd977np038a07x515axagkl",
                      "name": "first_radio_answer",
                      "value": "first_radio_answer",
                      "classifications": []
                    }
                  }
                ]
              },
              "15": {
                "objects": {
                  "cltd97cag01th14ys6fyyxyk9": {
                    "feature_id": "cltd97cag01th14ys6fyyxyk9",
                    "feature_schema_id": "cltd977no037j07x5d82dacbc",
                    "name": "bbox_video",
                    "value": "bbox_video",
                    "annotation_kind": "VideoBoundingBox",
                    "classifications": [],
                    "bounding_box": {
                      "top": 617.0,
                      "left": 1371.0,
                      "height": 419.0,
                      "width": 505.0
                    }
                  },
                  "cltd97cag01tp14ysz3my848t": {
                    "feature_id": "cltd97cag01tp14ysz3my848t",
                    "feature_schema_id": "cltd977np037n07x5d1buckz9",
                    "name": "line_video_frame",
                    "value": "line_video_frame",
                    "annotation_kind": "VideoPolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 680.0,
                        "y": 100.0
                      },
                      {
                        "x": 100.0,
                        "y": 190.0
                      }
                    ]
                  }
                },
                "classifications": [
                  {
                    "feature_id": "cltd97cag01tf14ysho6tfl7j",
                    "feature_schema_id": "cltd977np038907x5ah8l9ha7",
                    "name": "radio_class",
                    "value": "radio_class",
                    "radio_answer": {
                      "feature_id": "cltd97cag01tg14ys171a3qqr",
                      "feature_schema_id": "cltd977np038a07x515axagkl",
                      "name": "first_radio_answer",
                      "value": "first_radio_answer",
                      "classifications": []
                    }
                  }
                ]
              },
              "13": {
                "objects": {
                  "cltd97cag01th14ys6fyyxyk9": {
                    "feature_id": "cltd97cag01th14ys6fyyxyk9",
                    "feature_schema_id": "cltd977no037j07x5d82dacbc",
                    "name": "bbox_video",
                    "value": "bbox_video",
                    "annotation_kind": "VideoBoundingBox",
                    "classifications": [],
                    "bounding_box": {
                      "top": 617.0,
                      "left": 1371.0,
                      "height": 419.0,
                      "width": 505.0
                    }
                  },
                  "cltd97cag01ti14ys8wedf89d": {
                    "feature_id": "cltd97cag01ti14ys8wedf89d",
                    "feature_schema_id": "cltd977np037r07x512iteflz",
                    "name": "bbox_class",
                    "value": "bbox_class",
                    "annotation_kind": "VideoBoundingBox",
                    "classifications": [
                      {
                        "feature_id": "cltd97cag01tj14ysz455nenr",
                        "feature_schema_id": "cltd977np037s07x5csbodf5f",
                        "name": "checklist_class",
                        "value": "checklist_class",
                        "radio_answer": {
                          "feature_id": "cltd97cag01tl14ysnbe8mw75",
                          "feature_schema_id": "cltd977np037v07x54esmdmws",
                          "name": "second_checklist_answer",
                          "value": "second_checklist_answer",
                          "classifications": []
                        }
                      }
                    ],
                    "bounding_box": {
                      "top": 146.0,
                      "left": 98.0,
                      "height": 382.0,
                      "width": 341.0
                    }
                  },
                  "cltd97cag01tp14ysz3my848t": {
                    "feature_id": "cltd97cag01tp14ysz3my848t",
                    "feature_schema_id": "cltd977np037n07x5d1buckz9",
                    "name": "line_video_frame",
                    "value": "line_video_frame",
                    "annotation_kind": "VideoPolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 680.0,
                        "y": 100.0
                      },
                      {
                        "x": 100.0,
                        "y": 190.0
                      }
                    ]
                  }
                },
                "classifications": []
              },
              "14": {
                "objects": {
                  "cltd97cag01th14ys6fyyxyk9": {
                    "feature_id": "cltd97cag01th14ys6fyyxyk9",
                    "feature_schema_id": "cltd977no037j07x5d82dacbc",
                    "name": "bbox_video",
                    "value": "bbox_video",
                    "annotation_kind": "VideoBoundingBox",
                    "classifications": [],
                    "bounding_box": {
                      "top": 617.0000000000001,
                      "left": 1371.0,
                      "height": 418.9999999999999,
                      "width": 505.0
                    }
                  },
                  "cltd97cag01tp14ysz3my848t": {
                    "feature_id": "cltd97cag01tp14ysz3my848t",
                    "feature_schema_id": "cltd977np037n07x5d1buckz9",
                    "name": "line_video_frame",
                    "value": "line_video_frame",
                    "annotation_kind": "VideoPolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 680.0,
                        "y": 100.0
                      },
                      {
                        "x": 100.0,
                        "y": 190.0
                      }
                    ]
                  }
                },
                "classifications": []
              },
              "16": {
                "objects": {
                  "cltd97cag01th14ys6fyyxyk9": {
                    "feature_id": "cltd97cag01th14ys6fyyxyk9",
                    "feature_schema_id": "cltd977no037j07x5d82dacbc",
                    "name": "bbox_video",
                    "value": "bbox_video",
                    "annotation_kind": "VideoBoundingBox",
                    "classifications": [],
                    "bounding_box": {
                      "top": 617.0,
                      "left": 1371.0,
                      "height": 419.0,
                      "width": 505.0
                    }
                  },
                  "cltd97cag01tp14ysz3my848t": {
                    "feature_id": "cltd97cag01tp14ysz3my848t",
                    "feature_schema_id": "cltd977np037n07x5d1buckz9",
                    "name": "line_video_frame",
                    "value": "line_video_frame",
                    "annotation_kind": "VideoPolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 680.0,
                        "y": 100.0
                      },
                      {
                        "x": 100.0,
                        "y": 190.0
                      }
                    ]
                  }
                },
                "classifications": []
              },
              "17": {
                "objects": {
                  "cltd97cag01th14ys6fyyxyk9": {
                    "feature_id": "cltd97cag01th14ys6fyyxyk9",
                    "feature_schema_id": "cltd977no037j07x5d82dacbc",
                    "name": "bbox_video",
                    "value": "bbox_video",
                    "annotation_kind": "VideoBoundingBox",
                    "classifications": [],
                    "bounding_box": {
                      "top": 617.0,
                      "left": 1371.0,
                      "height": 419.0,
                      "width": 505.0
                    }
                  },
                  "cltd97cag01to14yslakirofl": {
                    "feature_id": "cltd97cag01to14yslakirofl",
                    "feature_schema_id": "cltd977no037l07x56wfu4536",
                    "name": "point_video",
                    "value": "point_video",
                    "annotation_kind": "VideoPoint",
                    "classifications": [],
                    "point": {
                      "x": 660.134,
                      "y": 407.926
                    }
                  },
                  "cltd97cag01tp14ysz3my848t": {
                    "feature_id": "cltd97cag01tp14ysz3my848t",
                    "feature_schema_id": "cltd977np037n07x5d1buckz9",
                    "name": "line_video_frame",
                    "value": "line_video_frame",
                    "annotation_kind": "VideoPolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 680.0,
                        "y": 100.0
                      },
                      {
                        "x": 100.0,
                        "y": 190.0
                      }
                    ]
                  }
                },
                "classifications": []
              },
              "18": {
                "objects": {
                  "cltd97cag01th14ys6fyyxyk9": {
                    "feature_id": "cltd97cag01th14ys6fyyxyk9",
                    "feature_schema_id": "cltd977no037j07x5d82dacbc",
                    "name": "bbox_video",
                    "value": "bbox_video",
                    "annotation_kind": "VideoBoundingBox",
                    "classifications": [],
                    "bounding_box": {
                      "top": 617.0,
                      "left": 1371.0,
                      "height": 419.0,
                      "width": 505.0
                    }
                  },
                  "cltd97cag01tp14ysz3my848t": {
                    "feature_id": "cltd97cag01tp14ysz3my848t",
                    "feature_schema_id": "cltd977np037n07x5d1buckz9",
                    "name": "line_video_frame",
                    "value": "line_video_frame",
                    "annotation_kind": "VideoPolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 680.0,
                        "y": 100.0
                      },
                      {
                        "x": 100.0,
                        "y": 190.0
                      }
                    ]
                  }
                },
                "classifications": []
              },
              "19": {
                "objects": {
                  "cltd97cag01th14ys6fyyxyk9": {
                    "feature_id": "cltd97cag01th14ys6fyyxyk9",
                    "feature_schema_id": "cltd977no037j07x5d82dacbc",
                    "name": "bbox_video",
                    "value": "bbox_video",
                    "annotation_kind": "VideoBoundingBox",
                    "classifications": [],
                    "bounding_box": {
                      "top": 617.0,
                      "left": 1371.0,
                      "height": 419.0,
                      "width": 505.0
                    }
                  },
                  "cltd97cag01tp14ysz3my848t": {
                    "feature_id": "cltd97cag01tp14ysz3my848t",
                    "feature_schema_id": "cltd977np037n07x5d1buckz9",
                    "name": "line_video_frame",
                    "value": "line_video_frame",
                    "annotation_kind": "VideoPolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 680.0,
                        "y": 100.0
                      },
                      {
                        "x": 100.0,
                        "y": 190.0
                      }
                    ]
                  }
                },
                "classifications": []
              },
              "10": {
                "objects": {
                  "cltd97cag01ti14ys8wedf89d": {
                    "feature_id": "cltd97cag01ti14ys8wedf89d",
                    "feature_schema_id": "cltd977np037r07x512iteflz",
                    "name": "bbox_class",
                    "value": "bbox_class",
                    "annotation_kind": "VideoBoundingBox",
                    "classifications": [],
                    "bounding_box": {
                      "top": 146.0,
                      "left": 98.0,
                      "height": 382.0,
                      "width": 341.0
                    }
                  },
                  "cltd97cag01tp14ysz3my848t": {
                    "feature_id": "cltd97cag01tp14ysz3my848t",
                    "feature_schema_id": "cltd977np037n07x5d1buckz9",
                    "name": "line_video_frame",
                    "value": "line_video_frame",
                    "annotation_kind": "VideoPolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 680.0,
                        "y": 100.0
                      },
                      {
                        "x": 100.0,
                        "y": 190.0
                      }
                    ]
                  }
                },
                "classifications": []
              },
              "11": {
                "objects": {
                  "cltd97cag01ti14ys8wedf89d": {
                    "feature_id": "cltd97cag01ti14ys8wedf89d",
                    "feature_schema_id": "cltd977np037r07x512iteflz",
                    "name": "bbox_class",
                    "value": "bbox_class",
                    "annotation_kind": "VideoBoundingBox",
                    "classifications": [
                      {
                        "feature_id": "cltd97cag01tj14ysz455nenr",
                        "feature_schema_id": "cltd977np037s07x5csbodf5f",
                        "name": "checklist_class",
                        "value": "checklist_class",
                        "radio_answer": {
                          "feature_id": "cltd97cag01tk14ysejipk2y6",
                          "feature_schema_id": "cltd977np037t07x5966ebxpe",
                          "name": "first_checklist_answer",
                          "value": "first_checklist_answer",
                          "classifications": []
                        }
                      }
                    ],
                    "bounding_box": {
                      "top": 146.0,
                      "left": 98.0,
                      "height": 382.0,
                      "width": 341.0
                    }
                  },
                  "cltd97cag01tp14ysz3my848t": {
                    "feature_id": "cltd97cag01tp14ysz3my848t",
                    "feature_schema_id": "cltd977np037n07x5d1buckz9",
                    "name": "line_video_frame",
                    "value": "line_video_frame",
                    "annotation_kind": "VideoPolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 680.0,
                        "y": 100.0
                      },
                      {
                        "x": 100.0,
                        "y": 190.0
                      }
                    ]
                  }
                },
                "classifications": []
              },
              "12": {
                "objects": {
                  "cltd97cag01ti14ys8wedf89d": {
                    "feature_id": "cltd97cag01ti14ys8wedf89d",
                    "feature_schema_id": "cltd977np037r07x512iteflz",
                    "name": "bbox_class",
                    "value": "bbox_class",
                    "annotation_kind": "VideoBoundingBox",
                    "classifications": [
                      {
                        "feature_id": "cltd97cag01tj14ysz455nenr",
                        "feature_schema_id": "cltd977np037s07x5csbodf5f",
                        "name": "checklist_class",
                        "value": "checklist_class",
                        "radio_answer": {
                          "feature_id": "cltd97cag01tk14ysejipk2y6",
                          "feature_schema_id": "cltd977np037t07x5966ebxpe",
                          "name": "first_checklist_answer",
                          "value": "first_checklist_answer",
                          "classifications": []
                        }
                      }
                    ],
                    "bounding_box": {
                      "top": 146.0,
                      "left": 98.0,
                      "height": 382.0,
                      "width": 341.0
                    }
                  },
                  "cltd97cag01tp14ysz3my848t": {
                    "feature_id": "cltd97cag01tp14ysz3my848t",
                    "feature_schema_id": "cltd977np037n07x5d1buckz9",
                    "name": "line_video_frame",
                    "value": "line_video_frame",
                    "annotation_kind": "VideoPolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 680.0,
                        "y": 100.0
                      },
                      {
                        "x": 100.0,
                        "y": 190.0
                      }
                    ]
                  }
                },
                "classifications": []
              },
              "22": {
                "objects": {
                  "cltd97cag01tm14ys8tm4x7yl": {
                    "feature_id": "cltd97cag01tm14ys8tm4x7yl",
                    "feature_schema_id": "cltd977no037j07x5d82dacbc",
                    "name": "bbox_video",
                    "value": "bbox_video",
                    "annotation_kind": "VideoBoundingBox",
                    "classifications": [],
                    "bounding_box": {
                      "top": 617.0,
                      "left": 1371.0,
                      "height": 419.0,
                      "width": 505.0
                    }
                  },
                  "cltd97cag01tn14yspqge5yph": {
                    "feature_id": "cltd97cag01tn14yspqge5yph",
                    "feature_schema_id": "cltd977no037j07x5d82dacbc",
                    "name": "bbox_video",
                    "value": "bbox_video",
                    "annotation_kind": "VideoBoundingBox",
                    "classifications": [],
                    "bounding_box": {
                      "top": 146.0,
                      "left": 98.0,
                      "height": 382.0,
                      "width": 341.0
                    }
                  }
                },
                "classifications": []
              },
              "23": {
                "objects": {
                  "cltd97cag01tm14ys8tm4x7yl": {
                    "feature_id": "cltd97cag01tm14ys8tm4x7yl",
                    "feature_schema_id": "cltd977no037j07x5d82dacbc",
                    "name": "bbox_video",
                    "value": "bbox_video",
                    "annotation_kind": "VideoBoundingBox",
                    "classifications": [],
                    "bounding_box": {
                      "top": 617.0,
                      "left": 1371.0,
                      "height": 419.0,
                      "width": 505.0000000000002
                    }
                  },
                  "cltd97cag01tn14yspqge5yph": {
                    "feature_id": "cltd97cag01tn14yspqge5yph",
                    "feature_schema_id": "cltd977no037j07x5d82dacbc",
                    "name": "bbox_video",
                    "value": "bbox_video",
                    "annotation_kind": "VideoBoundingBox",
                    "classifications": [],
                    "bounding_box": {
                      "top": 146.0,
                      "left": 98.0,
                      "height": 382.0,
                      "width": 341.00000000000006
                    }
                  }
                },
                "classifications": []
              },
              "24": {
                "objects": {
                  "cltd97cag01tm14ys8tm4x7yl": {
                    "feature_id": "cltd97cag01tm14ys8tm4x7yl",
                    "feature_schema_id": "cltd977no037j07x5d82dacbc",
                    "name": "bbox_video",
                    "value": "bbox_video",
                    "annotation_kind": "VideoBoundingBox",
                    "classifications": [],
                    "bounding_box": {
                      "top": 617.0,
                      "left": 1371.0,
                      "height": 419.0,
                      "width": 505.0
                    }
                  },
                  "cltd97cag01tn14yspqge5yph": {
                    "feature_id": "cltd97cag01tn14yspqge5yph",
                    "feature_schema_id": "cltd977no037j07x5d82dacbc",
                    "name": "bbox_video",
                    "value": "bbox_video",
                    "annotation_kind": "VideoBoundingBox",
                    "classifications": [],
                    "bounding_box": {
                      "top": 146.0,
                      "left": 98.0,
                      "height": 382.0,
                      "width": 341.0
                    }
                  },
                  "cltd97cag01tp14ysz3my848t": {
                    "feature_id": "cltd97cag01tp14ysz3my848t",
                    "feature_schema_id": "cltd977np037n07x5d1buckz9",
                    "name": "line_video_frame",
                    "value": "line_video_frame",
                    "annotation_kind": "VideoPolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 680.0,
                        "y": 100.0
                      },
                      {
                        "x": 100.0,
                        "y": 190.0
                      }
                    ]
                  },
                  "cltd97cag01u514ys2bfg9vvo": {
                    "feature_id": "cltd97cag01u514ys2bfg9vvo",
                    "feature_schema_id": "cltd977np037p07x55hu71x6k",
                    "name": "video_mask",
                    "value": "video_mask",
                    "annotation_kind": "VideoSegmentationMask",
                    "classifications": [],
                    "mask": {
                      "url": "https://api.labelbox.com/api/v1/projects/cltd977uu016y08xyggep8aiv/annotations/cltd97cag01u514ys2bfg9vvo/index/24/mask"
                    },
                    "composite_mask": {
                      "url": "https://api.labelbox.com/api/v1/tasks/clur7t5d501i707wfawrp050b/masks/cltd97a7h01tb14ysldafd9es/index/24",
                      "color_rgb": [
                        76,
                        104,
                        177
                      ]
                    }
                  },
                  "cltd97cah01u614ysvntxcyub": {
                    "feature_id": "cltd97cah01u614ysvntxcyub",
                    "feature_schema_id": "cltd977np037z07x5gclv41y4",
                    "name": "mask_with_text_subclass",
                    "value": "mask_with_text_subclass",
                    "annotation_kind": "VideoSegmentationMask",
                    "classifications": [],
                    "mask": {
                      "url": "https://api.labelbox.com/api/v1/projects/cltd977uu016y08xyggep8aiv/annotations/cltd97cah01u614ysvntxcyub/index/24/mask"
                    },
                    "composite_mask": {
                      "url": "https://api.labelbox.com/api/v1/tasks/clur7t5d501i707wfawrp050b/masks/cltd97a7h01tb14ysldafd9es/index/24",
                      "color_rgb": [
                        233,
                        218,
                        94
                      ]
                    }
                  },
                  "cltd97cah01u714ysbqbvqs6b": {
                    "feature_id": "cltd97cah01u714ysbqbvqs6b",
                    "feature_schema_id": "cltd977np037p07x55hu71x6k",
                    "name": "video_mask",
                    "value": "video_mask",
                    "annotation_kind": "VideoSegmentationMask",
                    "classifications": [],
                    "mask": {
                      "url": "https://api.labelbox.com/api/v1/projects/cltd977uu016y08xyggep8aiv/annotations/cltd97cah01u714ysbqbvqs6b/index/24/mask"
                    },
                    "composite_mask": {
                      "url": "https://api.labelbox.com/api/v1/tasks/clur7t5d501i707wfawrp050b/masks/cltd97a7h01tb14ysldafd9es/index/24",
                      "color_rgb": [
                        83,
                        152,
                        103
                      ]
                    }
                  }
                },
                "classifications": []
              },
              "25": {
                "objects": {
                  "cltd97cag01tm14ys8tm4x7yl": {
                    "feature_id": "cltd97cag01tm14ys8tm4x7yl",
                    "feature_schema_id": "cltd977no037j07x5d82dacbc",
                    "name": "bbox_video",
                    "value": "bbox_video",
                    "annotation_kind": "VideoBoundingBox",
                    "classifications": [],
                    "bounding_box": {
                      "top": 617.0,
                      "left": 1371.0,
                      "height": 419.0,
                      "width": 505.0
                    }
                  },
                  "cltd97cag01tn14yspqge5yph": {
                    "feature_id": "cltd97cag01tn14yspqge5yph",
                    "feature_schema_id": "cltd977no037j07x5d82dacbc",
                    "name": "bbox_video",
                    "value": "bbox_video",
                    "annotation_kind": "VideoBoundingBox",
                    "classifications": [],
                    "bounding_box": {
                      "top": 146.0,
                      "left": 98.0,
                      "height": 382.0,
                      "width": 341.0
                    }
                  },
                  "cltd97cag01tp14ysz3my848t": {
                    "feature_id": "cltd97cag01tp14ysz3my848t",
                    "feature_schema_id": "cltd977np037n07x5d1buckz9",
                    "name": "line_video_frame",
                    "value": "line_video_frame",
                    "annotation_kind": "VideoPolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 680.0,
                        "y": 99.99999999999999
                      },
                      {
                        "x": 99.99999999999999,
                        "y": 189.99999999999997
                      }
                    ]
                  }
                },
                "classifications": []
              },
              "26": {
                "objects": {
                  "cltd97cag01tm14ys8tm4x7yl": {
                    "feature_id": "cltd97cag01tm14ys8tm4x7yl",
                    "feature_schema_id": "cltd977no037j07x5d82dacbc",
                    "name": "bbox_video",
                    "value": "bbox_video",
                    "annotation_kind": "VideoBoundingBox",
                    "classifications": [],
                    "bounding_box": {
                      "top": 617.0,
                      "left": 1371.0,
                      "height": 419.0,
                      "width": 505.0
                    }
                  },
                  "cltd97cag01tn14yspqge5yph": {
                    "feature_id": "cltd97cag01tn14yspqge5yph",
                    "feature_schema_id": "cltd977no037j07x5d82dacbc",
                    "name": "bbox_video",
                    "value": "bbox_video",
                    "annotation_kind": "VideoBoundingBox",
                    "classifications": [],
                    "bounding_box": {
                      "top": 146.0,
                      "left": 98.0,
                      "height": 382.0,
                      "width": 341.0
                    }
                  },
                  "cltd97cag01tp14ysz3my848t": {
                    "feature_id": "cltd97cag01tp14ysz3my848t",
                    "feature_schema_id": "cltd977np037n07x5d1buckz9",
                    "name": "line_video_frame",
                    "value": "line_video_frame",
                    "annotation_kind": "VideoPolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 680.0,
                        "y": 100.0
                      },
                      {
                        "x": 100.0,
                        "y": 190.0
                      }
                    ]
                  },
                  "cltd97cah01ua14ysr6u1sh7w": {
                    "feature_id": "cltd97cah01ua14ysr6u1sh7w",
                    "feature_schema_id": "cltd977np037z07x5gclv41y4",
                    "name": "mask_with_text_subclass",
                    "value": "mask_with_text_subclass",
                    "annotation_kind": "VideoSegmentationMask",
                    "classifications": [],
                    "mask": {
                      "url": "https://api.labelbox.com/api/v1/projects/cltd977uu016y08xyggep8aiv/annotations/cltd97cah01ua14ysr6u1sh7w/index/26/mask"
                    },
                    "composite_mask": {
                      "url": "https://api.labelbox.com/api/v1/tasks/clur7t5d501i707wfawrp050b/masks/cltd97a7h01tb14ysldafd9es/index/26",
                      "color_rgb": [
                        226,
                        211,
                        41
                      ]
                    }
                  }
                },
                "classifications": []
              },
              "27": {
                "objects": {
                  "cltd97cag01tm14ys8tm4x7yl": {
                    "feature_id": "cltd97cag01tm14ys8tm4x7yl",
                    "feature_schema_id": "cltd977no037j07x5d82dacbc",
                    "name": "bbox_video",
                    "value": "bbox_video",
                    "annotation_kind": "VideoBoundingBox",
                    "classifications": [],
                    "bounding_box": {
                      "top": 617.0,
                      "left": 1371.0,
                      "height": 419.0,
                      "width": 505.0
                    }
                  },
                  "cltd97cag01tn14yspqge5yph": {
                    "feature_id": "cltd97cag01tn14yspqge5yph",
                    "feature_schema_id": "cltd977no037j07x5d82dacbc",
                    "name": "bbox_video",
                    "value": "bbox_video",
                    "annotation_kind": "VideoBoundingBox",
                    "classifications": [],
                    "bounding_box": {
                      "top": 146.0,
                      "left": 98.0,
                      "height": 382.0,
                      "width": 341.0
                    }
                  },
                  "cltd97cag01tp14ysz3my848t": {
                    "feature_id": "cltd97cag01tp14ysz3my848t",
                    "feature_schema_id": "cltd977np037n07x5d1buckz9",
                    "name": "line_video_frame",
                    "value": "line_video_frame",
                    "annotation_kind": "VideoPolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 680.0,
                        "y": 100.0
                      },
                      {
                        "x": 100.0,
                        "y": 190.0
                      }
                    ]
                  }
                },
                "classifications": []
              },
              "5": {
                "objects": {
                  "cltd97cag01tp14ysz3my848t": {
                    "feature_id": "cltd97cag01tp14ysz3my848t",
                    "feature_schema_id": "cltd977np037n07x5d1buckz9",
                    "name": "line_video_frame",
                    "value": "line_video_frame",
                    "annotation_kind": "VideoPolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 680.0,
                        "y": 100.0
                      },
                      {
                        "x": 100.0,
                        "y": 190.0
                      }
                    ]
                  }
                },
                "classifications": []
              },
              "6": {
                "objects": {
                  "cltd97cag01tp14ysz3my848t": {
                    "feature_id": "cltd97cag01tp14ysz3my848t",
                    "feature_schema_id": "cltd977np037n07x5d1buckz9",
                    "name": "line_video_frame",
                    "value": "line_video_frame",
                    "annotation_kind": "VideoPolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 680.0,
                        "y": 100.0
                      },
                      {
                        "x": 100.0,
                        "y": 190.0
                      }
                    ]
                  }
                },
                "classifications": []
              },
              "7": {
                "objects": {
                  "cltd97cag01tp14ysz3my848t": {
                    "feature_id": "cltd97cag01tp14ysz3my848t",
                    "feature_schema_id": "cltd977np037n07x5d1buckz9",
                    "name": "line_video_frame",
                    "value": "line_video_frame",
                    "annotation_kind": "VideoPolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 680.0,
                        "y": 100.0
                      },
                      {
                        "x": 100.0,
                        "y": 190.00000000000003
                      }
                    ]
                  }
                },
                "classifications": []
              },
              "8": {
                "objects": {
                  "cltd97cag01tp14ysz3my848t": {
                    "feature_id": "cltd97cag01tp14ysz3my848t",
                    "feature_schema_id": "cltd977np037n07x5d1buckz9",
                    "name": "line_video_frame",
                    "value": "line_video_frame",
                    "annotation_kind": "VideoPolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 680.0,
                        "y": 100.0
                      },
                      {
                        "x": 100.0,
                        "y": 190.0
                      }
                    ]
                  }
                },
                "classifications": []
              },
              "20": {
                "objects": {
                  "cltd97cag01tp14ysz3my848t": {
                    "feature_id": "cltd97cag01tp14ysz3my848t",
                    "feature_schema_id": "cltd977np037n07x5d1buckz9",
                    "name": "line_video_frame",
                    "value": "line_video_frame",
                    "annotation_kind": "VideoPolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 680.0,
                        "y": 100.0
                      },
                      {
                        "x": 100.0,
                        "y": 190.0
                      }
                    ]
                  },
                  "cltd97cag01u514ys2bfg9vvo": {
                    "feature_id": "cltd97cag01u514ys2bfg9vvo",
                    "feature_schema_id": "cltd977np037p07x55hu71x6k",
                    "name": "video_mask",
                    "value": "video_mask",
                    "annotation_kind": "VideoSegmentationMask",
                    "classifications": [],
                    "mask": {
                      "url": "https://api.labelbox.com/api/v1/projects/cltd977uu016y08xyggep8aiv/annotations/cltd97cag01u514ys2bfg9vvo/index/20/mask"
                    },
                    "composite_mask": {
                      "url": "https://api.labelbox.com/api/v1/tasks/clur7t5d501i707wfawrp050b/masks/cltd97a7h01tb14ysldafd9es/index/20",
                      "color_rgb": [
                        76,
                        104,
                        177
                      ]
                    }
                  },
                  "cltd97cah01u614ysvntxcyub": {
                    "feature_id": "cltd97cah01u614ysvntxcyub",
                    "feature_schema_id": "cltd977np037z07x5gclv41y4",
                    "name": "mask_with_text_subclass",
                    "value": "mask_with_text_subclass",
                    "annotation_kind": "VideoSegmentationMask",
                    "classifications": [],
                    "mask": {
                      "url": "https://api.labelbox.com/api/v1/projects/cltd977uu016y08xyggep8aiv/annotations/cltd97cah01u614ysvntxcyub/index/20/mask"
                    },
                    "composite_mask": {
                      "url": "https://api.labelbox.com/api/v1/tasks/clur7t5d501i707wfawrp050b/masks/cltd97a7h01tb14ysldafd9es/index/20",
                      "color_rgb": [
                        233,
                        218,
                        94
                      ]
                    }
                  },
                  "cltd97cah01u714ysbqbvqs6b": {
                    "feature_id": "cltd97cah01u714ysbqbvqs6b",
                    "feature_schema_id": "cltd977np037p07x55hu71x6k",
                    "name": "video_mask",
                    "value": "video_mask",
                    "annotation_kind": "VideoSegmentationMask",
                    "classifications": [],
                    "mask": {
                      "url": "https://api.labelbox.com/api/v1/projects/cltd977uu016y08xyggep8aiv/annotations/cltd97cah01u714ysbqbvqs6b/index/20/mask"
                    },
                    "composite_mask": {
                      "url": "https://api.labelbox.com/api/v1/tasks/clur7t5d501i707wfawrp050b/masks/cltd97a7h01tb14ysldafd9es/index/20",
                      "color_rgb": [
                        83,
                        152,
                        103
                      ]
                    }
                  }
                },
                "classifications": []
              },
              "28": {
                "objects": {
                  "cltd97cag01tp14ysz3my848t": {
                    "feature_id": "cltd97cag01tp14ysz3my848t",
                    "feature_schema_id": "cltd977np037n07x5d1buckz9",
                    "name": "line_video_frame",
                    "value": "line_video_frame",
                    "annotation_kind": "VideoPolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 680.0,
                        "y": 100.0
                      },
                      {
                        "x": 100.0,
                        "y": 190.0
                      }
                    ]
                  }
                },
                "classifications": []
              },
              "30": {
                "objects": {
                  "cltd97cag01tp14ysz3my848t": {
                    "feature_id": "cltd97cag01tp14ysz3my848t",
                    "feature_schema_id": "cltd977np037n07x5d1buckz9",
                    "name": "line_video_frame",
                    "value": "line_video_frame",
                    "annotation_kind": "VideoPolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 680.0,
                        "y": 100.0
                      },
                      {
                        "x": 100.0,
                        "y": 190.0
                      }
                    ]
                  }
                },
                "classifications": []
              },
              "31": {
                "objects": {
                  "cltd97cag01tp14ysz3my848t": {
                    "feature_id": "cltd97cag01tp14ysz3my848t",
                    "feature_schema_id": "cltd977np037n07x5d1buckz9",
                    "name": "line_video_frame",
                    "value": "line_video_frame",
                    "annotation_kind": "VideoPolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 680.0,
                        "y": 100.0
                      },
                      {
                        "x": 100.0,
                        "y": 190.0
                      }
                    ]
                  }
                },
                "classifications": []
              },
              "32": {
                "objects": {
                  "cltd97cag01tp14ysz3my848t": {
                    "feature_id": "cltd97cag01tp14ysz3my848t",
                    "feature_schema_id": "cltd977np037n07x5d1buckz9",
                    "name": "line_video_frame",
                    "value": "line_video_frame",
                    "annotation_kind": "VideoPolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 680.0,
                        "y": 100.0
                      },
                      {
                        "x": 100.0,
                        "y": 190.0
                      }
                    ]
                  }
                },
                "classifications": []
              },
              "33": {
                "objects": {
                  "cltd97cag01tp14ysz3my848t": {
                    "feature_id": "cltd97cag01tp14ysz3my848t",
                    "feature_schema_id": "cltd977np037n07x5d1buckz9",
                    "name": "line_video_frame",
                    "value": "line_video_frame",
                    "annotation_kind": "VideoPolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 680.0,
                        "y": 100.0
                      },
                      {
                        "x": 100.0,
                        "y": 190.0
                      }
                    ]
                  }
                },
                "classifications": []
              },
              "34": {
                "objects": {
                  "cltd97cag01tp14ysz3my848t": {
                    "feature_id": "cltd97cag01tp14ysz3my848t",
                    "feature_schema_id": "cltd977np037n07x5d1buckz9",
                    "name": "line_video_frame",
                    "value": "line_video_frame",
                    "annotation_kind": "VideoPolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 680.0,
                        "y": 100.0
                      },
                      {
                        "x": 100.0,
                        "y": 190.0
                      }
                    ]
                  }
                },
                "classifications": []
              },
              "36": {
                "objects": {
                  "cltd97cag01tp14ysz3my848t": {
                    "feature_id": "cltd97cag01tp14ysz3my848t",
                    "feature_schema_id": "cltd977np037n07x5d1buckz9",
                    "name": "line_video_frame",
                    "value": "line_video_frame",
                    "annotation_kind": "VideoPolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 680.0,
                        "y": 100.0
                      },
                      {
                        "x": 100.0,
                        "y": 190.0
                      }
                    ]
                  }
                },
                "classifications": []
              },
              "37": {
                "objects": {
                  "cltd97cag01tp14ysz3my848t": {
                    "feature_id": "cltd97cag01tp14ysz3my848t",
                    "feature_schema_id": "cltd977np037n07x5d1buckz9",
                    "name": "line_video_frame",
                    "value": "line_video_frame",
                    "annotation_kind": "VideoPolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 680.0,
                        "y": 100.0
                      },
                      {
                        "x": 100.0,
                        "y": 190.0
                      }
                    ]
                  }
                },
                "classifications": []
              },
              "38": {
                "objects": {
                  "cltd97cag01tp14ysz3my848t": {
                    "feature_id": "cltd97cag01tp14ysz3my848t",
                    "feature_schema_id": "cltd977np037n07x5d1buckz9",
                    "name": "line_video_frame",
                    "value": "line_video_frame",
                    "annotation_kind": "VideoPolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 680.0,
                        "y": 100.0
                      },
                      {
                        "x": 100.0,
                        "y": 190.0
                      }
                    ]
                  }
                },
                "classifications": []
              },
              "39": {
                "objects": {
                  "cltd97cag01tp14ysz3my848t": {
                    "feature_id": "cltd97cag01tp14ysz3my848t",
                    "feature_schema_id": "cltd977np037n07x5d1buckz9",
                    "name": "line_video_frame",
                    "value": "line_video_frame",
                    "annotation_kind": "VideoPolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 680.0,
                        "y": 100.0
                      },
                      {
                        "x": 100.0,
                        "y": 190.0
                      }
                    ]
                  }
                },
                "classifications": []
              },
              "40": {
                "objects": {
                  "cltd97cag01tp14ysz3my848t": {
                    "feature_id": "cltd97cag01tp14ysz3my848t",
                    "feature_schema_id": "cltd977np037n07x5d1buckz9",
                    "name": "line_video_frame",
                    "value": "line_video_frame",
                    "annotation_kind": "VideoPolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 680.0,
                        "y": 100.0
                      },
                      {
                        "x": 100.0,
                        "y": 190.0
                      }
                    ]
                  }
                },
                "classifications": []
              },
              "41": {
                "objects": {
                  "cltd97cag01tp14ysz3my848t": {
                    "feature_id": "cltd97cag01tp14ysz3my848t",
                    "feature_schema_id": "cltd977np037n07x5d1buckz9",
                    "name": "line_video_frame",
                    "value": "line_video_frame",
                    "annotation_kind": "VideoPolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 680.0,
                        "y": 100.0
                      },
                      {
                        "x": 100.0,
                        "y": 190.0
                      }
                    ]
                  }
                },
                "classifications": []
              },
              "42": {
                "objects": {
                  "cltd97cag01tp14ysz3my848t": {
                    "feature_id": "cltd97cag01tp14ysz3my848t",
                    "feature_schema_id": "cltd977np037n07x5d1buckz9",
                    "name": "line_video_frame",
                    "value": "line_video_frame",
                    "annotation_kind": "VideoPolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 680.0,
                        "y": 100.0
                      },
                      {
                        "x": 100.0,
                        "y": 190.0
                      }
                    ]
                  }
                },
                "classifications": []
              },
              "43": {
                "objects": {
                  "cltd97cag01tp14ysz3my848t": {
                    "feature_id": "cltd97cag01tp14ysz3my848t",
                    "feature_schema_id": "cltd977np037n07x5d1buckz9",
                    "name": "line_video_frame",
                    "value": "line_video_frame",
                    "annotation_kind": "VideoPolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 680.0,
                        "y": 100.0
                      },
                      {
                        "x": 100.0,
                        "y": 190.0
                      }
                    ]
                  }
                },
                "classifications": []
              },
              "44": {
                "objects": {
                  "cltd97cag01tp14ysz3my848t": {
                    "feature_id": "cltd97cag01tp14ysz3my848t",
                    "feature_schema_id": "cltd977np037n07x5d1buckz9",
                    "name": "line_video_frame",
                    "value": "line_video_frame",
                    "annotation_kind": "VideoPolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 680.0,
                        "y": 100.0
                      },
                      {
                        "x": 100.0,
                        "y": 190.0
                      }
                    ]
                  }
                },
                "classifications": []
              },
              "45": {
                "objects": {
                  "cltd97cag01tp14ysz3my848t": {
                    "feature_id": "cltd97cag01tp14ysz3my848t",
                    "feature_schema_id": "cltd977np037n07x5d1buckz9",
                    "name": "line_video_frame",
                    "value": "line_video_frame",
                    "annotation_kind": "VideoPolyline",
                    "classifications": [],
                    "line": [
                      {
                        "x": 680.0,
                        "y": 100.0
                      },
                      {
                        "x": 100.0,
                        "y": 190.0
                      }
                    ]
                  }
                },
                "classifications": []
              },
              "1": {
                "objects": {
                  "cltd97cah01u814ys0k41cttu": {
                    "feature_id": "cltd97cah01u814ys0k41cttu",
                    "feature_schema_id": "cltd977np037p07x55hu71x6k",
                    "name": "video_mask",
                    "value": "video_mask",
                    "annotation_kind": "VideoSegmentationMask",
                    "classifications": [],
                    "mask": {
                      "url": "https://api.labelbox.com/api/v1/projects/cltd977uu016y08xyggep8aiv/annotations/cltd97cah01u814ys0k41cttu/index/1/mask"
                    },
                    "composite_mask": {
                      "url": "https://api.labelbox.com/api/v1/tasks/clur7t5d501i707wfawrp050b/masks/cltd97a7h01tb14ysldafd9es/index/1",
                      "color_rgb": [
                        169,
                        248,
                        152
                      ]
                    }
                  },
                  "cltd97cah01u914ysy7t58lvy": {
                    "feature_id": "cltd97cah01u914ysy7t58lvy",
                    "feature_schema_id": "cltd977np037p07x55hu71x6k",
                    "name": "video_mask",
                    "value": "video_mask",
                    "annotation_kind": "VideoSegmentationMask",
                    "classifications": [],
                    "mask": {
                      "url": "https://api.labelbox.com/api/v1/projects/cltd977uu016y08xyggep8aiv/annotations/cltd97cah01u914ysy7t58lvy/index/1/mask"
                    },
                    "composite_mask": {
                      "url": "https://api.labelbox.com/api/v1/tasks/clur7t5d501i707wfawrp050b/masks/cltd97a7h01tb14ysldafd9es/index/1",
                      "color_rgb": [
                        227,
                        135,
                        126
                      ]
                    }
                  }
                },
                "classifications": []
              }
            },
            "segments": {
              "cltd97cag01td14ysxw2n8xkz": [
                [
                  29,
                  35
                ]
              ],
              "cltd97cag01te14ys1jdf1fku": [
                [
                  29,
                  35
                ]
              ],
              "cltd97cag01tg14ys171a3qqr": [
                [
                  9,
                  15
                ]
              ],
              "cltd97cag01th14ys6fyyxyk9": [
                [
                  13,
                  19
                ]
              ],
              "cltd97cag01ti14ys8wedf89d": [
                [
                  10,
                  13
                ]
              ],
              "cltd97cag01tm14ys8tm4x7yl": [
                [
                  22,
                  27
                ]
              ],
              "cltd97cag01tn14yspqge5yph": [
                [
                  22,
                  27
                ]
              ],
              "cltd97cag01to14yslakirofl": [
                [
                  17,
                  17
                ]
              ],
              "cltd97cag01tp14ysz3my848t": [
                [
                  5,
                  20
                ],
                [
                  24,
                  45
                ]
              ]
            },
            "key_frame_feature_map": {
              "cltd97cag01tc14yspm0lev2d": [
                29,
                35
              ],
              "cltd97cag01tp14ysz3my848t": [
                24,
                5,
                20,
                45
              ],
              "cltd97cag01tf14ysho6tfl7j": [
                9,
                15
              ],
              "cltd97cag01th14ys6fyyxyk9": [
                13,
                19
              ],
              "cltd97cag01ti14ys8wedf89d": [
                13,
                10
              ],
              "cltd97cag01to14yslakirofl": [
                17
              ],
              "cltd97cag01tm14ys8tm4x7yl": [
                22,
                27
              ],
              "cltd97cag01tn14yspqge5yph": [
                22,
                27
              ],
              "cltd97cag01u514ys2bfg9vvo": [
                24,
                20
              ],
              "cltd97cah01u614ysvntxcyub": [
                24,
                20
              ],
              "cltd97cah01u714ysbqbvqs6b": [
                24,
                20
              ],
              "cltd97cah01ua14ysr6u1sh7w": [
                26
              ],
              "cltd97cah01u814ys0k41cttu": [
                1
              ],
              "cltd97cah01u914ysy7t58lvy": [
                1
              ]
            },
            "classifications": [
              {
                "feature_id": "cltd97cag01tq14ysb4m3nyjo",
                "feature_schema_id": "cltd977nq039107x5dsml2knm",
                "name": "checklist_class_global",
                "value": "checklist_class_global",
                "checklist_answers": [
                  {
                    "feature_id": "cltd97cag01tr14ysozjbx0c6",
                    "feature_schema_id": "cltd977nq039207x5ei1605s1",
                    "name": "first_checklist_answer",
                    "value": "first_checklist_answer",
                    "classifications": []
                  },
                  {
                    "feature_id": "cltd97cag01ts14ys4vvxihnw",
                    "feature_schema_id": "cltd977nq039407x50qs3g61l",
                    "name": "second_checklist_answer",
                    "value": "second_checklist_answer",
                    "classifications": []
                  }
                ]
              },
              {
                "feature_id": "cltd97cag01tt14yscmqh2izy",
                "feature_schema_id": "cltd977nq038v07x57qz18lnj",
                "name": "radio_class_global",
                "value": "radio_class_global",
                "radio_answer": {
                  "feature_id": "cltd97cag01tu14ysq046ad6p",
                  "feature_schema_id": "cltd977nq038w07x5bxw3exp9",
                  "name": "first_radio_answer",
                  "value": "first_radio_answer",
                  "classifications": []
                }
              },
              {
                "feature_id": "cltd97cag01tv14ys1y3u4jre",
                "feature_schema_id": "cltd977nq038n07x52t8s0j1f",
                "name": "nested_checklist_question",
                "value": "nested_checklist_question",
                "checklist_answers": [
                  {
                    "feature_id": "cltd97cag01tw14ysieenbrzy",
                    "feature_schema_id": "cltd977nq038o07x528j39vc4",
                    "name": "first_checklist_answer",
                    "value": "first_checklist_answer",
                    "classifications": [
                      {
                        "feature_id": "cltd97cag01tx14ysaqs15mjh",
                        "feature_schema_id": "cltd977nq038p07x5bfuf7v56",
                        "name": "sub_checklist_question",
                        "value": "sub_checklist_question",
                        "checklist_answers": [
                          {
                            "feature_id": "cltd97cag01ty14yshujgka3h",
                            "feature_schema_id": "cltd977nq038q07x52m1h32mj",
                            "name": "first_sub_checklist_answer",
                            "value": "first_sub_checklist_answer",
                            "classifications": []
                          }
                        ]
                      }
                    ]
                  }
                ]
              },
              {
                "feature_id": "cltd97cag01tz14ys73p2w3oa",
                "feature_schema_id": "cltd977nq038f07x5bgrc2fm6",
                "name": "nested_radio_question",
                "value": "nested_radio_question",
                "radio_answer": {
                  "feature_id": "cltd97cag01u014ysho9dljue",
                  "feature_schema_id": "cltd977nq038g07x50ewqbbi3",
                  "name": "first_radio_answer",
                  "value": "first_radio_answer",
                  "classifications": [
                    {
                      "feature_id": "cltd97cag01u114ysktcruoro",
                      "feature_schema_id": "cltd977nq038h07x59zx07wfe",
                      "name": "sub_radio_question",
                      "value": "sub_radio_question",
                      "radio_answer": {
                        "feature_id": "cltd97cag01u214ysxi71u4qi",
                        "feature_schema_id": "cltd977nq038i07x5et0717w6",
                        "name": "first_sub_radio_answer",
                        "value": "first_sub_radio_answer",
                        "classifications": []
                      }
                    }
                  ]
                }
              },
              {
                "feature_id": "cltd97cag01u314ysx1h72f0h",
                "feature_schema_id": "cltd977nq039707x535ufeo39",
                "name": "free_text",
                "value": "free_text",
                "text_answer": {
                  "content": "sample text"
                }
              }
            ]
          }
        }
      ],
      "project_details": {
        "ontology_id": "cltd977mx037i07x52amua8iw",
        "task_name": "Done",
        "batch_id": "1026dd60-da52-11ee-9774-cfce344359c8",
        "batch_name": "first-batch-video-demo2",
        "workflow_status": "DONE",
        "priority": 5,
        "consensus_expected_label_count": 1,
        "workflow_history": []
      }
    }
  }
}
```

### Sample Model Run Export

```json JSON expandable theme={"theme":"css-variables"}
{
  "data_row": {
    "id": "cldtd2nai107407y5979xbu97",
    "global_key": "3548cfa1-1a42-40f9-a246-2aa7c696dcbe",
    "row_data": "https://storage.googleapis.com/labelbox-datasets/video-sample-data/sample-video-2.mp4",
    "details": {
      "dataset_id": "cldtd2mox11dz07wh3gkk9awx",
      "dataset_name": "video_prediction_demo",
      "created_at": "2023-02-06T22:04:54.000+00:00",
      "updated_at": "2023-02-06T22:04:54.000+00:00",
      "created_by": "labelbox.public.user@labelbox.com"
    }
  },
  "media_attributes": {
    "height": 1080,
    "width": 1920,
    "mime_type": "video/mp4",
    "frame_rate": 10,
    "frame_count": 100
  },
  "attachments": [],
  "metadata_fields": [],
  "experiments": {
    "a088408e-81af-0152-f2ca-0b870c7a2bfa": {
      "name": "video_model_run_8d7dbe3d-30e0-4c9c-becd-26afb2386bc9",
      "runs": {
        "a088408e-ce36-09c1-cbd9-d43c677f7773": {
          "name": "iteration 1",
          "run_data_row_id": "999a247e-0b57-415e-af7e-d3a87cc6559b",
          "labels": [
            {
              "label_kind": "Video",
              "version": "1.0.0",
              "id": "cldteelzh08us1331q4pzj2ke",
              "annotations": {
                "frames": {
                  "17": {
                    "objects": {
                      "12a89fb9-73e4-4c5e-93c1-a8727e51c122": {
                        "feature_id": "12a89fb9-73e4-4c5e-93c1-a8727e51c122",
                        "feature_schema_id": "cldtd2nmm107i07y53ii0d6ly",
                        "name": "point_video",
                        "value": "point_video",
                        "annotation_kind": "VideoPoint",
                        "classifications": [],
                        "point": {
                          "x": 660.134,
                          "y": 407.926
                        }
                      }
                    },
                    "classifications": []
                  },
                  "5": {
                    "objects": {
                      "878b4842-d57b-43e4-8327-a0c5295f2a84": {
                        "feature_id": "878b4842-d57b-43e4-8327-a0c5295f2a84",
                        "feature_schema_id": "cldtd2nmm107k07y577q807az",
                        "name": "line_video_frame",
                        "value": "line_video_frame",
                        "annotation_kind": "VideoPolyline",
                        "classifications": [],
                        "line": [
                          {
                            "x": 680.0,
                            "y": 100.0
                          },
                          {
                            "x": 100.0,
                            "y": 190.0
                          },
                          {
                            "x": 190.0,
                            "y": 220.0
                          }
                        ]
                      },
                      "9cd6cd8e-0511-42a9-bc2f-726cf5c8f983": {
                        "feature_id": "9cd6cd8e-0511-42a9-bc2f-726cf5c8f983",
                        "feature_schema_id": "cldtd2nml107a07y51qtydeoj",
                        "name": "bbox_video",
                        "value": "bbox_video",
                        "annotation_kind": "VideoBoundingBox",
                        "classifications": [],
                        "bounding_box": {
                          "top": 146.0,
                          "left": 98.0,
                          "height": 382.0,
                          "width": 341.0
                        }
                      }
                    },
                    "classifications": []
                  },
                  "12": {
                    "objects": {
                      "878b4842-d57b-43e4-8327-a0c5295f2a84": {
                        "feature_id": "878b4842-d57b-43e4-8327-a0c5295f2a84",
                        "feature_schema_id": "cldtd2nmm107k07y577q807az",
                        "name": "line_video_frame",
                        "value": "line_video_frame",
                        "annotation_kind": "VideoPolyline",
                        "classifications": [],
                        "line": [
                          {
                            "x": 680.0,
                            "y": 280.0
                          },
                          {
                            "x": 300.0,
                            "y": 380.0
                          },
                          {
                            "x": 400.0,
                            "y": 460.0
                          }
                        ]
                      }
                    },
                    "classifications": []
                  },
                  "20": {
                    "objects": {
                      "878b4842-d57b-43e4-8327-a0c5295f2a84": {
                        "feature_id": "878b4842-d57b-43e4-8327-a0c5295f2a84",
                        "feature_schema_id": "cldtd2nmm107k07y577q807az",
                        "name": "line_video_frame",
                        "value": "line_video_frame",
                        "annotation_kind": "VideoPolyline",
                        "classifications": [],
                        "line": [
                          {
                            "x": 680.0,
                            "y": 180.0
                          },
                          {
                            "x": 100.0,
                            "y": 200.0
                          },
                          {
                            "x": 200.0,
                            "y": 260.0
                          }
                        ]
                      }
                    },
                    "classifications": []
                  },
                  "24": {
                    "objects": {
                      "878b4842-d57b-43e4-8327-a0c5295f2a84": {
                        "feature_id": "878b4842-d57b-43e4-8327-a0c5295f2a84",
                        "feature_schema_id": "cldtd2nmm107k07y577q807az",
                        "name": "line_video_frame",
                        "value": "line_video_frame",
                        "annotation_kind": "VideoPolyline",
                        "classifications": [],
                        "line": [
                          {
                            "x": 300.0,
                            "y": 310.0
                          },
                          {
                            "x": 330.0,
                            "y": 430.0
                          }
                        ]
                      }
                    },
                    "classifications": []
                  },
                  "45": {
                    "objects": {
                      "878b4842-d57b-43e4-8327-a0c5295f2a84": {
                        "feature_id": "878b4842-d57b-43e4-8327-a0c5295f2a84",
                        "feature_schema_id": "cldtd2nmm107k07y577q807az",
                        "name": "line_video_frame",
                        "value": "line_video_frame",
                        "annotation_kind": "VideoPolyline",
                        "classifications": [],
                        "line": [
                          {
                            "x": 600.0,
                            "y": 810.0
                          },
                          {
                            "x": 900.0,
                            "y": 930.0
                          }
                        ]
                      }
                    },
                    "classifications": []
                  },
                  "1": {
                    "objects": {
                      "9cd6cd8e-0511-42a9-bc2f-726cf5c8f983": {
                        "feature_id": "9cd6cd8e-0511-42a9-bc2f-726cf5c8f983",
                        "feature_schema_id": "cldtd2nml107a07y51qtydeoj",
                        "name": "bbox_video",
                        "value": "bbox_video",
                        "annotation_kind": "VideoBoundingBox",
                        "classifications": [],
                        "bounding_box": {
                          "top": 146.0,
                          "left": 98.0,
                          "height": 382.0,
                          "width": 341.0
                        }
                      }
                    },
                    "classifications": []
                  }
                },
                "segments": {
                  "12a89fb9-73e4-4c5e-93c1-a8727e51c122": [
                    [
                      17,
                      17
                    ]
                  ],
                  "878b4842-d57b-43e4-8327-a0c5295f2a84": [
                    [
                      5,
                      5
                    ],
                    [
                      12,
                      12
                    ],
                    [
                      20,
                      20
                    ],
                    [
                      24,
                      24
                    ],
                    [
                      45,
                      45
                    ]
                  ],
                  "9cd6cd8e-0511-42a9-bc2f-726cf5c8f983": [
                    [
                      5,
                      5
                    ],
                    [
                      1,
                      1
                    ]
                  ]
                },
                "key_frame_feature_map": {
                  "12a89fb9-73e4-4c5e-93c1-a8727e51c122": [
                    17
                  ],
                  "878b4842-d57b-43e4-8327-a0c5295f2a84": [
                    5,
                    12,
                    20,
                    24,
                    45
                  ],
                  "9cd6cd8e-0511-42a9-bc2f-726cf5c8f983": [
                    5,
                    1
                  ]
                },
                "classifications": [
                  {
                    "feature_id": "3644a35a-8aaa-4905-a2d2-9abac1f3dacb",
                    "feature_schema_id": "cldtd2nmm107s07y57ddy5wnn",
                    "name": "checklist_question",
                    "value": "checklist_question",
                    "checklist_answers": [
                      {
                        "feature_id": "86805980-7632-4dd0-9669-299730927326",
                        "feature_schema_id": "cldtd2nmm107v07y5cmixhywh",
                        "name": "second_checklist_answer",
                        "value": "second_checklist_answer",
                        "classifications": []
                      },
                      {
                        "feature_id": "c5bfb5c1-aa9a-4a6a-b671-be7c20df4b19",
                        "feature_schema_id": "cldtd2nmm107t07y5cqdw94z7",
                        "name": "first_checklist_answer",
                        "value": "first_checklist_answer",
                        "classifications": []
                      }
                    ]
                  },
                  {
                    "feature_id": "3d302a82-cc91-41d9-8c69-0f1e2e66ea71",
                    "feature_schema_id": "cldtd2nmm107m07y5h45sg6av",
                    "name": "radio_question",
                    "value": "radio_question",
                    "radio_answer": {
                      "feature_id": "dab3bcfa-6c44-4f28-945b-1bc324edfab1",
                      "feature_schema_id": "cldtd2nmm107p07y542ff3hvd",
                      "name": "second_radio_answer",
                      "value": "second_radio_answer",
                      "classifications": []
                    }
                  },
                  {
                    "feature_id": "6dede674-18a6-4b95-abdd-13b3713f994a",
                    "feature_schema_id": "cldtd2nmm108607y5ghou4ma6",
                    "name": "nested_checklist_question",
                    "value": "nested_checklist_question",
                    "checklist_answers": [
                      {
                        "feature_id": "9f748713-ffb5-44db-a16e-396676dab6bc",
                        "feature_schema_id": "cldtd2nmm108707y50shn5996",
                        "name": "first_checklist_answer",
                        "value": "first_checklist_answer",
                        "classifications": [
                          {
                            "feature_id": "ca1f2173-38c6-4c23-96ba-8a3feb54e2c2",
                            "feature_schema_id": "cldtd2nmm108807y52ps9d5m4",
                            "name": "sub_checklist_question",
                            "value": "sub_checklist_question",
                            "checklist_answers": [
                              {
                                "feature_id": "2181d943-8a6a-4aef-aae0-4c28146e9008",
                                "feature_schema_id": "cldtd2nmm108907y5269ua7bz",
                                "name": "first_sub_checklist_answer",
                                "value": "first_sub_checklist_answer",
                                "classifications": []
                              }
                            ]
                          }
                        ]
                      }
                    ]
                  },
                  {
                    "feature_id": "b72dd664-93fc-46d1-8bd8-8dfe1fd2d7ef",
                    "feature_schema_id": "cldtd2nmm107y07y5fvhkb6c7",
                    "name": "nested_radio_question",
                    "value": "nested_radio_question",
                    "radio_answer": {
                      "feature_id": "5fc17086-601e-42e5-a5b1-a28eb96841ab",
                      "feature_schema_id": "cldtd2nmm107z07y59j8n62ym",
                      "name": "first_radio_answer",
                      "value": "first_radio_answer",
                      "classifications": [
                        {
                          "feature_id": "a74e9958-345f-4b42-925d-9d5938011cf6",
                          "feature_schema_id": "cldtd2nmm108007y5gs1y5eqv",
                          "name": "sub_radio_question",
                          "value": "sub_radio_question",
                          "radio_answer": {
                            "feature_id": "734a2cf1-571d-4855-869e-d92ff20a4cb4",
                            "feature_schema_id": "cldtd2nmm108107y53dvreuv2",
                            "name": "first_sub_radio_answer",
                            "value": "first_sub_radio_answer",
                            "classifications": []
                          }
                        }
                      ]
                    }
                  }
                ]
              }
            }
          ],
          "predictions": [
            {
              "label_kind": "Video",
              "version": "1.0.0",
              "id": "cldteelzh08us1331q4pzj2ke",
              "annotations": {
                "frames": {
                  "5": {
                    "objects": {
                      "1306f719-d588-4154-8335-792347bfdee2": {
                        "feature_id": "1306f719-d588-4154-8335-792347bfdee2",
                        "feature_schema_id": "cldtd2nmm107k07y577q807az",
                        "name": "line_video_frame",
                        "value": "line_video_frame",
                        "annotation_kind": "VideoPolyline",
                        "classifications": [],
                        "line": [
                          {
                            "x": 680.0,
                            "y": 100.0
                          },
                          {
                            "x": 100.0,
                            "y": 190.0
                          },
                          {
                            "x": 190.0,
                            "y": 220.0
                          }
                        ]
                      },
                      "a505f426-5a2e-4dea-b4fe-ab913e1d2a99": {
                        "feature_id": "a505f426-5a2e-4dea-b4fe-ab913e1d2a99",
                        "feature_schema_id": "cldtd2nml107a07y51qtydeoj",
                        "name": "bbox_video",
                        "value": "bbox_video",
                        "annotation_kind": "VideoBoundingBox",
                        "classifications": [],
                        "bounding_box": {
                          "top": 146.0,
                          "left": 98.0,
                          "height": 382.0,
                          "width": 341.0
                        }
                      }
                    },
                    "classifications": []
                  },
                  "12": {
                    "objects": {
                      "1306f719-d588-4154-8335-792347bfdee2": {
                        "feature_id": "1306f719-d588-4154-8335-792347bfdee2",
                        "feature_schema_id": "cldtd2nmm107k07y577q807az",
                        "name": "line_video_frame",
                        "value": "line_video_frame",
                        "annotation_kind": "VideoPolyline",
                        "classifications": [],
                        "line": [
                          {
                            "x": 680.0,
                            "y": 280.0
                          },
                          {
                            "x": 300.0,
                            "y": 380.0
                          },
                          {
                            "x": 400.0,
                            "y": 460.0
                          }
                        ]
                      }
                    },
                    "classifications": []
                  },
                  "20": {
                    "objects": {
                      "1306f719-d588-4154-8335-792347bfdee2": {
                        "feature_id": "1306f719-d588-4154-8335-792347bfdee2",
                        "feature_schema_id": "cldtd2nmm107k07y577q807az",
                        "name": "line_video_frame",
                        "value": "line_video_frame",
                        "annotation_kind": "VideoPolyline",
                        "classifications": [],
                        "line": [
                          {
                            "x": 680.0,
                            "y": 180.0
                          },
                          {
                            "x": 100.0,
                            "y": 200.0
                          },
                          {
                            "x": 200.0,
                            "y": 260.0
                          }
                        ]
                      }
                    },
                    "classifications": []
                  },
                  "24": {
                    "objects": {
                      "1306f719-d588-4154-8335-792347bfdee2": {
                        "feature_id": "1306f719-d588-4154-8335-792347bfdee2",
                        "feature_schema_id": "cldtd2nmm107k07y577q807az",
                        "name": "line_video_frame",
                        "value": "line_video_frame",
                        "annotation_kind": "VideoPolyline",
                        "classifications": [],
                        "line": [
                          {
                            "x": 300.0,
                            "y": 310.0
                          },
                          {
                            "x": 330.0,
                            "y": 430.0
                          }
                        ]
                      }
                    },
                    "classifications": []
                  },
                  "45": {
                    "objects": {
                      "1306f719-d588-4154-8335-792347bfdee2": {
                        "feature_id": "1306f719-d588-4154-8335-792347bfdee2",
                        "feature_schema_id": "cldtd2nmm107k07y577q807az",
                        "name": "line_video_frame",
                        "value": "line_video_frame",
                        "annotation_kind": "VideoPolyline",
                        "classifications": [],
                        "line": [
                          {
                            "x": 600.0,
                            "y": 810.0
                          },
                          {
                            "x": 900.0,
                            "y": 930.0
                          }
                        ]
                      }
                    },
                    "classifications": []
                  },
                  "17": {
                    "objects": {
                      "89b4636f-08cc-4fa2-832b-18bcda636722": {
                        "feature_id": "89b4636f-08cc-4fa2-832b-18bcda636722",
                        "feature_schema_id": "cldtd2nmm107i07y53ii0d6ly",
                        "name": "point_video",
                        "value": "point_video",
                        "annotation_kind": "VideoPoint",
                        "classifications": [],
                        "point": {
                          "x": 660.134,
                          "y": 407.926
                        }
                      }
                    },
                    "classifications": []
                  },
                  "1": {
                    "objects": {
                      "a505f426-5a2e-4dea-b4fe-ab913e1d2a99": {
                        "feature_id": "a505f426-5a2e-4dea-b4fe-ab913e1d2a99",
                        "feature_schema_id": "cldtd2nml107a07y51qtydeoj",
                        "name": "bbox_video",
                        "value": "bbox_video",
                        "annotation_kind": "VideoBoundingBox",
                        "classifications": [],
                        "bounding_box": {
                          "top": 146.0,
                          "left": 98.0,
                          "height": 382.0,
                          "width": 341.0
                        }
                      }
                    },
                    "classifications": []
                  }
                },
                "segments": {
                  "1306f719-d588-4154-8335-792347bfdee2": [
                    [
                      5,
                      5
                    ],
                    [
                      12,
                      12
                    ],
                    [
                      20,
                      20
                    ],
                    [
                      24,
                      24
                    ],
                    [
                      45,
                      45
                    ]
                  ],
                  "a505f426-5a2e-4dea-b4fe-ab913e1d2a99": [
                    [
                      5,
                      5
                    ],
                    [
                      1,
                      1
                    ]
                  ],
                  "89b4636f-08cc-4fa2-832b-18bcda636722": [
                    [
                      17,
                      17
                    ]
                  ]
                },
                "key_frame_feature_map": {
                  "1306f719-d588-4154-8335-792347bfdee2": [
                    5,
                    12,
                    20,
                    24,
                    45
                  ],
                  "a505f426-5a2e-4dea-b4fe-ab913e1d2a99": [
                    5,
                    1
                  ],
                  "89b4636f-08cc-4fa2-832b-18bcda636722": [
                    17
                  ]
                },
                "classifications": [
                  {
                    "feature_id": "5deccc93-922a-4fcd-bf70-0bd90249ed65",
                    "feature_schema_id": "cldtd2nmm108607y5ghou4ma6",
                    "name": "nested_checklist_question",
                    "value": "nested_checklist_question",
                    "checklist_answers": [
                      {
                        "feature_id": "3d9f3952-c303-42ed-91ba-b112927029c9",
                        "feature_schema_id": "cldtd2nmm108707y50shn5996",
                        "name": "first_checklist_answer",
                        "value": "first_checklist_answer",
                        "classifications": [
                          {
                            "feature_id": "fe4d0f5f-99e6-4abd-9401-b0ad04fe13ed",
                            "feature_schema_id": "cldtd2nmm108807y52ps9d5m4",
                            "name": "sub_checklist_question",
                            "value": "sub_checklist_question",
                            "checklist_answers": [
                              {
                                "feature_id": "05fca900-787a-4774-a011-0b6931d9a012",
                                "feature_schema_id": "cldtd2nmm108907y5269ua7bz",
                                "name": "first_sub_checklist_answer",
                                "value": "first_sub_checklist_answer",
                                "classifications": []
                              }
                            ]
                          }
                        ]
                      }
                    ]
                  },
                  {
                    "feature_id": "7a92c014-48e1-4367-8ff9-4ec146d9968e",
                    "feature_schema_id": "cldtd2nmm107y07y5fvhkb6c7",
                    "name": "nested_radio_question",
                    "value": "nested_radio_question",
                    "radio_answer": {
                      "feature_id": "9f35c500-be1d-4693-ae9a-71507931d558",
                      "feature_schema_id": "cldtd2nmm107z07y59j8n62ym",
                      "name": "first_radio_answer",
                      "value": "first_radio_answer",
                      "classifications": [
                        {
                          "feature_id": "d1269040-bf2c-4e8c-8fa6-58c3d6a7e343",
                          "feature_schema_id": "cldtd2nmm108007y5gs1y5eqv",
                          "name": "sub_radio_question",
                          "value": "sub_radio_question",
                          "radio_answer": {
                            "feature_id": "2fecd9bc-98ca-49bd-8db4-cb9d1be6a63d",
                            "feature_schema_id": "cldtd2nmm108107y53dvreuv2",
                            "name": "first_sub_radio_answer",
                            "value": "first_sub_radio_answer",
                            "classifications": []
                          }
                        }
                      ]
                    }
                  },
                  {
                    "feature_id": "95085369-6add-4ce1-9641-d6b690490eb3",
                    "feature_schema_id": "cldtd2nmm107s07y57ddy5wnn",
                    "name": "checklist_question",
                    "value": "checklist_question",
                    "checklist_answers": [
                      {
                        "feature_id": "53a98fa1-19ea-4541-b42f-8df7f7e3b2fc",
                        "feature_schema_id": "cldtd2nmm107v07y5cmixhywh",
                        "name": "second_checklist_answer",
                        "value": "second_checklist_answer",
                        "classifications": []
                      },
                      {
                        "feature_id": "6383f391-7ca6-4d99-a855-cdc2e2afa811",
                        "feature_schema_id": "cldtd2nmm107t07y5cqdw94z7",
                        "name": "first_checklist_answer",
                        "value": "first_checklist_answer",
                        "classifications": []
                      }
                    ]
                  },
                  {
                    "feature_id": "af2f95fe-6ce6-4e3a-83a8-fd11b1ae0cdc",
                    "feature_schema_id": "cldtd2nmm107m07y5h45sg6av",
                    "name": "radio_question",
                    "value": "radio_question",
                    "radio_answer": {
                      "feature_id": "71d910d5-df6b-4def-b68e-caced48dd25c",
                      "feature_schema_id": "cldtd2nmm107p07y542ff3hvd",
                      "name": "second_radio_answer",
                      "value": "second_radio_answer",
                      "classifications": []
                    }
                  }
                ]
              }
            }
          ]
        }
      }
    }
  },
  "models": {},
  "embeddings": [
    {
      "id": "c400000000000000000000000",
      "name": "Video Embedding V2 (Google MultiModal)",
      "dimensions": 1408,
      "is_custom": false,
      "values": [
        {
          "value": [
            -0.00489852577,
            0.0287129097 ...
                      ]
        }
      ]
    }
  ]
}
```

## Text

### Sample Project Export

```json JSON expandable theme={"theme":"css-variables"}
{
  "data_row": {
    "id": "clfh5lsju0bd90763h2o22vvq",
    "global_key": "lorem-ipsum.txt",
    "row_data": "https://storage.googleapis.com/labelbox-sample-datasets/nlp/lorem-ipsum.txt",
    "details": {
      "dataset_id": "clfh5lpul063507x4a16r5j6x",
      "dataset_name": "text_annotation_import_demo_dataset",
      "created_at": "2023-03-20T18:22:00.000+00:00",
      "updated_at": "2023-03-20T18:22:00.000+00:00",
      "last_activity_at": "2024-04-10T14:43:10.000+00:00",
      "created_by": "labelbox.public.user@labelbox.com"
    }
  },
  "media_attributes": {
    "mime_type": "text/plain"
  },
  "attachments": [
    {
      "type": "TEXT_URL",
      "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt",
      "id": "clfh5lsjv0bdd07638apx7bf5"
    }
  ],
  "metadata_fields": [],
   "embeddings": [
    {
      "id": "c300000000000000000000000",
      "name": "Text Embedding V2 (All-MPNet-base-v2)",
      "dimensions": 768,
      "is_custom": false,
      "values": [
        {
          "value": [
            0.04099537059664726,
            0.03790362924337387,
            -0.013698477298021317,
            0.038478486239910126,
            -0.01930154114961624,
            0.05670716613531113,
            0.02421298250555992 ...
	],
  "projects": {
    "clutx6nli00ks07x6efh6e1pw": {
      "name": "Text Annotation Import Demo",
      "labels": [
        {
          "label_kind": "Default",
          "version": "1.0.0",
          "id": "clutx70xi0oob148pgr7sz5v0",
          "label_details": {
            "created_at": "2024-04-10T14:43:11.000+00:00",
            "updated_at": "2024-04-10T14:43:11.000+00:00",
            "created_by": "aovalle@labelbox.com",
            "content_last_updated_at": "2024-04-10T14:43:10.580+00:00",
            "reviews": []
          },
          "performance_details": {
            "seconds_to_create": 0,
            "seconds_to_review": 0,
            "skipped": false
          },
          "annotations": {
            "objects": [
              {
                "feature_id": "clutx70x20onv148pb5ah49lb",
                "feature_schema_id": "clutx6n1g00r707y110ruh99w",
                "name": "named_entity",
                "value": "named_entity",
                "annotation_kind": "TextEntity",
                "classifications": [],
                "location": {
                  "start": 10,
                  "end": 20,
                  "token": "um dolor si"
                }
              }
            ],
            "classifications": [
              {
                "feature_id": "clutx70x20onw148p7mlgpq90",
                "feature_schema_id": "clutx6n1h00r907y16s49do9e",
                "name": "radio_question",
                "value": "radio_question",
                "radio_answer": {
                  "feature_id": "clutx70x20onx148px2679qsp",
                  "feature_schema_id": "clutx6n1h00ra07y100njafnj",
                  "name": "first_radio_answer",
                  "value": "first_radio_answer",
                  "classifications": []
                }
              },
              {
                "feature_id": "clutx70x20ony148p6gxk7ec8",
                "feature_schema_id": "clutx6n1h00rt07y16xe27hhv",
                "name": "checklist_question",
                "value": "checklist_question",
                "checklist_answers": [
                  {
                    "feature_id": "clutx70x20onz148pjjl5r8m5",
                    "feature_schema_id": "clutx6n1h00ru07y1hgvd3ekt",
                    "name": "first_checklist_answer",
                    "value": "first_checklist_answer",
                    "classifications": []
                  },
                  {
                    "feature_id": "clutx70x20oo0148p3w6es43a",
                    "feature_schema_id": "clutx6n1h00rw07y16t1v8tvi",
                    "name": "second_checklist_answer",
                    "value": "second_checklist_answer",
                    "classifications": []
                  },
                  {
                    "feature_id": "clutx70x20oo1148ps1lpz6dl",
                    "feature_schema_id": "clutx6n1h00ry07y1fkbk0pde",
                    "name": "third_checklist_answer",
                    "value": "third_checklist_answer",
                    "classifications": []
                  }
                ]
              },
              {
                "feature_id": "clutx70x20oo2148pz466o8ep",
                "feature_schema_id": "clutx6n1i00s107y18bywdwuh",
                "name": "free_text",
                "value": "free_text",
                "text_answer": {
                  "content": "sample text"
                }
              },
              {
                "feature_id": "clutx70x20oo3148poaesoyw7",
                "feature_schema_id": "clutx6n1h00rl07y12xox0uqk",
                "name": "nested_checklist_question",
                "value": "nested_checklist_question",
                "checklist_answers": [
                  {
                    "feature_id": "clutx70x20oo4148pgxhy7xr3",
                    "feature_schema_id": "clutx6n1h00rm07y15qdj4mms",
                    "name": "first_checklist_answer",
                    "value": "first_checklist_answer",
                    "classifications": [
                      {
                        "feature_id": "clutx70x20oo5148plvcm1yhy",
                        "feature_schema_id": "clutx6n1h00rn07y13xi0e1au",
                        "name": "sub_checklist_question",
                        "value": "sub_checklist_question",
                        "checklist_answers": [
                          {
                            "feature_id": "clutx70x20oo6148pcqjahziw",
                            "feature_schema_id": "clutx6n1h00ro07y164ddfda2",
                            "name": "first_sub_checklist_answer",
                            "value": "first_sub_checklist_answer",
                            "classifications": []
                          }
                        ]
                      }
                    ]
                  }
                ]
              },
              {
                "feature_id": "clutx70x20oo7148pkei2ymj3",
                "feature_schema_id": "clutx6n1h00rd07y188n4hd08",
                "name": "nested_radio_question",
                "value": "nested_radio_question",
                "radio_answer": {
                  "feature_id": "clutx70x20oo8148pjiiobfl5",
                  "feature_schema_id": "clutx6n1h00re07y1b7v1dd8z",
                  "name": "first_radio_answer",
                  "value": "first_radio_answer",
                  "classifications": [
                    {
                      "feature_id": "clutx70x20oo9148px2xcuanp",
                      "feature_schema_id": "clutx6n1h00rf07y18f8f45nu",
                      "name": "sub_radio_question",
                      "value": "sub_radio_question",
                      "radio_answer": {
                        "feature_id": "clutx70x20ooa148pqhs0hmot",
                        "feature_schema_id": "clutx6n1h00rg07y1fkkjgr7o",
                        "name": "first_sub_radio_answer",
                        "value": "first_sub_radio_answer",
                        "classifications": []
                      }
                    }
                  ]
                }
              }
            ],
            "relationships": []
          }
        }
      ],
      "project_details": {
        "ontology_id": "clutx6n0k00r607y1a0a9bkzj",
        "task_name": "Done",
        "batch_id": "9f1c1db0-f748-11ee-b74f-f180a0c567c8",
        "batch_name": "first-batch-text-demo",
        "workflow_status": "DONE",
        "priority": 5,
        "consensus_expected_label_count": 1,
        "workflow_history": []
      }
    }
  }
}
```

### Sample Model Run Export

```json JSON expandable theme={"theme":"css-variables"}
{
  "data_row": {
    "id": "clfh5lsju0bd90763h2o22vvq",
    "global_key": "lorem-ipsum.txt",
    "row_data": "https://storage.googleapis.com/labelbox-sample-datasets/nlp/lorem-ipsum.txt",
    "details": {
      "dataset_id": "clfh5lpul063507x4a16r5j6x",
      "dataset_name": "text_annotation_import_demo_dataset",
      "created_at": "2023-03-20T18:22:00.000+00:00",
      "updated_at": "2023-03-20T18:22:00.000+00:00",
      "created_by": "labelbox.public.user@labelbox.com"
    }
  },
  "media_attributes": {
    "mime_type": "text/plain"
  },
  "attachments": [
    {
      "type": "TEXT_URL",
      "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt",
      "id": "clfh5lsjv0bdd07638apx7bf5"
    }
  ],
  "metadata_fields": [],
  "experiments": {
    "a349da10-5d85-0c82-293d-8165aa611e7c": {
      "name": "text_model_run_8ad83b1f-c9c6-4ea4-83ce-a287393bb917",
      "runs": {
        "a349da10-aa0d-0879-711b-11f0cc035b65": {
          "name": "iteration 1",
          "run_data_row_id": "e1898158-c2a9-42fb-8c64-2c81beadafed",
          "labels": [
            {
              "label_kind": "Default",
              "version": "1.0.0",
              "id": "clr16pmfw0g4o140zkwys3f7u",
              "annotations": {
                "objects": [
                  {
                    "feature_id": "6d651dde-6567-0669-83e6-7a6677a5acb2",
                    "feature_schema_id": "clr16p71n018r07tuawb18jt6",
                    "name": "named_entity",
                    "value": "named_entity",
                    "annotation_kind": "TextEntity",
                    "classifications": [],
                    "location": {
                      "start": 10,
                      "end": 20
                    }
                  }
                ],
                "classifications": [
                  {
                    "feature_id": "08a38f86-5cd1-0feb-ae98-5731b2d2d3bf",
                    "feature_schema_id": "clr16p71p019f07tucv4g1cmm",
                    "name": "nested_checklist_question",
                    "value": "nested_checklist_question",
                    "checklist_answers": [
                      {
                        "feature_id": "3b97e87d-b196-0c21-a236-8b64f3f08086",
                        "feature_schema_id": "clr16p71p019g07tu9sca1dde",
                        "name": "first_checklist_answer",
                        "value": "first_checklist_answer",
                        "classifications": [
                          {
                            "feature_id": "8301a2fe-e898-044f-bc8b-a75e48b16f45",
                            "feature_schema_id": "clr16p71p019h07tu5j9o7b4o",
                            "name": "sub_checklist_question",
                            "value": "sub_checklist_question",
                            "checklist_answers": [
                              {
                                "feature_id": "baea7ccd-57a6-0c4f-bc66-1ddc2a8aa2d9",
                                "feature_schema_id": "clr16p71p019i07tu50tq966e",
                                "name": "first_sub_checklist_answer",
                                "value": "first_sub_checklist_answer",
                                "classifications": []
                              }
                            ]
                          }
                        ]
                      }
                    ]
                  },
                  {
                    "feature_id": "60fd8cc0-b5a6-013b-b02b-d91adafb45c1",
                    "feature_schema_id": "clr16p71o018t07tubkih92ua",
                    "name": "radio_question",
                    "value": "radio_question",
                    "radio_answer": {
                      "feature_id": "f0211ed1-4139-08ee-8c35-ea2bcad92b79",
                      "feature_schema_id": "clr16p71o018u07tu8bwc76w1",
                      "name": "first_radio_answer",
                      "value": "first_radio_answer",
                      "classifications": []
                    }
                  },
                  {
                    "feature_id": "73beac75-7e6c-0805-a148-2f6b7465f468",
                    "feature_schema_id": "clr16p71p019d07tufulw888j",
                    "name": "free_text",
                    "value": "free_text",
                    "text_answer": {
                      "content": "sample text"
                    }
                  },
                  {
                    "feature_id": "7a87c79e-6213-032c-859b-547e039ceff7",
                    "feature_schema_id": "clr16p71o018x07tuf7sj7c0x",
                    "name": "nested_radio_question",
                    "value": "nested_radio_question",
                    "radio_answer": {
                      "feature_id": "3aead946-dafd-0f41-8609-826d0d702d68",
                      "feature_schema_id": "clr16p71o018y07tu9oevflh8",
                      "name": "first_radio_answer",
                      "value": "first_radio_answer",
                      "classifications": [
                        {
                          "feature_id": "ab632136-83c1-0f3e-a4af-87da69b86624",
                          "feature_schema_id": "clr16p71o018z07tuas953oef",
                          "name": "sub_radio_question",
                          "value": "sub_radio_question",
                          "radio_answer": {
                            "feature_id": "2823619a-03bf-0b27-a12e-69f869d8e334",
                            "feature_schema_id": "clr16p71o019007tu8htocx3x",
                            "name": "first_sub_radio_answer",
                            "value": "first_sub_radio_answer",
                            "classifications": []
                          }
                        }
                      ]
                    }
                  },
                  {
                    "feature_id": "bd11b44d-3c8d-089d-9531-5177047736b7",
                    "feature_schema_id": "clr16p71p019507tu0x8d0vu4",
                    "name": "checklist_question",
                    "value": "checklist_question",
                    "checklist_answers": [
                      {
                        "feature_id": "360c8a23-3a74-0e75-9553-286b1b152100",
                        "feature_schema_id": "clr16p71p019a07tuetmgfe7c",
                        "name": "third_checklist_answer",
                        "value": "third_checklist_answer",
                        "classifications": []
                      },
                      {
                        "feature_id": "4e12f117-9f03-0a51-8b5d-721b88ce4be0",
                        "feature_schema_id": "clr16p71p019607tu4hkkbhxv",
                        "name": "first_checklist_answer",
                        "value": "first_checklist_answer",
                        "classifications": []
                      },
                      {
                        "feature_id": "501d328c-c0ea-0142-8809-ecd1b19961eb",
                        "feature_schema_id": "clr16p71p019807tu3ya04d5x",
                        "name": "second_checklist_answer",
                        "value": "second_checklist_answer",
                        "classifications": []
                      }
                    ]
                  }
                ],
                "relationships": []
              }
            }
          ],
          "predictions": [
            {
              "label_kind": "Default",
              "version": "1.0.0",
              "id": "clr16pmfw0g4o140zkwys3f7u",
              "annotations": {
                "objects": [
                  {
                    "feature_id": "4bafa398-69d8-4bf8-a3f0-834869ab3e5f",
                    "feature_schema_id": "clr16p71n018r07tuawb18jt6",
                    "name": "named_entity",
                    "value": "named_entity",
                    "annotation_kind": "TextEntity",
                    "classifications": [],
                    "location": {
                      "start": 10,
                      "end": 20
                    }
                  }
                ],
                "classifications": [
                  {
                    "feature_id": "31737a80-4a5c-4c40-8362-8d59ae6c2047",
                    "feature_schema_id": "clr16p71o018x07tuf7sj7c0x",
                    "name": "nested_radio_question",
                    "value": "nested_radio_question",
                    "radio_answer": {
                      "feature_id": "a00eca18-fb50-4130-a33b-d505ba8e3d04",
                      "feature_schema_id": "clr16p71o018y07tu9oevflh8",
                      "name": "first_radio_answer",
                      "value": "first_radio_answer",
                      "classifications": [
                        {
                          "feature_id": "94b4ee14-2837-4fb5-b097-4e59cb92b07f",
                          "feature_schema_id": "clr16p71o018z07tuas953oef",
                          "name": "sub_radio_question",
                          "value": "sub_radio_question",
                          "radio_answer": {
                            "feature_id": "a5863817-4c81-424a-8c4c-4cdec05508bf",
                            "feature_schema_id": "clr16p71o019007tu8htocx3x",
                            "name": "first_sub_radio_answer",
                            "value": "first_sub_radio_answer",
                            "classifications": []
                          }
                        }
                      ]
                    }
                  },
                  {
                    "feature_id": "741288f5-76ab-4cdf-9277-0f97f0e426fd",
                    "feature_schema_id": "clr16p71o018t07tubkih92ua",
                    "name": "radio_question",
                    "value": "radio_question",
                    "radio_answer": {
                      "feature_id": "64065994-7568-46e1-8bfc-bd96d20384cf",
                      "feature_schema_id": "clr16p71o018u07tu8bwc76w1",
                      "name": "first_radio_answer",
                      "value": "first_radio_answer",
                      "classifications": []
                    }
                  },
                  {
                    "feature_id": "a66cc378-aa78-4250-bde1-28da1310bedb",
                    "feature_schema_id": "clr16p71p019507tu0x8d0vu4",
                    "name": "checklist_question",
                    "value": "checklist_question",
                    "checklist_answers": [
                      {
                        "feature_id": "6db15123-a2dd-4722-8253-32a0c0236850",
                        "feature_schema_id": "clr16p71p019607tu4hkkbhxv",
                        "name": "first_checklist_answer",
                        "value": "first_checklist_answer",
                        "classifications": []
                      },
                      {
                        "feature_id": "8eb5629e-8ae1-44f1-b369-491c5c61ea02",
                        "feature_schema_id": "clr16p71p019a07tuetmgfe7c",
                        "name": "third_checklist_answer",
                        "value": "third_checklist_answer",
                        "classifications": []
                      },
                      {
                        "feature_id": "fdb2574a-fea7-4127-9ee1-b112126f117a",
                        "feature_schema_id": "clr16p71p019807tu3ya04d5x",
                        "name": "second_checklist_answer",
                        "value": "second_checklist_answer",
                        "classifications": []
                      }
                    ]
                  },
                  {
                    "feature_id": "a8386f96-ae13-45c0-bb35-033380f86dc5",
                    "feature_schema_id": "clr16p71p019d07tufulw888j",
                    "name": "free_text",
                    "value": "free_text",
                    "text_answer": {
                      "content": "sample text"
                    }
                  },
                  {
                    "feature_id": "db0150e9-ecb5-4bad-8350-c82446244657",
                    "feature_schema_id": "clr16p71p019f07tucv4g1cmm",
                    "name": "nested_checklist_question",
                    "value": "nested_checklist_question",
                    "checklist_answers": [
                      {
                        "feature_id": "84f14b81-301f-41fd-a842-6d48594a40fc",
                        "feature_schema_id": "clr16p71p019g07tu9sca1dde",
                        "name": "first_checklist_answer",
                        "value": "first_checklist_answer",
                        "classifications": [
                          {
                            "feature_id": "8367c018-cba4-40ff-85e5-c06b137773c3",
                            "feature_schema_id": "clr16p71p019h07tu5j9o7b4o",
                            "name": "sub_checklist_question",
                            "value": "sub_checklist_question",
                            "checklist_answers": [
                              {
                                "feature_id": "c1e2da03-c23d-4ffa-a25c-e6b6c13a1dae",
                                "feature_schema_id": "clr16p71p019i07tu50tq966e",
                                "name": "first_sub_checklist_answer",
                                "value": "first_sub_checklist_answer",
                                "classifications": []
                              }
                            ]
                          }
                        ]
                      }
                    ]
                  }
                ],
                "relationships": []
              }
            }
          ]
        }
      }
    }
  },
  "models": {},
  "embeddings": [
    {
      "id": "c300000000000000000000000",
      "name": "Text Embedding V2 (All-MPNet-base-v2)",
      "dimensions": 768,
      "is_custom": false,
      "values": [
        {
          "value": [
            0.04099537059664726,
            0.03790362924337387,
            -0.013698477298021317,
            0.038478486239910126,
            -0.01930154114961624,
            0.05670716613531113,
            0.02421298250555992 ...
	]

}
```

## Geospatial (tiled imagery)

### Sample Project Export

```json JSON expandable theme={"theme":"css-variables"}
{
    "data_row": {
        "id": "cldnf9zmt0d7n075ohgnvecul",
        "row_data": "{\"tileLayerUrl\":\"https://api.mapbox.com/styles/v1/mapbox/satellite-streets-v11/tiles/{z}/{x}/{y}?access_token=pk.eyJ1IjoibWFwYm94IiwiYSI6ImNpejY4NXVycTA2emYycXBndHRqcmZ3N3gifQ.rJcFIG214AriISLbB6B5aw\",\"bounds\":[[-122.31764674186705,37.87276155898985],[-122.31635199317932,37.87398109727749]],\"minZoom\":17,\"maxZoom\":23,\"maxNativeZoom\":null,\"epsg\":\"EPSG4326\",\"tileSize\":256,\"alternativeLayers\":[],\"version\":2}",
        "details": {
            "dataset_id": "cldnf9yvd041f070d33t94f41",
          	"dataset_name": "geotiff_noaa.json",
            "created_at": "2023-02-02",
            "updated_at": "2023-02-02",
            "created_by": "kyang@labelbox.com"
        }
    },
    "media_attributes": {
        "mime_type": "application/x-tms-geo"
    },
    "attachments": [],
    "metadata_fields": [],
    "projects": {
        "cldnfa25103kd081ph3edg7xp": {
            "project_name": "geospatial_project_demo",
            "labels": [
                {
                    "label_kind": "Default",
                    "version": "1.0.0",
                    "id": "cldnfc62h029b13ydq6bgvyak",
                    "label_details": {
                        "created_at": "2023-02-02T18:21:40+00:00",
                        "updated_at": "2023-02-02T18:21:40+00:00",
                        "created_by": "kyang@labelbox.com",
                        "reviews": []
                    },
                    "performance_details": {
                        "seconds_to_create": 0,
                        "seconds_to_review": 0,
                        "skipped": false
                    },
                    "annotations": {
                        "objects": [
                            {
                                "feature_id": "cldnfc61v028v13ydwgha3a1d",
                                "name": "point_geo",
                                "annotation_kind": "GeoPoint",
                                "classifications": [],
                                "geojson": {
                                    "type": "Point",
                                    "coordinates": [
                                        -122.31741,
                                        37.873557
                                    ]
                                }
                            },
                            {
                                "feature_id": "cldnfc61v028w13ydnbm7vl2q",
                                "name": "polyline_geo",
                                "annotation_kind": "GeoPolyline",
                                "classifications": [],
                                "geojson": {
                                    "type": "LineString",
                                    "coordinates": [
                                        [
                                            -122.317578,
                                            37.873963
                                        ],
                                        [
                                            -122.316398,
                                            37.873967
                                        ],
                                        [
                                            -122.31639,
                                            37.872779
                                        ]
                                    ]
                                }
                            },
                            {
                                "feature_id": "cldnfc61v028x13yd5swk0drd",
                                "name": "polygon_geo",
                                "annotation_kind": "GeoPolygon",
                                "classifications": [],
                                "geojson": {
                                    "type": "Polygon",
                                    "coordinates": [
                                        [
                                            [
                                                -122.316918,
                                                37.87329
                                            ],
                                            [
                                                -122.317102,
                                                37.873043
                                            ],
                                            [
                                                -122.316801,
                                                37.873036
                                            ],
                                            [
                                                -122.316918,
                                                37.87329
                                            ]
                                        ]
                                    ]
                                }
                            },
                            {
                                "feature_id": "cldnfc61v028y13yd41qozic7",
                                "name": "bbox_geo",
                                "annotation_kind": "GeoBoundingBox",
                                "classifications": [],
                                "geojson": {
                                    "type": "Polygon",
                                    "coordinates": [
                                        [
                                            [
                                                -122.317345,
                                                37.873713
                                            ],
                                            [
                                                -122.317345,
                                                37.873859
                                            ],
                                            [
                                                -122.31673,
                                                37.873859
                                            ],
                                            [
                                                -122.31673,
                                                37.873713
                                            ],
                                            [
                                                -122.317345,
                                                37.873713
                                            ]
                                        ]
                                    ]
                                }
                            },
                            {
                                "feature_id": "cldnfc61v029113ydvc788rx9",
                                "name": "bbox_checklist_geo",
                                "annotation_kind": "GeoBoundingBox",
                                "classifications": [
                                    {
                                        "feature_id": "cldnfc61v029213yd344u0cyk",
                                        "name": "checklist_class_name",
                                        "checklist_answers": [
                                            {
                                                "feature_id": "cldnfc61v029313yda92qm39k",
                                                "name": "first_checklist_answer",
                                                "classifications": []
                                            }
                                        ]
                                    }
                                ],
                                "geojson": {
                                    "type": "Polygon",
                                    "coordinates": [
                                        [
                                            [
                                                -122.317113,
                                                37.873402
                                            ],
                                            [
                                                -122.317113,
                                                37.873608
                                            ],
                                            [
                                                -122.316655,
                                                37.873608
                                            ],
                                            [
                                                -122.316655,
                                                37.873402
                                            ],
                                            [
                                                -122.317113,
                                                37.873402
                                            ]
                                        ]
                                    ]
                                }
                            },
                            {
                                "feature_id": "cldnfc61v029413yd5lgevg2c",
                                "name": "bbox_text_geo",
                                "annotation_kind": "GeoBoundingBox",
                                "classifications": [
                                    {
                                        "feature_id": "cldnfc61v029513ydmluqvqyt",
                                        "name": "free_text_geo",
                                        "text_answer": {
                                            "content": "sample text"
                                        }
                                    }
                                ],
                                "geojson": {
                                    "type": "Polygon",
                                    "coordinates": [
                                        [
                                            [
                                                -122.317508,
                                                37.873182
                                            ],
                                            [
                                                -122.317508,
                                                37.87338
                                            ],
                                            [
                                                -122.3171,
                                                37.87338
                                            ],
                                            [
                                                -122.3171,
                                                37.873182
                                            ],
                                            [
                                                -122.317508,
                                                37.873182
                                            ]
                                        ]
                                    ]
                                }
                            },
                            {
                                "feature_id": "cldnfc61v029a13ydmog1dhzw",
                                "name": "polygon_geo_2",
                                "annotation_kind": "GeoPolygon",
                                "classifications": [],
                                "geojson": {
                                    "type": "Polygon",
                                    "coordinates": [
                                        [
                                            [
                                                -122.316362,
                                                37.872966
                                            ],
                                            [
                                                -122.316362,
                                                37.872948
                                            ],
                                            [
                                                -122.316356,
                                                37.872948
                                            ],
                                            [
                                                -122.316356,
                                                37.872966
                                            ],
                                            [
                                                -122.316362,
                                                37.872966
                                            ]
                                        ]
                                    ]
                                }
                            }
                        ],
                        "classifications": [
                            {
                                "feature_id": "cldnfc61v028z13ydy0xcffbj",
                                "name": "radio_question_geo",
                                "radio_answer": {
                                    "feature_id": "cldnfc61v029013ydj55ek20y",
                                    "name": "first_radio_answer",
                                    "classifications": []
                                }
                            },
                            {
                                "feature_id": "cldnfc61v029613ydu76vcgbt",
                                "name": "checklist_question_geo",
                                "checklist_answers": [
                                    {
                                        "feature_id": "cldnfc61v029713yd5x16lyg3",
                                        "name": "first_checklist_answer",
                                        "classifications": []
                                    },
                                    {
                                        "feature_id": "cldnfc61v029813ydcsx4mc2t",
                                        "name": "second_checklist_answer",
                                        "classifications": []
                                    },
                                    {
                                        "feature_id": "cldnfc61v029913yd2smfzpft",
                                        "name": "third_checklist_answer",
                                        "classifications": []
                                    }
                                ]
                            }
                        ],
                        "relationships": []
                    }
                }
            ],
            "project_details": {
                "ontology_id": "cl8emhd3e7bll070k6gbicl97",
                "task_id": "fdc4b3d2-e0fe-41dc-bdef-0dd556870a20",
                "task_name": "Initial review task",
                "batch_id": "cc4e7070-f20e-11ed-9dd9-639c7e9f70ba",
                "batch_name": "geotiff_noaa.json-cl8emco177d5g07y8e06j4acp",
                "workflow_status": "IN_REVIEW",
                "priority": 5,
                "consensus_expected_label_count": 3,
                "workflow_history": [
                    {
                        "action": "Move",
                        "created_at": "2023-05-14T04:21:29.062+00:00",
                        "created_by": "ptancre+10@labelbox.com",
                        "previous_task_name": "Initial labeling task",
                        "previous_task_id": "0dde450a-93ba-0b77-b97e-de8f7f479cee",
                        "next_task_name": "Initial review task",
                        "next_task_id": "fdc4b3d2-e0fe-41dc-bdef-0dd556870a20"
                    },
                    {
                        "action": "Move",
                        "created_at": "2023-05-14T04:21:29.044+00:00",
                        "created_by": "ptancre+10@labelbox.com",
                        "next_task_name": "Initial labeling task",
                        "next_task_id": "0dde450a-93ba-0b77-b97e-de8f7f479cee"
                    }
                ]
            }
        }
    }
}
```

### Sample Model Run Export

```json JSON expandable theme={"theme":"css-variables"}
{
  "data_row": {
    "id": "clfh28su81agn07623rql6j44",
    "global_key": "slippy_maps",
    "row_data": "{\"tileLayerUrl\":\"https://api.mapbox.com/styles/v1/mapbox/satellite-streets-v11/tiles/{z}/{x}/{y}?access_token=pk.eyJ1IjoibWFwYm94IiwiYSI6ImNpejY4NXVycTA2emYycXBndHRqcmZ3N3gifQ.rJcFIG214AriISLbB6B5aw\",\"bounds\":[[-122.31764674186705,37.87276155898985],[-122.31635199317932,37.87398109727749]],\"minZoom\":17,\"maxZoom\":23,\"maxNativeZoom\":null,\"epsg\":\"EPSG4326\",\"tileSize\":256,\"alternativeLayers\":[],\"version\":2}",
    "details": {
      "dataset_id": "clfh28rib089g07134eqx03ew",
      "dataset_name": "geo_demo_dataset",
      "created_at": "2023-03-20T16:47:55.000+00:00",
      "updated_at": "2023-03-20T16:47:55.000+00:00",
      "created_by": "labelbox.public.user@labelbox.com"
    }
  },
  "media_attributes": {
    "mime_type": "application/x-tms-geo"
  },
  "attachments": [],
  "metadata_fields": [],
  "experiments": {
    "a15bda50-b901-02e0-07d6-e842d13b3f47": {
      "name": "geospatial_model_run_1f54f23a-0359-4f18-b3ec-808a6181a4fd",
      "runs": {
        "a15bda50-fd4e-06ff-8f9b-762516cf58c8": {
          "name": "iteration 1",
          "run_data_row_id": "247e2c89-0e82-4662-a757-5f7cfa07a8fd",
          "labels": [
            {
              "label_kind": "Default",
              "version": "1.0.0",
              "id": "clhs22mh25b4d142bagekzwlv",
              "annotations": {
                "objects": [
                  {
                    "feature_id": "12527b8c-e5c5-4cd5-9266-c6a1d90ef14c",
                    "feature_schema_id": "clhs21cpy0c6t07y1fof39vn7",
                    "name": "point_geo",
                    "value": "point_geo",
                    "annotation_kind": "GeoPoint",
                    "classifications": [],
                    "geojson": {
                      "type": "Point",
                      "coordinates": [
                        -122.31741,
                        37.873557
                      ]
                    }
                  },
                  {
                    "feature_id": "185de2b0-fca0-48fc-9f3c-982e1487ce13",
                    "feature_schema_id": "clhs21cpy0c7307y1dg174k0l",
                    "name": "bbox_checklist_geo",
                    "value": "bbox_checklist_geo",
                    "annotation_kind": "GeoBoundingBox",
                    "classifications": [
                      {
                        "feature_id": "d2809ef9-01ae-4ad7-a4d0-1fee6938e186",
                        "feature_schema_id": "clhs21cpy0c7407y1gl9g9ver",
                        "name": "checklist_class_name",
                        "value": "checklist_class_name",
                        "checklist_answers": [
                          {
                            "feature_id": "6cbc5158-ac53-445a-a6c1-8035b7a4cac1",
                            "feature_schema_id": "clhs21cpy0c7507y16idc9qjy",
                            "name": "first_checklist_answer",
                            "value": "first_checklist_answer",
                            "classifications": []
                          }
                        ]
                      }
                    ],
                    "geojson": {
                      "type": "Polygon",
                      "coordinates": [
                        [
                          [
                            -122.317113,
                            37.873608
                          ],
                          [
                            -122.317113,
                            37.873402
                          ],
                          [
                            -122.316655,
                            37.873402
                          ],
                          [
                            -122.316655,
                            37.873608
                          ],
                          [
                            -122.317113,
                            37.873608
                          ]
                        ]
                      ]
                    }
                  },
                  {
                    "feature_id": "78024089-436d-4a0e-aea9-f458ee66cb42",
                    "feature_schema_id": "clhs21cpy0c6v07y19w4zb3b7",
                    "name": "polyline_geo",
                    "value": "polyline_geo",
                    "annotation_kind": "GeoPolyline",
                    "classifications": [],
                    "geojson": {
                      "type": "LineString",
                      "coordinates": [
                        [
                          -122.317578,
                          37.873963
                        ],
                        [
                          -122.316398,
                          37.873967
                        ],
                        [
                          -122.31639,
                          37.872779
                        ]
                      ]
                    }
                  },
                  {
                    "feature_id": "97110d09-559b-46c9-abb4-7e5d5f578b3e",
                    "feature_schema_id": "clhs21cpy0c6x07y18t2hgvdp",
                    "name": "polygon_geo",
                    "value": "polygon_geo",
                    "annotation_kind": "GeoPolygon",
                    "classifications": [],
                    "geojson": {
                      "type": "Polygon",
                      "coordinates": [
                        [
                          [
                            -122.316918,
                            37.87329
                          ],
                          [
                            -122.317102,
                            37.873043
                          ],
                          [
                            -122.316801,
                            37.873036
                          ],
                          [
                            -122.316918,
                            37.87329
                          ]
                        ]
                      ]
                    }
                  },
                  {
                    "feature_id": "cbbd8796-d6ff-4238-9347-bb12b239ac74",
                    "feature_schema_id": "clhs21cpy0c7907y19etw75vg",
                    "name": "bbox_text_geo",
                    "value": "bbox_text_geo",
                    "annotation_kind": "GeoBoundingBox",
                    "classifications": [
                      {
                        "feature_id": "0492b8b4-217b-4fdb-bc93-1b0a30f5aa8b",
                        "feature_schema_id": "clhs21cpz0c7a07y16uczf4dx",
                        "name": "free_text_geo",
                        "value": "free_text_geo",
                        "text_answer": {
                          "content": "sample text"
                        }
                      }
                    ],
                    "geojson": {
                      "type": "Polygon",
                      "coordinates": [
                        [
                          [
                            -122.317508,
                            37.87338
                          ],
                          [
                            -122.317508,
                            37.873182
                          ],
                          [
                            -122.3171,
                            37.873182
                          ],
                          [
                            -122.3171,
                            37.87338
                          ],
                          [
                            -122.317508,
                            37.87338
                          ]
                        ]
                      ]
                    }
                  },
                  {
                    "feature_id": "d80e2563-002a-48b8-b6c6-50f7df1a4a8b",
                    "feature_schema_id": "clhs21cpy0c7107y1b07n6ef9",
                    "name": "bbox_geo",
                    "value": "bbox_geo",
                    "annotation_kind": "GeoBoundingBox",
                    "classifications": [],
                    "geojson": {
                      "type": "Polygon",
                      "coordinates": [
                        [
                          [
                            -122.317345,
                            37.873859
                          ],
                          [
                            -122.317345,
                            37.873713
                          ],
                          [
                            -122.31673,
                            37.873713
                          ],
                          [
                            -122.31673,
                            37.873859
                          ],
                          [
                            -122.317345,
                            37.873859
                          ]
                        ]
                      ]
                    }
                  },
                  {
                    "feature_id": "f6abee6b-8130-4033-a223-be428dd59c0a",
                    "feature_schema_id": "clhs21cpy0c6x07y18t2hgvdp",
                    "name": "polygon_geo",
                    "value": "polygon_geo",
                    "annotation_kind": "GeoPolygon",
                    "classifications": [],
                    "geojson": {
                      "type": "Polygon",
                      "coordinates": [
                        [
                          [
                            -122.31703,
                            37.873978
                          ],
                          [
                            -122.317024,
                            37.873935
                          ],
                          [
                            -122.316989,
                            37.873899
                          ],
                          [
                            -122.316979,
                            37.873859
                          ],
                          [
                            -122.316958,
                            37.873856
                          ],
                          [
                            -122.316955,
                            37.873816
                          ],
                          [
                            -122.31692,
                            37.873779
                          ],
                          [
                            -122.31691,
                            37.873734
                          ],
                          [
                            -122.316886,
                            37.873697
                          ],
                          [
                            -122.316855,
                            37.873685
                          ],
                          [
                            -122.31682,
                            37.873611
                          ],
                          [
                            -122.316765,
                            37.87355
                          ],
                          [
                            -122.316758,
                            37.873517
                          ],
                          [
                            -122.31671,
                            37.873489
                          ],
                          [
                            -122.31671,
                            37.873452
                          ],
                          [
                            -122.316679,
                            37.87344
                          ],
                          [
                            -122.316634,
                            37.873355
                          ],
                          [
                            -122.316603,
                            37.873348
                          ],
                          [
                            -122.3166,
                            37.873321
                          ],
                          [
                            -122.316545,
                            37.87329
                          ],
                          [
                            -122.316528,
                            37.873199
                          ],
                          [
                            -122.31649,
                            37.873183
                          ],
                          [
                            -122.31649,
                            37.873162
                          ],
                          [
                            -122.316417,
                            37.873092
                          ],
                          [
                            -122.316411,
                            37.873055
                          ],
                          [
                            -122.316397,
                            37.87304
                          ],
                          [
                            -122.316356,
                            37.87304
                          ],
                          [
                            -122.316356,
                            37.873981
                          ],
                          [
                            -122.31703,
                            37.873978
                          ]
                        ]
                      ]
                    }
                  }
                ],
                "classifications": [
                  {
                    "feature_id": "08c1417e-ac8f-4272-9f4f-8b523ae5e0d7",
                    "feature_schema_id": "clhs21cpz0c7l07y160gh5xdw",
                    "name": "radio_question_geo",
                    "value": "radio_question_geo",
                    "radio_answer": {
                      "feature_id": "8c72625a-e663-4241-a2c2-8082804a59e3",
                      "feature_schema_id": "clhs21cpz0c7m07y188gjbqbj",
                      "name": "first_radio_answer",
                      "value": "first_radio_answer",
                      "classifications": []
                    }
                  },
                  {
                    "feature_id": "a22dac3b-5c40-4f08-9723-d70e7bc4bff4",
                    "feature_schema_id": "clhs21cpz0c7d07y14vnl7zpi",
                    "name": "checklist_question_geo",
                    "value": "checklist_question_geo",
                    "checklist_answers": [
                      {
                        "feature_id": "1c4a1950-6961-42bf-b134-50596f136164",
                        "feature_schema_id": "clhs21cpz0c7e07y16k192u14",
                        "name": "first_checklist_answer",
                        "value": "first_checklist_answer",
                        "classifications": []
                      },
                      {
                        "feature_id": "963cc214-b134-4b0e-80d5-8e63bcc000ae",
                        "feature_schema_id": "clhs21cpz0c7g07y17yo0e2cn",
                        "name": "second_checklist_answer",
                        "value": "second_checklist_answer",
                        "classifications": []
                      },
                      {
                        "feature_id": "bfa3e08b-5f2d-468e-aa7c-a6c2a29f9f4f",
                        "feature_schema_id": "clhs21cpz0c7i07y1hytkftpk",
                        "name": "third_checklist_answer",
                        "value": "third_checklist_answer",
                        "classifications": []
                      }
                    ]
                  }
                ],
                "relationships": []
              }
            }
          ],
          "predictions": [
            {
              "label_kind": "Default",
              "version": "1.0.0",
              "id": "clhs22mh25b4d142bagekzwlv",
              "annotations": {
                "objects": [
                  {
                    "feature_id": "21d3b49c-0e17-4eb1-9824-2b05a23386d5",
                    "feature_schema_id": "clhs21cpy0c6v07y19w4zb3b7",
                    "name": "polyline_geo",
                    "value": "polyline_geo",
                    "annotation_kind": "GeoPolyline",
                    "classifications": [],
                    "geojson": {
                      "type": "LineString",
                      "coordinates": [
                        [
                          -122.317578,
                          37.873963
                        ],
                        [
                          -122.316398,
                          37.873967
                        ],
                        [
                          -122.31639,
                          37.872779
                        ]
                      ]
                    }
                  },
                  {
                    "feature_id": "4bcd8a06-cdc9-4450-b59e-d6601d185b88",
                    "feature_schema_id": "clhs21cpy0c6t07y1fof39vn7",
                    "name": "point_geo",
                    "value": "point_geo",
                    "annotation_kind": "GeoPoint",
                    "classifications": [],
                    "geojson": {
                      "type": "Point",
                      "coordinates": [
                        -122.31741,
                        37.873557
                      ]
                    }
                  },
                  {
                    "feature_id": "69d6cf96-1198-42e0-83a7-c75f0c945037",
                    "feature_schema_id": "clhs21cpy0c7107y1b07n6ef9",
                    "name": "bbox_geo",
                    "value": "bbox_geo",
                    "annotation_kind": "GeoBoundingBox",
                    "classifications": [],
                    "geojson": {
                      "type": "Polygon",
                      "coordinates": [
                        [
                          [
                            -122.317345,
                            37.873859
                          ],
                          [
                            -122.317345,
                            37.873713
                          ],
                          [
                            -122.31673,
                            37.873713
                          ],
                          [
                            -122.31673,
                            37.873859
                          ],
                          [
                            -122.317345,
                            37.873859
                          ]
                        ]
                      ]
                    }
                  },
                  {
                    "feature_id": "9090fb5d-09f3-4a5f-9114-9b3c4ce6c909",
                    "feature_schema_id": "clhs21cpy0c6x07y18t2hgvdp",
                    "name": "polygon_geo",
                    "value": "polygon_geo",
                    "annotation_kind": "GeoPolygon",
                    "classifications": [],
                    "geojson": {
                      "type": "Polygon",
                      "coordinates": [
                        [
                          [
                            -122.316918,
                            37.87329
                          ],
                          [
                            -122.317102,
                            37.873043
                          ],
                          [
                            -122.316801,
                            37.873036
                          ],
                          [
                            -122.316918,
                            37.87329
                          ]
                        ]
                      ]
                    }
                  },
                  {
                    "feature_id": "9ef3b0dc-b085-4f64-9018-264db30f9018",
                    "feature_schema_id": "clhs21cpy0c7307y1dg174k0l",
                    "name": "bbox_checklist_geo",
                    "value": "bbox_checklist_geo",
                    "annotation_kind": "GeoBoundingBox",
                    "classifications": [
                      {
                        "feature_id": "a77cd878-d15d-4628-ab08-9ecd192bbebc",
                        "feature_schema_id": "clhs21cpy0c7407y1gl9g9ver",
                        "name": "checklist_class_name",
                        "value": "checklist_class_name",
                        "checklist_answers": [
                          {
                            "feature_id": "5a6a23f8-39c7-409c-953f-732d3cf4cbec",
                            "feature_schema_id": "clhs21cpy0c7507y16idc9qjy",
                            "name": "first_checklist_answer",
                            "value": "first_checklist_answer",
                            "classifications": []
                          }
                        ]
                      }
                    ],
                    "geojson": {
                      "type": "Polygon",
                      "coordinates": [
                        [
                          [
                            -122.317113,
                            37.873608
                          ],
                          [
                            -122.317113,
                            37.873402
                          ],
                          [
                            -122.316655,
                            37.873402
                          ],
                          [
                            -122.316655,
                            37.873608
                          ],
                          [
                            -122.317113,
                            37.873608
                          ]
                        ]
                      ]
                    }
                  },
                  {
                    "feature_id": "f7df220a-27c0-4fc1-b67f-094b28efec3d",
                    "feature_schema_id": "clhs21cpy0c6z07y1a99ubplb",
                    "name": "polygon_geo_2",
                    "value": "polygon_geo_2",
                    "annotation_kind": "GeoPolygon",
                    "classifications": [],
                    "geojson": {
                      "type": "Polygon",
                      "coordinates": [
                        [
                          [
                            -122.31703,
                            37.873978
                          ],
                          [
                            -122.317024,
                            37.873935
                          ],
                          [
                            -122.316989,
                            37.873899
                          ],
                          [
                            -122.316979,
                            37.873859
                          ],
                          [
                            -122.316958,
                            37.873856
                          ],
                          [
                            -122.316955,
                            37.873816
                          ],
                          [
                            -122.31692,
                            37.873779
                          ],
                          [
                            -122.31691,
                            37.873734
                          ],
                          [
                            -122.316886,
                            37.873697
                          ],
                          [
                            -122.316855,
                            37.873685
                          ],
                          [
                            -122.31682,
                            37.873611
                          ],
                          [
                            -122.316765,
                            37.87355
                          ],
                          [
                            -122.316758,
                            37.873517
                          ],
                          [
                            -122.31671,
                            37.873489
                          ],
                          [
                            -122.31671,
                            37.873452
                          ],
                          [
                            -122.316679,
                            37.87344
                          ],
                          [
                            -122.316634,
                            37.873355
                          ],
                          [
                            -122.316603,
                            37.873348
                          ],
                          [
                            -122.3166,
                            37.873321
                          ],
                          [
                            -122.316545,
                            37.87329
                          ],
                          [
                            -122.316528,
                            37.873199
                          ],
                          [
                            -122.31649,
                            37.873183
                          ],
                          [
                            -122.31649,
                            37.873162
                          ],
                          [
                            -122.316417,
                            37.873092
                          ],
                          [
                            -122.316411,
                            37.873055
                          ],
                          [
                            -122.316397,
                            37.87304
                          ],
                          [
                            -122.316356,
                            37.87304
                          ],
                          [
                            -122.316356,
                            37.873981
                          ],
                          [
                            -122.31703,
                            37.873978
                          ]
                        ]
                      ]
                    }
                  },
                  {
                    "feature_id": "fbc72808-7f01-4cae-b53d-e868ac2e6b30",
                    "feature_schema_id": "clhs21cpy0c7907y19etw75vg",
                    "name": "bbox_text_geo",
                    "value": "bbox_text_geo",
                    "annotation_kind": "GeoBoundingBox",
                    "classifications": [
                      {
                        "feature_id": "cd12abb2-2c30-48be-b4ef-a8e3b649e998",
                        "feature_schema_id": "clhs21cpz0c7a07y16uczf4dx",
                        "name": "free_text_geo",
                        "value": "free_text_geo",
                        "text_answer": {
                          "content": "sample text"
                        }
                      }
                    ],
                    "geojson": {
                      "type": "Polygon",
                      "coordinates": [
                        [
                          [
                            -122.317508,
                            37.87338
                          ],
                          [
                            -122.317508,
                            37.873182
                          ],
                          [
                            -122.3171,
                            37.873182
                          ],
                          [
                            -122.3171,
                            37.87338
                          ],
                          [
                            -122.317508,
                            37.87338
                          ]
                        ]
                      ]
                    }
                  }
                ],
                "classifications": [
                  {
                    "feature_id": "413f0623-6bb6-4631-9c2c-dd95cd01ff0b",
                    "feature_schema_id": "clhs21cpz0c7d07y14vnl7zpi",
                    "name": "checklist_question_geo",
                    "value": "checklist_question_geo",
                    "checklist_answers": [
                      {
                        "feature_id": "27fe9828-16fc-446d-8ff9-563d1bda1df9",
                        "feature_schema_id": "clhs21cpz0c7e07y16k192u14",
                        "name": "first_checklist_answer",
                        "value": "first_checklist_answer",
                        "classifications": []
                      },
                      {
                        "feature_id": "8b97530e-96ce-4ce4-8232-e27c8b9bde4f",
                        "feature_schema_id": "clhs21cpz0c7g07y17yo0e2cn",
                        "name": "second_checklist_answer",
                        "value": "second_checklist_answer",
                        "classifications": []
                      },
                      {
                        "feature_id": "c8c5c1d3-c805-4cde-9522-254d390ca880",
                        "feature_schema_id": "clhs21cpz0c7i07y1hytkftpk",
                        "name": "third_checklist_answer",
                        "value": "third_checklist_answer",
                        "classifications": []
                      }
                    ]
                  },
                  {
                    "feature_id": "dd36f2ec-91be-4bf6-9f6e-7531598628c0",
                    "feature_schema_id": "clhs21cpz0c7l07y160gh5xdw",
                    "name": "radio_question_geo",
                    "value": "radio_question_geo",
                    "radio_answer": {
                      "feature_id": "06a746a8-78c0-4bb5-9f71-65e8303155b5",
                      "feature_schema_id": "clhs21cpz0c7m07y188gjbqbj",
                      "name": "first_radio_answer",
                      "value": "first_radio_answer",
                      "classifications": []
                    }
                  }
                ],
                "relationships": []
              }
            }
          ]
        }
      }
    }
  },
  "models": {}
}
```

## Document

### Sample Project Export

```json JSON expandable theme={"theme":"css-variables"}
{
  "data_row": {
    "id": "cluu0c5g13a9s0735cjhi0hb0",
    "global_key": "0801.3483_doc.pdf",
    "row_data": "https://storage.googleapis.com/labelbox-datasets/arxiv-pdf/data/99-word-token-pdfs/0801.3483.pdf",
    "details": {
      "dataset_id": "cluu0c389004l0723jubmv4xg",
      "dataset_name": "pdf_demo_dataset",
      "created_at": "2024-04-10T16:11:08.609+00:00",
      "updated_at": "2024-04-10T16:11:17.957+00:00",
      "last_activity_at": "2024-04-10T16:17:17.000+00:00",
      "created_by": "aovalle@labelbox.com"
    }
  },
  "media_attributes": {
    "mime_type": "application/pdf",
    "text_layer_url": "https://storage.labelbox.com/cl3ahv73w1891087qbwzs3edd%2F0bh0ihjc5370s9a31g5c0uulc-text-layer.json?Expires=1712852458451&KeyName=labelbox-assets-key-3&Signature=5r7Ky5JCM0qbvvkCCpI_1sIw2cQ"
  },
  "attachments": [],
  "metadata_fields": [],
  "projects": {
    "cluu0cl1901aa07zd6t2p7315": {
      "name": "PDF_annotation_demo",
      "labels": [
        {
          "label_kind": "Default",
          "version": "1.0.0",
          "id": "cluu0jznq01zg07wc0lu192t5",
          "label_details": {
            "created_at": "2024-04-10T16:17:17.000+00:00",
            "updated_at": "2024-04-10T16:17:17.000+00:00",
            "created_by": "aovalle@labelbox.com",
            "content_last_updated_at": "2024-04-10T16:17:17.277+00:00",
            "reviews": []
          },
          "performance_details": {
            "seconds_to_create": 1,
            "seconds_to_review": 0,
            "skipped": false
          },
          "annotations": {
            "objects": [
              {
                "feature_id": "cluu0k04w05hk07h76lv05sb0",
                "feature_schema_id": "cluu0ckta024507udcols6x8y",
                "name": "bounding_box",
                "value": "bounding_box",
                "annotation_kind": "DocumentBoundingBox",
                "classifications": [],
                "page_number": 1,
                "bounding_box": {
                  "top": 66.251,
                  "left": 96.424,
                  "height": 80.68099999999998,
                  "width": 82.65
                }
              },
              {
                "feature_id": "cluu0k04w05hl07h79bg9hhu0",
                "feature_schema_id": "cluu0cktb024h07ud57ra95o5",
                "name": "bbox_with_radio_subclass",
                "value": "bbox_with_radio_subclass",
                "annotation_kind": "DocumentBoundingBox",
                "classifications": [
                  {
                    "feature_id": "cluu0k04y05i207h7934rbph0",
                    "feature_schema_id": "cluu0cktb024i07udefqrfcf1",
                    "name": "sub_radio_question",
                    "value": "sub_radio_question",
                    "radio_answer": {
                      "feature_id": "cluu0k04y05i607h76nvj24ls",
                      "feature_schema_id": "cluu0cktb024j07ud89kieukk",
                      "name": "first_sub_radio_answer",
                      "value": "first_sub_radio_answer",
                      "classifications": [
                        {
                          "feature_id": "cluu0k04y05i807h75hpn55pv",
                          "feature_schema_id": "cluu0cktb024k07udf0jw3423",
                          "name": "second_sub_radio_question",
                          "value": "second_sub_radio_question",
                          "radio_answer": {
                            "feature_id": "cluu0k04y05ia07h74p8mgt8q",
                            "feature_schema_id": "cluu0cktb024l07ud7kw1gowr",
                            "name": "second_sub_radio_answer",
                            "value": "second_sub_radio_answer",
                            "classifications": []
                          }
                        }
                      ]
                    }
                  }
                ],
                "page_number": 1,
                "bounding_box": {
                  "top": 226.757,
                  "left": 317.271,
                  "height": 194.22899999999998,
                  "width": 249.38600000000002
                }
              },
              {
                "feature_id": "cluu0k04w05ho07h73jx24ol7",
                "name": "named_entity",
                "value": "named_entity",
                "annotation_kind": "DocumentEntityToken",
                "classifications": [],
                "location": {
                  "groups": [
                    {
                      "id": "1f54807e-b612-4f4a-9ca5-ee302ee9c659",
                      "page_number": 1,
                      "tokens": [
                        "839967fa-89a0-4ad1-896e-d25e5f4cb8e5",
                        "9853c27e-30e0-45e0-b461-c28da6573b48",
                        "e24f45c6-1efc-440f-a4dd-bb3eb522134f",
                        "3339e940-3f73-4b56-b695-474808dad1e5",
                        "a6f79ed8-fc20-4870-b9d1-aee3e92943bc",
                        "07ac4d2e-68bb-4c25-bcc0-b03a697c4138",
                        "7469a5c7-74f8-43c4-b1fb-1c89105a5c66",
                        "eff97919-2529-4b67-9640-20301d45a2d4"
                      ],
                      "text": "Organic charge transfer salts based on the donor"
                    }
                  ]
                }
              },
              {
                "feature_id": "cluu0k04w05hq07h726ie8atv",
                "feature_schema_id": "cluu0ckta024507udcols6x8y",
                "name": "bounding_box",
                "value": "bounding_box",
                "annotation_kind": "DocumentBoundingBox",
                "classifications": [],
                "page_number": 0,
                "bounding_box": {
                  "top": 135.3,
                  "left": 102.771,
                  "height": 109.84299999999999,
                  "width": 415.8
                }
              },
              {
                "feature_id": "cluu0k04w05hr07h7dsu225r3",
                "name": "named_entity",
                "value": "named_entity",
                "annotation_kind": "DocumentEntityToken",
                "classifications": [],
                "location": {
                  "groups": [
                    {
                      "id": "c41c90ad-4ef2-4c49-94cf-6b66b9db521e",
                      "page_number": 1,
                      "tokens": [
                        "db2d0b48-6deb-4013-95ea-e09fd9c09251",
                        "f6f32337-4375-410a-b49b-046b3a07f0be",
                        "327521e0-35e3-481c-940f-ff4548d41390",
                        "23745fb8-8640-4220-badf-1efdbcd15ecd",
                        "a41b5975-c531-4219-a979-70034eac3ce2",
                        "1b5f236c-9919-4f46-92a9-e9b3f42736c8",
                        "fc98ef37-9bad-4195-b837-93c5e9b989bc",
                        "0d352bcd-36e9-4d37-9625-3974e7aba2ec"
                      ],
                      "text": "the experimental investigations on this issue have not"
                    }
                  ]
                }
              },
              {
                "feature_id": "cluu0k04w05hs07h7ahsw7689",
                "feature_schema_id": "cluu0ckta024507udcols6x8y",
                "name": "bounding_box",
                "value": "bounding_box",
                "annotation_kind": "DocumentBoundingBox",
                "classifications": [],
                "page_number": 1,
                "bounding_box": {
                  "top": 68.875,
                  "left": 188.257,
                  "height": 80.68100000000001,
                  "width": 82.64999999999998
                }
              },
              {
                "feature_id": "cluu0k04w05ht07h7gdlucd5k",
                "name": "ner_with_checklist_subclass",
                "value": "ner_with_checklist_subclass",
                "annotation_kind": "DocumentEntityToken",
                "classifications": [
                  {
                    "feature_id": "cluu0k04y05hz07h7am3xc9s9",
                    "feature_schema_id": "cluu0cktb024c07ud1dtpcax0",
                    "name": "sub_checklist_question",
                    "value": "sub_checklist_question",
                    "checklist_answers": [
                      {
                        "feature_id": "cluu0k04y05i007h78cfagn14",
                        "feature_schema_id": "cluu0cktb024d07udhx513yld",
                        "name": "first_sub_checklist_answer",
                        "value": "first_sub_checklist_answer",
                        "classifications": []
                      }
                    ]
                  }
                ],
                "location": {
                  "groups": [
                    {
                      "id": "3e129182-09bf-47ee-a20a-6f9475a8bebb",
                      "page_number": 1,
                      "tokens": [
                        "b2265333-b4ea-43e1-9321-4a81f3c477b0",
                        "ceb2a276-d628-4005-847d-45fe9fcedfab",
                        "5624be95-c333-48b3-8232-28d754699ecb",
                        "65f5ee88-45b9-4a0d-810c-859fac7a85fe",
                        "1c067d79-c119-4fc2-92a1-ef409511cc71",
                        "2b9c4e8a-586f-40c9-82a0-aab3f8fa4dd6",
                        "707924a8-5368-4c6d-afb8-c4e5ab7e2b46",
                        "588436fd-23c6-40cf-87c4-d630740741af",
                        "ca87b137-ff15-4cf5-a74b-a0f5fc768e86"
                      ],
                      "text": "T. Sasaki , N. Yoneyama , and N. Kobayashi"
                    }
                  ]
                }
              },
              {
                "feature_id": "cluu0k04y05hv07h79ci47rgw",
                "name": "named_entity",
                "value": "named_entity",
                "annotation_kind": "DocumentEntityToken",
                "classifications": [],
                "location": {
                  "groups": [
                    {
                      "id": "e14a5896-7f3f-4a57-8cff-fa78523583d0",
                      "page_number": 1,
                      "tokens": [
                        "3158fc72-e758-4d39-ab05-2f785a51463f",
                        "0b9da1f7-3af8-428a-9fbb-1d8972d4865c",
                        "902c7e26-adbd-4b4a-89bf-3b7a28e0a9e0",
                        "719c028c-9f23-49e2-9cc6-ab88c9bfab11",
                        "31f7ab2f-2613-4074-ae92-554a61d199d5",
                        "2bdc2a66-de50-48b9-91c6-5628e9696476",
                        "8df51638-4fb1-4918-948e-5b081c8ac9bf",
                        "cd63cdf8-e805-4082-92d1-037faf1f9fa6",
                        "c796ca0e-55ff-4dfb-90ea-39ca260a2114",
                        "863c87ca-8cff-4a61-975e-df6d865b9028",
                        "79a87aed-444a-417d-a25e-0a63b76188c3",
                        "1fc66034-14ed-481f-a76c-5d9b86e75997"
                      ],
                      "text": "Metal - insulator ( MI ) transitions have been one of the"
                    }
                  ]
                }
              }
            ],
            "classifications": [
              {
                "feature_id": "cluu0k04w05hj07h71okedbdg",
                "feature_schema_id": "cluu0cktc024x07udc5fr0p96",
                "name": "checklist_question",
                "value": "checklist_question",
                "checklist_answers": [
                  {
                    "feature_id": "cluu0k04y05hy07h73gy3awz7",
                    "feature_schema_id": "cluu0cktc025007ud0d5w5o05",
                    "name": "second_checklist_answer",
                    "value": "second_checklist_answer",
                    "classifications": []
                  },
                  {
                    "feature_id": "cluu0k04y05i107h73vpphndb",
                    "feature_schema_id": "cluu0cktc024y07ud97zu7h1d",
                    "name": "first_checklist_answer",
                    "value": "first_checklist_answer",
                    "classifications": []
                  }
                ]
              },
              {
                "feature_id": "cluu0k04w05hm07h7f1m9c0pe",
                "feature_schema_id": "cluu0cktc025d07ud6kzzchyp",
                "name": "nested_checklist_question",
                "value": "nested_checklist_question",
                "checklist_answers": [
                  {
                    "feature_id": "cluu0k04y05i307h7gt9herg8",
                    "feature_schema_id": "cluu0cktc025e07udc46rfioc",
                    "name": "first_checklist_answer",
                    "value": "first_checklist_answer",
                    "classifications": [
                      {
                        "feature_id": "cluu0k04y05i707h7c8hv88jc",
                        "feature_schema_id": "cluu0cktc025f07ud8yhraccd",
                        "name": "sub_checklist_question",
                        "value": "sub_checklist_question",
                        "checklist_answers": [
                          {
                            "feature_id": "cluu0k04y05ib07h7an9s5rln",
                            "feature_schema_id": "cluu0cktc025g07udbpnd5578",
                            "name": "first_sub_checklist_answer",
                            "value": "first_sub_checklist_answer",
                            "classifications": []
                          }
                        ]
                      }
                    ]
                  }
                ]
              },
              {
                "feature_id": "cluu0k04w05hn07h7bf1dehdu",
                "feature_schema_id": "cluu0cktc025507udeiv10o12",
                "name": "nested_radio_question",
                "value": "nested_radio_question",
                "radio_answer": {
                  "feature_id": "cluu0k04y05i507h76d434ayj",
                  "feature_schema_id": "cluu0cktc025607udbgogb22u",
                  "name": "first_radio_answer",
                  "value": "first_radio_answer",
                  "classifications": [
                    {
                      "feature_id": "cluu0k04y05i907h74xuqh3c9",
                      "feature_schema_id": "cluu0cktc025707udh9h9et5l",
                      "name": "sub_radio_question",
                      "value": "sub_radio_question",
                      "radio_answer": {
                        "feature_id": "cluu0k04y05ic07h7fv1m1h7d",
                        "feature_schema_id": "cluu0cktc025807udcbpnbd9s",
                        "name": "first_sub_radio_answer",
                        "value": "first_sub_radio_answer",
                        "classifications": []
                      }
                    }
                  ]
                }
              },
              {
                "feature_id": "cluu0k04w05hp07h7fn0k9bdw",
                "feature_schema_id": "cluu0cktc025307udgntw3qxi",
                "name": "free_text",
                "value": "free_text",
                "text_answer": {
                  "content": "sample text"
                }
              },
              {
                "feature_id": "cluu0k04y05hw07h70s248skz",
                "feature_schema_id": "cluu0cktc024r07ud2x2oba9o",
                "name": "radio_question",
                "value": "radio_question",
                "radio_answer": {
                  "feature_id": "cluu0k04y05i407h7ehvn84w9",
                  "feature_schema_id": "cluu0cktc024s07udc81y6lx3",
                  "name": "first_radio_answer",
                  "value": "first_radio_answer",
                  "classifications": []
                }
              }
            ],
            "relationships": [
              {
                "feature_id": "cluu0k04w05hu07h71d8aaiqh",
                "feature_schema_id": "cluu0cktb024907udbo0qhco1",
                "name": "relationship",
                "value": "relationship",
                "annotation_kind": "DocumentUnidirectionalRelationship",
                "classifications": [],
                "unidirectional_relationship": {
                  "source": "cluu0k04w05ho07h73jx24ol7",
                  "target": "cluu0k04w05hr07h7dsu225r3"
                }
              },
              {
                "feature_id": "cluu0k04y05hx07h75lne7jh6",
                "feature_schema_id": "cluu0cktb024907udbo0qhco1",
                "name": "relationship",
                "value": "relationship",
                "annotation_kind": "DocumentUnidirectionalRelationship",
                "classifications": [],
                "unidirectional_relationship": {
                  "source": "cluu0k04w05hs07h7ahsw7689",
                  "target": "cluu0k04w05hk07h76lv05sb0"
                }
              }
            ]
          }
        }
      ],
      "project_details": {
        "ontology_id": "cluu0cksd024407udeatf87mf",
        "task_id": "0f42415e-708d-452b-ac10-8296b9744746",
        "task_name": "Initial review task",
        "batch_id": "fdf99590-f754-11ee-89ff-078b5fb2d273",
        "batch_name": "PDF_annotation_batch",
        "workflow_status": "IN_REVIEW",
        "priority": 5,
        "consensus_expected_label_count": 1,
        "workflow_history": [
          {
            "action": "Move",
            "created_at": "2024-04-10T16:17:17.681+00:00",
            "created_by": "aovalle@labelbox.com",
            "previous_task_name": "Initial labeling task",
            "previous_task_id": "63c893d3-a7f7-084e-9d56-329c60c58ae0",
            "next_task_name": "Initial review task",
            "next_task_id": "0f42415e-708d-452b-ac10-8296b9744746"
          },
          {
            "action": "Move",
            "created_at": "2024-04-10T16:17:17.675+00:00",
            "created_by": "aovalle@labelbox.com",
            "next_task_name": "Initial labeling task",
            "next_task_id": "63c893d3-a7f7-084e-9d56-329c60c58ae0"
          }
        ]
      }
    }
  }
}
```

### Sample Model Run Export

```json JSON expandable theme={"theme":"css-variables"}
{
  "data_row": {
    "id": "clfh1n29e1mh2078d3jt3b5cx",
    "global_key": "0801.3483.pdf",
    "row_data": "https://storage.googleapis.com/labelbox-datasets/arxiv-pdf/data/99-word-token-pdfs/0801.3483.pdf",
    "details": {
      "dataset_id": "clfh1n0um00wd071d2uf21fy6",
      "dataset_name": "pdf_demo_dataset",
      "created_at": "2023-03-20T16:31:01.000+00:00",
      "updated_at": "2023-03-20T16:31:01.000+00:00",
      "created_by": "labelbox.public.user@labelbox.com"
    }
  },
  "media_attributes": {
    "mime_type": "application/pdf",
    "text_layer_url": "https://storage.googleapis.com/labelbox-datasets/arxiv-pdf/data/99-word-token-pdfs/0801.3483-lb-textlayer.json"
  },
  "attachments": [],
  "metadata_fields": [],
  "experiments": {
    "a15bd76b-e29f-02f2-b909-0e20f007f735": {
      "name": "PDF_model_run_9c051c54-bb5b-47be-a0f4-16c909a9afb0",
      "runs": {
        "a15bd76c-41aa-0f04-c633-ee3e40b1d0bc": {
          "name": "iteration 1",
          "run_data_row_id": "dff29c2a-27c2-443b-95a8-402d9c2be311",
          "labels": [
            {
              "label_kind": "Default",
              "version": "1.0.0",
              "id": "clhs1ru105b3w142bd71v29u1",
              "annotations": {
                "objects": [
                  {
                    "feature_id": "2aa47146-4d81-4fcd-b513-7f7f5e275db7",
                    "feature_schema_id": "clhs1rh090buq07y15p2wd58d",
                    "name": "bounding_box",
                    "value": "bounding_box",
                    "annotation_kind": "DocumentBoundingBox",
                    "classifications": [],
                    "page_number": 0,
                    "bounding_box": {
                      "top": 135.3,
                      "left": 102.771,
                      "height": 109.84299999999999,
                      "width": 415.8
                    }
                  },
                  {
                    "feature_id": "40cd3ef3-7258-4da6-bc8b-d4f2eb12abfb",
                    "name": "ner_with_checklist_subclass",
                    "value": "ner_with_checklist_subclass",
                    "annotation_kind": "DocumentEntityToken",
                    "classifications": [
                      {
                        "feature_id": "4cf8dcba-32ab-4bf0-a97e-5a17d72335b5",
                        "feature_schema_id": "clhs1rh0a0buv07y1e0wobtgu",
                        "name": "sub_checklist_question",
                        "value": "sub_checklist_question",
                        "checklist_answers": [
                          {
                            "feature_id": "b139a807-7b9f-41f2-97e2-1dca6cd9f460",
                            "feature_schema_id": "clhs1rh0a0buw07y12tg3hm0x",
                            "name": "first_sub_checklist_answer",
                            "value": "first_sub_checklist_answer",
                            "classifications": []
                          }
                        ]
                      }
                    ],
                    "location": {
                      "groups": [
                        {
                          "id": "80e1e326-0194-4f38-845c-d9d22b0054fb",
                          "page_number": 1,
                          "tokens": [
                            "c2639311-4db1-4ad5-9a8d-1f2ab2c2dae8",
                            "2aa2f9df-df5a-4b3a-aa57-05aff053c147",
                            "aa725467-7b86-4629-9c70-a6391decb39b",
                            "6a4a0a69-f04e-44c0-b343-096aef413444",
                            "8dcce11c-c2ae-4274-b634-c950c8a7f333",
                            "85fccf77-474b-4707-85cc-592871d5c680",
                            "5d16be99-fc40-4228-99db-0a121a0478e7"
                          ],
                          "text": "T. Sasaki,* N. Yoneyama, and N. Kobayashi"
                        }
                      ]
                    }
                  },
                  {
                    "feature_id": "d2cd83b8-7a7b-4751-8b5d-dfa05be90bf5",
                    "name": "named_entity",
                    "value": "named_entity",
                    "annotation_kind": "DocumentEntityToken",
                    "classifications": [],
                    "location": {
                      "groups": [
                        {
                          "id": "2f4336f4-a07e-4e0a-a9e1-5629b03b719b",
                          "page_number": 1,
                          "tokens": [
                            "3f984bf3-1d61-44f5-b59a-9658a2e3440f",
                            "3bf00b56-ff12-4e52-8cc1-08dbddb3c3b8",
                            "6e1c3420-d4b7-4c5a-8fd6-ead43bf73d80",
                            "87a43d32-af76-4a1d-b262-5c5f4d5ace3a",
                            "e8606e8a-dfd9-4c49-a635-ad5c879c75d0",
                            "67c7c19e-4654-425d-bf17-2adb8cf02c30",
                            "149c5e80-3e07-49a7-ab2d-29ddfe6a38fa",
                            "b0e94071-2187-461e-8e76-96c58738a52c"
                          ],
                          "text": "Metal-insulator (MI) transitions have been one of the"
                        }
                      ]
                    }
                  },
                  {
                    "feature_id": "e166e286-1ead-438d-bb6e-0bc148b4c5f0",
                    "feature_schema_id": "clhs1rh0a0bv007y1fyxw0gyg",
                    "name": "bbox_with_radio_subclass",
                    "value": "bbox_with_radio_subclass",
                    "annotation_kind": "DocumentBoundingBox",
                    "classifications": [
                      {
                        "feature_id": "68838874-d209-4672-891f-10954a698292",
                        "feature_schema_id": "clhs1rh0a0bv107y14g5de74m",
                        "name": "sub_radio_question",
                        "value": "sub_radio_question",
                        "radio_answer": {
                          "feature_id": "8ce7710d-7a15-4d95-b9a9-a9a2bf733380",
                          "feature_schema_id": "clhs1rh0a0bv207y13bsndvuv",
                          "name": "first_sub_radio_answer",
                          "value": "first_sub_radio_answer",
                          "classifications": [
                            {
                              "feature_id": "bc3542d1-f373-4668-841b-15a4e9f8f67b",
                              "feature_schema_id": "clhs1rh0a0bv307y17teq5ob3",
                              "name": "second_sub_radio_question",
                              "value": "second_sub_radio_question",
                              "radio_answer": {
                                "feature_id": "6d200263-2065-4363-b962-1ee5a1296a8e",
                                "feature_schema_id": "clhs1rh0a0bv407y1d72h3bdp",
                                "name": "second_sub_radio_answer",
                                "value": "second_sub_radio_answer",
                                "classifications": []
                              }
                            }
                          ]
                        }
                      }
                    ],
                    "page_number": 1,
                    "bounding_box": {
                      "top": 226.757,
                      "left": 317.271,
                      "height": 194.22899999999998,
                      "width": 249.38600000000002
                    }
                  }
                ],
                "classifications": [
                  {
                    "feature_id": "2fbbc78f-d1df-428d-be26-da69a6516786",
                    "feature_schema_id": "clhs1rh0a0bva07y11vb9gjzu",
                    "name": "radio_question",
                    "value": "radio_question",
                    "radio_answer": {
                      "feature_id": "24924adc-cb1c-40fb-a945-364a2cb0782f",
                      "feature_schema_id": "clhs1rh0a0bvb07y134ka29be",
                      "name": "first_radio_answer",
                      "value": "first_radio_answer",
                      "classifications": []
                    }
                  },
                  {
                    "feature_id": "a10706c7-0e09-47d9-b273-ed61f6ee72c5",
                    "feature_schema_id": "clhs1rh0b0bvg07y1as0cfj9r",
                    "name": "checklist_question",
                    "value": "checklist_question",
                    "checklist_answers": [
                      {
                        "feature_id": "03c2feb6-bda3-4ee2-ad9f-faef038ee029",
                        "feature_schema_id": "clhs1rh0b0bvj07y1f1brb5f3",
                        "name": "second_checklist_answer",
                        "value": "second_checklist_answer",
                        "classifications": []
                      },
                      {
                        "feature_id": "3653a591-4915-450f-88e4-f5afc8cd1440",
                        "feature_schema_id": "clhs1rh0b0bvh07y19qgt6nsv",
                        "name": "first_checklist_answer",
                        "value": "first_checklist_answer",
                        "classifications": []
                      }
                    ]
                  },
                  {
                    "feature_id": "da14ab32-c007-4e86-aaff-a64e51ff2503",
                    "feature_schema_id": "clhs1rh0b0bvm07y1gpw00bw3",
                    "name": "free_text",
                    "value": "free_text",
                    "text_answer": {
                      "content": "sample text"
                    }
                  },
                  {
                    "feature_id": "de0e9826-a0fc-4603-a59a-0c356895759c",
                    "feature_schema_id": "clhs1rh0b0bvo07y1025n0p4a",
                    "name": "nested_radio_question",
                    "value": "nested_radio_question",
                    "radio_answer": {
                      "feature_id": "913b5925-7b37-43a0-9086-c7734491d60a",
                      "feature_schema_id": "clhs1rh0b0bvp07y1d1tbgpeg",
                      "name": "first_radio_answer",
                      "value": "first_radio_answer",
                      "classifications": [
                        {
                          "feature_id": "7c10d9b2-0e07-4723-abef-ad022db24b10",
                          "feature_schema_id": "clhs1rh0b0bvq07y13wux72rp",
                          "name": "sub_radio_question",
                          "value": "sub_radio_question",
                          "radio_answer": {
                            "feature_id": "ae62a004-f6b3-4436-916e-ce375cf8210e",
                            "feature_schema_id": "clhs1rh0b0bvr07y15n246t5c",
                            "name": "first_sub_radio_answer",
                            "value": "first_sub_radio_answer",
                            "classifications": []
                          }
                        }
                      ]
                    }
                  },
                  {
                    "feature_id": "ee0a7c2c-cb78-4474-b12a-82406ee415c5",
                    "feature_schema_id": "clhs1rh0c0bvw07y1hiyweequ",
                    "name": "nested_checklist_question",
                    "value": "nested_checklist_question",
                    "checklist_answers": [
                      {
                        "feature_id": "22d25a8a-0831-4499-8980-aaebbac1b4eb",
                        "feature_schema_id": "clhs1rh0c0bvx07y1er8ge5qk",
                        "name": "first_checklist_answer",
                        "value": "first_checklist_answer",
                        "classifications": [
                          {
                            "feature_id": "825ce160-5e9f-4926-aa63-ed17ad7fb534",
                            "feature_schema_id": "clhs1rh0c0bvy07y1hmkv360v",
                            "name": "sub_checklist_question",
                            "value": "sub_checklist_question",
                            "checklist_answers": [
                              {
                                "feature_id": "32cd713f-c272-4657-8d0e-910e24e90414",
                                "feature_schema_id": "clhs1rh0c0bvz07y1d1njge4e",
                                "name": "first_sub_checklist_answer",
                                "value": "first_sub_checklist_answer",
                                "classifications": []
                              }
                            ]
                          }
                        ]
                      }
                    ]
                  }
                ],
                "relationships": []
              }
            }
          ],
          "predictions": [
            {
              "label_kind": "Default",
              "version": "1.0.0",
              "id": "clhs1ru105b3w142bd71v29u1",
              "annotations": {
                "objects": [
                  {
                    "feature_id": "1d1a302a-2bf1-4a64-b2db-1283400fc8c6",
                    "feature_schema_id": "clhs1rh0a0bv007y1fyxw0gyg",
                    "name": "bbox_with_radio_subclass",
                    "value": "bbox_with_radio_subclass",
                    "annotation_kind": "DocumentBoundingBox",
                    "classifications": [
                      {
                        "feature_id": "f13c6571-b3c8-4584-886b-555e1808a955",
                        "feature_schema_id": "clhs1rh0a0bv107y14g5de74m",
                        "name": "sub_radio_question",
                        "value": "sub_radio_question",
                        "radio_answer": {
                          "feature_id": "610215da-b253-46f8-8a6d-7dd56b36e32c",
                          "feature_schema_id": "clhs1rh0a0bv207y13bsndvuv",
                          "name": "first_sub_radio_answer",
                          "value": "first_sub_radio_answer",
                          "classifications": [
                            {
                              "feature_id": "a1002a15-c1f1-43ce-9606-9b02fb4d3d77",
                              "feature_schema_id": "clhs1rh0a0bv307y17teq5ob3",
                              "name": "second_sub_radio_question",
                              "value": "second_sub_radio_question",
                              "radio_answer": {
                                "feature_id": "a225ff2c-dccc-4ad1-8dd5-d76f10c5aab2",
                                "feature_schema_id": "clhs1rh0a0bv407y1d72h3bdp",
                                "name": "second_sub_radio_answer",
                                "value": "second_sub_radio_answer",
                                "classifications": []
                              }
                            }
                          ]
                        }
                      }
                    ],
                    "page_number": 1,
                    "bounding_box": {
                      "top": 226.757,
                      "left": 317.271,
                      "height": 194.22899999999998,
                      "width": 249.38600000000002
                    }
                  },
                  {
                    "feature_id": "469420eb-12a7-401f-a5ea-142b24c622e3",
                    "feature_schema_id": "clhs1rh090buq07y15p2wd58d",
                    "name": "bounding_box",
                    "value": "bounding_box",
                    "annotation_kind": "DocumentBoundingBox",
                    "classifications": [],
                    "page_number": 0,
                    "bounding_box": {
                      "top": 135.3,
                      "left": 102.771,
                      "height": 109.84299999999999,
                      "width": 415.8
                    }
                  },
                  {
                    "feature_id": "d3c14b37-6b64-4c6d-91bc-446bf82dc0b5",
                    "name": "named_entity",
                    "value": "named_entity",
                    "annotation_kind": "DocumentEntityToken",
                    "classifications": [],
                    "location": {
                      "groups": [
                        {
                          "id": "2f4336f4-a07e-4e0a-a9e1-5629b03b719b",
                          "page_number": 1,
                          "tokens": [
                            "3f984bf3-1d61-44f5-b59a-9658a2e3440f",
                            "3bf00b56-ff12-4e52-8cc1-08dbddb3c3b8",
                            "6e1c3420-d4b7-4c5a-8fd6-ead43bf73d80",
                            "87a43d32-af76-4a1d-b262-5c5f4d5ace3a",
                            "e8606e8a-dfd9-4c49-a635-ad5c879c75d0",
                            "67c7c19e-4654-425d-bf17-2adb8cf02c30",
                            "149c5e80-3e07-49a7-ab2d-29ddfe6a38fa",
                            "b0e94071-2187-461e-8e76-96c58738a52c"
                          ],
                          "text": "Metal-insulator (MI) transitions have been one of the"
                        }
                      ]
                    }
                  },
                  {
                    "feature_id": "efe907c9-b84b-4015-b29b-3a1cae1655cc",
                    "name": "ner_with_checklist_subclass",
                    "value": "ner_with_checklist_subclass",
                    "annotation_kind": "DocumentEntityToken",
                    "classifications": [
                      {
                        "feature_id": "12fb5823-1a12-4de4-8613-8460cec5eaf1",
                        "feature_schema_id": "clhs1rh0a0buv07y1e0wobtgu",
                        "name": "sub_checklist_question",
                        "value": "sub_checklist_question",
                        "checklist_answers": [
                          {
                            "feature_id": "8be1ca80-1f98-48ad-ae65-d38ab941ef63",
                            "feature_schema_id": "clhs1rh0a0buw07y12tg3hm0x",
                            "name": "first_sub_checklist_answer",
                            "value": "first_sub_checklist_answer",
                            "classifications": []
                          }
                        ]
                      }
                    ],
                    "location": {
                      "groups": [
                        {
                          "id": "",
                          "page_number": 1,
                          "tokens": [],
                          "text": ""
                        }
                      ]
                    }
                  }
                ],
                "classifications": [
                  {
                    "feature_id": "35f2a9ed-e245-4654-94a5-9e7e48adff8d",
                    "feature_schema_id": "clhs1rh0b0bvo07y1025n0p4a",
                    "name": "nested_radio_question",
                    "value": "nested_radio_question",
                    "radio_answer": {
                      "feature_id": "70a52e6d-eda7-48f3-a1dd-65672c273318",
                      "feature_schema_id": "clhs1rh0b0bvp07y1d1tbgpeg",
                      "name": "first_radio_answer",
                      "value": "first_radio_answer",
                      "classifications": [
                        {
                          "feature_id": "5f87ca53-3637-4f26-9068-8645b6036646",
                          "feature_schema_id": "clhs1rh0b0bvq07y13wux72rp",
                          "name": "sub_radio_question",
                          "value": "sub_radio_question",
                          "radio_answer": {
                            "feature_id": "c652afc0-d24f-42ea-9a5a-dee14cc1d80d",
                            "feature_schema_id": "clhs1rh0b0bvr07y15n246t5c",
                            "name": "first_sub_radio_answer",
                            "value": "first_sub_radio_answer",
                            "classifications": []
                          }
                        }
                      ]
                    }
                  },
                  {
                    "feature_id": "73114ba5-3045-46e0-9323-468914afbdf0",
                    "feature_schema_id": "clhs1rh0b0bvg07y1as0cfj9r",
                    "name": "checklist_question",
                    "value": "checklist_question",
                    "checklist_answers": [
                      {
                        "feature_id": "666fc303-b30e-4647-a694-5c34dab4781f",
                        "feature_schema_id": "clhs1rh0b0bvj07y1f1brb5f3",
                        "name": "second_checklist_answer",
                        "value": "second_checklist_answer",
                        "classifications": []
                      },
                      {
                        "feature_id": "f8c4ce45-0e56-4ec2-bc5a-ce9364b471f8",
                        "feature_schema_id": "clhs1rh0b0bvh07y19qgt6nsv",
                        "name": "first_checklist_answer",
                        "value": "first_checklist_answer",
                        "classifications": []
                      }
                    ]
                  },
                  {
                    "feature_id": "911665a7-4901-4c31-a410-5c3a7a614c1d",
                    "feature_schema_id": "clhs1rh0c0bvw07y1hiyweequ",
                    "name": "nested_checklist_question",
                    "value": "nested_checklist_question",
                    "checklist_answers": [
                      {
                        "feature_id": "0341d434-c589-49d4-9a4a-b554647c2a44",
                        "feature_schema_id": "clhs1rh0c0bvx07y1er8ge5qk",
                        "name": "first_checklist_answer",
                        "value": "first_checklist_answer",
                        "classifications": [
                          {
                            "feature_id": "ee1c729c-d1f1-478e-98f8-baefa24dff28",
                            "feature_schema_id": "clhs1rh0c0bvy07y1hmkv360v",
                            "name": "sub_checklist_question",
                            "value": "sub_checklist_question",
                            "checklist_answers": [
                              {
                                "feature_id": "a2c0d47f-811c-4d70-b024-8514d67b206c",
                                "feature_schema_id": "clhs1rh0c0bvz07y1d1njge4e",
                                "name": "first_sub_checklist_answer",
                                "value": "first_sub_checklist_answer",
                                "classifications": []
                              }
                            ]
                          }
                        ]
                      }
                    ]
                  },
                  {
                    "feature_id": "a7f83569-636d-4993-81f0-6d4de144bb24",
                    "feature_schema_id": "clhs1rh0a0bva07y11vb9gjzu",
                    "name": "radio_question",
                    "value": "radio_question",
                    "radio_answer": {
                      "feature_id": "58624513-6ab0-452b-a9a0-72cebf648cd0",
                      "feature_schema_id": "clhs1rh0a0bvb07y134ka29be",
                      "name": "first_radio_answer",
                      "value": "first_radio_answer",
                      "classifications": []
                    }
                  },
                  {
                    "feature_id": "c7e3f324-7b53-47a3-aeb0-4217717fcfed",
                    "feature_schema_id": "clhs1rh0b0bvm07y1gpw00bw3",
                    "name": "free_text",
                    "value": "free_text",
                    "text_answer": {
                      "content": "sample text"
                    }
                  }
                ],
                "relationships": []
              }
            }
          ]
        }
      }
    }
  },
  "models": {}
}
```

## Audio

### Sample Project Export

```json JSON expandable theme={"theme":"css-variables"}
{
  "data_row": {
    "id": "clftvvbln01cb07343mpx8n1l",
    "global_key": "sample-audio-1.mp3",
    "row_data": "https://storage.googleapis.com/labelbox-datasets/audio-sample-data/sample-audio-1.mp3",
    "details": {
      "dataset_id": "clftvva210b5t070d0no12z0v",
      "dataset_name": "audio_annotation_import_demo_dataset",
      "created_at": "2023-03-29T16:10:29.000+00:00",
      "updated_at": "2024-02-10T20:36:56.174+00:00",
      "last_activity_at": "2023-10-17T20:33:05.000+00:00",
      "created_by": "labelbox.public.user@labelbox.com"
    }
  },
  "media_attributes": {
    "mime_type": "audio/mpeg"
  },
  "attachments": [],
  "metadata_fields": [],
  "projects": {
    "clnus6sqx08k607yd0d33gfld": {
      "name": "audio_project",
      "labels": [
        {
          "label_kind": "Default",
          "version": "1.0.0",
          "id": "clnus731g4gc6143uyfqfbjth",
          "label_details": {
            "created_at": "2023-10-17T20:33:05.000+00:00",
            "updated_at": "2023-10-17T20:33:05.000+00:00",
            "created_by": "aovalle@labelbox.com",
            "content_last_updated_at": "2023-10-17T20:33:05.001+00:00",
            "reviews": []
          },
          "performance_details": {
            "seconds_to_create": 0,
            "seconds_to_review": 0,
            "skipped": false
          },
          "annotations": {
            "objects": [],
            "classifications": [
              {
                "feature_id": "clnus73114gc0143urjlfk558",
                "feature_schema_id": "clnus6slu06xg07yjcg87apa0",
                "name": "text_audio",
                "value": "text_audio",
                "text_answer": {
                  "content": "free text audio annotation"
                }
              },
              {
                "feature_id": "clnus73114gc1143u7fzf4ebv",
                "feature_schema_id": "clnus6slu06xi07yj7m9d7rft",
                "name": "checklist_audio",
                "value": "checklist_audio",
                "checklist_answers": [
                  {
                    "feature_id": "clnus73114gc2143u6ah5wdhn",
                    "feature_schema_id": "clnus6slu06xj07yj9eb7dpvp",
                    "name": "first_checklist_answer",
                    "value": "first_checklist_answer",
                    "classifications": []
                  },
                  {
                    "feature_id": "clnus73114gc3143uq755vkzi",
                    "feature_schema_id": "clnus6slu06xl07yjdab4gxe8",
                    "name": "second_checklist_answer",
                    "value": "second_checklist_answer",
                    "classifications": []
                  }
                ]
              },
              {
                "feature_id": "clnus73114gc4143u2dux654s",
                "feature_schema_id": "clnus6slv06xo07yja7l47gtw",
                "name": "radio_audio",
                "value": "radio_audio",
                "radio_answer": {
                  "feature_id": "clnus73114gc5143u2jadnt6r",
                  "feature_schema_id": "clnus6slv06xr07yj24yp1gdk",
                  "name": "second_radio_answer",
                  "value": "second_radio_answer",
                  "classifications": []
                }
              }
            ],
            "relationships": []
          }
        }
      ],
      "project_details": {
        "ontology_id": "clnus6sl206xf07yj7md9cbg0",
        "task_name": "Done",
        "batch_id": "58ad3900-6d2c-11ee-a344-51974336e7ee",
        "batch_name": "first-batch-audio-demo",
        "workflow_status": "DONE",
        "priority": 5,
        "consensus_expected_label_count": 1,
        "workflow_history": []
      }
    }
  }
}
```

## HTML

### Sample Project Export

```json JSON expandable theme={"theme":"css-variables"}
{
  "data_row": {
    "id": "clfskiyrx0e0q078k40evbanc",
    "global_key": "sample_html_1.html",
    "row_data": "https://storage.googleapis.com/labelbox-datasets/html_sample_data/sample_html_1.html",
    "details": {
      "dataset_id": "clfskixq70ym7071scn9l16z0",
      "dataset_name": "html_annotation_import_demo_dataset",
      "created_at": "2023-03-28T18:05:11.000+00:00",
      "updated_at": "2024-01-19T23:05:36.279+00:00",
      "last_activity_at": "2023-04-24T20:19:59.000+00:00",
      "created_by": "labelbox.public.user@labelbox.com"
    }
  },
  "media_attributes": {
    "mime_type": "text/html"
  },
  "attachments": [],
  "metadata_fields": [],
  "projects": {
    "clgva7x400opk07x73jgs6p8o": {
      "name": "HTML Import Annotation Demo",
      "labels": [
        {
          "label_kind": "Default",
          "version": "1.0.0",
          "id": "clgva8bonhmg21401hcpq6tee",
          "label_details": {
            "created_at": "2023-04-24T20:19:59.000+00:00",
            "updated_at": "2023-04-24T20:19:59.000+00:00",
            "created_by": "aovalle@labelbox.com",
            "content_last_updated_at": "2023-04-24T20:19:59.466+00:00",
            "reviews": []
          },
          "performance_details": {
            "seconds_to_create": 0,
            "seconds_to_review": 0,
            "skipped": false
          },
          "annotations": {
            "objects": [],
            "classifications": [
              {
                "feature_id": "clgva8bo5hmfo140186f07jci",
                "feature_schema_id": "clgva7wwa0l9j07x51e4t0bq8",
                "name": "text_html",
                "value": "text_html",
                "text_answer": {
                  "content": "sample text"
                }
              },
              {
                "feature_id": "clgva8bo5hmfp14016xok5j73",
                "feature_schema_id": "clgva7wwa0l9l07x558vydexo",
                "name": "checklist_html",
                "value": "checklist_html",
                "checklist_answers": [
                  {
                    "feature_id": "clgva8bo5hmfq1401xnngji4s",
                    "feature_schema_id": "clgva7wwb0l9m07x5671qe9ej",
                    "name": "first_checklist_answer",
                    "value": "first_checklist_answer",
                    "classifications": []
                  },
                  {
                    "feature_id": "clgva8bo5hmfr1401w191mdo5",
                    "feature_schema_id": "clgva7wwb0l9o07x5h08p9572",
                    "name": "second_checklist_answer",
                    "value": "second_checklist_answer",
                    "classifications": []
                  }
                ]
              },
              {
                "feature_id": "clgva8bo5hmfs1401auduvxht",
                "feature_schema_id": "clgva7wwb0l9r07x54f9h826z",
                "name": "radio_html",
                "value": "radio_html",
                "radio_answer": {
                  "feature_id": "clgva8bo5hmft1401nhfbg7ja",
                  "feature_schema_id": "clgva7wwb0l9u07x55fyxa5g3",
                  "name": "second_radio_answer",
                  "value": "second_radio_answer",
                  "classifications": []
                }
              },
              {
                "feature_id": "clgva8bo5hmfu1401yuz7wvuy",
                "feature_schema_id": "clgva7wwb0l9x07x57si1gihw",
                "name": "nested_checklist_question",
                "value": "nested_checklist_question",
                "checklist_answers": [
                  {
                    "feature_id": "clgva8bo6hmfv14015d33m625",
                    "feature_schema_id": "clgva7wwb0l9y07x5dt6l44qv",
                    "name": "first_checklist_answer",
                    "value": "first_checklist_answer",
                    "classifications": [
                      {
                        "feature_id": "clgva8bo6hmfw1401u59hfep5",
                        "feature_schema_id": "clgva7wwb0l9z07x5bvtx0agv",
                        "name": "sub_checklist_question",
                        "value": "sub_checklist_question",
                        "checklist_answers": [
                          {
                            "feature_id": "clgva8bo6hmfx14018asstw29",
                            "feature_schema_id": "clgva7wwb0la007x515qu9lje",
                            "name": "first_sub_checklist_answer",
                            "value": "first_sub_checklist_answer",
                            "classifications": []
                          }
                        ]
                      }
                    ]
                  }
                ]
              },
              {
                "feature_id": "clgva8bo6hmfy1401bivtvjjt",
                "feature_schema_id": "clgva7wwb0la507x583lg7mvf",
                "name": "nested_radio_question",
                "value": "nested_radio_question",
                "radio_answer": {
                  "feature_id": "clgva8bo6hmfz1401y555xn92",
                  "feature_schema_id": "clgva7wwb0la607x5h39u2z31",
                  "name": "first_radio_answer",
                  "value": "first_radio_answer",
                  "classifications": [
                    {
                      "feature_id": "clgva8bo6hmg01401agmna01o",
                      "feature_schema_id": "clgva7wwb0la707x5hxwe16bo",
                      "name": "sub_radio_question",
                      "value": "sub_radio_question",
                      "radio_answer": {
                        "feature_id": "clgva8bo6hmg11401dzqw0od8",
                        "feature_schema_id": "clgva7wwb0la807x51792e6il",
                        "name": "first_sub_radio_answer",
                        "value": "first_sub_radio_answer",
                        "classifications": []
                      }
                    }
                  ]
                }
              }
            ],
            "relationships": []
          }
        }
      ],
      "project_details": {
        "ontology_id": "clgva7wto0l9i07x5emb8b4uo",
        "task_name": "Done",
        "batch_id": "5ba71600-e2dd-11ed-8ee3-e3123dc756e1",
        "batch_name": "first-batch-html-demo",
        "workflow_status": "DONE",
        "priority": 5,
        "consensus_expected_label_count": 1,
        "workflow_history": []
      }
    }
  }
}
```

### Sample Model Run Export

```json JSON expandable theme={"theme":"css-variables"}
{
  "data_row": {
    "id": "clfcvkqx708ba079n4nfo1drf",
    "global_key": "sample_html_2.html",
    "row_data": "https://storage.googleapis.com/labelbox-datasets/html_sample_data/sample_html_2.html",
    "details": {
      "dataset_id": "clfcvkpzc0v8l07yt9vhc16sv",
      "dataset_name": "html demo dataset",
      "created_at": "2023-03-17T18:30:11.000+00:00",
      "updated_at": "2024-01-19T22:55:40.282+00:00",
      "created_by": "labelbox.public.user@labelbox.com"
    }
  },
  "media_attributes": {
    "mime_type": "text/html"
  },
  "attachments": [],
  "metadata_fields": [],
  "experiments": {
    "a414cfe7-f6ab-094b-8307-68d5a7272397": {
      "name": "HTML_model_run_1d21d941-f392-41e9-85d4-a52b3a3b5bfd",
      "runs": {
        "a414cfe8-690d-0b10-6b9f-7e9bf6a7dd1c": {
          "name": "iteration 1",
          "run_data_row_id": "d2bdab47-af16-4052-a310-74fa324cda5f",
          "labels": [
            {
              "label_kind": "Default",
              "version": "1.0.0",
              "id": "cluu1x19z0out148p9vadgmcu",
              "annotations": {
                "objects": [],
                "classifications": [
                  {
                    "feature_id": "3625d4eb-ca0f-0385-9c74-c3538b40621c",
                    "feature_schema_id": "cluu1wl30002007wo1gnn8htz",
                    "name": "free_text",
                    "value": "free_text",
                    "text_answer": {
                      "content": "sample text"
                    }
                  },
                  {
                    "feature_id": "47d39827-416f-042f-a9f5-7d5bf45ba884",
                    "feature_schema_id": "cluu1wl2z001k07wo17r02qrr",
                    "name": "nested_radio_question",
                    "value": "nested_radio_question",
                    "radio_answer": {
                      "feature_id": "89c36638-ec65-095c-893a-49417a1611fc",
                      "feature_schema_id": "cluu1wl2z001l07wo6kll5fc0",
                      "name": "first_radio_answer",
                      "value": "first_radio_answer",
                      "classifications": [
                        {
                          "feature_id": "5de3b3eb-78a8-06e8-b420-491234d05743",
                          "feature_schema_id": "cluu1wl2z001m07wodiwh0hjp",
                          "name": "sub_radio_question",
                          "value": "sub_radio_question",
                          "radio_answer": {
                            "feature_id": "684b6ca5-e5d4-0684-991d-597e5ea76700",
                            "feature_schema_id": "cluu1wl2z001n07wo8ey5f1yw",
                            "name": "first_sub_radio_answer",
                            "value": "first_sub_radio_answer",
                            "classifications": []
                          }
                        }
                      ]
                    }
                  },
                  {
                    "feature_id": "9297d521-2b0a-04bd-b88b-b5512b85ec20",
                    "feature_schema_id": "cluu1wl2z001s07woc5xmcz68",
                    "name": "checklist_question",
                    "value": "checklist_question",
                    "checklist_answers": [
                      {
                        "feature_id": "add68a9d-01b1-039a-bbdf-c96f63d408e6",
                        "feature_schema_id": "cluu1wl30001v07wocp3hg8a0",
                        "name": "second_checklist_answer",
                        "value": "second_checklist_answer",
                        "classifications": []
                      },
                      {
                        "feature_id": "ce4eca3a-b2f5-0861-954f-ad7ff07c1a7f",
                        "feature_schema_id": "cluu1wl30001x07wobpdwgf4w",
                        "name": "third_checklist_answer",
                        "value": "third_checklist_answer",
                        "classifications": []
                      },
                      {
                        "feature_id": "fab6d119-206f-0b6d-a5a7-8a08e17ed8cf",
                        "feature_schema_id": "cluu1wl2z001t07wobsowcibp",
                        "name": "first_checklist_answer",
                        "value": "first_checklist_answer",
                        "classifications": []
                      }
                    ]
                  },
                  {
                    "feature_id": "9c1c6bad-ca20-08c9-8d1a-78907e65ebe4",
                    "feature_schema_id": "cluu1wl2z001g07wo5elxfwut",
                    "name": "radio_question",
                    "value": "radio_question",
                    "radio_answer": {
                      "feature_id": "28ea7b46-b891-0285-a8a3-3ba7a30cdeb4",
                      "feature_schema_id": "cluu1wl2z001h07wo760uc0sb",
                      "name": "first_radio_answer",
                      "value": "first_radio_answer",
                      "classifications": []
                    }
                  },
                  {
                    "feature_id": "d48dce83-f514-0728-9b5a-deb045493dc3",
                    "feature_schema_id": "cluu1wl30002207woa3u870v7",
                    "name": "nested_checklist_question",
                    "value": "nested_checklist_question",
                    "checklist_answers": [
                      {
                        "feature_id": "ca4f111b-c6f6-013a-a38e-0ce1ee77b370",
                        "feature_schema_id": "cluu1wl30002307wogixwdc5z",
                        "name": "first_checklist_answer",
                        "value": "first_checklist_answer",
                        "classifications": [
                          {
                            "feature_id": "6d08884d-ff25-0a41-a5aa-875e6385ea1e",
                            "feature_schema_id": "cluu1wl30002407woea6e102z",
                            "name": "sub_checklist_question",
                            "value": "sub_checklist_question",
                            "checklist_answers": [
                              {
                                "feature_id": "02db9eb9-6763-0a2c-92ee-cdf3b11b8de4",
                                "feature_schema_id": "cluu1wl30002507wo07av3u9e",
                                "name": "first_sub_checklist_answer",
                                "value": "first_sub_checklist_answer",
                                "classifications": []
                              }
                            ]
                          }
                        ]
                      }
                    ]
                  }
                ],
                "relationships": []
              }
            }
          ],
          "predictions": [
            {
              "label_kind": "Default",
              "version": "1.0.0",
              "id": "cluu1x19z0out148p9vadgmcu",
              "annotations": {
                "objects": [],
                "classifications": [
                  {
                    "feature_id": "162ea9d9-c6e8-48e9-b4fe-1beb5ec35e71",
                    "feature_schema_id": "cluu1wl2z001k07wo17r02qrr",
                    "name": "nested_radio_question",
                    "value": "nested_radio_question",
                    "radio_answer": {
                      "feature_id": "ecdb63be-bd8b-4768-ab51-6399e38c3fd8",
                      "feature_schema_id": "cluu1wl2z001l07wo6kll5fc0",
                      "name": "first_radio_answer",
                      "value": "first_radio_answer",
                      "classifications": [
                        {
                          "feature_id": "9bdb8546-6ee7-4993-bea7-9de7d1d4f0d5",
                          "feature_schema_id": "cluu1wl2z001m07wodiwh0hjp",
                          "name": "sub_radio_question",
                          "value": "sub_radio_question",
                          "radio_answer": {
                            "feature_id": "a0466f11-c940-4217-9589-56f261425e43",
                            "feature_schema_id": "cluu1wl2z001n07wo8ey5f1yw",
                            "name": "first_sub_radio_answer",
                            "value": "first_sub_radio_answer",
                            "classifications": []
                          }
                        }
                      ]
                    }
                  },
                  {
                    "feature_id": "2b7e8e1b-a55d-4ac6-80e2-629bc64c5b97",
                    "feature_schema_id": "cluu1wl30002007wo1gnn8htz",
                    "name": "free_text",
                    "value": "free_text",
                    "text_answer": {
                      "content": "sample text"
                    }
                  },
                  {
                    "feature_id": "347dd2d9-7729-4ebd-970a-c4eedb1d41cc",
                    "feature_schema_id": "cluu1wl30002207woa3u870v7",
                    "name": "nested_checklist_question",
                    "value": "nested_checklist_question",
                    "checklist_answers": [
                      {
                        "feature_id": "1a17cdf4-2159-4c21-b1d8-7b9490d8bb14",
                        "feature_schema_id": "cluu1wl30002307wogixwdc5z",
                        "name": "first_checklist_answer",
                        "value": "first_checklist_answer",
                        "classifications": [
                          {
                            "feature_id": "ab20eb98-3002-4108-a541-37741a9a1531",
                            "feature_schema_id": "cluu1wl30002407woea6e102z",
                            "name": "sub_checklist_question",
                            "value": "sub_checklist_question",
                            "checklist_answers": [
                              {
                                "feature_id": "31e09418-2b8e-4759-bd71-4f88302398ce",
                                "feature_schema_id": "cluu1wl30002507wo07av3u9e",
                                "name": "first_sub_checklist_answer",
                                "value": "first_sub_checklist_answer",
                                "classifications": []
                              }
                            ]
                          }
                        ]
                      }
                    ]
                  },
                  {
                    "feature_id": "70f54814-79c0-4d66-84c5-531e8b53e657",
                    "feature_schema_id": "cluu1wl2z001g07wo5elxfwut",
                    "name": "radio_question",
                    "value": "radio_question",
                    "radio_answer": {
                      "feature_id": "214cb3db-da72-4692-9fa4-c86e2933b11e",
                      "feature_schema_id": "cluu1wl2z001h07wo760uc0sb",
                      "name": "first_radio_answer",
                      "value": "first_radio_answer",
                      "classifications": []
                    }
                  },
                  {
                    "feature_id": "baa18a09-6626-48cb-9b4c-7d75aa7d8c57",
                    "feature_schema_id": "cluu1wl2z001s07woc5xmcz68",
                    "name": "checklist_question",
                    "value": "checklist_question",
                    "checklist_answers": [
                      {
                        "feature_id": "c1cf3d18-238f-4817-9fed-93c14954c153",
                        "feature_schema_id": "cluu1wl30001v07wocp3hg8a0",
                        "name": "second_checklist_answer",
                        "value": "second_checklist_answer",
                        "classifications": []
                      },
                      {
                        "feature_id": "cf4606ab-1a5b-4c50-87a3-1f2b5dfc9547",
                        "feature_schema_id": "cluu1wl30001x07wobpdwgf4w",
                        "name": "third_checklist_answer",
                        "value": "third_checklist_answer",
                        "classifications": []
                      },
                      {
                        "feature_id": "f4f28750-5a7f-4da0-a081-b4da8eacb667",
                        "feature_schema_id": "cluu1wl2z001t07wobsowcibp",
                        "name": "first_checklist_answer",
                        "value": "first_checklist_answer",
                        "classifications": []
                      }
                    ]
                  }
                ],
                "relationships": []
              }
            }
          ]
        }
      }
    }
  },
  "models": {}
}
```

## Conversational

### Sample Project Export

```json JSON expandable theme={"theme":"css-variables"}
{
  "data_row": {
    "id": "clfbnskoxb9q9076fbyjngwhm",
    "global_key": "conversation-1.json",
    "row_data": "https://storage.googleapis.com/labelbox-developer-testing-assets/conversational_text/1000-conversations/conversation-1.json",
    "details": {
      "dataset_id": "clfbnsjaj05hr07z605db0w3b",
      "dataset_name": "conversational_annotation_import_demo_dataset",
      "created_at": "2023-03-16T22:04:33.000+00:00",
      "updated_at": "2024-01-18T19:10:53.352+00:00",
      "last_activity_at": "2024-04-10T16:33:23.000+00:00",
      "created_by": "labelbox.public.user@labelbox.com"
    }
  },
  "media_attributes": {
    "mime_type": "application/vnd.labelbox.conversational",
    "labelable_ids": [
      "0",
      "2",
      "4",
      "6",
      "8",
      "10",
      "12",
      "14",
      "16",
      "18"
    ],
    "message_count": 20
  },
  "attachments": [],
  "metadata_fields": [],
  "projects": {
    "cluu14fwi01g0070321g9c1xi": {
      "name": "Conversational Text Annotation Import Demo",
      "labels": [
        {
          "label_kind": "Default",
          "version": "1.0.0",
          "id": "cluu14r490oud148pdk29bwes",
          "label_details": {
            "created_at": "2024-04-10T16:33:23.000+00:00",
            "updated_at": "2024-04-10T16:33:23.000+00:00",
            "created_by": "aovalle@labelbox.com",
            "content_last_updated_at": "2024-04-10T16:33:23.063+00:00",
            "reviews": []
          },
          "performance_details": {
            "seconds_to_create": 0,
            "seconds_to_review": 0,
            "skipped": false
          },
          "annotations": {
            "objects": [
              {
                "feature_id": "cluu14r3p0oty148p5d2isiki",
                "feature_schema_id": "cluu14fnh01f40703aigj669b",
                "name": "ner",
                "value": "ner",
                "annotation_kind": "ConversationalTextEntity",
                "classifications": [],
                "conversational_location": {
                  "message_id": "4",
                  "location": {
                    "start": 0,
                    "end": 8
                  }
                }
              }
            ],
            "classifications": [
              {
                "feature_id": "cluu14r3p0otz148p09nol3tq",
                "feature_schema_id": "cluu14fnh01f60703b4uadjbd",
                "name": "text_convo",
                "value": "text_convo",
                "message_id": "0",
                "conversational_text_answer": {
                  "content": "the answer to the text questions right here"
                }
              },
              {
                "feature_id": "cluu14r3p0ou0148ptekhuz0a",
                "feature_schema_id": "cluu14fnh01f8070385tv57vy",
                "name": "checklist_convo",
                "value": "checklist_convo",
                "message_id": "2",
                "conversational_checklist_answers": [
                  {
                    "feature_id": "cluu14r3p0ou1148p8pb3s49a",
                    "feature_schema_id": "cluu14fnh01f90703cx3j6gqu",
                    "name": "first_checklist_answer",
                    "value": "first_checklist_answer",
                    "classifications": []
                  },
                  {
                    "feature_id": "cluu14r3p0ou2148ph8tyhtp1",
                    "feature_schema_id": "cluu14fnh01fb07030q2m5hqc",
                    "name": "second_checklist_answer",
                    "value": "second_checklist_answer",
                    "classifications": []
                  }
                ]
              },
              {
                "feature_id": "cluu14r3p0ou3148p6d15q82b",
                "feature_schema_id": "cluu14fnh01fe0703abp5fjgx",
                "name": "radio_convo",
                "value": "radio_convo",
                "message_id": "0",
                "conversational_radio_answer": {
                  "feature_id": "cluu14r3p0ou4148pc028jtzk",
                  "feature_schema_id": "cluu14fnh01ff070383rh4ryk",
                  "name": "first_radio_answer",
                  "value": "first_radio_answer",
                  "classifications": []
                }
              },
              {
                "feature_id": "cluu14r3p0ou5148piuv74m17",
                "feature_schema_id": "cluu14fni01fs0703ax77277l",
                "name": "nested_radio_question",
                "value": "nested_radio_question",
                "radio_answer": {
                  "feature_id": "cluu14r3p0ou6148p0b4azzcu",
                  "feature_schema_id": "cluu14fni01ft07033xic99k3",
                  "name": "first_radio_answer",
                  "value": "first_radio_answer",
                  "classifications": [
                    {
                      "feature_id": "cluu14r3p0ou7148pgmcqsf11",
                      "feature_schema_id": "cluu14fni01fu0703c3iu2c0a",
                      "name": "sub_radio_question",
                      "value": "sub_radio_question",
                      "radio_answer": {
                        "feature_id": "cluu14r3p0ou8148pka3hvx26",
                        "feature_schema_id": "cluu14fni01fv07031r8i190o",
                        "name": "first_sub_radio_answer",
                        "value": "first_sub_radio_answer",
                        "classifications": []
                      }
                    }
                  ]
                }
              },
              {
                "feature_id": "cluu14r3p0ou9148puacr13og",
                "feature_schema_id": "cluu14fni01fk07033aeye6ht",
                "name": "nested_checklist_question",
                "value": "nested_checklist_question",
                "message_id": "10",
                "conversational_checklist_answers": [
                  {
                    "feature_id": "cluu14r3p0oua148pttkqa3fe",
                    "feature_schema_id": "cluu14fni01fl0703eg9y2qgt",
                    "name": "first_checklist_answer",
                    "value": "first_checklist_answer",
                    "classifications": [
                      {
                        "feature_id": "cluu14r3p0oub148p3hld5bh9",
                        "feature_schema_id": "cluu14fni01fm0703b3mwa6ot",
                        "name": "sub_checklist_question",
                        "value": "sub_checklist_question",
                        "checklist_answers": [
                          {
                            "feature_id": "cluu14r3p0ouc148pfpiwlzht",
                            "feature_schema_id": "cluu14fni01fn0703apfk8j9q",
                            "name": "first_sub_checklist_answer",
                            "value": "first_sub_checklist_answer",
                            "classifications": []
                          }
                        ]
                      }
                    ]
                  }
                ]
              }
            ],
            "relationships": []
          }
        }
      ],
      "project_details": {
        "ontology_id": "cluu14fmp01f307039z1h2c64",
        "task_name": "Done",
        "batch_id": "04e9d4c0-f758-11ee-8be6-db43b18899b6",
        "batch_name": "first-batch-convo-demo",
        "workflow_status": "DONE",
        "priority": 5,
        "consensus_expected_label_count": 1,
        "workflow_history": []
      }
    }
  }
}
```

### Sample Model Run Export

```json JSON expandable theme={"theme":"css-variables"}
{
  "data_row": {
    "id": "clfbnskoxb9q9076fbyjngwhm",
    "global_key": "conversation-1.json",
    "row_data": "https://storage.googleapis.com/labelbox-developer-testing-assets/conversational_text/1000-conversations/conversation-1.json",
    "details": {
      "dataset_id": "clfbnsjaj05hr07z605db0w3b",
      "dataset_name": "conversational_annotation_import_demo_dataset",
      "created_at": "2023-03-16T22:04:33.000+00:00",
      "updated_at": "2024-01-18T19:10:53.352+00:00",
      "created_by": "labelbox.public.user@labelbox.com"
    }
  },
  "media_attributes": {
    "mime_type": "application/vnd.labelbox.conversational",
    "labelable_ids": [
      "0",
      "2",
      "4",
      "6",
      "8",
      "10",
      "12",
      "14",
      "16",
      "18"
    ],
    "message_count": 20
  },
  "attachments": [],
  "metadata_fields": [],
  "experiments": {
    "a3143b06-742d-0576-d722-c187a5aad1b3": {
      "name": "Conversational_model_run_9985d371-1b0b-4a5b-8c94-4a6167b20090",
      "runs": {
        "a3143b06-adc7-086f-79a5-d251a0054cb8": {
          "name": "iteration 1",
          "run_data_row_id": "4f22e94f-71da-45ac-bab7-f95daf04e7a1",
          "labels": [
            {
              "label_kind": "Default",
              "version": "1.0.0",
              "id": "clq10ymdubt0u1447v6s77glu",
              "annotations": {
                "objects": [
                  {
                    "feature_id": "e063fe4e-4675-0988-9f81-659c83dd1b43",
                    "feature_schema_id": "clq10ybnl067t07ztbavefdsy",
                    "name": "ner",
                    "value": "ner",
                    "annotation_kind": "ConversationalTextEntity",
                    "classifications": [],
                    "conversational_location": {
                      "message_id": "4",
                      "location": {
                        "start": 0,
                        "end": 8
                      }
                    }
                  }
                ],
                "classifications": [
                  {
                    "feature_id": "169f1fee-5db4-02c9-a232-dfb245a158ec",
                    "feature_schema_id": "clq10ybnl068907ztf1htgtoc",
                    "name": "nested_checklist_question",
                    "value": "nested_checklist_question",
                    "message_id": "10",
                    "conversational_checklist_answers": [
                      {
                        "feature_id": "c1c5898b-f1ce-02c7-b806-930ac8140923",
                        "feature_schema_id": "clq10ybnl068a07ztcnyzbhsn",
                        "name": "first_checklist_answer",
                        "value": "first_checklist_answer",
                        "classifications": [
                          {
                            "feature_id": "1fc2b7e8-a292-02ad-81b6-10cb9bdd56c4",
                            "feature_schema_id": "clq10ybnl068b07zth87h8c34",
                            "name": "sub_checklist_question",
                            "value": "sub_checklist_question",
                            "checklist_answers": [
                              {
                                "feature_id": "de42a672-041f-03f3-bbd9-e771e332030a",
                                "feature_schema_id": "clq10ybnm068c07zt53sdcs0m",
                                "name": "first_sub_checklist_answer",
                                "value": "first_sub_checklist_answer",
                                "classifications": []
                              }
                            ]
                          }
                        ]
                      }
                    ]
                  },
                  {
                    "feature_id": "8adc0320-6d61-0011-bda0-80842b3733da",
                    "feature_schema_id": "clq10ybnm068h07ztaugh4uma",
                    "name": "nested_radio_question",
                    "value": "nested_radio_question",
                    "radio_answer": {
                      "feature_id": "475fb394-f606-0223-b625-48045ebf7208",
                      "feature_schema_id": "clq10ybnm068i07zt1xxv25it",
                      "name": "first_radio_answer",
                      "value": "first_radio_answer",
                      "classifications": [
                        {
                          "feature_id": "a61d83fd-4c47-0e88-9ae1-da3eb100eea0",
                          "feature_schema_id": "clq10ybnm068j07ztcqf794w4",
                          "name": "sub_radio_question",
                          "value": "sub_radio_question",
                          "radio_answer": {
                            "feature_id": "6dbde4c3-3a1e-0348-a5ac-7787a8ac0461",
                            "feature_schema_id": "clq10ybnm068k07ztdx9dca3i",
                            "name": "first_sub_radio_answer",
                            "value": "first_sub_radio_answer",
                            "classifications": []
                          }
                        }
                      ]
                    }
                  },
                  {
                    "feature_id": "96dce41f-f1c1-0822-9848-39c098b449f5",
                    "feature_schema_id": "clq10ybnl067v07ztb8f91r4u",
                    "name": "text_convo",
                    "value": "text_convo",
                    "message_id": "0",
                    "conversational_text_answer": {
                      "content": "the answer to the text questions are right here"
                    }
                  },
                  {
                    "feature_id": "ba7c519c-c618-01ba-ac4b-fdbbd8137bf0",
                    "feature_schema_id": "clq10ybnl068307ztd255hndf",
                    "name": "radio_convo",
                    "value": "radio_convo",
                    "message_id": "0",
                    "conversational_radio_answer": {
                      "feature_id": "ec0fc3fb-1948-0ff9-8bc6-0377721e4f0e",
                      "feature_schema_id": "clq10ybnl068407zt1x572rym",
                      "name": "first_radio_answer",
                      "value": "first_radio_answer",
                      "classifications": []
                    }
                  },
                  {
                    "feature_id": "fe3a901b-8e62-01fe-a792-10b15ec7fa0d",
                    "feature_schema_id": "clq10ybnl067x07zt0u67b9h3",
                    "name": "checklist_convo",
                    "value": "checklist_convo",
                    "message_id": "2",
                    "conversational_checklist_answers": [
                      {
                        "feature_id": "699a5673-bb4a-06d3-9b9e-19237e11f96d",
                        "feature_schema_id": "clq10ybnl068007zt6ct25775",
                        "name": "second_checklist_answer",
                        "value": "second_checklist_answer",
                        "classifications": []
                      },
                      {
                        "feature_id": "da192bd6-84ec-0acf-b875-d1ade525f466",
                        "feature_schema_id": "clq10ybnl067y07zt6gvjczsg",
                        "name": "first_checklist_answer",
                        "value": "first_checklist_answer",
                        "classifications": []
                      }
                    ]
                  }
                ],
                "relationships": []
              }
            }
          ],
          "predictions": [
            {
              "label_kind": "Default",
              "version": "1.0.0",
              "id": "clq10ymdubt0u1447v6s77glu",
              "annotations": {
                "objects": [
                  {
                    "feature_id": "9c8f89a7-7211-4b33-a3c5-a3dab59f0407",
                    "feature_schema_id": "clq10ybnl067t07ztbavefdsy",
                    "name": "ner",
                    "value": "ner",
                    "annotation_kind": "ConversationalTextEntity",
                    "classifications": [],
                    "conversational_location": {
                      "message_id": "4",
                      "location": {
                        "start": 0,
                        "end": 8
                      }
                    }
                  }
                ],
                "classifications": [
                  {
                    "feature_id": "20cda99c-d0f7-4e59-8f43-a8b4f3929e2f",
                    "feature_schema_id": "clq10ybnl068307ztd255hndf",
                    "name": "radio_convo",
                    "value": "radio_convo",
                    "message_id": "0",
                    "conversational_radio_answer": {
                      "feature_id": "ec1ec509-79b6-48e7-973c-9c06e754630f",
                      "feature_schema_id": "clq10ybnl068407zt1x572rym",
                      "name": "first_radio_answer",
                      "value": "first_radio_answer",
                      "classifications": []
                    }
                  },
                  {
                    "feature_id": "49e4667e-ed14-400b-9b75-b8b89bfb6f11",
                    "feature_schema_id": "clq10ybnm068h07ztaugh4uma",
                    "name": "nested_radio_question",
                    "value": "nested_radio_question",
                    "radio_answer": {
                      "feature_id": "723be4ec-fa0c-4a67-9df0-a18f56f7bbd8",
                      "feature_schema_id": "clq10ybnm068i07zt1xxv25it",
                      "name": "first_radio_answer",
                      "value": "first_radio_answer",
                      "classifications": [
                        {
                          "feature_id": "f28f3d0a-9428-4b45-8de1-4ec6714da726",
                          "feature_schema_id": "clq10ybnm068j07ztcqf794w4",
                          "name": "sub_radio_question",
                          "value": "sub_radio_question",
                          "radio_answer": {
                            "feature_id": "82808a1e-a004-4965-bb77-4ab9e499c747",
                            "feature_schema_id": "clq10ybnm068k07ztdx9dca3i",
                            "name": "first_sub_radio_answer",
                            "value": "first_sub_radio_answer",
                            "classifications": []
                          }
                        }
                      ]
                    }
                  },
                  {
                    "feature_id": "988e5b9a-9a6d-4338-96bf-901475b4a2c0",
                    "feature_schema_id": "clq10ybnl068907ztf1htgtoc",
                    "name": "nested_checklist_question",
                    "value": "nested_checklist_question",
                    "message_id": "10",
                    "conversational_checklist_answers": [
                      {
                        "feature_id": "9ea5984d-fb21-48e4-96c8-ce19162b5850",
                        "feature_schema_id": "clq10ybnl068a07ztcnyzbhsn",
                        "name": "first_checklist_answer",
                        "value": "first_checklist_answer",
                        "classifications": [
                          {
                            "feature_id": "bf896185-5a77-410e-a4cd-abf0b07122fa",
                            "feature_schema_id": "clq10ybnl068b07zth87h8c34",
                            "name": "sub_checklist_question",
                            "value": "sub_checklist_question",
                            "checklist_answers": [
                              {
                                "feature_id": "0812ee3d-7f5d-480b-82a2-4e965a48fc49",
                                "feature_schema_id": "clq10ybnm068c07zt53sdcs0m",
                                "name": "first_sub_checklist_answer",
                                "value": "first_sub_checklist_answer",
                                "classifications": []
                              }
                            ]
                          }
                        ]
                      }
                    ]
                  },
                  {
                    "feature_id": "cb4730df-2e22-45db-bc74-fd439d577571",
                    "feature_schema_id": "clq10ybnl067x07zt0u67b9h3",
                    "name": "checklist_convo",
                    "value": "checklist_convo",
                    "message_id": "2",
                    "conversational_checklist_answers": [
                      {
                        "feature_id": "8184feca-970c-4ba7-bfbd-4d0eb5849a90",
                        "feature_schema_id": "clq10ybnl068007zt6ct25775",
                        "name": "second_checklist_answer",
                        "value": "second_checklist_answer",
                        "classifications": []
                      },
                      {
                        "feature_id": "dc40cfef-aa6f-436d-a5b5-0d56bbd1741e",
                        "feature_schema_id": "clq10ybnl067y07zt6gvjczsg",
                        "name": "first_checklist_answer",
                        "value": "first_checklist_answer",
                        "classifications": []
                      }
                    ]
                  },
                  {
                    "feature_id": "db7109ed-e380-4389-ab2f-98e72b4c2afe",
                    "feature_schema_id": "clq10ybnl067v07ztb8f91r4u",
                    "name": "text_convo",
                    "value": "text_convo",
                    "message_id": "0",
                    "conversational_text_answer": {
                      "content": "the answer to the text questions are right here"
                    }
                  }
                ],
                "relationships": []
              }
            }
          ]
        }
      }
    }
  },
  "models": {}
}
```

## LLM Data Generation

### Sample Project Export

```json JSON expandable theme={"theme":"css-variables"}
{
  "data_row": {
    "id": "clpvnouh04uyy0723mmru42qn",
    "global_key": "clpvnou2v03js07xsghfo2nzc",
    "row_data": "{\"type\":\"application/llm.prompt-response-creation\",\"version\":1}",
    "details": {
      "dataset_id": "clpvnou0z004c0724pd4cmw8g",
      "dataset_name": "test-humans-generate-prompts-reponses-dataset",
      "created_at": "2023-12-07T20:34:06.540+00:00",
      "updated_at": "2023-12-07T20:34:06.815+00:00",
      "last_activity_at": "2024-04-10T15:21:31.000+00:00",
      "created_by": "aovalle@labelbox.com"
    }
  },
  "media_attributes": {
    "mime_type": "application/llm.prompt-response-creation"
  },
  "attachments": [],
  "metadata_fields": [],
  "projects": {
    "clpvnotzb03jo07xs48r7ewka": {
      "name": "Andrea-test-humans-generate-prompts-responses",
      "labels": [
        {
          "label_kind": "Default",
          "version": "1.0.0",
          "id": "clutyida4041a07h4a8ojbu1g",
          "label_details": {
            "created_at": "2024-04-10T15:21:31.000+00:00",
            "updated_at": "2024-04-10T15:21:31.000+00:00",
            "created_by": "aovalle@labelbox.com",
            "content_last_updated_at": "2024-04-10T15:21:31.137+00:00",
            "reviews": []
          },
          "performance_details": {
            "seconds_to_create": 28,
            "seconds_to_review": 0,
            "skipped": false
          },
          "annotations": {
            "objects": [],
            "classifications": [
              {
                "feature_id": "clutykado00013b6rw65zj51e",
                "feature_schema_id": "clutyjef000xk07wfeurhc2qb",
                "name": "Is this a shirt?",
                "value": "Is this a shirt?",
                "text_answer": {
                  "content": "Potentially this is a shirt, but keep in mind this is not a good quality prompt"
                }
              },
              {
                "feature_id": "clutykap700033b6rmmlkpo6u",
                "feature_schema_id": "clutyjef000xm07wf5ym80ud6",
                "name": "Yes",
                "value": "yes",
                "radio_answer": {
                  "feature_id": "clutykap700023b6rhktauyzc",
                  "feature_schema_id": "clutyjef000xn07wf6smvc54b",
                  "name": "Red shirt",
                  "value": "red_shirt",
                  "classifications": []
                }
              }
            ],
            "relationships": []
          }
        }
      ],
      "project_details": {
        "ontology_id": "clpvnqsqz01kv07zydvbscxzq",
        "task_id": "14b02ec0-71f3-4d1f-b720-c6318e9a9346",
        "task_name": "Initial review task",
        "batch_id": "00870bc0-9540-11ee-a202-8d3b90bd1707",
        "batch_name": "batch_clpvnotzb03jo07xs48r7ewka",
        "workflow_status": "IN_REVIEW",
        "priority": 5,
        "consensus_expected_label_count": 2,
        "workflow_history": [
          {
            "action": "Move",
            "created_at": "2024-04-10T15:21:31.517+00:00",
            "created_by": "aovalle@labelbox.com",
            "previous_task_name": "Initial labeling task",
            "previous_task_id": "1d0062f9-dfd9-0f86-baed-823235868a8c",
            "next_task_name": "Initial review task",
            "next_task_id": "14b02ec0-71f3-4d1f-b720-c6318e9a9346"
          },
          {
            "action": "Move",
            "created_at": "2024-04-10T15:21:31.506+00:00",
            "created_by": "aovalle@labelbox.com",
            "next_task_name": "Initial labeling task",
            "next_task_id": "1d0062f9-dfd9-0f86-baed-823235868a8c"
          }
        ]
      }
    }
  }
}
```

## LLM Human Preference

### Sample Project Export

```json JSON expandable theme={"theme":"css-variables"}
{
    "data_row": {
        "id": "clpiqgsvn4lxr079096g5mt9v",
        "global_key": "0914c7da-5999-464b-b621-054a5247cf48",
        "row_data": "https://storage.googleapis.com/labelbox-developer-testing-assets/Pranoy/shopping_3.json",
        "details": {
            "dataset_id": "clpiqgrsr00f60737sqeiaf4v",
            "dataset_name": "pairwise_demo_42454af2-b7b5-45c0-a25a-90f97efd49be",
            "created_at": "2023-11-28T19:30:49.729+00:00",
            "updated_at": "2023-11-28T19:30:50.609+00:00",
            "last_activity_at": "2023-11-28T23:11:08.000+00:00",
            "created_by": "user@labelbox.com"
        }
    },
    "media_attributes": {
        "mime_type": "application/vnd.labelbox.conversational",
        "labelable_ids": [
            "message-1",
            "message-3"
        ],
        "message_count": 4
    },
    "attachments": [],
    "metadata_fields": [],
    "projects": {
        "clpiy2itz08vg0705akh60zc1": {
            "name": "Test LLM Human Preference",
            "labels": [
                {
                    "label_kind": "Default",
                    "version": "1.0.0",
                    "id": "clpiyah63071f07e4756yer2d",
                    "label_details": {
                        "created_at": "2023-11-28T23:11:08.000+00:00",
                        "updated_at": "2023-11-28T23:11:07.000+00:00",
                        "created_by": "user@labelbox.com",
                        "content_last_updated_at": "2023-11-28T23:11:07.568+00:00",
                        "reviews": []
                    },
                    "annotations": {
                        "objects": [],
                        "classifications": [
                            {
                                "feature_id": "clpiybl7h000g3b6mil1amjme",
                                "feature_schema_id": "clpiqh0v104mq0729homqffup",
                                "name": "Choose the best response",
                                "radio_answer": {
                                    "feature_id": "clpiybl7h000f3b6m03fn8ry3",
                                    "feature_schema_id": "clpiqh0v104mt0729atflgvvw",
                                    "name": "Response B",
                                    "classifications": []
                                }
                            },
                            {
                                "feature_id": "clpiybzp3000h3b6m9uowaci4",
                                "feature_schema_id": "clpiqh0v104my072907gt7id0",
                                "name": "Provide a reason for your choice",
                                "text_answer": {
                                    "content": "Direct approach that is convenient"
                                }
                            },
                            {
                                "feature_id": "clpiyc10q000j3b6m01i2d1pv",
                                "feature_schema_id": "clpiya8h507dx072efo9kb97j",
                                "name": "Evaluations done",
                                "checklist_answers": [
                                    {
                                        "feature_id": "clpiyc10q000i3b6mhhvhdzqs",
                                        "name": "Relevance",
                                        "classifications": []
                                    },
                                    {
                                        "feature_id": "clpiyc1ie000l3b6m3kpff0lp",
                                        "name": "Usefulness",
                                        "classifications": []
                                    }
                                ]
                            }
                        ],
                        "relationships": []
                    }
                }
            ],
            "project_details": {
                "ontology_id": "clpiqh0ud04mp0729adeoccsl",
                "task_id": "33e77b4d-424f-49ba-8792-628c1cb746b7",
                "task_name": "Initial review task",
                "batch_id": "66882c10-8e42-11ee-8f8d-71857f06db2f",
                "batch_name": "test_llm_human_preference_batch_1",
                "workflow_status": "IN_REVIEW",
                "priority": 5,
                "consensus_expected_label_count": 1,
                "workflow_history": [
                    {
                        "action": "Move",
                        "created_at": "2023-11-28T23:11:07.654+00:00",
                        "created_by": "user@labelbox.com",
                        "previous_task_name": "Initial labeling task",
                        "previous_task_id": "8f9917a7-6030-0914-befc-ac06f59a7ec9",
                        "next_task_name": "Initial review task",
                        "next_task_id": "33e77b4d-424f-49ba-8792-628c1cb746b7"
                    },
                    {
                        "action": "Move",
                        "created_at": "2023-11-28T23:11:07.631+00:00",
                        "created_by": "user@labelbox.com",
                        "next_task_name": "Initial labeling task",
                        "next_task_id": "8f9917a7-6030-0914-befc-ac06f59a7ec9"
                    }
                ]
            }
        }
    }
}
```

## Model Chat Evaluation

### Sample Project Export

```json JSON expandable theme={"theme":"css-variables"}
  {
    "data_row": {
      "id": "clwrm93488cpd0734f740r9lr",
      "global_key": "clwrm92sb0grg07wncr6dhihw",
      "row_data": "{\"type\":\"application/vnd.labelbox.conversational.model-chat-evaluation\",\"version\":2,\"actors\":{\"clwrt16zz00023b6pg6hlhtub\":{\"role\":\"human\",\"metadata\":{\"name\":\"User\"}},\"clwrt17a900043b6py2tepemd\":{\"role\":\"model\",\"metadata\":{\"modelConfigId\":\"e606593e-def3-4662-9737-3fc4ac412dd3\"}},\"clwrt17a900053b6p0xtyv1im\":{\"role\":\"model\",\"metadata\":{\"modelConfigId\":\"8fcda464-a0c7-4300-a9fa-c9a5cf82cab2\"}}},\"messages\":{\"clwrt16zz00033b6p6ud7w233\":{\"actorId\":\"clwrt16zz00023b6pg6hlhtub\",\"content\":[{\"type\":\"text\",\"content\":\"Is there a car in this image?\"},{\"type\":\"fileData\",\"fileUri\":\"https://c7.staticflickr.com/3/2447/3799589580_4b73283f92_o.jpg\",\"mimeType\":\"image/jpeg\"}],\"childMessageIds\":[\"clwrt1fw700063b6pe8zpo1xp\",\"clwrt1fw700073b6p8k3vkh1x\"]},\"clwrt1fw700063b6pe8zpo1xp\":{\"actorId\":\"clwrt17a900043b6py2tepemd\",\"content\":[{\"type\":\"text\",\"content\":\"Yes, there is a car in this image. It appears to be a white hatchback parked in a parking lot.\"}],\"childMessageIds\":[]},\"clwrt1fw700073b6p8k3vkh1x\":{\"actorId\":\"clwrt17a900053b6p0xtyv1im\",\"content\":[{\"type\":\"text\",\"content\":\" Yes, there is a white Volkswagen Golf in the image.\"}],\"childMessageIds\":[]}},\"rootMessageIds\":[\"clwrt16zz00033b6p6ud7w233\"]}",
      "details": {
        "dataset_id": "clwkf82ul001l0755mhckcmuy",
        "dataset_name": "Model evaluation 101",
        "created_at": "2024-05-29T09:20:43.342+00:00",
        "updated_at": "2024-05-29T12:31:52.027+00:00",
        "last_activity_at": "2024-05-29T12:33:21.203+00:00",
        "created_by": "user@labelbox.com"
      }
    },
    "media_attributes": {
      "asset_type": "conversational",
      "mime_type": "application/vnd.labelbox.conversational.model-chat-evaluation",
      "labelable_ids": [
        "clwrt16zz00033b6p6ud7w233",
        "clwrt1fw700063b6pe8zpo1xp",
        "clwrt1fw700073b6p8k3vkh1x"
      ],
      "message_count": 3
    },
    "attachments": [],
    "metadata_fields": [],
    "projects": {
      "clwrm92jc0grc07wncq654q5s": {
        "name": "Multi modal chat evalution",
        "labels": [
          {
            "label_kind": "Default",
            "version": "1.0.0",
            "id": "clwrsyrdp02mr07x5378j3ysr",
            "label_details": {
              "created_at": "2024-05-29T12:33:21.000+00:00",
              "updated_at": "2024-05-29T12:33:21.000+00:00",
              "created_by": "user@labelbox.com",
              "content_last_updated_at": "2024-05-29T12:33:20.787+00:00",
              "reviews": []
            },
            "performance_details": {
              "seconds_to_create": 250,
              "seconds_to_review": 0,
              "skipped": false
            },
            "annotations": {
              "objects": [
                {
                  "feature_id": "clwrt25ar00093b6po9g7fiu3",
                  "feature_schema_id": "clwkfbcnn01kl07z68er64ffi",
                  "name": "Rank given prompt",
                  "annotation_kind": "MessageRanking",
                  "classifications": [],
                  "ranked_messages": {
                    "parent_message_id": "clwrt16zz00033b6p6ud7w233",
                    "ranked_messages": [
                      {
                        "message_id": "clwrt1fw700063b6pe8zpo1xp",
                        "order": 1,
                        "model_config_id": "e606593e-def3-4662-9737-3fc4ac412dd3"
                      },
                      {
                        "message_id": "clwrt1fw700073b6p8k3vkh1x",
                        "order": 2,
                        "model_config_id": "8fcda464-a0c7-4300-a9fa-c9a5cf82cab2"
                      }
                    ]
                  }
                },
                {
                  "feature_id": "clwrt2dru000b3b6pkj3i5zxk",
                  "feature_schema_id": "clwknckmp030v07xp6kow09xd",
                  "name": "Subject relevance",
                  "annotation_kind": "MessageMultiSelection",
                  "classifications": [],
                  "selected_messages": {
                    "parent_message_id": "clwrt16zz00033b6p6ud7w233",
                    "messages": [
                      {
                        "message_id": "clwrt1fw700063b6pe8zpo1xp",
                        "model_config_id": "e606593e-def3-4662-9737-3fc4ac412dd3"
                      }
                    ]
                  }
                }
              ],
              "classifications": [],
              "relationships": []
            }
          }
        ],
        "project_details": {
          "ontology_id": "clwkn6leb034b07waeolfb7tl",
          "task_id": "0c392f64-bc65-44e6-8848-ae5a12000328",
          "task_name": "Initial review task",
          "batch_id": "c2267a60-1d9c-11ef-9edf-ed6f8eede535",
          "batch_name": "batch_clwrm92jc0grc07wncq654q5s_clwrm9eph006d0750b3s5b2cr",
          "workflow_status": "IN_REVIEW",
          "priority": 5,
          "consensus_expected_label_count": 2,
          "workflow_history": [
            {
              "action": "Move",
              "created_at": "2024-05-29T12:33:21.160+00:00",
              "created_by": "user@labelbox.com",
              "previous_task_name": "Initial labeling task",
              "previous_task_id": "ebf41e2b-e231-0e43-adee-92252c1858dd",
              "next_task_name": "Initial review task",
              "next_task_id": "0c392f64-bc65-44e6-8848-ae5a12000328"
            },
            {
              "action": "Move",
              "created_at": "2024-05-29T12:33:21.151+00:00",
              "created_by": "user@labelbox.com",
              "next_task_name": "Initial labeling task",
              "next_task_id": "ebf41e2b-e231-0e43-adee-92252c1858dd"
            }
          ]
        },
        "project_model_configs": {
          "models": {
            "5d9647e7-aafe-4275-944d-c2e9b122105a": {
              "name": "Google Gemini Pro Vision",
              "created_at": "2023-12-13T21:51:42.657+00:00"
            },
            "036aeb03-b0b4-442e-a6e1-161382bc1228": {
              "name": "OpenAI GPT-4o",
              "created_at": "2024-05-23T23:56:03.078+00:00"
            }
          },
          "model_configs": {
            "e606593e-def3-4662-9737-3fc4ac412dd3": {
              "name": "GPT_4o_multi_modal",
              "model_id": "036aeb03-b0b4-442e-a6e1-161382bc1228",
              "inference_params": {
                "top_p": 1,
                "detail": "high",
                "temperature": 0,
                "use_text_only": false,
                "max_new_tokens": 1024,
                "use_image_only": false,
                "use_attachments": false
              }
            },
            "8fcda464-a0c7-4300-a9fa-c9a5cf82cab2": {
              "name": "Gemini_pro_vison_multi_modal",
              "model_id": "5d9647e7-aafe-4275-944d-c2e9b122105a",
              "inference_params": {
                "use_attachments": false
              }
            }
          },
          "project_model_configs": [
            {
              "id": "clwrsxr1y009i07v93w2h45qm",
              "model_id": "036aeb03-b0b4-442e-a6e1-161382bc1228",
              "model_config_id": "e606593e-def3-4662-9737-3fc4ac412dd3"
            },
            {
              "id": "clwrsyext00ar07v5czpo7w2a",
              "model_id": "5d9647e7-aafe-4275-944d-c2e9b122105a",
              "model_config_id": "8fcda464-a0c7-4300-a9fa-c9a5cf82cab2"
            }
          ]
        },
        "project_tags": []
      }
    }
```


# Export text annotations
Source: https://docs.labelbox.com/horizon/developer-guides/export-text-annotations

How to export text annotations and sample export formats.

<CardGroup>
  <Card title="Open In Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/exports/export_data.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/exports/export_data.ipynb" />
</CardGroup>

## Export JSON annotations

<CodeGroup>
  ```python Export theme={"theme":"css-variables"}
  # Set the export params to include/exclude certain fields. 
  export_params= {
    "attachments": True,
    "metadata_fields": True,
    "data_row_details": True,
    "project_details": True,
    "label_details": True,
    "performance_details": True,
    "interpolated_frames": True
  }

  # Note: Filters follow AND logic, so typically using one filter is sufficient.
  filters= {
    "last_activity_at": ["2000-01-01 00:00:00", "2050-01-01 00:00:00"],
    "workflow_status": "<wkf-status>"
  }

  export_task = project.export(params=export_params, filters=filters)
  export_task.wait_till_done()

  # Stream the export using a callback function
  def json_stream_handler(output: labelbox.BufferedJsonConverterOutput):
    print(output.json)

  export_task.get_buffered_stream(stream_type=labelbox.StreamType.RESULT).start(stream_handler=json_stream_handler)

  # Collect all exported data into a list
  export_json = [data_row.json for data_row in export_task.get_buffered_stream()]

  print("file size: ", export_task.get_total_file_size(stream_type=lb.StreamType.RESULT))
  print("line count: ", export_task.get_total_lines(stream_type=lb.StreamType.RESULT))
       
  ```
</CodeGroup>

## Annotation export formats

### Text entity

To include the `token` field in your export, set `data_row_details` to `True` when exporting the data.

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "clutx70x20onv148pb5ah49lb",
    "feature_schema_id": "clutx6n1g00r707y110ruh99w",
    "name": "named_entity",
    "value": "named_entity",
    "annotation_kind": "TextEntity",
    "classifications": [],
    "location": {
      "start": 10,
      "end": 20,
      "token": "um dolor si"
    }
  }
  ```
</CodeGroup>

### Radio

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "clutx70x20onw148p7mlgpq90",
    "feature_schema_id": "clutx6n1h00r907y16s49do9e",
    "name": "radio_question",
    "value": "radio_question",
    "radio_answer": {
      "feature_id": "clutx70x20onx148px2679qsp",
      "feature_schema_id": "clutx6n1h00ra07y100njafnj",
      "name": "first_radio_answer",
      "value": "first_radio_answer",
      "classifications": []
    }
  }
  ```
</CodeGroup>

### Checklist

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "clutx70x20ony148p6gxk7ec8",
    "feature_schema_id": "clutx6n1h00rt07y16xe27hhv",
    "name": "checklist_question",
    "value": "checklist_question",
    "checklist_answers": [
      {
        "feature_id": "clutx70x20onz148pjjl5r8m5",
        "feature_schema_id": "clutx6n1h00ru07y1hgvd3ekt",
        "name": "first_checklist_answer",
        "value": "first_checklist_answer",
        "classifications": []
      },
      {
        "feature_id": "clutx70x20oo0148p3w6es43a",
        "feature_schema_id": "clutx6n1h00rw07y16t1v8tvi",
        "name": "second_checklist_answer",
        "value": "second_checklist_answer",
        "classifications": []
      },
      {
        "feature_id": "clutx70x20oo1148ps1lpz6dl",
        "feature_schema_id": "clutx6n1h00ry07y1fkbk0pde",
        "name": "third_checklist_answer",
        "value": "third_checklist_answer",
        "classifications": []
      }
    ]
  }
  ```
</CodeGroup>

### Free-form text

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "clpsmujyn4z13143s2iw9n15v",
    "feature_schema_id": "clpsmu6sz003v07zp2vie3zd8",
    "name": "free_text",
    "text_answer": {
      "content": "sample text"
    }
  }
  ```
</CodeGroup>

## Sample project export

<CodeGroup>
  ```json JSON expandable theme={"theme":"css-variables"}
  {
    "data_row": {
      "id": "clfh5lsju0bd90763h2o22vvq",
      "global_key": "lorem-ipsum.txt",
      "row_data": "https://storage.googleapis.com/labelbox-sample-datasets/nlp/lorem-ipsum.txt",
      "details": {
        "dataset_id": "clfh5lpul063507x4a16r5j6x",
        "dataset_name": "text_annotation_import_demo_dataset",
        "created_at": "2023-03-20T18:22:00.000+00:00",
        "updated_at": "2023-03-20T18:22:00.000+00:00",
        "last_activity_at": "2024-04-10T14:43:10.000+00:00",
        "created_by": "[email protected]"
      }
    },
    "media_attributes": {
      "mime_type": "text/plain"
    },
    "attachments": [
      {
        "type": "TEXT_URL",
        "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt",
        "id": "clfh5lsjv0bdd07638apx7bf5"
      }
    ],
    "metadata_fields": [],
     "embeddings": [
      {
        "id": "c300000000000000000000000",
        "name": "Text Embedding V2 (All-MPNet-base-v2)",
        "dimensions": 768,
        "is_custom": false,
        "values": [
          {
            "value": [
              0.04099537059664726,
              0.03790362924337387,
              -0.013698477298021317,
              0.038478486239910126,
              -0.01930154114961624,
              0.05670716613531113,
              0.02421298250555992 ...
  	],
    "projects": {
      "clutx6nli00ks07x6efh6e1pw": {
        "name": "Text Annotation Import Demo",
        "labels": [
          {
            "label_kind": "Default",
            "version": "1.0.0",
            "id": "clutx70xi0oob148pgr7sz5v0",
            "label_details": {
              "created_at": "2024-04-10T14:43:11.000+00:00",
              "updated_at": "2024-04-10T14:43:11.000+00:00",
              "created_by": "[email protected]",
              "content_last_updated_at": "2024-04-10T14:43:10.580+00:00",
              "reviews": []
            },
            "performance_details": {
              "seconds_to_create": 0,
              "seconds_to_review": 0,
              "skipped": false
            },
            "annotations": {
              "objects": [
                {
                  "feature_id": "clutx70x20onv148pb5ah49lb",
                  "feature_schema_id": "clutx6n1g00r707y110ruh99w",
                  "name": "named_entity",
                  "value": "named_entity",
                  "annotation_kind": "TextEntity",
                  "classifications": [],
                  "location": {
                    "start": 10,
                    "end": 20,
                    "token": "um dolor si"
                  }
                }
              ],
              "classifications": [
                {
                  "feature_id": "clutx70x20onw148p7mlgpq90",
                  "feature_schema_id": "clutx6n1h00r907y16s49do9e",
                  "name": "radio_question",
                  "value": "radio_question",
                  "radio_answer": {
                    "feature_id": "clutx70x20onx148px2679qsp",
                    "feature_schema_id": "clutx6n1h00ra07y100njafnj",
                    "name": "first_radio_answer",
                    "value": "first_radio_answer",
                    "classifications": []
                  }
                },
                {
                  "feature_id": "clutx70x20ony148p6gxk7ec8",
                  "feature_schema_id": "clutx6n1h00rt07y16xe27hhv",
                  "name": "checklist_question",
                  "value": "checklist_question",
                  "checklist_answers": [
                    {
                      "feature_id": "clutx70x20onz148pjjl5r8m5",
                      "feature_schema_id": "clutx6n1h00ru07y1hgvd3ekt",
                      "name": "first_checklist_answer",
                      "value": "first_checklist_answer",
                      "classifications": []
                    },
                    {
                      "feature_id": "clutx70x20oo0148p3w6es43a",
                      "feature_schema_id": "clutx6n1h00rw07y16t1v8tvi",
                      "name": "second_checklist_answer",
                      "value": "second_checklist_answer",
                      "classifications": []
                    },
                    {
                      "feature_id": "clutx70x20oo1148ps1lpz6dl",
                      "feature_schema_id": "clutx6n1h00ry07y1fkbk0pde",
                      "name": "third_checklist_answer",
                      "value": "third_checklist_answer",
                      "classifications": []
                    }
                  ]
                },
                {
                  "feature_id": "clutx70x20oo2148pz466o8ep",
                  "feature_schema_id": "clutx6n1i00s107y18bywdwuh",
                  "name": "free_text",
                  "value": "free_text",
                  "text_answer": {
                    "content": "sample text"
                  }
                },
                {
                  "feature_id": "clutx70x20oo3148poaesoyw7",
                  "feature_schema_id": "clutx6n1h00rl07y12xox0uqk",
                  "name": "nested_checklist_question",
                  "value": "nested_checklist_question",
                  "checklist_answers": [
                    {
                      "feature_id": "clutx70x20oo4148pgxhy7xr3",
                      "feature_schema_id": "clutx6n1h00rm07y15qdj4mms",
                      "name": "first_checklist_answer",
                      "value": "first_checklist_answer",
                      "classifications": [
                        {
                          "feature_id": "clutx70x20oo5148plvcm1yhy",
                          "feature_schema_id": "clutx6n1h00rn07y13xi0e1au",
                          "name": "sub_checklist_question",
                          "value": "sub_checklist_question",
                          "checklist_answers": [
                            {
                              "feature_id": "clutx70x20oo6148pcqjahziw",
                              "feature_schema_id": "clutx6n1h00ro07y164ddfda2",
                              "name": "first_sub_checklist_answer",
                              "value": "first_sub_checklist_answer",
                              "classifications": []
                            }
                          ]
                        }
                      ]
                    }
                  ]
                },
                {
                  "feature_id": "clutx70x20oo7148pkei2ymj3",
                  "feature_schema_id": "clutx6n1h00rd07y188n4hd08",
                  "name": "nested_radio_question",
                  "value": "nested_radio_question",
                  "radio_answer": {
                    "feature_id": "clutx70x20oo8148pjiiobfl5",
                    "feature_schema_id": "clutx6n1h00re07y1b7v1dd8z",
                    "name": "first_radio_answer",
                    "value": "first_radio_answer",
                    "classifications": [
                      {
                        "feature_id": "clutx70x20oo9148px2xcuanp",
                        "feature_schema_id": "clutx6n1h00rf07y18f8f45nu",
                        "name": "sub_radio_question",
                        "value": "sub_radio_question",
                        "radio_answer": {
                          "feature_id": "clutx70x20ooa148pqhs0hmot",
                          "feature_schema_id": "clutx6n1h00rg07y1fkkjgr7o",
                          "name": "first_sub_radio_answer",
                          "value": "first_sub_radio_answer",
                          "classifications": []
                        }
                      }
                    ]
                  }
                }
              ],
              "relationships": []
            }
          }
        ],
        "project_details": {
          "ontology_id": "clutx6n0k00r607y1a0a9bkzj",
          "task_name": "Done",
          "batch_id": "9f1c1db0-f748-11ee-b74f-f180a0c567c8",
          "batch_name": "first-batch-text-demo",
          "workflow_status": "DONE",
          "priority": 5,
          "consensus_expected_label_count": 1,
          "workflow_history": []
        },
        "project_tags": []
      }
    }
  }
  ```
</CodeGroup>

## Sample model run export

<CodeGroup>
  ```json JSON expandable theme={"theme":"css-variables"}
  {
    "data_row": {
      "id": "clfh5lsju0bd90763h2o22vvq",
      "global_key": "lorem-ipsum.txt",
      "row_data": "https://storage.googleapis.com/labelbox-sample-datasets/nlp/lorem-ipsum.txt",
      "details": {
        "dataset_id": "clfh5lpul063507x4a16r5j6x",
        "dataset_name": "text_annotation_import_demo_dataset",
        "created_at": "2023-03-20T18:22:00.000+00:00",
        "updated_at": "2023-03-20T18:22:00.000+00:00",
        "created_by": "[email protected]"
      }
    },
    "media_attributes": {
      "mime_type": "text/plain"
    },
    "attachments": [
      {
        "type": "TEXT_URL",
        "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt",
        "id": "clfh5lsjv0bdd07638apx7bf5"
      }
    ],
    "metadata_fields": [],
    "experiments": {
      "a349da10-5d85-0c82-293d-8165aa611e7c": {
        "name": "text_model_run_8ad83b1f-c9c6-4ea4-83ce-a287393bb917",
        "runs": {
          "a349da10-aa0d-0879-711b-11f0cc035b65": {
            "name": "iteration 1",
            "run_data_row_id": "e1898158-c2a9-42fb-8c64-2c81beadafed",
            "labels": [
              {
                "label_kind": "Default",
                "version": "1.0.0",
                "id": "clr16pmfw0g4o140zkwys3f7u",
                "annotations": {
                  "objects": [
                    {
                      "feature_id": "6d651dde-6567-0669-83e6-7a6677a5acb2",
                      "feature_schema_id": "clr16p71n018r07tuawb18jt6",
                      "name": "named_entity",
                      "value": "named_entity",
                      "annotation_kind": "TextEntity",
                      "classifications": [],
                      "location": {
                        "start": 10,
                        "end": 20
                      }
                    }
                  ],
                  "classifications": [
                    {
                      "feature_id": "08a38f86-5cd1-0feb-ae98-5731b2d2d3bf",
                      "feature_schema_id": "clr16p71p019f07tucv4g1cmm",
                      "name": "nested_checklist_question",
                      "value": "nested_checklist_question",
                      "checklist_answers": [
                        {
                          "feature_id": "3b97e87d-b196-0c21-a236-8b64f3f08086",
                          "feature_schema_id": "clr16p71p019g07tu9sca1dde",
                          "name": "first_checklist_answer",
                          "value": "first_checklist_answer",
                          "classifications": [
                            {
                              "feature_id": "8301a2fe-e898-044f-bc8b-a75e48b16f45",
                              "feature_schema_id": "clr16p71p019h07tu5j9o7b4o",
                              "name": "sub_checklist_question",
                              "value": "sub_checklist_question",
                              "checklist_answers": [
                                {
                                  "feature_id": "baea7ccd-57a6-0c4f-bc66-1ddc2a8aa2d9",
                                  "feature_schema_id": "clr16p71p019i07tu50tq966e",
                                  "name": "first_sub_checklist_answer",
                                  "value": "first_sub_checklist_answer",
                                  "classifications": []
                                }
                              ]
                            }
                          ]
                        }
                      ]
                    },
                    {
                      "feature_id": "60fd8cc0-b5a6-013b-b02b-d91adafb45c1",
                      "feature_schema_id": "clr16p71o018t07tubkih92ua",
                      "name": "radio_question",
                      "value": "radio_question",
                      "radio_answer": {
                        "feature_id": "f0211ed1-4139-08ee-8c35-ea2bcad92b79",
                        "feature_schema_id": "clr16p71o018u07tu8bwc76w1",
                        "name": "first_radio_answer",
                        "value": "first_radio_answer",
                        "classifications": []
                      }
                    },
                    {
                      "feature_id": "73beac75-7e6c-0805-a148-2f6b7465f468",
                      "feature_schema_id": "clr16p71p019d07tufulw888j",
                      "name": "free_text",
                      "value": "free_text",
                      "text_answer": {
                        "content": "sample text"
                      }
                    },
                    {
                      "feature_id": "7a87c79e-6213-032c-859b-547e039ceff7",
                      "feature_schema_id": "clr16p71o018x07tuf7sj7c0x",
                      "name": "nested_radio_question",
                      "value": "nested_radio_question",
                      "radio_answer": {
                        "feature_id": "3aead946-dafd-0f41-8609-826d0d702d68",
                        "feature_schema_id": "clr16p71o018y07tu9oevflh8",
                        "name": "first_radio_answer",
                        "value": "first_radio_answer",
                        "classifications": [
                          {
                            "feature_id": "ab632136-83c1-0f3e-a4af-87da69b86624",
                            "feature_schema_id": "clr16p71o018z07tuas953oef",
                            "name": "sub_radio_question",
                            "value": "sub_radio_question",
                            "radio_answer": {
                              "feature_id": "2823619a-03bf-0b27-a12e-69f869d8e334",
                              "feature_schema_id": "clr16p71o019007tu8htocx3x",
                              "name": "first_sub_radio_answer",
                              "value": "first_sub_radio_answer",
                              "classifications": []
                            }
                          }
                        ]
                      }
                    },
                    {
                      "feature_id": "bd11b44d-3c8d-089d-9531-5177047736b7",
                      "feature_schema_id": "clr16p71p019507tu0x8d0vu4",
                      "name": "checklist_question",
                      "value": "checklist_question",
                      "checklist_answers": [
                        {
                          "feature_id": "360c8a23-3a74-0e75-9553-286b1b152100",
                          "feature_schema_id": "clr16p71p019a07tuetmgfe7c",
                          "name": "third_checklist_answer",
                          "value": "third_checklist_answer",
                          "classifications": []
                        },
                        {
                          "feature_id": "4e12f117-9f03-0a51-8b5d-721b88ce4be0",
                          "feature_schema_id": "clr16p71p019607tu4hkkbhxv",
                          "name": "first_checklist_answer",
                          "value": "first_checklist_answer",
                          "classifications": []
                        },
                        {
                          "feature_id": "501d328c-c0ea-0142-8809-ecd1b19961eb",
                          "feature_schema_id": "clr16p71p019807tu3ya04d5x",
                          "name": "second_checklist_answer",
                          "value": "second_checklist_answer",
                          "classifications": []
                        }
                      ]
                    }
                  ],
                  "relationships": []
                }
              }
            ],
            "predictions": [
              {
                "label_kind": "Default",
                "version": "1.0.0",
                "id": "clr16pmfw0g4o140zkwys3f7u",
                "annotations": {
                  "objects": [
                    {
                      "feature_id": "4bafa398-69d8-4bf8-a3f0-834869ab3e5f",
                      "feature_schema_id": "clr16p71n018r07tuawb18jt6",
                      "name": "named_entity",
                      "value": "named_entity",
                      "annotation_kind": "TextEntity",
                      "classifications": [],
                      "location": {
                        "start": 10,
                        "end": 20
                      }
                    }
                  ],
                  "classifications": [
                    {
                      "feature_id": "31737a80-4a5c-4c40-8362-8d59ae6c2047",
                      "feature_schema_id": "clr16p71o018x07tuf7sj7c0x",
                      "name": "nested_radio_question",
                      "value": "nested_radio_question",
                      "radio_answer": {
                        "feature_id": "a00eca18-fb50-4130-a33b-d505ba8e3d04",
                        "feature_schema_id": "clr16p71o018y07tu9oevflh8",
                        "name": "first_radio_answer",
                        "value": "first_radio_answer",
                        "classifications": [
                          {
                            "feature_id": "94b4ee14-2837-4fb5-b097-4e59cb92b07f",
                            "feature_schema_id": "clr16p71o018z07tuas953oef",
                            "name": "sub_radio_question",
                            "value": "sub_radio_question",
                            "radio_answer": {
                              "feature_id": "a5863817-4c81-424a-8c4c-4cdec05508bf",
                              "feature_schema_id": "clr16p71o019007tu8htocx3x",
                              "name": "first_sub_radio_answer",
                              "value": "first_sub_radio_answer",
                              "classifications": []
                            }
                          }
                        ]
                      }
                    },
                    {
                      "feature_id": "741288f5-76ab-4cdf-9277-0f97f0e426fd",
                      "feature_schema_id": "clr16p71o018t07tubkih92ua",
                      "name": "radio_question",
                      "value": "radio_question",
                      "radio_answer": {
                        "feature_id": "64065994-7568-46e1-8bfc-bd96d20384cf",
                        "feature_schema_id": "clr16p71o018u07tu8bwc76w1",
                        "name": "first_radio_answer",
                        "value": "first_radio_answer",
                        "classifications": []
                      }
                    },
                    {
                      "feature_id": "a66cc378-aa78-4250-bde1-28da1310bedb",
                      "feature_schema_id": "clr16p71p019507tu0x8d0vu4",
                      "name": "checklist_question",
                      "value": "checklist_question",
                      "checklist_answers": [
                        {
                          "feature_id": "6db15123-a2dd-4722-8253-32a0c0236850",
                          "feature_schema_id": "clr16p71p019607tu4hkkbhxv",
                          "name": "first_checklist_answer",
                          "value": "first_checklist_answer",
                          "classifications": []
                        },
                        {
                          "feature_id": "8eb5629e-8ae1-44f1-b369-491c5c61ea02",
                          "feature_schema_id": "clr16p71p019a07tuetmgfe7c",
                          "name": "third_checklist_answer",
                          "value": "third_checklist_answer",
                          "classifications": []
                        },
                        {
                          "feature_id": "fdb2574a-fea7-4127-9ee1-b112126f117a",
                          "feature_schema_id": "clr16p71p019807tu3ya04d5x",
                          "name": "second_checklist_answer",
                          "value": "second_checklist_answer",
                          "classifications": []
                        }
                      ]
                    },
                    {
                      "feature_id": "a8386f96-ae13-45c0-bb35-033380f86dc5",
                      "feature_schema_id": "clr16p71p019d07tufulw888j",
                      "name": "free_text",
                      "value": "free_text",
                      "text_answer": {
                        "content": "sample text"
                      }
                    },
                    {
                      "feature_id": "db0150e9-ecb5-4bad-8350-c82446244657",
                      "feature_schema_id": "clr16p71p019f07tucv4g1cmm",
                      "name": "nested_checklist_question",
                      "value": "nested_checklist_question",
                      "checklist_answers": [
                        {
                          "feature_id": "84f14b81-301f-41fd-a842-6d48594a40fc",
                          "feature_schema_id": "clr16p71p019g07tu9sca1dde",
                          "name": "first_checklist_answer",
                          "value": "first_checklist_answer",
                          "classifications": [
                            {
                              "feature_id": "8367c018-cba4-40ff-85e5-c06b137773c3",
                              "feature_schema_id": "clr16p71p019h07tu5j9o7b4o",
                              "name": "sub_checklist_question",
                              "value": "sub_checklist_question",
                              "checklist_answers": [
                                {
                                  "feature_id": "c1e2da03-c23d-4ffa-a25c-e6b6c13a1dae",
                                  "feature_schema_id": "clr16p71p019i07tu50tq966e",
                                  "name": "first_sub_checklist_answer",
                                  "value": "first_sub_checklist_answer",
                                  "classifications": []
                                }
                              ]
                            }
                          ]
                        }
                      ]
                    }
                  ],
                  "relationships": []
                }
              }
            ]
          }
        }
      }
    },
    "models": {},
    "embeddings": [
      {
        "id": "c300000000000000000000000",
        "name": "Text Embedding V2 (All-MPNet-base-v2)",
        "dimensions": 768,
        "is_custom": false,
        "values": [
          {
            "value": [
              0.04099537059664726,
              0.03790362924337387,
              -0.013698477298021317,
              0.038478486239910126,
              -0.01930154114961624,
              0.05670716613531113,
              0.02421298250555992 ...
  	]
  }
  ```
</CodeGroup>


# Export video annotations
Source: https://docs.labelbox.com/horizon/developer-guides/export-video-annotations

How to export video annotations and sample export formats.

<CardGroup>
  <Card title="Open In Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/exports/export_data.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/exports/export_data.ipynb" />
</CardGroup>

## Export as JSON

<CodeGroup>
  ```python Export theme={"theme":"css-variables"}
  # Set the export params to include/exclude certain fields.
  export_params= {
    "attachments": True,
    "metadata_fields": True,
    "data_row_details": True,
    "project_details": True,
    "label_details": True,
    "performance_details": True,
    "interpolated_frames": True
  }

  # Note: Filters follow AND logic, so typically using one filter is sufficient.

  filters= {
    "last_activity_at": ["2000-01-01 00:00:00", "2050-01-01 00:00:00"],
    "workflow_status": "<wkf-status>"
  }

  export_task = project.export(params=export_params, filters=filters)
  export_task.wait_till_done()

  # Stream the export using a callback function

  def json_stream_handler(output: labelbox.BufferedJsonConverterOutput):
    print(output.json)

  export_task.get_buffered_stream(stream_type=labelbox.StreamType.RESULT).start(stream_handler=json_stream_handler)

  # Collect all exported data into a list

  export_json = [data_row.json for data_row in export_task.get_buffered_stream()]

  print("file size: ", export_task.get_total_file_size(stream_type=lb.StreamType.RESULT))
  print("line count: ", export_task.get_total_lines(stream_type=lb.StreamType.RESULT))

  ```
</CodeGroup>

## Annotation export formats

### Bounding box

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "cltd97cag01th14ys6fyyxyk9",
    "feature_schema_id": "cltd977no037j07x5d82dacbc",
    "name": "bbox_video",
    "value": "bbox_video",
    "annotation_kind": "VideoBoundingBox",
    "classifications": [],
    "bounding_box": {
      "top": 617.0,
      "left": 1371.0,
      "height": 419.0,
      "width": 505.0
    }
  }
  ```
</CodeGroup>

### Point

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "cltd97cag01to14yslakirofl",
    "feature_schema_id": "cltd977no037l07x56wfu4536",
    "name": "point_video",
    "value": "point_video",
    "annotation_kind": "VideoPoint",
    "classifications": [],
    "point": {
      "x": 660.134,
      "y": 407.926
    }
  }
  ```
</CodeGroup>

### Polyline

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
   {
     "feature_id": "cltd97cag01tp14ysz3my848t",
     "feature_schema_id": "cltd977np037n07x5d1buckz9",
     "name": "line_video_frame",
     "value": "line_video_frame",
     "annotation_kind": "VideoPolyline",
     "classifications": [],
     "line": [
       {
         "x": 680.0,
         "y": 100.0
       },
       {
         "x": 100.0,
         "y": 190.0
       }
     ]
   }
  ```
</CodeGroup>

### Mask

Regardless of which [tool](/horizon/guides/image-editor#segmentation-mask) is used to create a segmentation mask, the annotation will appear in the following form in the export. Whether you use AutoSegment, the brush tool, the pen tool, or the fill tool, all masks take the same format in exports.

#### Composite masks

[Open this Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/exports/composite_mask_export.ipynb) for an interactive tutorial on exporting composite masks.

A composite mask is a cloud-hosted PNG file that contains all raster segmentation mask instances on a labeled data row (one composite mask per label).

When you export data rows containing segmentation masks, you can use `composite_mask.url` to access all segmentation masks on that data row. To access a single instance of a segmentation mask, use the `color_rgb` values. The `color_rgb` value represents the color of a single annotation or instance mask in a label.

##### Sample composite masks export

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "clsivokxg00043b70xuf7x5tz",
    "feature_schema_id": "clsgbcaqw000j071t2ga23jnc",
    "name": "car",
    "value": "car",
    "annotation_kind": "ImageSegmentationMask",
    "classifications": [],
    "mask": {
      "url": "https://api.labelbox.com/api/v1/projects/clsivnieg050k074i2jqaa0c3/annotations/clsivokxg00043b70xuf7x5tz/index/1/mask"
    },
    "composite_mask": {
      // This URL contains all mask instances on the data row
      "url": "https://api.labelbox.com/api/v1/tasks/clsiw88n205mw072ke4n78y5o/masks/clsivok5f00023b70r1a4fgzy",
      // This RGB reflects a single mask instance
      "color_rgb": [
        31,
        185,
        101
      ]
    }
  }
  ```
</CodeGroup>

#### Access mask data

To access the mask data, you must pass your Labelbox API key stored in `client.headers` in an API request.

When you grab a URL from the mask annotation in the export, the `project_id` and `feature_id` will already be in place, as shown above. Here, we provide the framework for structuring a URL with any project ID and feature ID.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  import urllib.request
  from PIL import Image

  # Provide a project ID and feature ID if using instance masks. Alternatively, replace the entire mask_url with a URL grabbed from your export.

  project_id = ""
  feature_id = ""

  mask_url = f"https://api.labelbox.com/api/v1/projects/{project_id}/annotations/{feature_id}/index/1/mask"

  # Instead of providing the individual variables, we recommend utilizing the URL from export for composite mask

  composite_mask_url = f"https://api.labelbox.com/api/v1/tasks/{task_id}/masks/{composite_mask_id}/index/1"

  # Make the API request

  req = urllib.request.Request(mask_url, headers=client.headers)

  # Optionally, print the image of the mask

  image = Image.open(urllib.request.urlopen(req))
  image.show()

  ```
</CodeGroup>

### Classification - Radio

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "cltd97cag01tf14ysho6tfl7j",
    "feature_schema_id": "cltd977np038907x5ah8l9ha7",
    "name": "radio_class",
    "value": "radio_class",
    "radio_answer": {
      "feature_id": "cltd97cag01tg14ys171a3qqr",
      "feature_schema_id": "cltd977np038a07x515axagkl",
      "name": "first_radio_answer",
      "value": "first_radio_answer",
      "classifications": []
    }
  }
  ```
</CodeGroup>

### Classification - Checklist

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "cltd97cag01tq14ysb4m3nyjo",
    "feature_schema_id": "cltd977nq039107x5dsml2knm",
    "name": "checklist_class_global",
    "value": "checklist_class_global",
    "checklist_answers": [
      {
        "feature_id": "cltd97cag01tr14ysozjbx0c6",
        "feature_schema_id": "cltd977nq039207x5ei1605s1",
        "name": "first_checklist_answer",
        "value": "first_checklist_answer",
        "classifications": []
      },
      {
        "feature_id": "cltd97cag01ts14ys4vvxihnw",
        "feature_schema_id": "cltd977nq039407x50qs3g61l",
        "name": "second_checklist_answer",
        "value": "second_checklist_answer",
        "classifications": []
      }
    ]
  }
  ```
</CodeGroup>

### Classification - Free-form text

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "feature_id": "cltd97cag01u314ysx1h72f0h",
    "feature_schema_id": "cltd977nq039707x535ufeo39",
    "name": "free_text",
    "value": "free_text",
    "text_answer": {
      "content": "sample text"
    }
  }
  ```
</CodeGroup>

## Sample project export

<CodeGroup>
  ```json JSON expandable theme={"theme":"css-variables"}
  {
    "data_row": {
      "id": "clp1jolki07uy0725flku4bmv",
      "global_key": "sample-video-jellyfish.mp4",
      "row_data": "https://storage.googleapis.com/labelbox-datasets/video-sample-data/sample-video-2.mp4",
      "details": {
        "dataset_id": "clp1jokqp000p07307gtfy1w3",
        "dataset_name": "video_demo_dataset",
        "created_at": "2023-11-16T18:48:51.208+00:00",
        "updated_at": "2023-11-16T18:48:51.801+00:00",
        "last_activity_at": "2024-04-08T17:09:54.000+00:00",
        "created_by": "[email protected]"
      }
    },
    "media_attributes": {
      "height": 1080,
      "width": 1920,
      "mime_type": "video/mp4",
      "frame_rate": 10,
      "frame_count": 100
    },
    "attachments": [],
    "metadata_fields": [],
     "embeddings": [
      {
        "id": "c400000000000000000000000",
        "name": "Video Embedding V2 (Google MultiModal)",
        "dimensions": 1408,
        "is_custom": false,
        "values": [
          {
            "value": [
              -0.00489852577,
              0.0287129097 ...
            ]
          }
        ]
      }
    ],
    "projects": {
      "cltd977uu016y08xyggep8aiv": {
        "name": "Video Annotation Import Demo",
        "labels": [
          {
            "label_kind": "Video",
            "version": "1.0.0",
            "id": "cltd97caz01ub14ysq8rf4x8w",
            "label_details": {
              "created_at": "2024-03-04T18:07:33.000+00:00",
              "updated_at": "2024-03-04T18:07:33.000+00:00",
              "created_by": "[email protected]",
              "content_last_updated_at": "2024-03-04T18:07:33.369+00:00",
              "reviews": []
            },
            "performance_details": {
              "seconds_to_create": 0,
              "seconds_to_review": 0,
              "skipped": false
            },
            "annotations": {
              "frames": {
                "29": {
                  "objects": {
                    "cltd97cag01tp14ysz3my848t": {
                      "feature_id": "cltd97cag01tp14ysz3my848t",
                      "feature_schema_id": "cltd977np037n07x5d1buckz9",
                      "name": "line_video_frame",
                      "value": "line_video_frame",
                      "annotation_kind": "VideoPolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 680.0,
                          "y": 100.0
                        },
                        {
                          "x": 100.0,
                          "y": 190.0
                        }
                      ]
                    }
                  },
                  "classifications": [
                    {
                      "feature_id": "cltd97cag01tc14yspm0lev2d",
                      "feature_schema_id": "cltd977np038307x55dlf3rgs",
                      "name": "checklist_class",
                      "value": "checklist_class",
                      "checklist_answers": [
                        {
                          "feature_id": "cltd97cag01td14ysxw2n8xkz",
                          "feature_schema_id": "cltd977np038407x51hcvdad7",
                          "name": "first_checklist_answer",
                          "value": "first_checklist_answer",
                          "classifications": []
                        },
                        {
                          "feature_id": "cltd97cag01te14ys1jdf1fku",
                          "feature_schema_id": "cltd977np038607x58epu4nb5",
                          "name": "second_checklist_answer",
                          "value": "second_checklist_answer",
                          "classifications": []
                        }
                      ]
                    }
                  ]
                },
                "35": {
                  "objects": {
                    "cltd97cag01tp14ysz3my848t": {
                      "feature_id": "cltd97cag01tp14ysz3my848t",
                      "feature_schema_id": "cltd977np037n07x5d1buckz9",
                      "name": "line_video_frame",
                      "value": "line_video_frame",
                      "annotation_kind": "VideoPolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 680.0,
                          "y": 100.0
                        },
                        {
                          "x": 100.0,
                          "y": 190.0
                        }
                      ]
                    }
                  },
                  "classifications": [
                    {
                      "feature_id": "cltd97cag01tc14yspm0lev2d",
                      "feature_schema_id": "cltd977np038307x55dlf3rgs",
                      "name": "checklist_class",
                      "value": "checklist_class",
                      "checklist_answers": [
                        {
                          "feature_id": "cltd97cag01td14ysxw2n8xkz",
                          "feature_schema_id": "cltd977np038407x51hcvdad7",
                          "name": "first_checklist_answer",
                          "value": "first_checklist_answer",
                          "classifications": []
                        },
                        {
                          "feature_id": "cltd97cag01te14ys1jdf1fku",
                          "feature_schema_id": "cltd977np038607x58epu4nb5",
                          "name": "second_checklist_answer",
                          "value": "second_checklist_answer",
                          "classifications": []
                        }
                      ]
                    }
                  ]
                },
                "9": {
                  "objects": {
                    "cltd97cag01tp14ysz3my848t": {
                      "feature_id": "cltd97cag01tp14ysz3my848t",
                      "feature_schema_id": "cltd977np037n07x5d1buckz9",
                      "name": "line_video_frame",
                      "value": "line_video_frame",
                      "annotation_kind": "VideoPolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 680.0,
                          "y": 100.00000000000001
                        },
                        {
                          "x": 100.00000000000001,
                          "y": 190.0
                        }
                      ]
                    }
                  },
                  "classifications": [
                    {
                      "feature_id": "cltd97cag01tf14ysho6tfl7j",
                      "feature_schema_id": "cltd977np038907x5ah8l9ha7",
                      "name": "radio_class",
                      "value": "radio_class",
                      "radio_answer": {
                        "feature_id": "cltd97cag01tg14ys171a3qqr",
                        "feature_schema_id": "cltd977np038a07x515axagkl",
                        "name": "first_radio_answer",
                        "value": "first_radio_answer",
                        "classifications": []
                      }
                    }
                  ]
                },
                "15": {
                  "objects": {
                    "cltd97cag01th14ys6fyyxyk9": {
                      "feature_id": "cltd97cag01th14ys6fyyxyk9",
                      "feature_schema_id": "cltd977no037j07x5d82dacbc",
                      "name": "bbox_video",
                      "value": "bbox_video",
                      "annotation_kind": "VideoBoundingBox",
                      "classifications": [],
                      "bounding_box": {
                        "top": 617.0,
                        "left": 1371.0,
                        "height": 419.0,
                        "width": 505.0
                      }
                    },
                    "cltd97cag01tp14ysz3my848t": {
                      "feature_id": "cltd97cag01tp14ysz3my848t",
                      "feature_schema_id": "cltd977np037n07x5d1buckz9",
                      "name": "line_video_frame",
                      "value": "line_video_frame",
                      "annotation_kind": "VideoPolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 680.0,
                          "y": 100.0
                        },
                        {
                          "x": 100.0,
                          "y": 190.0
                        }
                      ]
                    }
                  },
                  "classifications": [
                    {
                      "feature_id": "cltd97cag01tf14ysho6tfl7j",
                      "feature_schema_id": "cltd977np038907x5ah8l9ha7",
                      "name": "radio_class",
                      "value": "radio_class",
                      "radio_answer": {
                        "feature_id": "cltd97cag01tg14ys171a3qqr",
                        "feature_schema_id": "cltd977np038a07x515axagkl",
                        "name": "first_radio_answer",
                        "value": "first_radio_answer",
                        "classifications": []
                      }
                    }
                  ]
                },
                "13": {
                  "objects": {
                    "cltd97cag01th14ys6fyyxyk9": {
                      "feature_id": "cltd97cag01th14ys6fyyxyk9",
                      "feature_schema_id": "cltd977no037j07x5d82dacbc",
                      "name": "bbox_video",
                      "value": "bbox_video",
                      "annotation_kind": "VideoBoundingBox",
                      "classifications": [],
                      "bounding_box": {
                        "top": 617.0,
                        "left": 1371.0,
                        "height": 419.0,
                        "width": 505.0
                      }
                    },
                    "cltd97cag01ti14ys8wedf89d": {
                      "feature_id": "cltd97cag01ti14ys8wedf89d",
                      "feature_schema_id": "cltd977np037r07x512iteflz",
                      "name": "bbox_class",
                      "value": "bbox_class",
                      "annotation_kind": "VideoBoundingBox",
                      "classifications": [
                        {
                          "feature_id": "cltd97cag01tj14ysz455nenr",
                          "feature_schema_id": "cltd977np037s07x5csbodf5f",
                          "name": "checklist_class",
                          "value": "checklist_class",
                          "radio_answer": {
                            "feature_id": "cltd97cag01tl14ysnbe8mw75",
                            "feature_schema_id": "cltd977np037v07x54esmdmws",
                            "name": "second_checklist_answer",
                            "value": "second_checklist_answer",
                            "classifications": []
                          }
                        }
                      ],
                      "bounding_box": {
                        "top": 146.0,
                        "left": 98.0,
                        "height": 382.0,
                        "width": 341.0
                      }
                    },
                    "cltd97cag01tp14ysz3my848t": {
                      "feature_id": "cltd97cag01tp14ysz3my848t",
                      "feature_schema_id": "cltd977np037n07x5d1buckz9",
                      "name": "line_video_frame",
                      "value": "line_video_frame",
                      "annotation_kind": "VideoPolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 680.0,
                          "y": 100.0
                        },
                        {
                          "x": 100.0,
                          "y": 190.0
                        }
                      ]
                    }
                  },
                  "classifications": []
                },
                "14": {
                  "objects": {
                    "cltd97cag01th14ys6fyyxyk9": {
                      "feature_id": "cltd97cag01th14ys6fyyxyk9",
                      "feature_schema_id": "cltd977no037j07x5d82dacbc",
                      "name": "bbox_video",
                      "value": "bbox_video",
                      "annotation_kind": "VideoBoundingBox",
                      "classifications": [],
                      "bounding_box": {
                        "top": 617.0000000000001,
                        "left": 1371.0,
                        "height": 418.9999999999999,
                        "width": 505.0
                      }
                    },
                    "cltd97cag01tp14ysz3my848t": {
                      "feature_id": "cltd97cag01tp14ysz3my848t",
                      "feature_schema_id": "cltd977np037n07x5d1buckz9",
                      "name": "line_video_frame",
                      "value": "line_video_frame",
                      "annotation_kind": "VideoPolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 680.0,
                          "y": 100.0
                        },
                        {
                          "x": 100.0,
                          "y": 190.0
                        }
                      ]
                    }
                  },
                  "classifications": []
                },
                "16": {
                  "objects": {
                    "cltd97cag01th14ys6fyyxyk9": {
                      "feature_id": "cltd97cag01th14ys6fyyxyk9",
                      "feature_schema_id": "cltd977no037j07x5d82dacbc",
                      "name": "bbox_video",
                      "value": "bbox_video",
                      "annotation_kind": "VideoBoundingBox",
                      "classifications": [],
                      "bounding_box": {
                        "top": 617.0,
                        "left": 1371.0,
                        "height": 419.0,
                        "width": 505.0
                      }
                    },
                    "cltd97cag01tp14ysz3my848t": {
                      "feature_id": "cltd97cag01tp14ysz3my848t",
                      "feature_schema_id": "cltd977np037n07x5d1buckz9",
                      "name": "line_video_frame",
                      "value": "line_video_frame",
                      "annotation_kind": "VideoPolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 680.0,
                          "y": 100.0
                        },
                        {
                          "x": 100.0,
                          "y": 190.0
                        }
                      ]
                    }
                  },
                  "classifications": []
                },
                "17": {
                  "objects": {
                    "cltd97cag01th14ys6fyyxyk9": {
                      "feature_id": "cltd97cag01th14ys6fyyxyk9",
                      "feature_schema_id": "cltd977no037j07x5d82dacbc",
                      "name": "bbox_video",
                      "value": "bbox_video",
                      "annotation_kind": "VideoBoundingBox",
                      "classifications": [],
                      "bounding_box": {
                        "top": 617.0,
                        "left": 1371.0,
                        "height": 419.0,
                        "width": 505.0
                      }
                    },
                    "cltd97cag01to14yslakirofl": {
                      "feature_id": "cltd97cag01to14yslakirofl",
                      "feature_schema_id": "cltd977no037l07x56wfu4536",
                      "name": "point_video",
                      "value": "point_video",
                      "annotation_kind": "VideoPoint",
                      "classifications": [],
                      "point": {
                        "x": 660.134,
                        "y": 407.926
                      }
                    },
                    "cltd97cag01tp14ysz3my848t": {
                      "feature_id": "cltd97cag01tp14ysz3my848t",
                      "feature_schema_id": "cltd977np037n07x5d1buckz9",
                      "name": "line_video_frame",
                      "value": "line_video_frame",
                      "annotation_kind": "VideoPolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 680.0,
                          "y": 100.0
                        },
                        {
                          "x": 100.0,
                          "y": 190.0
                        }
                      ]
                    }
                  },
                  "classifications": []
                },
                "18": {
                  "objects": {
                    "cltd97cag01th14ys6fyyxyk9": {
                      "feature_id": "cltd97cag01th14ys6fyyxyk9",
                      "feature_schema_id": "cltd977no037j07x5d82dacbc",
                      "name": "bbox_video",
                      "value": "bbox_video",
                      "annotation_kind": "VideoBoundingBox",
                      "classifications": [],
                      "bounding_box": {
                        "top": 617.0,
                        "left": 1371.0,
                        "height": 419.0,
                        "width": 505.0
                      }
                    },
                    "cltd97cag01tp14ysz3my848t": {
                      "feature_id": "cltd97cag01tp14ysz3my848t",
                      "feature_schema_id": "cltd977np037n07x5d1buckz9",
                      "name": "line_video_frame",
                      "value": "line_video_frame",
                      "annotation_kind": "VideoPolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 680.0,
                          "y": 100.0
                        },
                        {
                          "x": 100.0,
                          "y": 190.0
                        }
                      ]
                    }
                  },
                  "classifications": []
                },
                "19": {
                  "objects": {
                    "cltd97cag01th14ys6fyyxyk9": {
                      "feature_id": "cltd97cag01th14ys6fyyxyk9",
                      "feature_schema_id": "cltd977no037j07x5d82dacbc",
                      "name": "bbox_video",
                      "value": "bbox_video",
                      "annotation_kind": "VideoBoundingBox",
                      "classifications": [],
                      "bounding_box": {
                        "top": 617.0,
                        "left": 1371.0,
                        "height": 419.0,
                        "width": 505.0
                      }
                    },
                    "cltd97cag01tp14ysz3my848t": {
                      "feature_id": "cltd97cag01tp14ysz3my848t",
                      "feature_schema_id": "cltd977np037n07x5d1buckz9",
                      "name": "line_video_frame",
                      "value": "line_video_frame",
                      "annotation_kind": "VideoPolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 680.0,
                          "y": 100.0
                        },
                        {
                          "x": 100.0,
                          "y": 190.0
                        }
                      ]
                    }
                  },
                  "classifications": []
                },
                "10": {
                  "objects": {
                    "cltd97cag01ti14ys8wedf89d": {
                      "feature_id": "cltd97cag01ti14ys8wedf89d",
                      "feature_schema_id": "cltd977np037r07x512iteflz",
                      "name": "bbox_class",
                      "value": "bbox_class",
                      "annotation_kind": "VideoBoundingBox",
                      "classifications": [],
                      "bounding_box": {
                        "top": 146.0,
                        "left": 98.0,
                        "height": 382.0,
                        "width": 341.0
                      }
                    },
                    "cltd97cag01tp14ysz3my848t": {
                      "feature_id": "cltd97cag01tp14ysz3my848t",
                      "feature_schema_id": "cltd977np037n07x5d1buckz9",
                      "name": "line_video_frame",
                      "value": "line_video_frame",
                      "annotation_kind": "VideoPolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 680.0,
                          "y": 100.0
                        },
                        {
                          "x": 100.0,
                          "y": 190.0
                        }
                      ]
                    }
                  },
                  "classifications": []
                },
                "11": {
                  "objects": {
                    "cltd97cag01ti14ys8wedf89d": {
                      "feature_id": "cltd97cag01ti14ys8wedf89d",
                      "feature_schema_id": "cltd977np037r07x512iteflz",
                      "name": "bbox_class",
                      "value": "bbox_class",
                      "annotation_kind": "VideoBoundingBox",
                      "classifications": [
                        {
                          "feature_id": "cltd97cag01tj14ysz455nenr",
                          "feature_schema_id": "cltd977np037s07x5csbodf5f",
                          "name": "checklist_class",
                          "value": "checklist_class",
                          "radio_answer": {
                            "feature_id": "cltd97cag01tk14ysejipk2y6",
                            "feature_schema_id": "cltd977np037t07x5966ebxpe",
                            "name": "first_checklist_answer",
                            "value": "first_checklist_answer",
                            "classifications": []
                          }
                        }
                      ],
                      "bounding_box": {
                        "top": 146.0,
                        "left": 98.0,
                        "height": 382.0,
                        "width": 341.0
                      }
                    },
                    "cltd97cag01tp14ysz3my848t": {
                      "feature_id": "cltd97cag01tp14ysz3my848t",
                      "feature_schema_id": "cltd977np037n07x5d1buckz9",
                      "name": "line_video_frame",
                      "value": "line_video_frame",
                      "annotation_kind": "VideoPolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 680.0,
                          "y": 100.0
                        },
                        {
                          "x": 100.0,
                          "y": 190.0
                        }
                      ]
                    }
                  },
                  "classifications": []
                },
                "12": {
                  "objects": {
                    "cltd97cag01ti14ys8wedf89d": {
                      "feature_id": "cltd97cag01ti14ys8wedf89d",
                      "feature_schema_id": "cltd977np037r07x512iteflz",
                      "name": "bbox_class",
                      "value": "bbox_class",
                      "annotation_kind": "VideoBoundingBox",
                      "classifications": [
                        {
                          "feature_id": "cltd97cag01tj14ysz455nenr",
                          "feature_schema_id": "cltd977np037s07x5csbodf5f",
                          "name": "checklist_class",
                          "value": "checklist_class",
                          "radio_answer": {
                            "feature_id": "cltd97cag01tk14ysejipk2y6",
                            "feature_schema_id": "cltd977np037t07x5966ebxpe",
                            "name": "first_checklist_answer",
                            "value": "first_checklist_answer",
                            "classifications": []
                          }
                        }
                      ],
                      "bounding_box": {
                        "top": 146.0,
                        "left": 98.0,
                        "height": 382.0,
                        "width": 341.0
                      }
                    },
                    "cltd97cag01tp14ysz3my848t": {
                      "feature_id": "cltd97cag01tp14ysz3my848t",
                      "feature_schema_id": "cltd977np037n07x5d1buckz9",
                      "name": "line_video_frame",
                      "value": "line_video_frame",
                      "annotation_kind": "VideoPolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 680.0,
                          "y": 100.0
                        },
                        {
                          "x": 100.0,
                          "y": 190.0
                        }
                      ]
                    }
                  },
                  "classifications": []
                },
                "22": {
                  "objects": {
                    "cltd97cag01tm14ys8tm4x7yl": {
                      "feature_id": "cltd97cag01tm14ys8tm4x7yl",
                      "feature_schema_id": "cltd977no037j07x5d82dacbc",
                      "name": "bbox_video",
                      "value": "bbox_video",
                      "annotation_kind": "VideoBoundingBox",
                      "classifications": [],
                      "bounding_box": {
                        "top": 617.0,
                        "left": 1371.0,
                        "height": 419.0,
                        "width": 505.0
                      }
                    },
                    "cltd97cag01tn14yspqge5yph": {
                      "feature_id": "cltd97cag01tn14yspqge5yph",
                      "feature_schema_id": "cltd977no037j07x5d82dacbc",
                      "name": "bbox_video",
                      "value": "bbox_video",
                      "annotation_kind": "VideoBoundingBox",
                      "classifications": [],
                      "bounding_box": {
                        "top": 146.0,
                        "left": 98.0,
                        "height": 382.0,
                        "width": 341.0
                      }
                    }
                  },
                  "classifications": []
                },
                "23": {
                  "objects": {
                    "cltd97cag01tm14ys8tm4x7yl": {
                      "feature_id": "cltd97cag01tm14ys8tm4x7yl",
                      "feature_schema_id": "cltd977no037j07x5d82dacbc",
                      "name": "bbox_video",
                      "value": "bbox_video",
                      "annotation_kind": "VideoBoundingBox",
                      "classifications": [],
                      "bounding_box": {
                        "top": 617.0,
                        "left": 1371.0,
                        "height": 419.0,
                        "width": 505.0000000000002
                      }
                    },
                    "cltd97cag01tn14yspqge5yph": {
                      "feature_id": "cltd97cag01tn14yspqge5yph",
                      "feature_schema_id": "cltd977no037j07x5d82dacbc",
                      "name": "bbox_video",
                      "value": "bbox_video",
                      "annotation_kind": "VideoBoundingBox",
                      "classifications": [],
                      "bounding_box": {
                        "top": 146.0,
                        "left": 98.0,
                        "height": 382.0,
                        "width": 341.00000000000006
                      }
                    }
                  },
                  "classifications": []
                },
                "24": {
                  "objects": {
                    "cltd97cag01tm14ys8tm4x7yl": {
                      "feature_id": "cltd97cag01tm14ys8tm4x7yl",
                      "feature_schema_id": "cltd977no037j07x5d82dacbc",
                      "name": "bbox_video",
                      "value": "bbox_video",
                      "annotation_kind": "VideoBoundingBox",
                      "classifications": [],
                      "bounding_box": {
                        "top": 617.0,
                        "left": 1371.0,
                        "height": 419.0,
                        "width": 505.0
                      }
                    },
                    "cltd97cag01tn14yspqge5yph": {
                      "feature_id": "cltd97cag01tn14yspqge5yph",
                      "feature_schema_id": "cltd977no037j07x5d82dacbc",
                      "name": "bbox_video",
                      "value": "bbox_video",
                      "annotation_kind": "VideoBoundingBox",
                      "classifications": [],
                      "bounding_box": {
                        "top": 146.0,
                        "left": 98.0,
                        "height": 382.0,
                        "width": 341.0
                      }
                    },
                    "cltd97cag01tp14ysz3my848t": {
                      "feature_id": "cltd97cag01tp14ysz3my848t",
                      "feature_schema_id": "cltd977np037n07x5d1buckz9",
                      "name": "line_video_frame",
                      "value": "line_video_frame",
                      "annotation_kind": "VideoPolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 680.0,
                          "y": 100.0
                        },
                        {
                          "x": 100.0,
                          "y": 190.0
                        }
                      ]
                    },
                    "cltd97cag01u514ys2bfg9vvo": {
                      "feature_id": "cltd97cag01u514ys2bfg9vvo",
                      "feature_schema_id": "cltd977np037p07x55hu71x6k",
                      "name": "video_mask",
                      "value": "video_mask",
                      "annotation_kind": "VideoSegmentationMask",
                      "classifications": [],
                      "mask": {
                        "url": "https://api.labelbox.com/api/v1/projects/cltd977uu016y08xyggep8aiv/annotations/cltd97cag01u514ys2bfg9vvo/index/24/mask"
                      },
                      "composite_mask": {
                        "url": "https://api.labelbox.com/api/v1/tasks/clur7t5d501i707wfawrp050b/masks/cltd97a7h01tb14ysldafd9es/index/24",
                        "color_rgb": [
                          76,
                          104,
                          177
                        ]
                      }
                    },
                    "cltd97cah01u614ysvntxcyub": {
                      "feature_id": "cltd97cah01u614ysvntxcyub",
                      "feature_schema_id": "cltd977np037z07x5gclv41y4",
                      "name": "mask_with_text_subclass",
                      "value": "mask_with_text_subclass",
                      "annotation_kind": "VideoSegmentationMask",
                      "classifications": [],
                      "mask": {
                        "url": "https://api.labelbox.com/api/v1/projects/cltd977uu016y08xyggep8aiv/annotations/cltd97cah01u614ysvntxcyub/index/24/mask"
                      },
                      "composite_mask": {
                        "url": "https://api.labelbox.com/api/v1/tasks/clur7t5d501i707wfawrp050b/masks/cltd97a7h01tb14ysldafd9es/index/24",
                        "color_rgb": [
                          233,
                          218,
                          94
                        ]
                      }
                    },
                    "cltd97cah01u714ysbqbvqs6b": {
                      "feature_id": "cltd97cah01u714ysbqbvqs6b",
                      "feature_schema_id": "cltd977np037p07x55hu71x6k",
                      "name": "video_mask",
                      "value": "video_mask",
                      "annotation_kind": "VideoSegmentationMask",
                      "classifications": [],
                      "mask": {
                        "url": "https://api.labelbox.com/api/v1/projects/cltd977uu016y08xyggep8aiv/annotations/cltd97cah01u714ysbqbvqs6b/index/24/mask"
                      },
                      "composite_mask": {
                        "url": "https://api.labelbox.com/api/v1/tasks/clur7t5d501i707wfawrp050b/masks/cltd97a7h01tb14ysldafd9es/index/24",
                        "color_rgb": [
                          83,
                          152,
                          103
                        ]
                      }
                    }
                  },
                  "classifications": []
                },
                "25": {
                  "objects": {
                    "cltd97cag01tm14ys8tm4x7yl": {
                      "feature_id": "cltd97cag01tm14ys8tm4x7yl",
                      "feature_schema_id": "cltd977no037j07x5d82dacbc",
                      "name": "bbox_video",
                      "value": "bbox_video",
                      "annotation_kind": "VideoBoundingBox",
                      "classifications": [],
                      "bounding_box": {
                        "top": 617.0,
                        "left": 1371.0,
                        "height": 419.0,
                        "width": 505.0
                      }
                    },
                    "cltd97cag01tn14yspqge5yph": {
                      "feature_id": "cltd97cag01tn14yspqge5yph",
                      "feature_schema_id": "cltd977no037j07x5d82dacbc",
                      "name": "bbox_video",
                      "value": "bbox_video",
                      "annotation_kind": "VideoBoundingBox",
                      "classifications": [],
                      "bounding_box": {
                        "top": 146.0,
                        "left": 98.0,
                        "height": 382.0,
                        "width": 341.0
                      }
                    },
                    "cltd97cag01tp14ysz3my848t": {
                      "feature_id": "cltd97cag01tp14ysz3my848t",
                      "feature_schema_id": "cltd977np037n07x5d1buckz9",
                      "name": "line_video_frame",
                      "value": "line_video_frame",
                      "annotation_kind": "VideoPolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 680.0,
                          "y": 99.99999999999999
                        },
                        {
                          "x": 99.99999999999999,
                          "y": 189.99999999999997
                        }
                      ]
                    }
                  },
                  "classifications": []
                },
                "26": {
                  "objects": {
                    "cltd97cag01tm14ys8tm4x7yl": {
                      "feature_id": "cltd97cag01tm14ys8tm4x7yl",
                      "feature_schema_id": "cltd977no037j07x5d82dacbc",
                      "name": "bbox_video",
                      "value": "bbox_video",
                      "annotation_kind": "VideoBoundingBox",
                      "classifications": [],
                      "bounding_box": {
                        "top": 617.0,
                        "left": 1371.0,
                        "height": 419.0,
                        "width": 505.0
                      }
                    },
                    "cltd97cag01tn14yspqge5yph": {
                      "feature_id": "cltd97cag01tn14yspqge5yph",
                      "feature_schema_id": "cltd977no037j07x5d82dacbc",
                      "name": "bbox_video",
                      "value": "bbox_video",
                      "annotation_kind": "VideoBoundingBox",
                      "classifications": [],
                      "bounding_box": {
                        "top": 146.0,
                        "left": 98.0,
                        "height": 382.0,
                        "width": 341.0
                      }
                    },
                    "cltd97cag01tp14ysz3my848t": {
                      "feature_id": "cltd97cag01tp14ysz3my848t",
                      "feature_schema_id": "cltd977np037n07x5d1buckz9",
                      "name": "line_video_frame",
                      "value": "line_video_frame",
                      "annotation_kind": "VideoPolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 680.0,
                          "y": 100.0
                        },
                        {
                          "x": 100.0,
                          "y": 190.0
                        }
                      ]
                    },
                    "cltd97cah01ua14ysr6u1sh7w": {
                      "feature_id": "cltd97cah01ua14ysr6u1sh7w",
                      "feature_schema_id": "cltd977np037z07x5gclv41y4",
                      "name": "mask_with_text_subclass",
                      "value": "mask_with_text_subclass",
                      "annotation_kind": "VideoSegmentationMask",
                      "classifications": [],
                      "mask": {
                        "url": "https://api.labelbox.com/api/v1/projects/cltd977uu016y08xyggep8aiv/annotations/cltd97cah01ua14ysr6u1sh7w/index/26/mask"
                      },
                      "composite_mask": {
                        "url": "https://api.labelbox.com/api/v1/tasks/clur7t5d501i707wfawrp050b/masks/cltd97a7h01tb14ysldafd9es/index/26",
                        "color_rgb": [
                          226,
                          211,
                          41
                        ]
                      }
                    }
                  },
                  "classifications": []
                },
                "27": {
                  "objects": {
                    "cltd97cag01tm14ys8tm4x7yl": {
                      "feature_id": "cltd97cag01tm14ys8tm4x7yl",
                      "feature_schema_id": "cltd977no037j07x5d82dacbc",
                      "name": "bbox_video",
                      "value": "bbox_video",
                      "annotation_kind": "VideoBoundingBox",
                      "classifications": [],
                      "bounding_box": {
                        "top": 617.0,
                        "left": 1371.0,
                        "height": 419.0,
                        "width": 505.0
                      }
                    },
                    "cltd97cag01tn14yspqge5yph": {
                      "feature_id": "cltd97cag01tn14yspqge5yph",
                      "feature_schema_id": "cltd977no037j07x5d82dacbc",
                      "name": "bbox_video",
                      "value": "bbox_video",
                      "annotation_kind": "VideoBoundingBox",
                      "classifications": [],
                      "bounding_box": {
                        "top": 146.0,
                        "left": 98.0,
                        "height": 382.0,
                        "width": 341.0
                      }
                    },
                    "cltd97cag01tp14ysz3my848t": {
                      "feature_id": "cltd97cag01tp14ysz3my848t",
                      "feature_schema_id": "cltd977np037n07x5d1buckz9",
                      "name": "line_video_frame",
                      "value": "line_video_frame",
                      "annotation_kind": "VideoPolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 680.0,
                          "y": 100.0
                        },
                        {
                          "x": 100.0,
                          "y": 190.0
                        }
                      ]
                    }
                  },
                  "classifications": []
                },
                "5": {
                  "objects": {
                    "cltd97cag01tp14ysz3my848t": {
                      "feature_id": "cltd97cag01tp14ysz3my848t",
                      "feature_schema_id": "cltd977np037n07x5d1buckz9",
                      "name": "line_video_frame",
                      "value": "line_video_frame",
                      "annotation_kind": "VideoPolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 680.0,
                          "y": 100.0
                        },
                        {
                          "x": 100.0,
                          "y": 190.0
                        }
                      ]
                    }
                  },
                  "classifications": []
                },
                "6": {
                  "objects": {
                    "cltd97cag01tp14ysz3my848t": {
                      "feature_id": "cltd97cag01tp14ysz3my848t",
                      "feature_schema_id": "cltd977np037n07x5d1buckz9",
                      "name": "line_video_frame",
                      "value": "line_video_frame",
                      "annotation_kind": "VideoPolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 680.0,
                          "y": 100.0
                        },
                        {
                          "x": 100.0,
                          "y": 190.0
                        }
                      ]
                    }
                  },
                  "classifications": []
                },
                "7": {
                  "objects": {
                    "cltd97cag01tp14ysz3my848t": {
                      "feature_id": "cltd97cag01tp14ysz3my848t",
                      "feature_schema_id": "cltd977np037n07x5d1buckz9",
                      "name": "line_video_frame",
                      "value": "line_video_frame",
                      "annotation_kind": "VideoPolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 680.0,
                          "y": 100.0
                        },
                        {
                          "x": 100.0,
                          "y": 190.00000000000003
                        }
                      ]
                    }
                  },
                  "classifications": []
                },
                "8": {
                  "objects": {
                    "cltd97cag01tp14ysz3my848t": {
                      "feature_id": "cltd97cag01tp14ysz3my848t",
                      "feature_schema_id": "cltd977np037n07x5d1buckz9",
                      "name": "line_video_frame",
                      "value": "line_video_frame",
                      "annotation_kind": "VideoPolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 680.0,
                          "y": 100.0
                        },
                        {
                          "x": 100.0,
                          "y": 190.0
                        }
                      ]
                    }
                  },
                  "classifications": []
                },
                "20": {
                  "objects": {
                    "cltd97cag01tp14ysz3my848t": {
                      "feature_id": "cltd97cag01tp14ysz3my848t",
                      "feature_schema_id": "cltd977np037n07x5d1buckz9",
                      "name": "line_video_frame",
                      "value": "line_video_frame",
                      "annotation_kind": "VideoPolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 680.0,
                          "y": 100.0
                        },
                        {
                          "x": 100.0,
                          "y": 190.0
                        }
                      ]
                    },
                    "cltd97cag01u514ys2bfg9vvo": {
                      "feature_id": "cltd97cag01u514ys2bfg9vvo",
                      "feature_schema_id": "cltd977np037p07x55hu71x6k",
                      "name": "video_mask",
                      "value": "video_mask",
                      "annotation_kind": "VideoSegmentationMask",
                      "classifications": [],
                      "mask": {
                        "url": "https://api.labelbox.com/api/v1/projects/cltd977uu016y08xyggep8aiv/annotations/cltd97cag01u514ys2bfg9vvo/index/20/mask"
                      },
                      "composite_mask": {
                        "url": "https://api.labelbox.com/api/v1/tasks/clur7t5d501i707wfawrp050b/masks/cltd97a7h01tb14ysldafd9es/index/20",
                        "color_rgb": [
                          76,
                          104,
                          177
                        ]
                      }
                    },
                    "cltd97cah01u614ysvntxcyub": {
                      "feature_id": "cltd97cah01u614ysvntxcyub",
                      "feature_schema_id": "cltd977np037z07x5gclv41y4",
                      "name": "mask_with_text_subclass",
                      "value": "mask_with_text_subclass",
                      "annotation_kind": "VideoSegmentationMask",
                      "classifications": [],
                      "mask": {
                        "url": "https://api.labelbox.com/api/v1/projects/cltd977uu016y08xyggep8aiv/annotations/cltd97cah01u614ysvntxcyub/index/20/mask"
                      },
                      "composite_mask": {
                        "url": "https://api.labelbox.com/api/v1/tasks/clur7t5d501i707wfawrp050b/masks/cltd97a7h01tb14ysldafd9es/index/20",
                        "color_rgb": [
                          233,
                          218,
                          94
                        ]
                      }
                    },
                    "cltd97cah01u714ysbqbvqs6b": {
                      "feature_id": "cltd97cah01u714ysbqbvqs6b",
                      "feature_schema_id": "cltd977np037p07x55hu71x6k",
                      "name": "video_mask",
                      "value": "video_mask",
                      "annotation_kind": "VideoSegmentationMask",
                      "classifications": [],
                      "mask": {
                        "url": "https://api.labelbox.com/api/v1/projects/cltd977uu016y08xyggep8aiv/annotations/cltd97cah01u714ysbqbvqs6b/index/20/mask"
                      },
                      "composite_mask": {
                        "url": "https://api.labelbox.com/api/v1/tasks/clur7t5d501i707wfawrp050b/masks/cltd97a7h01tb14ysldafd9es/index/20",
                        "color_rgb": [
                          83,
                          152,
                          103
                        ]
                      }
                    }
                  },
                  "classifications": []
                },
                "28": {
                  "objects": {
                    "cltd97cag01tp14ysz3my848t": {
                      "feature_id": "cltd97cag01tp14ysz3my848t",
                      "feature_schema_id": "cltd977np037n07x5d1buckz9",
                      "name": "line_video_frame",
                      "value": "line_video_frame",
                      "annotation_kind": "VideoPolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 680.0,
                          "y": 100.0
                        },
                        {
                          "x": 100.0,
                          "y": 190.0
                        }
                      ]
                    }
                  },
                  "classifications": []
                },
                "30": {
                  "objects": {
                    "cltd97cag01tp14ysz3my848t": {
                      "feature_id": "cltd97cag01tp14ysz3my848t",
                      "feature_schema_id": "cltd977np037n07x5d1buckz9",
                      "name": "line_video_frame",
                      "value": "line_video_frame",
                      "annotation_kind": "VideoPolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 680.0,
                          "y": 100.0
                        },
                        {
                          "x": 100.0,
                          "y": 190.0
                        }
                      ]
                    }
                  },
                  "classifications": []
                },
                "31": {
                  "objects": {
                    "cltd97cag01tp14ysz3my848t": {
                      "feature_id": "cltd97cag01tp14ysz3my848t",
                      "feature_schema_id": "cltd977np037n07x5d1buckz9",
                      "name": "line_video_frame",
                      "value": "line_video_frame",
                      "annotation_kind": "VideoPolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 680.0,
                          "y": 100.0
                        },
                        {
                          "x": 100.0,
                          "y": 190.0
                        }
                      ]
                    }
                  },
                  "classifications": []
                },
                "32": {
                  "objects": {
                    "cltd97cag01tp14ysz3my848t": {
                      "feature_id": "cltd97cag01tp14ysz3my848t",
                      "feature_schema_id": "cltd977np037n07x5d1buckz9",
                      "name": "line_video_frame",
                      "value": "line_video_frame",
                      "annotation_kind": "VideoPolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 680.0,
                          "y": 100.0
                        },
                        {
                          "x": 100.0,
                          "y": 190.0
                        }
                      ]
                    }
                  },
                  "classifications": []
                },
                "33": {
                  "objects": {
                    "cltd97cag01tp14ysz3my848t": {
                      "feature_id": "cltd97cag01tp14ysz3my848t",
                      "feature_schema_id": "cltd977np037n07x5d1buckz9",
                      "name": "line_video_frame",
                      "value": "line_video_frame",
                      "annotation_kind": "VideoPolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 680.0,
                          "y": 100.0
                        },
                        {
                          "x": 100.0,
                          "y": 190.0
                        }
                      ]
                    }
                  },
                  "classifications": []
                },
                "34": {
                  "objects": {
                    "cltd97cag01tp14ysz3my848t": {
                      "feature_id": "cltd97cag01tp14ysz3my848t",
                      "feature_schema_id": "cltd977np037n07x5d1buckz9",
                      "name": "line_video_frame",
                      "value": "line_video_frame",
                      "annotation_kind": "VideoPolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 680.0,
                          "y": 100.0
                        },
                        {
                          "x": 100.0,
                          "y": 190.0
                        }
                      ]
                    }
                  },
                  "classifications": []
                },
                "36": {
                  "objects": {
                    "cltd97cag01tp14ysz3my848t": {
                      "feature_id": "cltd97cag01tp14ysz3my848t",
                      "feature_schema_id": "cltd977np037n07x5d1buckz9",
                      "name": "line_video_frame",
                      "value": "line_video_frame",
                      "annotation_kind": "VideoPolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 680.0,
                          "y": 100.0
                        },
                        {
                          "x": 100.0,
                          "y": 190.0
                        }
                      ]
                    }
                  },
                  "classifications": []
                },
                "37": {
                  "objects": {
                    "cltd97cag01tp14ysz3my848t": {
                      "feature_id": "cltd97cag01tp14ysz3my848t",
                      "feature_schema_id": "cltd977np037n07x5d1buckz9",
                      "name": "line_video_frame",
                      "value": "line_video_frame",
                      "annotation_kind": "VideoPolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 680.0,
                          "y": 100.0
                        },
                        {
                          "x": 100.0,
                          "y": 190.0
                        }
                      ]
                    }
                  },
                  "classifications": []
                },
                "38": {
                  "objects": {
                    "cltd97cag01tp14ysz3my848t": {
                      "feature_id": "cltd97cag01tp14ysz3my848t",
                      "feature_schema_id": "cltd977np037n07x5d1buckz9",
                      "name": "line_video_frame",
                      "value": "line_video_frame",
                      "annotation_kind": "VideoPolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 680.0,
                          "y": 100.0
                        },
                        {
                          "x": 100.0,
                          "y": 190.0
                        }
                      ]
                    }
                  },
                  "classifications": []
                },
                "39": {
                  "objects": {
                    "cltd97cag01tp14ysz3my848t": {
                      "feature_id": "cltd97cag01tp14ysz3my848t",
                      "feature_schema_id": "cltd977np037n07x5d1buckz9",
                      "name": "line_video_frame",
                      "value": "line_video_frame",
                      "annotation_kind": "VideoPolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 680.0,
                          "y": 100.0
                        },
                        {
                          "x": 100.0,
                          "y": 190.0
                        }
                      ]
                    }
                  },
                  "classifications": []
                },
                "40": {
                  "objects": {
                    "cltd97cag01tp14ysz3my848t": {
                      "feature_id": "cltd97cag01tp14ysz3my848t",
                      "feature_schema_id": "cltd977np037n07x5d1buckz9",
                      "name": "line_video_frame",
                      "value": "line_video_frame",
                      "annotation_kind": "VideoPolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 680.0,
                          "y": 100.0
                        },
                        {
                          "x": 100.0,
                          "y": 190.0
                        }
                      ]
                    }
                  },
                  "classifications": []
                },
                "41": {
                  "objects": {
                    "cltd97cag01tp14ysz3my848t": {
                      "feature_id": "cltd97cag01tp14ysz3my848t",
                      "feature_schema_id": "cltd977np037n07x5d1buckz9",
                      "name": "line_video_frame",
                      "value": "line_video_frame",
                      "annotation_kind": "VideoPolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 680.0,
                          "y": 100.0
                        },
                        {
                          "x": 100.0,
                          "y": 190.0
                        }
                      ]
                    }
                  },
                  "classifications": []
                },
                "42": {
                  "objects": {
                    "cltd97cag01tp14ysz3my848t": {
                      "feature_id": "cltd97cag01tp14ysz3my848t",
                      "feature_schema_id": "cltd977np037n07x5d1buckz9",
                      "name": "line_video_frame",
                      "value": "line_video_frame",
                      "annotation_kind": "VideoPolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 680.0,
                          "y": 100.0
                        },
                        {
                          "x": 100.0,
                          "y": 190.0
                        }
                      ]
                    }
                  },
                  "classifications": []
                },
                "43": {
                  "objects": {
                    "cltd97cag01tp14ysz3my848t": {
                      "feature_id": "cltd97cag01tp14ysz3my848t",
                      "feature_schema_id": "cltd977np037n07x5d1buckz9",
                      "name": "line_video_frame",
                      "value": "line_video_frame",
                      "annotation_kind": "VideoPolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 680.0,
                          "y": 100.0
                        },
                        {
                          "x": 100.0,
                          "y": 190.0
                        }
                      ]
                    }
                  },
                  "classifications": []
                },
                "44": {
                  "objects": {
                    "cltd97cag01tp14ysz3my848t": {
                      "feature_id": "cltd97cag01tp14ysz3my848t",
                      "feature_schema_id": "cltd977np037n07x5d1buckz9",
                      "name": "line_video_frame",
                      "value": "line_video_frame",
                      "annotation_kind": "VideoPolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 680.0,
                          "y": 100.0
                        },
                        {
                          "x": 100.0,
                          "y": 190.0
                        }
                      ]
                    }
                  },
                  "classifications": []
                },
                "45": {
                  "objects": {
                    "cltd97cag01tp14ysz3my848t": {
                      "feature_id": "cltd97cag01tp14ysz3my848t",
                      "feature_schema_id": "cltd977np037n07x5d1buckz9",
                      "name": "line_video_frame",
                      "value": "line_video_frame",
                      "annotation_kind": "VideoPolyline",
                      "classifications": [],
                      "line": [
                        {
                          "x": 680.0,
                          "y": 100.0
                        },
                        {
                          "x": 100.0,
                          "y": 190.0
                        }
                      ]
                    }
                  },
                  "classifications": []
                },
                "1": {
                  "objects": {
                    "cltd97cah01u814ys0k41cttu": {
                      "feature_id": "cltd97cah01u814ys0k41cttu",
                      "feature_schema_id": "cltd977np037p07x55hu71x6k",
                      "name": "video_mask",
                      "value": "video_mask",
                      "annotation_kind": "VideoSegmentationMask",
                      "classifications": [],
                      "mask": {
                        "url": "https://api.labelbox.com/api/v1/projects/cltd977uu016y08xyggep8aiv/annotations/cltd97cah01u814ys0k41cttu/index/1/mask"
                      },
                      "composite_mask": {
                        "url": "https://api.labelbox.com/api/v1/tasks/clur7t5d501i707wfawrp050b/masks/cltd97a7h01tb14ysldafd9es/index/1",
                        "color_rgb": [
                          169,
                          248,
                          152
                        ]
                      }
                    },
                    "cltd97cah01u914ysy7t58lvy": {
                      "feature_id": "cltd97cah01u914ysy7t58lvy",
                      "feature_schema_id": "cltd977np037p07x55hu71x6k",
                      "name": "video_mask",
                      "value": "video_mask",
                      "annotation_kind": "VideoSegmentationMask",
                      "classifications": [],
                      "mask": {
                        "url": "https://api.labelbox.com/api/v1/projects/cltd977uu016y08xyggep8aiv/annotations/cltd97cah01u914ysy7t58lvy/index/1/mask"
                      },
                      "composite_mask": {
                        "url": "https://api.labelbox.com/api/v1/tasks/clur7t5d501i707wfawrp050b/masks/cltd97a7h01tb14ysldafd9es/index/1",
                        "color_rgb": [
                          227,
                          135,
                          126
                        ]
                      }
                    }
                  },
                  "classifications": []
                }
              },
              "segments": {
                "cltd97cag01td14ysxw2n8xkz": [
                  [
                    29,
                    35
                  ]
                ],
                "cltd97cag01te14ys1jdf1fku": [
                  [
                    29,
                    35
                  ]
                ],
                "cltd97cag01tg14ys171a3qqr": [
                  [
                    9,
                    15
                  ]
                ],
                "cltd97cag01th14ys6fyyxyk9": [
                  [
                    13,
                    19
                  ]
                ],
                "cltd97cag01ti14ys8wedf89d": [
                  [
                    10,
                    13
                  ]
                ],
                "cltd97cag01tm14ys8tm4x7yl": [
                  [
                    22,
                    27
                  ]
                ],
                "cltd97cag01tn14yspqge5yph": [
                  [
                    22,
                    27
                  ]
                ],
                "cltd97cag01to14yslakirofl": [
                  [
                    17,
                    17
                  ]
                ],
                "cltd97cag01tp14ysz3my848t": [
                  [
                    5,
                    20
                  ],
                  [
                    24,
                    45
                  ]
                ]
              },
              "key_frame_feature_map": {
                "cltd97cag01tc14yspm0lev2d": [
                  29,
                  35
                ],
                "cltd97cag01tp14ysz3my848t": [
                  24,
                  5,
                  20,
                  45
                ],
                "cltd97cag01tf14ysho6tfl7j": [
                  9,
                  15
                ],
                "cltd97cag01th14ys6fyyxyk9": [
                  13,
                  19
                ],
                "cltd97cag01ti14ys8wedf89d": [
                  13,
                  10
                ],
                "cltd97cag01to14yslakirofl": [
                  17
                ],
                "cltd97cag01tm14ys8tm4x7yl": [
                  22,
                  27
                ],
                "cltd97cag01tn14yspqge5yph": [
                  22,
                  27
                ],
                "cltd97cag01u514ys2bfg9vvo": [
                  24,
                  20
                ],
                "cltd97cah01u614ysvntxcyub": [
                  24,
                  20
                ],
                "cltd97cah01u714ysbqbvqs6b": [
                  24,
                  20
                ],
                "cltd97cah01ua14ysr6u1sh7w": [
                  26
                ],
                "cltd97cah01u814ys0k41cttu": [
                  1
                ],
                "cltd97cah01u914ysy7t58lvy": [
                  1
                ]
              },
              "classifications": [
                {
                  "feature_id": "cltd97cag01tq14ysb4m3nyjo",
                  "feature_schema_id": "cltd977nq039107x5dsml2knm",
                  "name": "checklist_class_global",
                  "value": "checklist_class_global",
                  "checklist_answers": [
                    {
                      "feature_id": "cltd97cag01tr14ysozjbx0c6",
                      "feature_schema_id": "cltd977nq039207x5ei1605s1",
                      "name": "first_checklist_answer",
                      "value": "first_checklist_answer",
                      "classifications": []
                    },
                    {
                      "feature_id": "cltd97cag01ts14ys4vvxihnw",
                      "feature_schema_id": "cltd977nq039407x50qs3g61l",
                      "name": "second_checklist_answer",
                      "value": "second_checklist_answer",
                      "classifications": []
                    }
                  ]
                },
                {
                  "feature_id": "cltd97cag01tt14yscmqh2izy",
                  "feature_schema_id": "cltd977nq038v07x57qz18lnj",
                  "name": "radio_class_global",
                  "value": "radio_class_global",
                  "radio_answer": {
                    "feature_id": "cltd97cag01tu14ysq046ad6p",
                    "feature_schema_id": "cltd977nq038w07x5bxw3exp9",
                    "name": "first_radio_answer",
                    "value": "first_radio_answer",
                    "classifications": []
                  }
                },
                {
                  "feature_id": "cltd97cag01tv14ys1y3u4jre",
                  "feature_schema_id": "cltd977nq038n07x52t8s0j1f",
                  "name": "nested_checklist_question",
                  "value": "nested_checklist_question",
                  "checklist_answers": [
                    {
                      "feature_id": "cltd97cag01tw14ysieenbrzy",
                      "feature_schema_id": "cltd977nq038o07x528j39vc4",
                      "name": "first_checklist_answer",
                      "value": "first_checklist_answer",
                      "classifications": [
                        {
                          "feature_id": "cltd97cag01tx14ysaqs15mjh",
                          "feature_schema_id": "cltd977nq038p07x5bfuf7v56",
                          "name": "sub_checklist_question",
                          "value": "sub_checklist_question",
                          "checklist_answers": [
                            {
                              "feature_id": "cltd97cag01ty14yshujgka3h",
                              "feature_schema_id": "cltd977nq038q07x52m1h32mj",
                              "name": "first_sub_checklist_answer",
                              "value": "first_sub_checklist_answer",
                              "classifications": []
                            }
                          ]
                        }
                      ]
                    }
                  ]
                },
                {
                  "feature_id": "cltd97cag01tz14ys73p2w3oa",
                  "feature_schema_id": "cltd977nq038f07x5bgrc2fm6",
                  "name": "nested_radio_question",
                  "value": "nested_radio_question",
                  "radio_answer": {
                    "feature_id": "cltd97cag01u014ysho9dljue",
                    "feature_schema_id": "cltd977nq038g07x50ewqbbi3",
                    "name": "first_radio_answer",
                    "value": "first_radio_answer",
                    "classifications": [
                      {
                        "feature_id": "cltd97cag01u114ysktcruoro",
                        "feature_schema_id": "cltd977nq038h07x59zx07wfe",
                        "name": "sub_radio_question",
                        "value": "sub_radio_question",
                        "radio_answer": {
                          "feature_id": "cltd97cag01u214ysxi71u4qi",
                          "feature_schema_id": "cltd977nq038i07x5et0717w6",
                          "name": "first_sub_radio_answer",
                          "value": "first_sub_radio_answer",
                          "classifications": []
                        }
                      }
                    ]
                  }
                },
                {
                  "feature_id": "cltd97cag01u314ysx1h72f0h",
                  "feature_schema_id": "cltd977nq039707x535ufeo39",
                  "name": "free_text",
                  "value": "free_text",
                  "text_answer": {
                    "content": "sample text"
                  }
                }
              ]
            }
          }
        ],
        "project_details": {
          "ontology_id": "cltd977mx037i07x52amua8iw",
          "task_name": "Done",
          "batch_id": "1026dd60-da52-11ee-9774-cfce344359c8",
          "batch_name": "first-batch-video-demo2",
          "workflow_status": "DONE",
          "priority": 5,
          "consensus_expected_label_count": 1,
          "workflow_history": []
        },
        "project_tags": []
      }
    }
  }
  ```
</CodeGroup>

## Sample model run export

<CodeGroup>
  ```json JSON expandable theme={"theme":"css-variables"}
  {
    "data_row": {
      "id": "cldtd2nai107407y5979xbu97",
      "global_key": "3548cfa1-1a42-40f9-a246-2aa7c696dcbe",
      "row_data": "https://storage.googleapis.com/labelbox-datasets/video-sample-data/sample-video-2.mp4",
      "details": {
        "dataset_id": "cldtd2mox11dz07wh3gkk9awx",
        "dataset_name": "video_prediction_demo",
        "created_at": "2023-02-06T22:04:54.000+00:00",
        "updated_at": "2023-02-06T22:04:54.000+00:00",
        "created_by": "[email protected]"
      }
    },
    "media_attributes": {
      "height": 1080,
      "width": 1920,
      "mime_type": "video/mp4",
      "frame_rate": 10,
      "frame_count": 100
    },
    "attachments": [],
    "metadata_fields": [],
    "experiments": {
      "a088408e-81af-0152-f2ca-0b870c7a2bfa": {
        "name": "video_model_run_8d7dbe3d-30e0-4c9c-becd-26afb2386bc9",
        "runs": {
          "a088408e-ce36-09c1-cbd9-d43c677f7773": {
            "name": "iteration 1",
            "run_data_row_id": "999a247e-0b57-415e-af7e-d3a87cc6559b",
            "labels": [
              {
                "label_kind": "Video",
                "version": "1.0.0",
                "id": "cldteelzh08us1331q4pzj2ke",
                "annotations": {
                  "frames": {
                    "17": {
                      "objects": {
                        "12a89fb9-73e4-4c5e-93c1-a8727e51c122": {
                          "feature_id": "12a89fb9-73e4-4c5e-93c1-a8727e51c122",
                          "feature_schema_id": "cldtd2nmm107i07y53ii0d6ly",
                          "name": "point_video",
                          "value": "point_video",
                          "annotation_kind": "VideoPoint",
                          "classifications": [],
                          "point": {
                            "x": 660.134,
                            "y": 407.926
                          }
                        }
                      },
                      "classifications": []
                    },
                    "5": {
                      "objects": {
                        "878b4842-d57b-43e4-8327-a0c5295f2a84": {
                          "feature_id": "878b4842-d57b-43e4-8327-a0c5295f2a84",
                          "feature_schema_id": "cldtd2nmm107k07y577q807az",
                          "name": "line_video_frame",
                          "value": "line_video_frame",
                          "annotation_kind": "VideoPolyline",
                          "classifications": [],
                          "line": [
                            {
                              "x": 680.0,
                              "y": 100.0
                            },
                            {
                              "x": 100.0,
                              "y": 190.0
                            },
                            {
                              "x": 190.0,
                              "y": 220.0
                            }
                          ]
                        },
                        "9cd6cd8e-0511-42a9-bc2f-726cf5c8f983": {
                          "feature_id": "9cd6cd8e-0511-42a9-bc2f-726cf5c8f983",
                          "feature_schema_id": "cldtd2nml107a07y51qtydeoj",
                          "name": "bbox_video",
                          "value": "bbox_video",
                          "annotation_kind": "VideoBoundingBox",
                          "classifications": [],
                          "bounding_box": {
                            "top": 146.0,
                            "left": 98.0,
                            "height": 382.0,
                            "width": 341.0
                          }
                        }
                      },
                      "classifications": []
                    },
                    "12": {
                      "objects": {
                        "878b4842-d57b-43e4-8327-a0c5295f2a84": {
                          "feature_id": "878b4842-d57b-43e4-8327-a0c5295f2a84",
                          "feature_schema_id": "cldtd2nmm107k07y577q807az",
                          "name": "line_video_frame",
                          "value": "line_video_frame",
                          "annotation_kind": "VideoPolyline",
                          "classifications": [],
                          "line": [
                            {
                              "x": 680.0,
                              "y": 280.0
                            },
                            {
                              "x": 300.0,
                              "y": 380.0
                            },
                            {
                              "x": 400.0,
                              "y": 460.0
                            }
                          ]
                        }
                      },
                      "classifications": []
                    },
                    "20": {
                      "objects": {
                        "878b4842-d57b-43e4-8327-a0c5295f2a84": {
                          "feature_id": "878b4842-d57b-43e4-8327-a0c5295f2a84",
                          "feature_schema_id": "cldtd2nmm107k07y577q807az",
                          "name": "line_video_frame",
                          "value": "line_video_frame",
                          "annotation_kind": "VideoPolyline",
                          "classifications": [],
                          "line": [
                            {
                              "x": 680.0,
                              "y": 180.0
                            },
                            {
                              "x": 100.0,
                              "y": 200.0
                            },
                            {
                              "x": 200.0,
                              "y": 260.0
                            }
                          ]
                        }
                      },
                      "classifications": []
                    },
                    "24": {
                      "objects": {
                        "878b4842-d57b-43e4-8327-a0c5295f2a84": {
                          "feature_id": "878b4842-d57b-43e4-8327-a0c5295f2a84",
                          "feature_schema_id": "cldtd2nmm107k07y577q807az",
                          "name": "line_video_frame",
                          "value": "line_video_frame",
                          "annotation_kind": "VideoPolyline",
                          "classifications": [],
                          "line": [
                            {
                              "x": 300.0,
                              "y": 310.0
                            },
                            {
                              "x": 330.0,
                              "y": 430.0
                            }
                          ]
                        }
                      },
                      "classifications": []
                    },
                    "45": {
                      "objects": {
                        "878b4842-d57b-43e4-8327-a0c5295f2a84": {
                          "feature_id": "878b4842-d57b-43e4-8327-a0c5295f2a84",
                          "feature_schema_id": "cldtd2nmm107k07y577q807az",
                          "name": "line_video_frame",
                          "value": "line_video_frame",
                          "annotation_kind": "VideoPolyline",
                          "classifications": [],
                          "line": [
                            {
                              "x": 600.0,
                              "y": 810.0
                            },
                            {
                              "x": 900.0,
                              "y": 930.0
                            }
                          ]
                        }
                      },
                      "classifications": []
                    },
                    "1": {
                      "objects": {
                        "9cd6cd8e-0511-42a9-bc2f-726cf5c8f983": {
                          "feature_id": "9cd6cd8e-0511-42a9-bc2f-726cf5c8f983",
                          "feature_schema_id": "cldtd2nml107a07y51qtydeoj",
                          "name": "bbox_video",
                          "value": "bbox_video",
                          "annotation_kind": "VideoBoundingBox",
                          "classifications": [],
                          "bounding_box": {
                            "top": 146.0,
                            "left": 98.0,
                            "height": 382.0,
                            "width": 341.0
                          }
                        }
                      },
                      "classifications": []
                    }
                  },
                  "segments": {
                    "12a89fb9-73e4-4c5e-93c1-a8727e51c122": [
                      [
                        17,
                        17
                      ]
                    ],
                    "878b4842-d57b-43e4-8327-a0c5295f2a84": [
                      [
                        5,
                        5
                      ],
                      [
                        12,
                        12
                      ],
                      [
                        20,
                        20
                      ],
                      [
                        24,
                        24
                      ],
                      [
                        45,
                        45
                      ]
                    ],
                    "9cd6cd8e-0511-42a9-bc2f-726cf5c8f983": [
                      [
                        5,
                        5
                      ],
                      [
                        1,
                        1
                      ]
                    ]
                  },
                  "key_frame_feature_map": {
                    "12a89fb9-73e4-4c5e-93c1-a8727e51c122": [
                      17
                    ],
                    "878b4842-d57b-43e4-8327-a0c5295f2a84": [
                      5,
                      12,
                      20,
                      24,
                      45
                    ],
                    "9cd6cd8e-0511-42a9-bc2f-726cf5c8f983": [
                      5,
                      1
                    ]
                  },
                  "classifications": [
                    {
                      "feature_id": "3644a35a-8aaa-4905-a2d2-9abac1f3dacb",
                      "feature_schema_id": "cldtd2nmm107s07y57ddy5wnn",
                      "name": "checklist_question",
                      "value": "checklist_question",
                      "checklist_answers": [
                        {
                          "feature_id": "86805980-7632-4dd0-9669-299730927326",
                          "feature_schema_id": "cldtd2nmm107v07y5cmixhywh",
                          "name": "second_checklist_answer",
                          "value": "second_checklist_answer",
                          "classifications": []
                        },
                        {
                          "feature_id": "c5bfb5c1-aa9a-4a6a-b671-be7c20df4b19",
                          "feature_schema_id": "cldtd2nmm107t07y5cqdw94z7",
                          "name": "first_checklist_answer",
                          "value": "first_checklist_answer",
                          "classifications": []
                        }
                      ]
                    },
                    {
                      "feature_id": "3d302a82-cc91-41d9-8c69-0f1e2e66ea71",
                      "feature_schema_id": "cldtd2nmm107m07y5h45sg6av",
                      "name": "radio_question",
                      "value": "radio_question",
                      "radio_answer": {
                        "feature_id": "dab3bcfa-6c44-4f28-945b-1bc324edfab1",
                        "feature_schema_id": "cldtd2nmm107p07y542ff3hvd",
                        "name": "second_radio_answer",
                        "value": "second_radio_answer",
                        "classifications": []
                      }
                    },
                    {
                      "feature_id": "6dede674-18a6-4b95-abdd-13b3713f994a",
                      "feature_schema_id": "cldtd2nmm108607y5ghou4ma6",
                      "name": "nested_checklist_question",
                      "value": "nested_checklist_question",
                      "checklist_answers": [
                        {
                          "feature_id": "9f748713-ffb5-44db-a16e-396676dab6bc",
                          "feature_schema_id": "cldtd2nmm108707y50shn5996",
                          "name": "first_checklist_answer",
                          "value": "first_checklist_answer",
                          "classifications": [
                            {
                              "feature_id": "ca1f2173-38c6-4c23-96ba-8a3feb54e2c2",
                              "feature_schema_id": "cldtd2nmm108807y52ps9d5m4",
                              "name": "sub_checklist_question",
                              "value": "sub_checklist_question",
                              "checklist_answers": [
                                {
                                  "feature_id": "2181d943-8a6a-4aef-aae0-4c28146e9008",
                                  "feature_schema_id": "cldtd2nmm108907y5269ua7bz",
                                  "name": "first_sub_checklist_answer",
                                  "value": "first_sub_checklist_answer",
                                  "classifications": []
                                }
                              ]
                            }
                          ]
                        }
                      ]
                    },
                    {
                      "feature_id": "b72dd664-93fc-46d1-8bd8-8dfe1fd2d7ef",
                      "feature_schema_id": "cldtd2nmm107y07y5fvhkb6c7",
                      "name": "nested_radio_question",
                      "value": "nested_radio_question",
                      "radio_answer": {
                        "feature_id": "5fc17086-601e-42e5-a5b1-a28eb96841ab",
                        "feature_schema_id": "cldtd2nmm107z07y59j8n62ym",
                        "name": "first_radio_answer",
                        "value": "first_radio_answer",
                        "classifications": [
                          {
                            "feature_id": "a74e9958-345f-4b42-925d-9d5938011cf6",
                            "feature_schema_id": "cldtd2nmm108007y5gs1y5eqv",
                            "name": "sub_radio_question",
                            "value": "sub_radio_question",
                            "radio_answer": {
                              "feature_id": "734a2cf1-571d-4855-869e-d92ff20a4cb4",
                              "feature_schema_id": "cldtd2nmm108107y53dvreuv2",
                              "name": "first_sub_radio_answer",
                              "value": "first_sub_radio_answer",
                              "classifications": []
                            }
                          }
                        ]
                      }
                    }
                  ]
                }
              }
            ],
            "predictions": [
              {
                "label_kind": "Video",
                "version": "1.0.0",
                "id": "cldteelzh08us1331q4pzj2ke",
                "annotations": {
                  "frames": {
                    "5": {
                      "objects": {
                        "1306f719-d588-4154-8335-792347bfdee2": {
                          "feature_id": "1306f719-d588-4154-8335-792347bfdee2",
                          "feature_schema_id": "cldtd2nmm107k07y577q807az",
                          "name": "line_video_frame",
                          "value": "line_video_frame",
                          "annotation_kind": "VideoPolyline",
                          "classifications": [],
                          "line": [
                            {
                              "x": 680.0,
                              "y": 100.0
                            },
                            {
                              "x": 100.0,
                              "y": 190.0
                            },
                            {
                              "x": 190.0,
                              "y": 220.0
                            }
                          ]
                        },
                        "a505f426-5a2e-4dea-b4fe-ab913e1d2a99": {
                          "feature_id": "a505f426-5a2e-4dea-b4fe-ab913e1d2a99",
                          "feature_schema_id": "cldtd2nml107a07y51qtydeoj",
                          "name": "bbox_video",
                          "value": "bbox_video",
                          "annotation_kind": "VideoBoundingBox",
                          "classifications": [],
                          "bounding_box": {
                            "top": 146.0,
                            "left": 98.0,
                            "height": 382.0,
                            "width": 341.0
                          }
                        }
                      },
                      "classifications": []
                    },
                    "12": {
                      "objects": {
                        "1306f719-d588-4154-8335-792347bfdee2": {
                          "feature_id": "1306f719-d588-4154-8335-792347bfdee2",
                          "feature_schema_id": "cldtd2nmm107k07y577q807az",
                          "name": "line_video_frame",
                          "value": "line_video_frame",
                          "annotation_kind": "VideoPolyline",
                          "classifications": [],
                          "line": [
                            {
                              "x": 680.0,
                              "y": 280.0
                            },
                            {
                              "x": 300.0,
                              "y": 380.0
                            },
                            {
                              "x": 400.0,
                              "y": 460.0
                            }
                          ]
                        }
                      },
                      "classifications": []
                    },
                    "20": {
                      "objects": {
                        "1306f719-d588-4154-8335-792347bfdee2": {
                          "feature_id": "1306f719-d588-4154-8335-792347bfdee2",
                          "feature_schema_id": "cldtd2nmm107k07y577q807az",
                          "name": "line_video_frame",
                          "value": "line_video_frame",
                          "annotation_kind": "VideoPolyline",
                          "classifications": [],
                          "line": [
                            {
                              "x": 680.0,
                              "y": 180.0
                            },
                            {
                              "x": 100.0,
                              "y": 200.0
                            },
                            {
                              "x": 200.0,
                              "y": 260.0
                            }
                          ]
                        }
                      },
                      "classifications": []
                    },
                    "24": {
                      "objects": {
                        "1306f719-d588-4154-8335-792347bfdee2": {
                          "feature_id": "1306f719-d588-4154-8335-792347bfdee2",
                          "feature_schema_id": "cldtd2nmm107k07y577q807az",
                          "name": "line_video_frame",
                          "value": "line_video_frame",
                          "annotation_kind": "VideoPolyline",
                          "classifications": [],
                          "line": [
                            {
                              "x": 300.0,
                              "y": 310.0
                            },
                            {
                              "x": 330.0,
                              "y": 430.0
                            }
                          ]
                        }
                      },
                      "classifications": []
                    },
                    "45": {
                      "objects": {
                        "1306f719-d588-4154-8335-792347bfdee2": {
                          "feature_id": "1306f719-d588-4154-8335-792347bfdee2",
                          "feature_schema_id": "cldtd2nmm107k07y577q807az",
                          "name": "line_video_frame",
                          "value": "line_video_frame",
                          "annotation_kind": "VideoPolyline",
                          "classifications": [],
                          "line": [
                            {
                              "x": 600.0,
                              "y": 810.0
                            },
                            {
                              "x": 900.0,
                              "y": 930.0
                            }
                          ]
                        }
                      },
                      "classifications": []
                    },
                    "17": {
                      "objects": {
                        "89b4636f-08cc-4fa2-832b-18bcda636722": {
                          "feature_id": "89b4636f-08cc-4fa2-832b-18bcda636722",
                          "feature_schema_id": "cldtd2nmm107i07y53ii0d6ly",
                          "name": "point_video",
                          "value": "point_video",
                          "annotation_kind": "VideoPoint",
                          "classifications": [],
                          "point": {
                            "x": 660.134,
                            "y": 407.926
                          }
                        }
                      },
                      "classifications": []
                    },
                    "1": {
                      "objects": {
                        "a505f426-5a2e-4dea-b4fe-ab913e1d2a99": {
                          "feature_id": "a505f426-5a2e-4dea-b4fe-ab913e1d2a99",
                          "feature_schema_id": "cldtd2nml107a07y51qtydeoj",
                          "name": "bbox_video",
                          "value": "bbox_video",
                          "annotation_kind": "VideoBoundingBox",
                          "classifications": [],
                          "bounding_box": {
                            "top": 146.0,
                            "left": 98.0,
                            "height": 382.0,
                            "width": 341.0
                          }
                        }
                      },
                      "classifications": []
                    }
                  },
                  "segments": {
                    "1306f719-d588-4154-8335-792347bfdee2": [
                      [
                        5,
                        5
                      ],
                      [
                        12,
                        12
                      ],
                      [
                        20,
                        20
                      ],
                      [
                        24,
                        24
                      ],
                      [
                        45,
                        45
                      ]
                    ],
                    "a505f426-5a2e-4dea-b4fe-ab913e1d2a99": [
                      [
                        5,
                        5
                      ],
                      [
                        1,
                        1
                      ]
                    ],
                    "89b4636f-08cc-4fa2-832b-18bcda636722": [
                      [
                        17,
                        17
                      ]
                    ]
                  },
                  "key_frame_feature_map": {
                    "1306f719-d588-4154-8335-792347bfdee2": [
                      5,
                      12,
                      20,
                      24,
                      45
                    ],
                    "a505f426-5a2e-4dea-b4fe-ab913e1d2a99": [
                      5,
                      1
                    ],
                    "89b4636f-08cc-4fa2-832b-18bcda636722": [
                      17
                    ]
                  },
                  "classifications": [
                    {
                      "feature_id": "5deccc93-922a-4fcd-bf70-0bd90249ed65",
                      "feature_schema_id": "cldtd2nmm108607y5ghou4ma6",
                      "name": "nested_checklist_question",
                      "value": "nested_checklist_question",
                      "checklist_answers": [
                        {
                          "feature_id": "3d9f3952-c303-42ed-91ba-b112927029c9",
                          "feature_schema_id": "cldtd2nmm108707y50shn5996",
                          "name": "first_checklist_answer",
                          "value": "first_checklist_answer",
                          "classifications": [
                            {
                              "feature_id": "fe4d0f5f-99e6-4abd-9401-b0ad04fe13ed",
                              "feature_schema_id": "cldtd2nmm108807y52ps9d5m4",
                              "name": "sub_checklist_question",
                              "value": "sub_checklist_question",
                              "checklist_answers": [
                                {
                                  "feature_id": "05fca900-787a-4774-a011-0b6931d9a012",
                                  "feature_schema_id": "cldtd2nmm108907y5269ua7bz",
                                  "name": "first_sub_checklist_answer",
                                  "value": "first_sub_checklist_answer",
                                  "classifications": []
                                }
                              ]
                            }
                          ]
                        }
                      ]
                    },
                    {
                      "feature_id": "7a92c014-48e1-4367-8ff9-4ec146d9968e",
                      "feature_schema_id": "cldtd2nmm107y07y5fvhkb6c7",
                      "name": "nested_radio_question",
                      "value": "nested_radio_question",
                      "radio_answer": {
                        "feature_id": "9f35c500-be1d-4693-ae9a-71507931d558",
                        "feature_schema_id": "cldtd2nmm107z07y59j8n62ym",
                        "name": "first_radio_answer",
                        "value": "first_radio_answer",
                        "classifications": [
                          {
                            "feature_id": "d1269040-bf2c-4e8c-8fa6-58c3d6a7e343",
                            "feature_schema_id": "cldtd2nmm108007y5gs1y5eqv",
                            "name": "sub_radio_question",
                            "value": "sub_radio_question",
                            "radio_answer": {
                              "feature_id": "2fecd9bc-98ca-49bd-8db4-cb9d1be6a63d",
                              "feature_schema_id": "cldtd2nmm108107y53dvreuv2",
                              "name": "first_sub_radio_answer",
                              "value": "first_sub_radio_answer",
                              "classifications": []
                            }
                          }
                        ]
                      }
                    },
                    {
                      "feature_id": "95085369-6add-4ce1-9641-d6b690490eb3",
                      "feature_schema_id": "cldtd2nmm107s07y57ddy5wnn",
                      "name": "checklist_question",
                      "value": "checklist_question",
                      "checklist_answers": [
                        {
                          "feature_id": "53a98fa1-19ea-4541-b42f-8df7f7e3b2fc",
                          "feature_schema_id": "cldtd2nmm107v07y5cmixhywh",
                          "name": "second_checklist_answer",
                          "value": "second_checklist_answer",
                          "classifications": []
                        },
                        {
                          "feature_id": "6383f391-7ca6-4d99-a855-cdc2e2afa811",
                          "feature_schema_id": "cldtd2nmm107t07y5cqdw94z7",
                          "name": "first_checklist_answer",
                          "value": "first_checklist_answer",
                          "classifications": []
                        }
                      ]
                    },
                    {
                      "feature_id": "af2f95fe-6ce6-4e3a-83a8-fd11b1ae0cdc",
                      "feature_schema_id": "cldtd2nmm107m07y5h45sg6av",
                      "name": "radio_question",
                      "value": "radio_question",
                      "radio_answer": {
                        "feature_id": "71d910d5-df6b-4def-b68e-caced48dd25c",
                        "feature_schema_id": "cldtd2nmm107p07y542ff3hvd",
                        "name": "second_radio_answer",
                        "value": "second_radio_answer",
                        "classifications": []
                      }
                    }
                  ]
                }
              }
            ]
          }
        }
      }
    },
    "models": {},
    "embeddings": [
      {
        "id": "c400000000000000000000000",
        "name": "Video Embedding V2 (Google MultiModal)",
        "dimensions": 1408,
        "is_custom": false,
        "values": [
          {
            "value": [
              -0.00489852577,
              0.0287129097 ...
                        ]
          }
        ]
      }
    ]
  }
  ```
</CodeGroup>


# Exporting to CSV
Source: https://docs.labelbox.com/horizon/developer-guides/exporting-to-csv

How-to guide on exporting Labelbox data to a CSV or Pandas-friendly format.

<CardGroup>
  <Card title="Open In Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/exports/exporting_to_csv.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/exports/exporting_to_csv.ipynb" />
</CardGroup>

Creating a CSV file for your Labelbox data can be difficult, especially if you want to include information on the annotations associated with your label. This guide will provide examples and show the process to get your Labelbox export to a CSV or [Pandas](https://pandas.pydata.org/) friendly format.

<Info>
  ### Information

  This guide will assume you have a basic understanding of Python data structures and interacting with Labelbox exports.
</Info>

## Before you start

The below imports are needed to use the code examples in this section.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  import labelbox as lb
  import labelbox.types as lb_types
  import uuid
  from pprint import pprint
  import csv
  import pandas as pd
  ```
</CodeGroup>

Replace the value of `API_KEY` with a valid [API key](/horizon/developer-guides/create-api-key) to connect to the Labelbox client.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  API_KEY = None
  client = lb.Client(API_KEY)
  ```
</CodeGroup>

## Create or select example project

The below steps will set up a project that can be used for this demo. Please feel free to delete the code block below and uncomment the code block that fetches your own project directly. For more information on this setup, visit our [quickstart](/horizon/developer-guides/quick-start) guide.

### Create Project

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Create dataset with image data row
  global_key = str(uuid.uuid4())

  test_img_url = {
      "row_data":
          "https://storage.googleapis.com/labelbox-datasets/image_sample_data/2560px-Kitano_Street_Kobe01s5s4110.jpeg",
      "global_key":
          global_key,
  }

  dataset = client.create_dataset(name="image-demo-dataset")
  task = dataset.create_data_rows([test_img_url])
  task.wait_till_done()
  print("Errors:", task.errors)
  print("Failed data rows:", task.failed_data_rows)

  # Create ontology
  ontology_builder = lb.OntologyBuilder(
      classifications=[  # List of Classification objects
          lb.Classification(
              class_type=lb.Classification.Type.RADIO,
              name="radio_question",
              options=[
                  lb.Option(value="first_radio_answer"),
                  lb.Option(value="second_radio_answer"),
              ],
          ),
          lb.Classification(
              class_type=lb.Classification.Type.CHECKLIST,
              name="checklist_question",
              options=[
                  lb.Option(value="first_checklist_answer"),
                  lb.Option(value="second_checklist_answer"),
              ],
          ),
          lb.Classification(class_type=lb.Classification.Type.TEXT,
                            name="free_text"),
          lb.Classification(
              class_type=lb.Classification.Type.RADIO,
              name="nested_radio_question",
              options=[
                  lb.Option(
                      "first_radio_answer",
                      options=[
                          lb.Classification(
                              class_type=lb.Classification.Type.RADIO,
                              name="sub_radio_question",
                              options=[lb.Option("first_sub_radio_answer")],
                          )
                      ],
                  )
              ],
          ),
      ],
      tools=[  # List of Tool objects
          lb.Tool(tool=lb.Tool.Type.BBOX, name="bounding_box"),
          lb.Tool(
              tool=lb.Tool.Type.BBOX,
              name="bbox_with_radio_subclass",
              classifications=[
                  lb.Classification(
                      class_type=lb.Classification.Type.RADIO,
                      name="sub_radio_question",
                      options=[lb.Option(value="tool_first_sub_radio_answer")],
                  ),
              ],
          ),
      ],
  )

  ontology = client.create_ontology(
      "Image CSV Demo Ontology",
      ontology_builder.asdict(),
      media_type=lb.MediaType.Image,
  )

  # Set up project and connect ontology
  project = client.create_project(name="Image Annotation Import Demo",
                                  media_type=lb.MediaType.Image)
  project.connect_ontology(ontology)

  # Send data row towards our project
  batch = project.create_batch(
      "image-demo-batch",
      global_keys=[
          global_key
      ],  # paginated collection of data row objects, list of data row ids or global keys
      priority=1,
  )

  print(f"Batch: {batch}")

  # Create a label and imported it towards our project
  radio_annotation = lb_types.ClassificationAnnotation(
      name="radio_question",
      value=lb_types.Radio(answer=lb_types.ClassificationAnswer(
          name="second_radio_answer")),
  )
  checklist_annotation = lb_types.ClassificationAnnotation(
      name="checklist_question",
      value=lb_types.Checklist(answer=[
          lb_types.ClassificationAnswer(name="first_checklist_answer"),
          lb_types.ClassificationAnswer(name="second_checklist_answer"),
      ]),
  )
  text_annotation = lb_types.ClassificationAnnotation(
      name="free_text",
      value=lb_types.Text(answer="sample text"),
  )
  nested_radio_annotation = lb_types.ClassificationAnnotation(
      name="nested_radio_question",
      value=lb_types.Radio(answer=lb_types.ClassificationAnswer(
          name="first_radio_answer",
          classifications=[
              lb_types.ClassificationAnnotation(
                  name="sub_radio_question",
                  value=lb_types.Radio(answer=lb_types.ClassificationAnswer(
                      name="first_sub_radio_answer")),
              )
          ],
      )),
  )
  bbox_annotation = lb_types.ObjectAnnotation(
      name="bounding_box",
      value=lb_types.Rectangle(
          start=lb_types.Point(x=1690, y=977),
          end=lb_types.Point(x=1915, y=1307),
      ),
  )
  bbox_with_radio_subclass_annotation = lb_types.ObjectAnnotation(
      name="bbox_with_radio_subclass",
      value=lb_types.Rectangle(
          start=lb_types.Point(x=541, y=933),  # x = left, y = top
          end=lb_types.Point(x=871, y=1124),  # x= left + width , y = top + height
      ),
      classifications=[
          lb_types.ClassificationAnnotation(
              name="sub_radio_question",
              value=lb_types.Radio(answer=lb_types.ClassificationAnswer(
                  name="tool_first_sub_radio_answer")),
          )
      ],
  )

  label = []
  annotations = [
      radio_annotation,
      nested_radio_annotation,
      checklist_annotation,
      text_annotation,
      bbox_annotation,
      bbox_with_radio_subclass_annotation,
  ]

  label.append(
      lb_types.Label(data={"global_key": global_key}, annotations=annotations))

  upload_job = lb.LabelImport.create_from_objects(
      client=client,
      project_id=project.uid,
      name="label_import_job" + str(uuid.uuid4()),
      labels=label,
  )

  upload_job.wait_until_done()
  print("Errors:", upload_job.errors)
  print("Status of uploads: ", upload_job.statuses)
  ```
</CodeGroup>

### Select project

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # PROJECT_ID = None
  # project = client.get_project(PROJECT_ID)
  ```
</CodeGroup>

## CSV format overview

To convert our Labelbox JSON data to a format more CSV-friendly, we must first define the needed structure of our JSON. A common format that is versatile for both the built-in Python CSV writer and Pandas is as follows:

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  [
      {"<column_1>":"<answer_1", "<column_2>":"<answer_2" ..},

      {"<column_1>":"<answer_1", "<column_2>":"<answer_2" ..},
      ..

  ]
  ```
</CodeGroup>

Essentially, we need to get our JSON data towards a list of Python dictionaries, with each Python dictionary representing one row, each key representing a column, and each value being an individual cell of our CSV table. Once we have our data in this format, creating Pandas DataFrames or writing our CSV file is trivial. The tricky part is getting Labelbox to export JSON to this format.

## Labelbox JSON format

Labelbox JSON format is centralized at the individual data row of your export. This format allows expandability when things evolve and provides a centralized view of fields such as metadata or data row details. The main labels are located inside the project key and can be nested, making it difficult to parse. For complete samples of our project export format, visit our [export overview](/horizon/developer-guides/export-overview) page.

To get Labelbox export JSON format to our CSV format, we established we must do the following:

1. Establish our base data row columns (project\_id, data\_row\_id, global\_key etc)
2. Create our columns for label fields (label detail and annotations we care about)
3. Define our functions and strategy used to parse through our data
4. Setting up our main data row handler function
5. Export our data
6. Convert to our desired format

## Step 1: Establish our base columns

We first establish our base columns that represent individual data row details. Typically, this column's information can be received from within one or two levels of a Labelbox export per data row.

Please modify the below columns if you want to include more. You must update the code later in this guide to pick up any additional columns.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  data_row_base_columns = [
      "Data Row ID",
      "Global Key",
      "External ID",
      "Project ID",
  ]
  ```
</CodeGroup>

## Step 2: Create our columns for label fields

In this step, we define the label details base columns we want to include in our CSV. In this case, we will use the following:

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  label_base_columns = ["Label ID", "Created By", "Skipped"]
  ```
</CodeGroup>

We then must establish the annotations we want to include in our columns. The order of our list matters since that is the order in which our columns will be presented. You can approach getting the annotations in a list in a number of ways, including hard defining the columns. We will be mapping between feature\_schema\_ids and our column name. The reason for introducing this mapping is the annotation name can be the same in certain situations, but feature\_schema\_ids are completely unique. This also allows you to change the column names to something other than what is included in the ontology. In the code below, I will be recursively going through the ontology we created to get our feature\_schema\_ids and column names based on the names of the features. In the next step of this guide, we will provide more information on recursion in the context of parsing through JSON or Python dictionaries.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  def get_classification_features(classifications: list, class_list=[]) -> None:
      """Finds classification features inside an ontology recursively and returns them in a list"""
      for classification in classifications:
          if "name" in classification:
              class_list.append({
                  "feature_schema_id": classification["featureSchemaId"],
                  "column_name": classification["instructions"],
              })
          if "options" in classification:
              get_classification_features(classification["options"], class_list)
      return class_list


  def get_tool_features(tools: list) -> None:
      """Creates list of tool names from ontology"""
      tool_list = []
      for tool in tools:
          tool_list.append({
              "feature_schema_id": tool["featureSchemaId"],
              "column_name": tool["name"],
          })
          if "classifications" in tool:
              tool_list = get_classification_features(tool["classifications"],
                                                      tool_list)
      return tool_list
  ```
</CodeGroup>

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Get ontology from project and normalized towards python dictionary
  ontology = project.ontology().normalized

  class_annotation_columns = get_classification_features(
    ontology["classifications"])
  tool_annotation_columns = get_tool_features(ontology["tools"])
  ```
</CodeGroup>

## Step 3: Define our functions and strategy used to parse through our data

Now that our columns are defined, we must develop a strategy for navigating our export data. Review this [sample export](/horizon/developer-guides/export-image-annotations#sample-project-export) to follow along. While creating our columns, it is always best to first check if a key exists in your data row before populating a column. This is especially important for optional fields. In this demo, we will populate the value `None` for anything absent, resulting in a blank cell in our CSV.

### Data row detail base columns

The data row details can be accessed within a depth of one or two keys. Below is a function we will use to access the columns we defined. The parameters are the data row, the dictionary row used to make our list, and our base columns list.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  def get_base_data_row_columns(data_row: dict[str:str], csv_row: dict[str:str],
                                base_columns: list[str]) -> dict[str:str]:
      for base_column in base_columns:
          if base_column == "Data Row ID":
              csv_row[base_column] = data_row["data_row"]["id"]

          elif base_column == "Global Key":
              if ("global_key"
                      in data_row["data_row"]):  # Check if global key exists
                  csv_row[base_column] = data_row["data_row"]["global_key"]
              else:
                  csv_row[base_column] = (
                      None  # If global key does not exist on data row set cell to None. This will create a blank cell on your csv
                  )

          elif base_column == "External ID":
              if ("external_id"
                      in data_row["data_row"]):  # Check if external_id exists
                  csv_row[base_column] = data_row["data_row"]["external_id"]
              else:
                  csv_row[base_column] = (
                      None  # If external id does not exist on data row set cell to None. This will create a blank cell on your csv
                  )

          elif base_column == "Project ID":
              csv_row[base_column] = project.uid
      return csv_row
  ```
</CodeGroup>

### Label detail base columns

The label details are similar to data row details but exist at our export's label level. Later in the guide we will demonstrate how to get our exported data row at this level. The function below shows the process of obtaining the details we defined above. The parameters are the label, the dictionary row we will modify, and the label detail column list we created.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  def get_base_label_columns(label: dict[str:str], csv_row: dict[str:str],
                             label_base_columns: list[str]) -> dict[str:str]:
      for label_base_column in label_base_columns:
          if label_base_column == "Label ID":
              csv_row[label_base_column] = label["id"]

        elif label_base_column == "Created By":
            if (
                    "label_details" in label
            ):  # Check if label details is present. This field can be omitted in export.
                csv_row[label_base_column] = label[
                    "label_details"]["created_by"]
            else:
                csv_row[label_base_column] = None

        elif label_base_column == "Skipped":
            if (
                    "performance_details" in label
            ):  # Check if performance details are present. This field can be omitted in export.
                csv_row[label_base_column] = label["performance_details"][
                    "skipped"]
            else:
                csv_row[label_base_column] = None

    return csv_row
  ```
</CodeGroup>

### Label annotation columns

The label annotations are the final columns we will need to obtain. Obtaining these fields is more challenging than our approach for our detail columns. Suppose we attempt to obtain the fields with conditional statements and hard-defined paths. In that case, we will run into issues as each label can have annotations in different orders, at different depths, or not present. This will quickly create a mess, especially when we want our methods to work for multiple ontology. The best and cleanest way of obtaining these annotations inside our export data is through a recursive function.

#### Recursion

A recursive function can be defined as a routine that calls itself directly or indirectly. They solve problems by solving smaller instances of the same problem. This technique is commonly used in programming to solve problems that can be broken down into simpler, similar subproblems. Our sub-problem, in this case, is obtaining each individual annotation. A recursive function is divided into two components:

* **Base case:** This is our termination condition that prevents the function from calling itself indefinitely.
* **Recursive case:** The function calls itself with the modified arguments in the recursive case. The recursive case should move closer to the base case with each iteration.

For our example, our base case will be either the annotation exists on the label (return the value/answer), or it does not (return `None`). Our recursive case would be finding more classifications to parse.

In the code block below, I will highlight a few important details about our function. Essentially, we will be navigating through our JSON file by moving one classification key at a time until we find our annotation or, if everything has been searched, returning `None`, which will populate a blank cell on our CSV table.

#### Tools

Tools are not nested, but they can have nested classifications we will use or `get_feature_answers` function below to find the nested classification. Since tools are at the base level of a label and each tool has a different value key name, we will only be searching for bounding boxes for this tutorial. If you want to include other tools, reference our export guide for your data type and find the appropriate key to add on.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  def get_feature_answers(feature: str,
                          annotations: list[dict[str:str]]) -> None | str:
      """Returns answer of feature provided by navigating through a label's annotation list. Will return None if answer is not found.

      Args:
          feature (str): feature we are searching
          classifications (list[dict[str:str]]): annotation list that we will be searching for our feature with.

      Returns:
          None | str: The answer/value of the feature returns None if nothing is found
      """
      for annotation in annotations:
          print(annotation)
          if (annotation["feature_schema_id"] == feature["feature_schema_id"]
             ):  # Base conditions (found feature)
              if "text_answer" in annotation:
                  return annotation["text_answer"]["content"]
              if "radio_answer" in annotation:
                  return annotation["radio_answer"]["value"]
              if "checklist_answers" in annotation:
                  # Since classifications can have more then one answer. This is set up to combine all classifications separated by a comma. Feel free to modify.
                  return ", ".join([
                      check_list_ans["value"]
                      for check_list_ans in annotation["checklist_answers"]
                  ])
              if "bounding_box" in annotation:
                  return annotation["bounding_box"]
              # Add more tools here with similar pattern as above

          # Recursion cases (found more classifications to search through)
          if "radio_answer" in annotation:
              if len(annotation["radio_answer"]["classifications"]) > 0:
                  value = get_feature_answers(
                      feature, annotation["radio_answer"]["classifications"]
                  )  # Call function again return value if answer found
                  if value:
                      return value
          if "checklist_answers" in annotation:
              for checklist_ans in annotation["checklist_answers"]:
                  if len(checklist_ans["classifications"]) > 0:
                      value = get_feature_answers(
                          feature, checklist_ans["classifications"])
                      if value:
                          return value
          if ("classifications"
                  in annotation):  # case for if tool has classifications
              if len(annotation["classifications"]) > 0:
                  value = get_feature_answers(feature,
                                              annotation["classifications"])
                  if value:
                      return value

      return None  # Base case if searched through classifications and nothing was found (end of JSON). This can be omitted but included to visualize
  ```
</CodeGroup>

## Step 4: Setting up our main data row handler function

Before exporting, we need to set up our main data row handler. This function will be fed straight into our export. This function will put everything together and connect all the pieces. We will also be defining our global dictionary list that will be used to create our CSVs. The output parameter represents each data row.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  GLOBAL_CSV_LIST = []

  def main(output: lb.BufferedJsonConverterOutput):

    # Navigate to our label list
    labels = output.json["projects"][project.uid]["labels"]
    for label in labels:
        # Define our CSV "row"
        csv_row = dict()

        # Start with data row base columns
        csv_row = get_base_data_row_columns(output.json, csv_row,
                                            data_row_base_columns)

        # Add our label details
        csv_row = get_base_label_columns(label, csv_row, label_base_columns)

        # Add classification features
        for classification in class_annotation_columns:
            csv_row[classification["column_name"]] = get_feature_answers(
                classification, label["annotations"]["classifications"])

        # Add tools features
        for tool in tool_annotation_columns:
            csv_row[tool["column_name"]] = get_feature_answers(
                tool, label["annotations"]["objects"])

        # Append to global csv list
        GLOBAL_CSV_LIST.append(csv_row)
  ```
</CodeGroup>

## Step 5: Export our data

We are ready to export now that we have defined functions and strategies. Below, we export directly from our project and feed in the main function we created above.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Params required to obtain all fields we need
  params = {"performance_details": True, "label_details": True}

  export_task = project.export(params=params)
  export_task.wait_till_done()

  # Stream the export using a callback function
  def json_stream_handler(output: labelbox.BufferedJsonConverterOutput):
    print(output.json)

  export_task.get_buffered_stream(stream_type=labelbox.StreamType.RESULT).start(stream_handler=json_stream_handler)

  # Collect all exported data into a list
  export_json = [data_row.json for data_row in export_task.get_buffered_stream()]
  ```
</CodeGroup>

If everything went through correctly, you should see your `GLOBAL_CSV_LIST` printed out below with all your "rows" filled out.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  pprint(GLOBAL_CSV_LIST)
  ```
</CodeGroup>

## Step 6: Convert to our desired format

The hard part is now completed!🚀 Now that you have your export in a flattened format, you can easily convert to a CSV or a Pandas DataFrame!

### Option A: CSV writer

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  with open("file.csv", "w", newline="") as csvfile:
      # Columns
      fieldnames = (data_row_base_columns + label_base_columns +
                    [name["column_name"] for name in class_annotation_columns] +
                    [name["column_name"] for name in tool_annotation_columns])
      writer = csv.DictWriter(csvfile, fieldnames=fieldnames)

      writer.writeheader()

      for row in GLOBAL_CSV_LIST:
          writer.writerow(row)
  ```
</CodeGroup>

### Option B: Pandas DataFrame

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  columns = (data_row_base_columns + label_base_columns +
             [name["column_name"] for name in class_annotation_columns] +
             [name["column_name"] for name in tool_annotation_columns])
  pd.DataFrame(GLOBAL_CSV_LIST, columns=columns)
  ```
</CodeGroup>


# Features
Source: https://docs.labelbox.com/horizon/developer-guides/feature

Developer guide for creating and modifying features via the Python SDK.

View [Features](/horizon/guides/working-with-features) guide page for feature schema definitions and details.

## Create a feature

**Features** define the **tools** and **classifications** used to annotate your data. To create features for use in an ontology, use the `client.create_feature_schema()` method.

<CodeGroup>
  ```python Tool theme={"theme":"css-variables"}
  from labelbox import Tool

  # Defining tool features
  # required is an optional field defaults to False if not specified
  bbox_tool = Tool(tool=Tool.Type.BBOX, name="dog_box", required=True)
  poly_tool = Tool(tool=Tool.Type.POLYGON, name="dog_poly")
  seg_tool = Tool(tool=Tool.Type.SEGMENTATION, name="dog_seg")
  point_tool = Tool(tool=Tool.Type.POINT, name="dog_center")
  line_tool = Tool(tool=Tool.Type.LINE, name="dog_orientation")
  ner_tool = Tool(tool=Tool.Type.NER, name="dog_reference", required=True)

  # Creating feature schema for each defined tool
  feature_schema_bbox = client.create_feature_schema(bbox_tool.asdict())
  feature_schema_poly = client.create_feature_schema(poly_tool.asdict())
  feature_schema_seg= client.create_feature_schema(seg_tool.asdict())
  feature_schema_tool = client.create_feature_schema(point_tool.asdict())
  feature_schema_line = client.create_feature_schema(line_tool.asdict())
  feature_schema_ner = client.create_feature_schema(ner_tool.asdict())
  ```

  ```python Classification theme={"theme":"css-variables"}

  # Defining classification features
  # required is an optional field defaults to False if not specified
  from labelbox import Classification, Option

  text_classification = Classification(class_type=Classification.Type.TEXT,
                                       name="dog_name", required=True)
  radio_classification = Classification(class_type=Classification.Type.RADIO,
                                        name="dog_breed",
                                        options=[Option("poodle")], required=True)
  checklist_classification = Classification(
      class_type=Classification.Type.CHECKLIST,
      name="background",
      options=[Option("at_park"), Option("has_leash")])
  nested_classification = Classification(
      class_type=Classification.Type.CHECKLIST,
      name="Appliance Features",
      options=[
          Option(
              "HasWarranty",
              options=[  # Nested options
                  Option("1 Year"),
                  Option("2 Years"),
                  Option("5 Years")
              ]
          ),
          Option("EnergyEfficient"),
          Option("SmartHomeCompatible")
      ]
  )

  # Creating feature schema for each defined classification
  feature_schema_text = client.create_feature_schema(text_classification.asdict())
  feature_schema_radio = client.create_feature_schema(radio_classification.asdict())
  feature_schema_checklist = client.create_feature_schema(checklist_classification.asdict())
  feature_schema_nested = client.create_feature_schema(nested_classification.asdict())
  ```
</CodeGroup>

After creating features and schemas, you can add them to ontologies. To learn how to upsert features into an ontology, see [Ontology](/horizon/developer-guides/ontology#create-ontology-from-new-features).

## Set a`UIMode` for a feature

You can set a `UIMode` for your classification, which works similarly like switching the `dropdown` toggle inside the platform. The `UIMode` has the following two options:

1. `Classification.UIMode.SEARCHABLE` allows the feature to be searched inside a dropdown menu, equivalent to enabling the dropdown toggle.
2. `Classification.UIMode.HOTKEY` gives each answer option a dedicated hotkey, equivalent to disabling the dropdown toggle.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  radio_classification = Classification(class_type=Classification.Type.RADIO,
                                        name="dog_breed",
                                        options=[Option("poodle")],
                                        required=True,
                                        ui_mode=Classification.UIMode.SEARCHABLE)

  checklist_classification = Classification(class_type=Classification.Type.CHECKLIST,
                                          name="background",
                                          options=[Option("at_park"), Option("has_leash")]
                                          ui_mode=Classification.UIMode.HOTKEY)
  ```
</CodeGroup>

## Set a  `likert scale` for a feature

You can set the parameter `is_likert_scale` for your radio classification if all option values are integers.

```python theme={"theme":"css-variables"}
classification_features = [
    Classification(
        class_type=lb.Classification.Type.RADIO,
        name="Quality Issues",
        options=[
            Option(value="1", label="Blurry"),
            Option(value="2", label="Distorted"),
            Option(value="3", label="OutOfFocus"),
        ],
        required=True,
        is_likert_scale=True,
    )
]
```

## Get a feature

You can get the feature schema by name or schema id. Only top-level feature schemas are supported.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  from labelbox import client

  client = Client(api_key="<YOUR_API_KEY>")

  ## Search feature by name in your org
  regulatory_sign_feature_schema = next(client.get_feature_schemas("regulatory-sign"))
  classification_feature = next(client.get_feature_schemas("Quality Issues"))

  ## Get feature by feature schema ID. You can get this from the UI
  regulatory_sign_feature_schema = client.get_feature_schema("FEATURE_SCHEMA_ID")

  print(regulatory_sign_feature_schema)
  print(classification_feature)
  ```
</CodeGroup>

## Update feature schema name

Updates the title/name of a feature schema. Only top-level feature schemas are supported.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  client.update_feature_schema_title("<feature_id>", "New Title")
  ```
</CodeGroup>

## Delete or archive a feature in an ontology

Deletes or archives a feature schema from an ontology. If the feature schema is a root-level node with associated labels, it will be *archived*. If the feature schema is a nested node in the ontology without associated labels, it will be deleted. If the feature schema is a nested ontology node with associated labels, it will neither be deleted nor archived.

To *archive* a feature means you can unarchive it later on and retrieve annotations made with this feature. If you delete a feature, the feature and its associated annotations cannot be recovered.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  client.delete_feature_schema_from_ontology(ontology_id="<ontology_id>", feature_schema_id="<feature_schema_id>")
  ```
</CodeGroup>

## Unarchive a feature in an ontology

Unarchives a feature schema node in an ontology. Only root-level feature schema nodes can be unarchived.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  client.unarchive_feature_schema_node(
    ontology_id="<ontology_id>",
    root_feature_schema_id="<root_feature_schema_id>"
  )
  ```
</CodeGroup>

## Check whether a feature is archived

Returns `True` if a feature schema is archived in the specified ontology, returns `False` otherwise.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  result = client.is_feature_schema_archived(
    ontology_id="<ontology_id>",
    feature_schema_id="<feature_schema_id>"
  )

  # True or False
  print(result)
  ```
</CodeGroup>


# Foundry
Source: https://docs.labelbox.com/horizon/developer-guides/foundry-app-developer-guide

Developer guide for creating and managing Foundry applications.

<CardGroup>
  <Card title="Open in Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/foundry/object_detection.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/foundry/object_detection.ipynb" />
</CardGroup>

Foundry apps help automate data labeling and enrichment. Here, we should how to use the Labelbox SDK to manage Foundry apps and how to use a Foundry app to predict and import annotations into a dataset.

## Create a Foundry app

A Foundry app (short for Foundry application) helps automate data labeling and enrichment. Once you’ve created a Foundry app, you can run it repeatedly against new data.

To create a Foundry app, use the Labelbox Model to create [your Foundry app](/horizon/guides/foundry-apps#create-foundry-apps).

Once the app is created, you can use the Labelbox SDK to run your app and manage its results.

## Run a Foundry app

You can use the Labelbox Model to run Foundry apps. Results are displayed like any [other model run](/horizon/guides/foundry-view-predictions#view-model-runs).

When running a Foundry app, you must provide an app ID. You can find the app ID by going to the **Model** tab and selecting **Apps**. When you select a Foundry app, the app ID will appear in the top right corner of the screen.

In addition to the app ID, the method also requires a `DataRowIdentifiables`object and a unique name.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  task = client.run_foundry_app(model_run_name=f"Amazon-{str(uuid.uuid4())}",
                                data_rows=lb.GlobalKeys(
                                    [global_key] # Provide a list of global keys
                                    ),
                                app_id=AMAZON_REKOGNITION_APP_ID)

  task.wait_till_done()

  print(f"Errors: {task.errors}")

  #Obtain model run ID from task
  MODEL_RUN_ID = task.metadata["modelRunId"]

  ```
</CodeGroup>

## Send Foundry annotations to Annotate

When you send predictions to **Annotate** from **Catalog**, you may choose to include or exclude certain parameters.

### Parameters

| Parameter | Required? | Description |
| - | - | - |
| `predictions_ontology_mapping` | Required | A dictionary containing the mapping of the model's ontology feature schema IDs to the project's ontology feature schema IDs. See [Send predictions to Annotate](/horizon/guides/foundry-annotate-predictions#resolving-issues) for more information on where to access your predictions map. |
| `exclude_data_rows_in_project` | Optional | Excludes data rows that are already in the project. |
| `override_existing_annotations_rule` | Optional | The strategy defines how to handle conflicts in classifications between the existing data rows in the project and incoming predictions from the source model run or annotations from the source project. Options include: -`ConflictResolutionStrategy.KeepExisting` (DEFAULT) -`ConflictResolutionStrategy.OverrideWithPredictions` -`ConflictResolutionStrategy.OverrideWithAnnotations` |
| `batch_priority` | Optional | Assigned priority to the batch of data rows (1-5). |

### Sample script

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  model_run = client.get_model_run("<model_run_id")

  send_to_annotations_params = {
    "predictions_ontology_mapping": PREDICTIONS_ONTOLOGY_MAPPING,
    "exclude_data_rows_in_project": False,
    "override_existing_annotations_rule": ConflictResolutionStrategy.OverrideWithPredictions,
    "batch_priority": 5,
  }

  task = model_run.send_to_annotate_from_model(
    destination_project_id=project.uid,
    task_queue_id=None, #ID of workflow task, set ID to None if you want to convert pre-labels to ground truths or obtain task queue id through project.task_queues().
    batch_name="Foundry Demo Batch",
    data_rows=lb.GlobalKeys(
        [global_key] # Provide a list of global keys from foundry app task
    ),
    params=send_to_annotations_params
    )

  task.wait_till_done()

  print(f"Errors: {task.errors}")
  ```
</CodeGroup>

## Example: Run and import annotations from a Foundry app

Regardless of what model you use from Foundry, the workflow through the SDK is similar. Step 6 (Map ontology through the UI) is where the process can slightly differ.

An [interactive tutorial](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/foundry/object_detection.ipynb) is also available as a [Colab notebook](https://research.google.com/colaboratory/faq.html#whats-colaboratory); it shows shows how to run and import annotations from a Foundry app.

### Before you start

You will need to import these libraries to use the code examples in this section.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  import labelbox as lb
  from labelbox.schema.conflict_resolution_strategy import ConflictResolutionStrategy
  import uuid
  ```
</CodeGroup>

### Replace with your API key

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  API_KEY = ""
  client = lb.Client(API_KEY)
  ```
</CodeGroup>

### Step 1: Import data rows into Catalog

You must have data rows in Catalog before you can run them through Foundry. In this example, we are using an image data row.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # send a sample image as data row for a dataset
  global_key = str(uuid.uuid4())

  test_img_url = {
  "row_data":
  "https://storage.googleapis.com/labelbox-datasets/image_sample_data/2560px-Kitano_Street_Kobe01s5s4110.jpeg",
  "global_key":
  global_key
  }

  dataset = client.create_dataset(name="foundry-demo-dataset")
  task = dataset.create_data_rows([test_img_url])
  task.wait_till_done()

  print(f"Errors: {task.errors}")
  print(f"Failed data rows: {task.failed_data_rows}")

  ```
</CodeGroup>

### Step 2: Set up ontology

Your project should have an ontology configured the tools and classifications supported for your model and selected data type.

To illustrate, [Amazon Rekognition](https://aws.amazon.com/rekognition) only supports object detection scenarios. Consequently, your ontology must include a bounding box annotation. for your ontology since it only supports object detection.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Create ontology with two bounding boxes that is included with Amazon Rekognition: Car and Person
  ontology_builder = lb.OntologyBuilder(
      classifications=[],
      tools=[
          lb.Tool(tool=lb.Tool.Type.BBOX, name="Car"),
          lb.Tool(tool=lb.Tool.Type.BBOX, name="Person")
      ]
  )

  ontology = client.create_ontology("Image Bounding Box Annotation Demo Foundry",
                                  ontology_builder.asdict(),
                                  media_type=lb.MediaType.Image)
  ```
</CodeGroup>

### Step 3: Create a labeling project

Connect the ontology to the labeling project.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  project = client.create_project(name="Foundry Image Demo",
                                  media_type=lb.MediaType.Image)

  project.connect_ontology(ontology)

  ```
</CodeGroup>

### Step 4: Create new Foundry app & copy App ID

To use the Labelbox app to create Foundry apps:

<Steps>
  <Step>
    From the main menu, select **Model** and then select **App** from the **Create** menu.
  </Step>

  <Step>
    Select **Amazon Rekognition**, name your Foundry app, and then select **Proceed**.
  </Step>

  <Step>
    Customize your parameters and then select **Save & Create**.
  </Step>

  <Step>
    In **Model**, select \**Aps* and then select your new Foundry app.
  </Step>

  <Step>
    Use the **Copy** button to copy the App ID to the clipboard.
  </Step>

  <Step>
    Paste the copied App ID into your code:
  </Step>
</Steps>

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  #Select your foundry application inside the UI and copy the APP ID from the top right corner
  AMAZON_REKOGNITION_APP_ID = "<App ID goes here>"
  ```
</CodeGroup>

Foundry apps can only be created using the app; you cannot use the SDK to create Foundry apps.

### Step 5: Run Foundry app on data rows

This step generates annotations that can later be reused as pre-labels or ground in a project.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  task = client.run_foundry_app(model_run_name=f"Amazon-{str(uuid.uuid4())}",
                                data_rows=lb.GlobalKeys(
                                    [global_key] # Provide a list of global keys
                                    ),
                                app_id=AMAZON_REKOGNITION_APP_ID)

  task.wait_till_done()

  print(f"Errors: {task.errors}") v

  #Obtain model run ID from task
  MODEL_RUN_ID = task.metadata["modelRunId"]

  ```
</CodeGroup>

### Step 6: Map ontology through the UI

Predictions may not directly map to ontology features. When this happens, you need to map predictions to your ontology. To do this using the Labelbox app:

<Steps>
  <Step>
    Use **Catalog** to select the dataset for your model run and then select the data rows Select **Select all** in the top right corner.
  </Step>

  <Step>
    Select **Manage selection** > **Send to Annotate**.
  </Step>

  <Step>
    Select your project from the **Project** menu.
  </Step>

  <Step>
    When sending annotations for **Annotate** review, you typically select a workflow step. This isn't necessary for this example.
  </Step>

  <Step>
    Place a checkmark next to **Include model predictions** and then select **Map**.
  </Step>

  <Step>
    Select the incoming ontology and matching ontology features for *Car* and *Person*.
  </Step>

  <Step>
    When the features are mapped, select **Copy ontology mapping as JSON**.
  </Step>

  <Step>
    Paste your copied JSON into the definition of `PREDICTIONS_ONTOLOGY_MAPPING`

    <CodeGroup>
      ```python Python theme={"theme":"css-variables"}
      # Copy map ontology through the UI then paste JSON here
      PREDICTIONS_ONTOLOGY_MAPPING = {}
      ```
    </CodeGroup>
  </Step>

  <Step>
    In a production workflow, you would typically save your configuration. You can skip this step for the sake of this example.
  </Step>
</Steps>

While you cannot the SDK to map predictions to ontology features, you can use it to send predictions to **Annotate**.

### Step 7: Send annotations from **Catalog** to **Annotate**

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  model_run = client.get_model_run(MODEL_RUN_ID)

  send_to_annotations_params = {
    "predictions_ontology_mapping": PREDICTIONS_ONTOLOGY_MAPPING,
    "exclude_data_rows_in_project": False,
    "override_existing_annotations_rule": ConflictResolutionStrategy.OverrideWithPredictions,
    "batch_priority": 5,
  }

  task = model_run.send_to_annotate_from_model(
    destination_project_id=project.uid,
    task_queue_id=None, #ID of workflow task, set ID to None if you want to convert pre-labels to ground truths or obtain task queue id through project.task_queues().
    batch_name="Foundry Demo Batch",
    data_rows=lb.GlobalKeys(
        [global_key] # Provide a list of global keys from foundry app task
    ),
    params=send_to_annotations_params
    )

  task.wait_till_done()

  print(f"Errors: {task.errors}")
  ```
</CodeGroup>


# Overview
Source: https://docs.labelbox.com/horizon/developer-guides/getting-started

Description of Labelbox-Python SDK and our developer guides.

Welcome to Labelbox-Python SDK developer guides! The Labelbox-Python SDK is an open-source project that provides access to the Labelbox API and can automate many actions and workflows in the Labelbox app. The SDK's source code can be found on our [GitHub page](https://github.com/Labelbox/labelbox-python). For a complete granular API reference, visit our [readthedocs](https://labelbox-python.readthedocs.io/en/latest/) page.

<Warning>
  ### SDK version deprecation

  SDK versions earlier than 3.68 are no longer supported. Please update to a supported version to maintain compatibility.
</Warning>

<Card title="Connect an AI assistant" icon="plug" href="/horizon/guides/labelbox-mcp">
  Connect an MCP client to Labelbox for project insights and approved actions. No SDK installation is required.
</Card>

## Installation

To install, run either `pip install labelbox` or`pip install "labelbox[data]"` in your command line.

<CodeGroup>
  ```shell shell theme={"theme":"css-variables"}
  pip install "labelbox[data]"
  # or
  pip install labelbox
  ```
</CodeGroup>

| Module | Description |
| - | - |
| `"labelbox[data]"` | Installs all [required dependencies](https://github.com/Labelbox/labelbox-python/blob/develop/requirements.lock) (libraries, tools to manipulate annotations, and more.) |
| `labelbox` | Installs all required libraries for client-only functionality |

For more in-depth installation information, please visit our [installation guide](/horizon/developer-guides/install-python-sdk).

## How to use our developer guides

Our developer documentation is designed to help you understand the different methods in the Labelbox-Python SDK. You are encouraged to run the code blocks locally as you read through the guides. Some guides include links to Google Colab Notebooks and our GitHub repository at the top of the page. These links allow you to run Python code quickly, demonstrating how specific methods work and providing example workflows.

## Best practices

To get the best experience from the SDK, we recommend the following:

1. Always use bulk operations. Unless you are testing or working on small datasets, use bulk operations for faster and more reliable performance.
   * `create_data_rows` instead of `create_data_row`
   * `project.export()` instead of `project.labels()`
   * `Label.bulk_delete(labels)` instead of `[label.delete() for label in labels]`
2. Use `get_one()` and `get_many()` methods to get elements of a paginated collection instead of `list`.
   * `list(dataset.data_rows())[0]` is going to query for every single `data_row` in your dataset.
3. Make sure to use the latest version of the SDK.
   * Labelbox is a rapidly evolving company, and we are constantly adding new features and optimizations.

## GraphQL API

<Warning>
  ### GraphQL APIs are not meant to be consumed directly

  GraphQL APIs may be updated or deprecated at any time without notice. For this reason, Labelbox strongly advises that customers use the Python SDK for all development.
</Warning>

## How to contribute

To learn how to contribute towards the Labelbox Python SDK and our example notebooks, refer to our dedicated [contributing guide](https://github.com/Labelbox/labelbox-python/blob/develop/CONTRIBUTING.md) .

For feedback or to report any issues with our documentation, please email [docs@labelbox.com](mailto:docs@labelbox.com) .


# Hugging Face integration
Source: https://docs.labelbox.com/horizon/developer-guides/hugging-face-integration

Describes how to combine the Python SDK and HuggingFace Transformer models to improve similarity search.

<CardGroup>
  <Card title="Open In Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/integrations/huggingface/huggingface_custom_embeddings.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/integrations/huggingface/huggingface_custom_embeddings.ipynb" />
</CardGroup>

You can improve similarity search for your data rows by using the Labelbox Python SDK and Hugging Face Transformer models to custom embeddings.

This lets you use off-the-shelf models to create search terms that lets you compare custom embedding and model performance.


# Import a labeled dataset (images)
Source: https://docs.labelbox.com/horizon/developer-guides/import-a-labeled-dataset-images

How to import a labeled dataset

## Overview

In this tutorial, you will:

* Create a dataset in Labelbox
* Import custom metadata and ground truth

# End-to-end example: import ground truth

## Import libraries

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  import labelbox as lb
  from labelbox.schema.data_row_metadata import DataRowMetadataField, DataRowMetadataKind
  import datetime
  import random
  import os
  import json
  from PIL import Image
  from labelbox.schema.ontology import OntologyBuilder, Tool
  import requests
  from tqdm.notebook import tqdm
  import uuid
  from labelbox.data.annotation_types import Label, ImageData, ObjectAnnotation, Rectangle
  ```
</CodeGroup>

## Set up Labelbox client

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Initialize the Labelbox client
  API_KEY = "" # Place API key
  client = lb.Client(API_KEY)
  ```
</CodeGroup>

## Download a public dataset

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Function to download files
  def download_files(filemap):
      path, uri = filemap
      if not os.path.exists(path):
          response = requests.get(uri, stream=True)
          with open(path, 'wb') as f:
              for chunk in response.iter_content(chunk_size=8192):
                  f.write(chunk)
      return path
  ```
</CodeGroup>

#### Download data rows and annotations

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Download data rows and annotations
  DATA_ROWS_URL = "https://storage.googleapis.com/labelbox-datasets/VHR_geospatial/geospatial_datarows.json"
  ANNOTATIONS_URL = "https://storage.googleapis.com/labelbox-datasets/VHR_geospatial/geospatial_annotations.json"
  download_files(("data_rows.json", DATA_ROWS_URL))
  download_files(("annotations.json", ANNOTATIONS_URL))
  ```
</CodeGroup>

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  with open('data_rows.json', 'r') as fp:
      data_rows = json.load(fp)

  with open('annotations.json', 'r') as fp:
      annotations = json.load(fp)
  ```
</CodeGroup>

## Create a dataset

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Create a new dataset
  dataset = client.create_dataset(name="Geospatial vessel detection")
  print(f"Created dataset with ID: {dataset.uid}")
  ```
</CodeGroup>

## Import Data Rows and metadata

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Here is an example of adding two metadata fields to your Data Rows: a "captureDateTime" field with datetime value, and a "tag" field with string value
  metadata_ontology = client.get_data_row_metadata_ontology()
  datetime_schema_id = metadata_ontology.reserved_by_name["captureDateTime"].uid
  tag_schema_id = metadata_ontology.reserved_by_name["tag"].uid
  tag_items = ["WorldView-1", "WorldView-2", "WorldView-3", "WorldView-4"]

  for datarow in tqdm(data_rows):
      dt = datetime.datetime.utcnow() + datetime.timedelta(days=random.random()*30) # this is random datetime value
      tag_item = random.choice(tag_items) # this is a random tag value

      # Option 1: Specify metadata with a list of DataRowMetadataField. This is the recommended option since it comes with validation for metadata fields.
      metadata_fields = [
                         DataRowMetadataField(schema_id=datetime_schema_id, value=dt),
                         DataRowMetadataField(schema_id=tag_schema_id, value=tag_item)
                         ]

      # Option 2: Uncomment to try. Alternatively, you can specify the metadata fields with dictionary format without declaring the DataRowMetadataField objects. It is equivalent to Option 1.
      # metadata_fields = [
      #                    {"schema_id": datetime_schema_id, "value": dt},
      #                    {"schema_id": tag_schema_id, "value": tag_item}
      #                    ]

      datarow["metadata_fields"] = metadata_fields
  ```
</CodeGroup>

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  task = dataset.create_data_rows(data_rows)
  task.wait_till_done()
  print(f"Failed data rows: {task.failed_data_rows}")
  print(f"Errors: {task.errors}")

  if task.errors:
      for error in task.errors:
          if 'Duplicate global key' in error['message'] and dataset.row_count == 0:
              # If the global key already  exists in the workspace the dataset will be created empty, so we can delete it.
              print(f"Deleting empty dataset: {dataset}")
              dataset.delete()
  ```
</CodeGroup>

## Examine a data row

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  datarow = next(dataset.data_rows())
  print(datarow)
  ```
</CodeGroup>

## Set up a labeling project

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}

  # Initialize the OntologyBuilder
  ontology_builder = OntologyBuilder()

  # Assuming 'annotations' is defined and contains the necessary data
  for category in annotations['categories']:
      print(category['name'])
      # Add tools to the ontology builder
      ontology_builder.add_tool(Tool(tool=Tool.Type.BBOX, name=category['name']))

  # Create the ontology in Labelbox
  ontology = client.create_ontology("Vessel Detection Ontology",
                                    ontology_builder.asdict(),
                                    media_type=lb.MediaType.Image)
  print(f"Created ontology with ID: {ontology.uid}")

  # Create a project and connect the ontology
  project = client.create_project(name="Vessel Detection", media_type=lb.MediaType.Image)
  project.connect_ontology(ontology=ontology)
  print(f"Created project with ID: {project.uid}")
  ```
</CodeGroup>

## Send a batch of data rows to the project

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}

  # Minimal Export parameters focused solely on data row IDs
  export_params = {
      "data_row_details": True  # Only export data row details
  }

  # Initiate the streamable export task from catalog
  dataset = client.get_dataset(dataset.uid)  # Update with the actual dataset ID
  export_task = dataset.export(params=export_params)
  export_task.wait_till_done()
  print(export_task)

  data_rows = []

  # Stream the export using a callback function
  def json_stream_handler(output: labelbox.BufferedJsonConverterOutput):
    print(output.json)

  export_task.get_buffered_stream(stream_type=labelbox.StreamType.RESULT).start(stream_handler=json_stream_handler)

  # Collect all exported data into a list
  export_json = [data_row.json for data_row in export_task.get_buffered_stream()]

  # Randomly select 200 Data Rows (or fewer if the dataset has less than 200 data rows)
  sampled_data_rows = random.sample(data_rows, min(len(data_rows), 200))

  # Create a new batch in the project and add the sampled data rows
  batch = project.create_batch(
      "Initial batch",  # name of the batch
      sampled_data_rows,  # list of Data Rows
      1  # priority between 1-5
  )
  print(f"Created batch with ID: {batch.uid}")
  ```
</CodeGroup>

## Create annotations payload

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Set export parameters focused on data row details
  export_params = {
      "data_row_details": True,  # Only export data row details
      "batch_ids": [batch.uid],  # Optional: Include batch ids to filter by specific batches
  }

  # Initialize the streamable export task from project
  export_task = project.export(params=export_params)
  export_task.wait_till_done()

  data_rows = []

  # Stream the export using a callback function
  def json_stream_handler(output: labelbox.BufferedJsonConverterOutput):
    print(output.json)

  export_task.get_buffered_stream(stream_type=labelbox.StreamType.RESULT).start(stream_handler=json_stream_handler)

  # Collect all exported data into a list
  export_json = [data_row.json for data_row in export_task.get_buffered_stream()]

  labels = []
  for datarow in data_rows:
      annotations_list = []
      # Access the 'data_row' dictionary first
      data_row_dict = datarow['data_row']
      folder = data_row_dict['external_id'].split("/")[0]
      id = data_row_dict['external_id'].split("/")[1]

      if folder == "positive_image_set":
          for image in annotations['images']:
              if image['file_name'] == id:
                  for annotation in annotations['annotations']:
                      if annotation['image_id'] == image['id']:
                          bbox = annotation['bbox']
                          category_id = annotation['category_id'] - 1
                          class_name = None
                          ontology = ontology_builder.asdict()  # Get the ontology dictionary
                          for category in ontology['tools']:
                              if category['name'] == annotations['categories'][category_id]['name']:
                                  class_name = category['name']
                                  break
                          if class_name:
                              annotations_list.append(ObjectAnnotation(
                                  name=class_name,
                                  value=Rectangle(start=Point(x=bbox[0], y=bbox[1]), end=Point(x=bbox[2]+bbox[0], y=bbox[3]+bbox[1]))
                              ))
      image_data = ImageData(uid=data_row_dict['id'])
      labels.append(Label(data=image_data, annotations=annotations_list))
  ```
</CodeGroup>

## Import ground truth annotations

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  upload_job = lb.LabelImport.create_from_objects(
      client=client,
      project_id=project.uid,
      name=f"label_import_job_{str(uuid.uuid4())}",
      labels=labels
  )

  # Wait for the upload to finish and print the results
  upload_job.wait_until_done()

  print(f"Errors: {upload_job.errors}")
  print(f"Status of uploads: {upload_job.statuses}")
  ```
</CodeGroup>


# Import a labeled dataset (text)
Source: https://docs.labelbox.com/horizon/developer-guides/import-a-labeled-dataset-text



## Initialize Huggingface dataset, models and pipeline

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  datasets = load_dataset("Babelscape/wikineural")
  tokenizer = AutoTokenizer.from_pretrained("dslim/bert-large-NER")
  model = AutoModelForTokenClassification.from_pretrained("dslim/bert-large-NER")

  ## Use GPU if it exists
  gpu_info = !nvidia-smi
  gpu_info = '\n'.join(gpu_info)
  if gpu_info.find('failed') >= 0:
    print('Not connected to a GPU')
    nlp = pipeline("ner", model=model, tokenizer=tokenizer)
    embedding_model = SentenceTransformer('sentence-transformers/bert-base-nli-mean-tokens')

  else:
    print(gpu_info)
    nlp = pipeline("ner", model=model, tokenizer=tokenizer, device=0)
    embedding_model = SentenceTransformer('sentence-transformers/bert-base-nli-mean-tokens', device=0)
  ```
</CodeGroup>

## Set up client and configure parameters

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  ## Enter your API key here
  LB_API_KEY = "YOUR_API_KEY"

  client = labelbox.Client(LB_API_KEY)
  ## Set batch size for batching data rows and annotation bulk import. 500-1000 is recommended size.
  BATCH_SIZE = 500

  ## Set max number of data rows to import. WikiNeural dataset has ~1.1M data rows
  MAX_DATA_ROW_LIMIT = 2000
  ```
</CodeGroup>

## Functions

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  def create_ner_objects(class_name, st, en):
    named_enity = TextEntity(start=st,end=en)
    named_enity_annotation = ObjectAnnotation(value=named_enity, name=class_name)
    return named_enity_annotation

  def generate_predictions(datarow):
    external_id = datarow["external_id"]
    dataset_name = external_id.split("_")[0] + "_" + external_id.split("_")[1]
    datarow_index = int(external_id.split("_")[2].split(".")[0])
    uid = datarow['id']
    text_data_row =  datasets[dataset_name][datarow_index]
    tokens = text_data_row["tokens"]
    tokenized_input = tokenizer(tokens, is_split_into_words=True)
    sentence = tokenizer.decode(tokenized_input["input_ids"], skip_special_tokens=True)
    annotations = []

    ## Generate prediction
    predictions = nlp(sentence)

    ## process predictions and compute text entities
    try:
      for item in predictions:
        score = item['score']
        if score > 0.99:
          entity = item['entity']
          start = item['start']
          end = item['end']
          index = predictions.index(item)

          if entity =="B-PER":
            for next_item in predictions[index+1:]:
              if next_item['entity']=="I-PER":
                end = next_item['end']
              else:
                break
            annotations.append(create_ner_objects("PER", start, end-1))

          if entity =="B-ORG":
            for next_item in predictions[index+1:]:
              if next_item['entity']=="I-ORG":
                end = next_item['end']
              else:
                break
            annotations.append(create_ner_objects("ORG", start, end-1))

          if entity =="B-LOC":
            for next_item in predictions[index+1:]:
              if next_item['entity']=="I-LOC":
                end = next_item['end']
              else:
                break
            annotations.append(create_ner_objects("LOC", start, end-1))

          if entity =="B-MISC":
            for next_item in predictions[index+1:]:
              if next_item['entity']=="I-MISC":
                end = next_item['end']
              else:
                break
            annotations.append(create_ner_objects("MISC", start, end-1))

    except Exception as e:
      print(e)

    text_data = TextData(uid=uid)
    return text_data, annotations

  def create_data_rows_payload(payload):
    data_row_content = None
    label_content = None

    try:
      h, text_data_row, lang = payload
      file_name = lang + "_" + str(h) +'.txt'

      tokens = text_data_row["tokens"]
      tokenized_input = tokenizer(tokens, is_split_into_words=True)
      sentence = tokenizer.decode(tokenized_input["input_ids"], skip_special_tokens=True)

      embeddings = embedding_model.encode(sentence)
      embeddings_metadata = DataRowMetadataField(
            schema_id=embedding_field.uid,
            ## Labelbox currently only supports custom embedding of 128 max length
            value=embeddings[:128].tolist(),
          )

      language_metadata = DataRowMetadataField(
            schema_id=language_field.uid,
              value=lang,
          )
      metadata_payload = [language_metadata, embeddings_metadata]

      data_row_content = {DataRow.row_data: "gs://labelbox-datasets/wiki_neural_text_ner/"+file_name, DataRow.external_id: file_name, DataRow.metadata_fields: metadata_payload}

    except Exception as e:
      print(e)

    return data_row_content, label_content
  ```
</CodeGroup>

## Create a dataset

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  dataset = client.create_dataset(name="WikiNEuRal Text NER")
  ```
</CodeGroup>

## Set up ontology

This example requires a custom metadata string field named `language`. For help, see [Custom fields](/horizon/guides/datarow-metadata#custom-fields).

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  metadata_ontology = client.get_data_row_metadata_ontology()
  ## Create a custom metadata schema called language of string kind: /horizon/guides/datarow-metadata#custom-fields
  language_field = metadata_ontology.custom_by_name["language"]
  embedding_field = metadata_ontology.reserved_by_name["embedding"]
  ```
</CodeGroup>

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  ontology = OntologyBuilder()
  PER = Tool(tool = Tool.Type.NER, name = "PER")
  ontology.add_tool(PER)
  ORG = Tool(tool = Tool.Type.NER, name = "ORG")
  ontology.add_tool(ORG)
  LOC = Tool(tool = Tool.Type.NER, name = "LOC")
  ontology.add_tool(LOC)
  MISC = Tool(tool = Tool.Type.NER, name = "MISC")
  ontology.add_tool(MISC)

  ontology = client.create_ontology("WikiNEuRal Text NER", ontology.asdict())
  ```
</CodeGroup>

## Set up a labeling project

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  project = client.create_project(name = "WikiNEuRal Text NER", media_type=MediaType.Text)
  project.connect_ontology(ontology)
  ontology_from_project = OntologyBuilder.from_project(project)
  project.update(queue_mode=project.QueueMode.Batch)
  ```
</CodeGroup>

## Process and create data row payload in batches

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  tuples = []
  for item in datasets:
    # if item == "train_en":
    for h, text_data_row in enumerate(datasets[item]):
      tuples.append((h, text_data_row, item))

  if MAX_DATA_ROW_LIMIT !=None:
    tuples = random.sample(tuples, MAX_DATA_ROW_LIMIT)
  ```
</CodeGroup>

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  chunked_tuples = list()
  for i in range(0, len(tuples), BATCH_SIZE):
      chunked_tuples.append(tuples[i:i+BATCH_SIZE])
  ```
</CodeGroup>

## Main iterator loop to import data

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  for chunk in chunked_tuples:
    start_time = time.time()
    current_index = chunked_tuples.index(chunk)
    print("Executing {} of {} iteration".format(current_index, len(chunked_tuples)))

    ## Generate data row payload
    data_rows = []
    for item in tqdm.tqdm(chunk):
      datarow,label = create_data_rows_payload(item)
      data_rows.append(datarow)

    ## Create data rows in Labelbox
    task = dataset.create_data_rows(data_rows)
    task.wait_till_done()
    print(task)

    ## Submit a batch of the recently created data rows
    batch_datarows = []
    for item in task.result:
      batch_datarows.append(item['id'])

    batch = project.create_batch(
      str(current_index) + "_" + str(binascii.b2a_hex(os.urandom(5))), # name of the batch
      batch_datarows, # list of Data Rows
      1 # priority between 1-5
    )

    ## Generate model predictions
    ground_truth_list = LabelList()
    results = []

    for item in tqdm.tqdm(task.result):
      result = generate_predictions(item)
      ground_truth_list.append(Label(
            data=result[0],
            annotations = result[1]
        ))

    ## Convert model predictions to NDJSON format
    ground_truth_list.assign_feature_schema_ids(OntologyBuilder.from_project(project))
    ground_truth_ndjson = list(NDJsonConverter.serialize(ground_truth_list))

    ## Upload model predictions as ground truth
    upload_task = LabelImport.create_from_objects(client, project.uid, f"upload-job-{uuid4()}", ground_truth_ndjson)
    upload_task.wait_until_done()
    print(upload_task.errors)

    print(str((time.time() - start_time))+" seconds")
  ```
</CodeGroup>


# Import audio annotations
Source: https://docs.labelbox.com/horizon/developer-guides/import-audio-annotations

Developer guide for importing annotations on audio data and sample import formats.

<CardGroup>
  <Card title="Open In Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/annotation_import/audio.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/annotation_import/audio.ipynb" />
</CardGroup>

## Overview

To import annotations in Labelbox, you need to create an annotations payload. In this section, we provide this payload for every supported annotation type.

### Annotation payload types

Labelbox supports two formats for the annotations payload:

* Python annotation types (recommended)

  * Provides a seamless transition between third-party platforms, machine learning pipelines, and Labelbox.
  * Allows you to build annotations locally with local file paths, numpy arrays, or URLs
  * Easily convert Python Annotation Type format to NDJSON format to quickly import annotations to Labelbox
  * Supports one-level nested classification (radio, checklist, or free-form text) under a tool or classification annotation.

* JSON

  * Skips formatting annotation payload in the Labelbox Python annotation type
  * Supports any levels of nested classification (radio, checklist, or free-form text) under a tool or classification annotation.

### Label import types

Labelbox additionally supports two types of label imports:

* [Model-assisted labeling (MAL)](/horizon/guides/model-assisted-labeling)
  * This workflow allows you to import computer-generated predictions (or simply annotations created outside of Labelbox) as pre-labels on an asset.
* [Ground truth](https://www.google.com/url?q=https%3A%2F%2Fdocs.labelbox.com%2Fdocs%2Fimport-ground-truth)
  * This workflow functionality allows you to bulk import your ground truth annotations from an external or third-party labeling system into Labelbox *Annotate*. Using the label import API to import external data is a useful way to consolidate and migrate all annotations into Labelbox as a single source of truth.

## Supported Annotations

The following annotations are supported for an audio data row:

* Radio
* Checklist
* Free-form text
* Temporal classifications (milliseconds): text, radio, checklist

## Classifications

### Radio (single choice)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  radio_annotation = lb_types.ClassificationAnnotation(
      name="radio_question",
      value=lb_types.Radio(answer =
          lb_types.ClassificationAnswer(name = "first_radio_answer")
      )
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  radio_annotation_ndjson = {
    "name": "radio_question",
    "answer": {"name": "first_radio_answer"}
  }
  ```
</CodeGroup>

### Checklist (multiple choice)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  checklist_annotation = lb_types.ClassificationAnnotation(
      name="checklist_question",
      value=lb_types.Checklist(answer = [
          lb_types.ClassificationAnswer(name = "first_checklist_answer"),
          lb_types.ClassificationAnswer(name = "second_checklist_answer"),
          lb_types.ClassificationAnswer(name = "third_checklist_answer")
      ])
    )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  checklist_annotation_ndjson = {
    "name": "checklist_question",
    "answer": [
      {"name": "first_checklist_answer"},
      {"name": "second_checklist_answer"},
      {"name": "third_checklist_answer"},
    ]
  }
  ```
</CodeGroup>

### Free-form text

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  text_annotation = lb_types.ClassificationAnnotation(
      name = "free_text",
      value = lb_types.Text(answer="sample text")
  )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  text_annotation_ndjson = {
    "name": "free_text",
    "answer": "sample text",
  }
  ```
</CodeGroup>

### Nested classifications

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  nested_radio_annotation = lb_types.ClassificationAnnotation(
    name="nested_radio_question",
    value=lb_types.Radio(
      answer=lb_types.ClassificationAnswer(
        name="first_radio_answer",
        classifications=[
          lb_types.ClassificationAnnotation(
            name="sub_radio_question",
            value=lb_types.Radio(
              answer=lb_types.ClassificationAnswer(
                name="first_sub_radio_answer"
              )
            )
          )
        ]
      )
    )
  )

  nested_checklist_annotation = lb_types.ClassificationAnnotation(
    name="nested_checklist_question",
    value=lb_types.Checklist(
      answer=[lb_types.ClassificationAnswer(
        name="first_checklist_answer",
        classifications=[
          lb_types.ClassificationAnnotation(
            name="sub_checklist_question",
            value=lb_types.Checklist(
              answer=[lb_types.ClassificationAnswer(
              name="first_sub_checklist_answer"
            )]
          ))
        ]
      )]
    )
  )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  nested_radio_annotation_ndjson= {
    "name": "nested_radio_question",
    "answer": {
        "name": "first_radio_answer",
        "classifications": [
          {
            "name": "sub_radio_question",
            "answer": {"name": "first_sub_radio_answer"}
          }
        ]
      }
  }

  nested_checklist_annotation_ndjson = {
    "name": "nested_checklist_question",
    "answer": [{
        "name": "first_checklist_answer",
        "classifications" : [
          {
            "name": "sub_checklist_question",
            "answer": {"name": "first_sub_checklist_answer"}
          }
        ]
    }]
  }
  ```
</CodeGroup>

## Temporal classifications (milliseconds)

Use temporal classifications to attach **time-based** classification labels to an audio asset. Temporal ranges are represented in **milliseconds** from the start of the asset.

When using NDJSON, temporal classifications use a **unified** `answer: [...]` list structure:

* For **temporal radio**, `answer` is a **single-item list** (unlike global radio, which uses an object).
* For **temporal checklist**, `answer` is a **multi-item list**.
* For **temporal text**, `answer` is a list of `{ "value": <text>, "frames": [{"start": <ms>, "end": <ms>}] }`.

Temporal classifications support nested hierarchies (for example: Text > Text > Text, or Radio > Radio > Radio).

### Free-form text (temporal)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  temporal_text_annotation = lb_types.TemporalClassificationText(
    name="transcription",
    value=[
      (1000, 1100, "Hello"),
      (1500, 2400, "How can I help you?"),
    ],
  )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  temporal_text_annotation_ndjson = {
    "name": "transcription",
    "answer": [
      { "value": "Hello", "frames": [{"start": 1000, "end": 1100}] },
      { "value": "How can I help you?", "frames": [{"start": 1500, "end": 2400}] }
    ]
  }
  ```
</CodeGroup>

### Radio (temporal)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  temporal_radio_annotation = lb_types.TemporalClassificationQuestion(
    name="speaker",
    value=[
      lb_types.TemporalClassificationAnswer(
        name="user",
        frames=[(200, 1600)],
      )
    ],
  )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  temporal_radio_annotation_ndjson = {
    "name": "speaker",
    "answer": [
      { "name": "user", "frames": [{"start": 200, "end": 1600}] }
    ]
  }
  ```
</CodeGroup>

### Checklist (temporal)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  temporal_checklist_annotation = lb_types.TemporalClassificationQuestion(
    name="audio_quality",
    value=[
      lb_types.TemporalClassificationAnswer(
        name="background_noise",
        frames=[(0, 1500), (2000, 3000)],
      ),
      lb_types.TemporalClassificationAnswer(
        name="echo",
        frames=[(2200, 2900)],
      ),
    ],
  )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  temporal_checklist_annotation_ndjson = {
    "name": "audio_quality",
    "answer": [
      {
        "name": "background_noise",
        "frames": [{"start": 0, "end": 1500}, {"start": 2000, "end": 3000}]
      },
      {
        "name": "echo",
        "frames": [{"start": 2200, "end": 2900}]
      }
    ]
  }
  ```
</CodeGroup>

### Nested classifications (temporal)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  nested_temporal_radio_annotation = lb_types.TemporalClassificationQuestion(
    name="speaker",
    value=[
      lb_types.TemporalClassificationAnswer(
        name="user",
        frames=[(200, 1600)],
        classifications=[
          lb_types.TemporalClassificationQuestion(
            name="tone",
            value=[
              lb_types.TemporalClassificationAnswer(
                name="professional",
                frames=[(1000, 1600)],
                classifications=[
                  lb_types.TemporalClassificationQuestion(
                    name="clarity",
                    value=[
                      lb_types.TemporalClassificationAnswer(
                        name="clear",
                        frames=[(1300, 1600)],
                      )
                    ],
                  )
                ],
              )
            ],
          )
        ],
      )
    ],
  )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  nested_temporal_radio_annotation_ndjson = {
    "name": "speaker",
    "answer": [
      {
        "name": "user",
        "frames": [{"start": 200, "end": 1600}],
        "classifications": [
          {
            "name": "tone",
            "answer": [
              {
                "name": "professional",
                "frames": [{"start": 1000, "end": 1600}],
                "classifications": [
                  {
                    "name": "clarity",
                    "answer": [
                      { "name": "clear", "frames": [{"start": 1300, "end": 1600}] }
                    ]
                  }
                ]
              }
            ]
          }
        ]
      }
    ]
  }
  ```
</CodeGroup>

## Example: Import pre-labels or ground truths

The steps to import annotations as pre-labels (machine-assisted learning) are similar to those to import annotations as ground truth labels. However, they vary slightly, and we will describe the differences for each scenario.

### Before you start

The below imports are needed to use the code examples in this section.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  import labelbox as lb
  import uuid
  import labelbox.types as lb_types
  ```
</CodeGroup>

Replace the value of `API_KEY` with a valid [API key](/horizon/developer-guides/create-api-key) to connect to the Labelbox client.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  API_KEY = None
  client = lb.Client(API_KEY)
  ```
</CodeGroup>

### Step 1: Import data rows

Data rows must first be uploaded to **Catalog** to attach annotations.

This example shows how to create a data row in **Catalog** by attaching it to a [dataset](/horizon/developer-guides/dataset) .

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  global_key = "sample-audio-1.mp3"

  asset = {
    "row_data": "https://storage.googleapis.com/labelbox-datasets/audio-sample-data/sample-audio-1.mp3",
    "global_key": global_key
  }

  dataset = client.create_dataset(name="audio_annotation_import_demo_dataset")
  task = dataset.create_data_rows([asset])

  task.wait_till_done()

  print("Errors:", task.errors)
  print("Failed data rows: ", task.failed_data_rows)

  ```
</CodeGroup>

### Step 2: Set up ontology

Your project ontology should support the tools and classifications required by your annotations. To ensure accurate schema feature mapping, the value used as the `name` parameter should match the value of the `name` field in your annotation.

For example, when we created an annotation above, we provided a name`annotation_name`. Now, when we set up our ontology, we must ensure that the name of our bounding box tool is also `anotations_name`. The same alignment must hold true for the other tools and classifications we create in our ontology.

This example shows how to create an ontology containing all supported [annotation types](#supported-annotations) .

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  ontology_builder = lb.OntologyBuilder(
    classifications=[
      lb.Classification(
        class_type=lb.Classification.Type.TEXT,
        name="text_audio"),
      lb.Classification(
        class_type=lb.Classification.Type.CHECKLIST,
        name="checklist_audio",
        options=[
          lb.Option(value="first_checklist_answer"),
          lb.Option(value="second_checklist_answer")
        ]
      ),
      lb.Classification(
        class_type=lb.Classification.Type.RADIO,
        name="radio_audio",
        options=[
          lb.Option(value="first_radio_answer"),
          lb.Option(value="second_radio_answer")
        ]
      ),
      # Temporal classifications for audio require INDEX scope
      lb.Classification(
        class_type=lb.Classification.Type.TEXT,
        name="transcription",
        scope=lb.Classification.Scope.INDEX
      ),
    ]
  )

  ontology = client.create_ontology("Ontology Audio Annotations",
                                    ontology_builder.asdict(),
                                    media_type=lb.MediaType.Audio)
  ```
</CodeGroup>

### Step 3: Set Up a Labeling Project

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Create Labelbox project
  project = client.create_project(name="audio_project",
                                      media_type=lb.MediaType.Audio)

  # Setup your ontology
  project.connect_ontology(ontology)
  ```
</CodeGroup>

### Step 4: Send Data Rows to Project

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Create a batch to send to your MAL project
  batch = project.create_batch(
    "first-batch-audio-demo", # Each batch in a project must have a unique name
    global_keys=[global_key], # Paginated collection of data row objects, list of data row ids or global keys
    priority=5 # priority between 1(Highest) - 5(lowest)
  )

  print("Batch: ", batch)
  ```
</CodeGroup>

### Step 5: Create annotation payloads

For help understanding annotation payloads, see [overview](#overview). To declare payloads, you can use Python annotation types (*preferred*) or NDJSON objects. For annotations that you want to import as ground truth labels, you can also specify [benchmarks](/horizon/guides/benchmark) using the `is_benchmark_reference` flag.

These examples demonstrate each format and how to compose annotations into labels attached to data rows.

<CodeGroup>
  ```python Python Annotation Payload theme={"theme":"css-variables"}
  label = []
  label.append(
    lb_types.Label(
      data={"global_key" : global_key },
      annotations=[
        text_annotation,
        checklist_annotation,
        radio_annotation,
        temporal_text_annotation,
        temporal_radio_annotation,
        temporal_checklist_annotation
      ],
      # Optional: set the label as a benchmark
      # Only supported for groud truth imports
      is_benchmark_reference = True
    )
  )
  ```

  ```python NDJSON Payload theme={"theme":"css-variables"}
  label_ndjson = []
  for annotations in [text_annotation_ndjson,
                      checklist_annotation_ndjson,
                      radio_annotation_ndjson,
                      temporal_text_annotation_ndjson,
                      temporal_radio_annotation_ndjson,
                      temporal_checklist_annotation_ndjson]:
    annotations.update({
        'dataRow': {
            'globalKey': global_key
        }
    })
    label_ndjson.append(annotations)
  ```
</CodeGroup>

### Step 6: Import annotation payload

For prelabeled (model-assisted labeling) scenarios, pass your payload as the value of the `predictions` parameter. For ground truths, pass the payload to the `labels` parameter.

#### Option A: Upload as [prelabels (model assisted labeling)](/horizon/guides/model-assisted-labeling)

This option is helpful for speeding up the initial labeling process and reducing the manual labeling workload for high-volume datasets.

<CodeGroup>
  ```python MAL import theme={"theme":"css-variables"}
  # Upload MAL label for this data row in project
  upload_job = lb.MALPredictionImport.create_from_objects(
      client = client,
      project_id = project.uid,
      name="mal_job"+str(uuid.uuid4()),
      predictions=label
  )

  print(f"Errors: {upload_job.errors}", )
  print(f"Status of uploads: {upload_job.statuses}")
  ```
</CodeGroup>

#### Option B: Upload to a labeling project as [ground truth](/horizon/guides/import-ground-truth)

This option is helpful for loading high-confidence labels from another platform or previous projects that just need review rather than manual labeling effort.

<CodeGroup>
  ```python Label import theme={"theme":"css-variables"}
  # Upload label for this data row in project
  upload_job = lb.LabelImport.create_from_objects(
      client = client,
      project_id = project.uid,
      name="label_import_job"+str(uuid.uuid4()),
      labels=label
  )

  print(f"Errors: {upload_job.errors}", )
  print(f"Status of uploads: {upload_job.statuses}")
  ```
</CodeGroup>


# Import audio data
Source: https://docs.labelbox.com/horizon/developer-guides/import-audio-data

How to import audio data and sample import formats.

## Specifications

File formats: MP3, WAV Import methods:

* IAM Delegated Access
* Signed URLs (`https` URLs only)
* Direct upload of local files(256 MB max file size) Note: Direct upload currently does not support adding additional metadata and attachments, see below Python example.

When importing audio data to Labelbox, your JSON file must include the following information for each audio file.

## Parameters

| Parameter | Required | Description |
| - | - | - |
| `row_data` | Yes | `https` path to a cloud-hosted audio. For IAM Delegated Access, this URL must be in [virtual-hosted-style format](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#virtual-hosted-style-access). For older regions, your S3 bucket may be in `https://<bucket-name>.s3.<region>.amazonaws.com/<key>` format. If your object URLs are formatted this way, make sure they are in the virtual-hosted-style format before importing. |
| `global_key` | No | Unique user-generated file name or ID for the file. [Global keys](/horizon/developer-guides/data-row-global-keys) are enforced to be unique in your org. Data rows will not be imported if its global keys are duplicated to existing data rows. |
| `media_type` | No | `"AUDIO"` (optional media type to provide better validation and error messaging) |
| `metadata_fields` | No | See [Metadata](/horizon/guides/datarow-metadata). |
| `attachments` | No | See [Attachments](/horizon/guides/label-data) and [Asset overlays](/horizon/guides/label-data). |

## Import format

<CodeGroup>
  ```json Delegated Access URL theme={"theme":"css-variables"}
  [
    {
      "row_data": "https://lb-test-data.s3.us-west-1.amazonaws.com/audio-samples/sample-audio-1.mp3",
      "global_key": "https://lb-test-data.s3.us-west-1.amazonaws.com/audio-samples/sample-audio-1.mp3",
      "media_type": "AUDIO",
      "metadata_fields": [{"name": "<metadata_field_name>", "value": "tag_string"}],
      "attachments": [{"type": "VIDEO", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/drone_video.mp4" }]
    },
    {
      "row_data": "https://lb-test-data.s3.us-west-1.amazonaws.com/audio-samples/sample-audio-2.mp3",
      "global_key": "https://lb-test-data.s3.us-west-1.amazonaws.com/audio-samples/sample-audio-2.mp3",
      "media_type": "AUDIO",
      "metadata_fields": [{"name": "<metadata_field_name>", "value": "tag_string"}],
      "attachments": [{"type": "TEXT_URL", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt"}]
    }
  ]
  ```

  ```json Standard URL theme={"theme":"css-variables"}
  [
    {
      "row_data": "https://storage.googleapis.com/labelbox-datasets/audio-sample-data/sample-audio-1.mp3",
      "global_key": "https://storage.googleapis.com/labelbox-datasets/audio-sample-data/sample-audio-1.mp3",
      "media_type": "AUDIO",
      "metadata_fields": [{"schema_id": "cko8s9r5v0001h2dk9elqdidh", "value": "tag_string"}],
      "attachments": [{"type": "VIDEO", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/drone_video.mp4" }]
    },
    {
      "row_data": "https://storage.googleapis.com/labelbox-datasets/audio-sample-data/sample-audio-2.mp3",
      "global_key": "https://storage.googleapis.com/labelbox-datasets/audio-sample-data/sample-audio-2.mp3",
      "media_type": "AUDIO",
      "metadata_fields": [{"schema_id": "cko8s9r5v0001h2dk9elqdidh", "value": "tag_string"}],
      "attachments": [{"type": "TEXT_URL", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt"}]
    }
  ]
  ```
</CodeGroup>

## Python example

<CodeGroup>
  ```python bulk import example theme={"theme":"css-variables"}
  from labelbox import Client
  from uuid import uuid4 ## to generate unique IDs
  import datetime

  client = Client(api_key="<YOUR_API_KEY>")

  dataset = client.create_dataset(name="Bulk import example - Audio")

  assets = [
    {
      "row_data": "https://storage.googleapis.com/labelbox-datasets/audio-sample-data/sample-audio-1.mp3",
      "global_key": "https://storage.googleapis.com/labelbox-datasets/audio-sample-data/sample-audio-1.mp3",
      "media_type": "AUDIO",
      "metadata_fields": [{"name": "<metadata_field_name>", "value": "tag_string"}],
      "attachments": [{"type": "VIDEO", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/drone_video.mp4" }]
    },
    {
      "row_data": "https://storage.googleapis.com/labelbox-datasets/audio-sample-data/sample-audio-2.mp3",
      "global_key": "https://storage.googleapis.com/labelbox-datasets/audio-sample-data/sample-audio-2.mp3",
      "media_type": "AUDIO",
      "metadata_fields": [{"name": "<metadata_field_name>", "value": "tag_string"}],
      "attachments": [{"type": "TEXT_URL", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt"}]
    }
  ]

  task = dataset.create_data_rows(assets)
  task.wait_till_done()
  print(task.errors)
  ```

  ```python local files theme={"theme":"css-variables"}
  local_file_paths = ['path/to/local/file1', 'path/to/local/file1'] # limit: 15k files

  new_dataset = client.create_dataset(name = "Local files upload")

  try:
      task = new_dataset.create_data_rows(local_file_paths)
      task.wait_till_done()
  except Exception as err:
      print(f'Error while creating labelbox dataset -  Error: {err}')
  ```
</CodeGroup>


# Import conversation & model response data
Source: https://docs.labelbox.com/horizon/developer-guides/import-conversation-model-response-data

How to import conversation text data & model response data.

## Specifications

File format: JSON in [our conversation format](#sample-conversation-json) Encoding: UTF-8 (Note: The Editor does not process any special character sequences like HTML Entities, Unicode Escape Sequence, or colon emoji aliases.)

Import methods:

* IAM Delegated Access
* Signed URLs (`https` URLs only)

When importing conversation or thread data to Labelbox, your JSON file should include the following information for each data row.

| Parameter | Required | Description |
| - | - | - |
| `row_data` | Yes | `https` path to a cloud-hosted conversational text JSON file. See the section below for details on our conversation format. For IAM Delegated Access, this URL must be in [virtual-hosted-style format](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#virtual-hosted-style-access). For older regions, your S3 bucket may be in `https://<bucket-name>.s3.<region>.amazonaws.com/<key>` format. If your object URLs are formatted this way, make sure they are in the virtual-hosted-style format before importing. |
| `global_key` | No | Unique user-generated file name or ID for the file. [Global keys](/horizon/developer-guides/data-row-global-keys) are enforced to be unique in your org. Data rows will not be imported if their global keys are duplicated to existing data rows. |
| `media_type` | No | `"CONVERSATIONAL"` (optional media type to provide better validation and error messaging) |
| `metadata_fields` | No | See [Metadata](/horizon/guides/datarow-metadata). |

## Import format

<CodeGroup>
  ```json json theme={"theme":"css-variables"}
  [
      {
        "row_data": "https://storage.googleapis.com/labelbox-datasets/conversational-sample-data/pairwise_shopping_1.json",
        "global_key": "global_key_1"
      },
      {
          "row_data": "https://storage.googleapis.com/labelbox-datasets/conversational-sample-data/pairwise_shopping_2.json",
          "global_key": "global_key_2"
      },
      {
          "row_data": "https://storage.googleapis.com/labelbox-datasets/conversational-sample-data/pairwise_shopping_3.json",
          "global_key": "global_key_3"
      }
  ]
  ```
</CodeGroup>

## Python example

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Generate dummy global keys
  global_key_1 = str(uuid.uuid4())
  global_key_2 = str(uuid.uuid4())
  global_key_3 = str(uuid.uuid4())

  # Create a dataset
  dataset = client.create_dataset(
      name="pairwise_demo_"+str(uuid.uuid4()),
      iam_integration=None
  )
  # Upload data rows
  task = dataset.create_data_rows([
      {
        "row_data": "https://storage.googleapis.com/labelbox-datasets/conversational-sample-data/pairwise_shopping_1.json",
        "global_key": global_key_1
      },
      {
          "row_data": "https://storage.googleapis.com/labelbox-datasets/conversational-sample-data/pairwise_shopping_2.json",
          "global_key": global_key_2
      },
      {
          "row_data": "https://storage.googleapis.com/labelbox-datasets/conversational-sample-data/pairwise_shopping_3.json",
          "global_key": global_key_3
      }
    ])
  task.wait_till_done()
  print("Errors:",task.errors)
  print("Failed data rows:", task.failed_data_rows)
  ```
</CodeGroup>

## Model Response Conversation JSON format

Each data row should have its own cloud-hosted JSON file. Each JSON file should include the following.

| Parameter | Required | Description |
| - | - | - |
| `type` | Yes | This should always be populated with `application/vnd.labelbox.conversational` |
| `version` | Yes | This should be populated with `1` |
| `messages` | Yes | This is the array of [Message Objects](/horizon/developer-guides/import-conversational-text-data#message-object) that makes up the conversation. Please see below for the fields that must be included for each message. The current maximum is 250 messages in a conversation |
| `modelOutputs` | Yes | This is the array of [Model outputs](/horizon/developer-guides/import-conversational-text-data#model-outputs-object) that makes up the model output to the conversation. See the section below for fields that must be included. |

### Message object

| Parameter | Required | Description |
| - | - | - |
| `messageId` | Yes | This is the external ID for the message. This must be unique within the conversation file and will be returned on the exports to help match annotations to message. |
| `timestampUsec` | No | The unix epoch timestamp of the message. |
| `content` | Yes | This field contains the text string of the conversation. This must be under 10000 characters. |
| `user` | Yes | This field is used to describe the person that is speaking. The user is defined by two fields: userId and name. The name field is used to show the user name in the editor. `{ "userId": "ID", "name": "Name" }` |
| `align` | No | This field Controls the alignment and indentation of the text string in the editor. If left blank, we will default to `left`. Options:`left``0-left-indent``1-left-indent``2-left-indent``3-left-indent``4-left-indent``5-left-indent``right``0-right-indent``1-right-indent``2-right-indent``3-right-indent``4-right-indent``5-right-indent` |

### Model outputs object

| Parameter | Required | Description |
| - | - | - |
| `title` | Yes | Use this field to name the model response |
| `content` | Yes | This field displays the content of the model output |
| `modelConfigName` | Yes | Use this field to specify the corresponding model config name |

## Sample Conversation JSON

<Info>
  ### Sample note

  This sample contains `modelOutputs`field which is unique to importing data rows for LLM human preference.
</Info>

<CodeGroup>
  ```json json theme={"theme":"css-variables"}
   {
    "type": "application/vnd.labelbox.conversational",
    "version": 1,
    "messages": [
      {
        "messageId": "message-0",
        "timestampUsec": 1530718491,
        "content": "Hi! How can I help?",
        "user": {
          "userId": "Bot 002",
          "name": "Bot"
        },
        "align": "left",
        "canLabel": false
      },
      {
        "messageId": "message-1",
        "timestampUsec": 1530718503,
        "content": "I just bought a vacuum cleaner from you guys like a week ago and it's already broken!!",
        "user": {
          "userId": "User 00686",
          "name": "User"
        },
        "align": "right",
        "canLabel": true
      }

    ],
    "modelOutputs": [
      {
        "title": "Response A",
        "content": "I'm really sorry to hear that your vacuum cleaner is not working as expected. We certainly understand how frustrating this can be, especially with a recent purchase. I assure you, we're here to help!\n\n To proceed with resolving this issue, could you please provide some details about the problem you're experiencing with the vacuum cleaner? Additionally, if you could share your order number or receipt details, that would be very helpful. We will definitely look into the available options for you, which may include a replacement or a refund, depending on the specific circumstances and our return policy.\n\n Your satisfaction is our priority, and we'll do our best to resolve this matter promptly and to your satisfaction.",
        "modelConfigName": "GPT-3.5 with temperature 0"
      },
      {
        "title": "Response B",
        "content": "I'm sorry about the issue with your vacuum cleaner. Please send us the order number or receipt details so we can quickly assist you with a replacement. Your satisfaction is important to us!",
        "modelConfigName": "Fine Tuned GPT-3.5 with demo data"
      }
    ]
  }
  ```
</CodeGroup>


# Import conversational text annotations
Source: https://docs.labelbox.com/horizon/developer-guides/import-conversational-text-annotations

Developer guide for importing annotations on conversational text data and sample import formats.

<CardGroup>
  <Card title="Open In Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/annotation_import/conversational.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/annotation_import/conversational.ipynb" />
</CardGroup>

## Overview

To import annotations in Labelbox, you need to create an annotations payload. In this section, we provide this payload for every supported annotation type.

### Annotation payload types

Labelbox supports two formats for the annotations payload:

* Python annotation types (recommended)
  * Provides a seamless transition between third-party platforms, machine learning pipelines, and Labelbox.
  * Allows you to build annotations locally with local file paths, numpy arrays, or URLs
  * Easily convert Python Annotation Type format to NDJSON format to quickly import annotations to Labelbox
  * Supports one-level nested classification (radio, checklist, or free-form text) under a tool or classification annotation.
* JSON
  * Skips formatting annotation payload in the Labelbox Python annotation type
  * Supports any levels of nested classification (radio, checklist, or free-form text) under a tool or classification annotation.

### Label import types

Labelbox additionally supports two types of label imports:

* [Model-assisted labeling (MAL)](/horizon/guides/model-assisted-labeling)
  * This workflow allows you to import computer-generated predictions (or simply annotations created outside of Labelbox) as pre-labels on an asset.
* [Ground truth](https://www.google.com/url?q=https%3A%2F%2Fdocs.labelbox.com%2Fdocs%2Fimport-ground-truth)
  * This workflow functionality allows you to bulk import your ground truth annotations from an external or third-party labeling system into Labelbox *Annotate*. Using the label import API to import external data is a useful way to consolidate and migrate all annotations into Labelbox as a single source of truth.

## Supported annotations

The following annotations are supported for a conversational text data row:

* Radio
* Checklist
* Free-form text
* Entity
* Relationship

<Info>
  ### Annotations can be message or global based

  Classification annotations can be supported both message and global based while tool annotations are supported only message specific. Removing the message\_id key inside a conversation text classification annotation results in the annotation becoming global.
</Info>

## Classifications

### Radio (single-choice, message-based)

<CodeGroup>
  ```python Python Annotation theme={"theme":"css-variables"}
  radio_annotation = lb_types.ClassificationAnnotation(
      name="radio_convo",
      value=lb_types.Radio(answer = lb_types.ClassificationAnswer(name = "first_radio_answer")),
      message_id="0" # Remove argument if importing annotation as a global classification (not message-based)
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  radio_annotation_ndjson = {
      "name": "radio_convo",
      "answer": {
          "name": "first_radio_answer"
      },
      "messageId": "0", # Remove argument if importing global classifications
  }
  ```
</CodeGroup>

### Checklist (multi-choice, message-based)

<CodeGroup>
  ```python Python Annotation theme={"theme":"css-variables"}
  checklist_annotation= lb_types.ClassificationAnnotation(
    name="checklist_convo", # must match your ontology feature"s name
    value=lb_types.Checklist(
        answer = [
          lb_types.ClassificationAnswer(
              name = "first_checklist_answer"
          ),
          lb_types.ClassificationAnswer(
              name = "second_checklist_answer"
          )
        ]
      ),
    message_id="2" # Remove argument if importing annotation as a global classification (not message-based)
   )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  checklist_annotation_ndjson = {
      "name": "checklist_convo",
      "answers": [
          {"name": "first_checklist_answer"},
          {"name": "second_checklist_answer"}
      ],
      "messageId": "2" # Remove argument if importing annotation as a global classification (not message-based)
  }
  ```
</CodeGroup>

### Free-form text (message-based)

<CodeGroup>
  ```python Python Annotation theme={"theme":"css-variables"}
  text_annotation = lb_types.ClassificationAnnotation(
      name="text_convo",
      value=lb_types.Text(answer="the answer to the text questions right here"),
      message_id="0" # Remove argument if importing annotation as a global classification (not message-based)
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  text_annotation_ndjson = {
      "name": "text_convo",
      "answer": "the answer to the text questions right here",
      "messageId": "0" # Remove argument if importing annotation as a global classification (not message-based)
  }
  ```
</CodeGroup>

## Tools

### Entity (message-based)

<CodeGroup>
  ```python Python Annotation theme={"theme":"css-variables"}
  ner_annotation = lb_types.ObjectAnnotation(
      name="ner",
      value=lb_types.ConversationEntity(
          start=0,
          end=8,
          message_id="4"
      )
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  ner_annotation = {
      "name": "ner",
      "location": {
        "start": 0,
        "end": 8
      },
      "messageId": "4" # this should match the message
  }
  ```
</CodeGroup>

## Relationships (message-based)

<CodeGroup>
  ```python Python Annotation theme={"theme":"css-variables"}
  ner_source = lb_types.ObjectAnnotation(
      name="ner",
      value=lb_types.ConversationEntity(
          start=16,
          end=26,
          message_id="4"
      )
  )
  ner_target = lb_types.ObjectAnnotation(
      name="ner",
      value=lb_types.ConversationEntity(
          start=29, 
          end=34, 
          message_id="4"
      )
  )

  ner_relationship = lb_types.RelationshipAnnotation(
      name="relationship",
      value=lb_types.Relationship(
          source=ner_source, # Alternatively, specify a source_ontology_name
          target=ner_target,
          type=lb_types.Relationship.Type.UNIDIRECTIONAL,
      ))
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  uuid_source = str(uuid.uuid4())
  uuid_target = str(uuid.uuid4())

  ner_source_ndjson = {
          "uuid": uuid_source,
          "name": "ner",
          "location": {
              "start": 16,
              "end": 26
          },
          "messageId": "4"
      }

  ner_target_ndjson = {
          "uuid": uuid_target,
          "name": "ner",
          "location": {
              "start": 29,
              "end": 34
          },
          "messageId": "4"
      }

  ner_relationship_annotation_ndjson = {
      "name": "relationship",
      "relationship": {
        "source": uuid_source, #UUID reference to the source annotation
        "target": uuid_target, # UUID reference to the target annotation
        "type": "bidirectional"
      }
  }
  ```
</CodeGroup>

## Example: Import pre-labels or ground truths

The steps to import annotations as pre-labels (machine-assisted learning) are similar to those to import annotations as ground truth labels. However, they vary slightly, and we will describe the differences for each scenario.

### Before you start

The below imports are needed to use the code examples in this section.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  import labelbox as lb
  import labelbox.types as lb_types
  import uuid
  import json
  import numpy as np
  ```
</CodeGroup>

Replace the value of `API_KEY` with a valid [API key](/horizon/developer-guides/create-api-key) to connect to the Labelbox client.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  API_KEY = None
  client = lb.Client(API_KEY)
  ```
</CodeGroup>

### Step 1: Import data rows

Data rows must first be uploaded to **Catalog** to attach annotations.

This example shows how to create a data row in **Catalog** by attaching it to a [dataset](/horizon/developer-guides/dataset) .

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Create one Labelbox dataset

  global_key = "conversation-1.json"

  asset = {
      "row_data": "https://storage.googleapis.com/labelbox-developer-testing-assets/conversational_text/1000-conversations/conversation-1.json",
      "global_key": global_key
  }

  dataset = client.create_dataset(name="conversational_annotation_import_demo_dataset")
  task = dataset.create_data_rows([asset])
  task.wait_till_done()
  print("Errors:", task.errors)
  print("Failed data rows: ", task.failed_data_rows)
  ```
</CodeGroup>

### Step 2: Set up ontology

Your project ontology should support the tools and classifications required by your annotations. To ensure accurate schema feature mapping, the value used as the `name` parameter should match the value of the `name` field in your annotation.

For example, when we created an annotation above, we provided a name`annotation_name`. Now, when we set up our ontology, we must ensure that the name of our bounding box tool is also `anotations_name`. The same alignment must hold true for the other tools and classifications we create in our ontology.

This example shows how to create an ontology containing all supported [annotation types](#supported-annotations) .

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  ontology_builder = lb.OntologyBuilder(
    tools=[
      lb.Tool(tool=lb.Tool.Type.NER,name="ner"),
      lb.Tool(tool=lb.Tool.Type.RELATIONSHIP,name="relationship")
      ],
    classifications=[
      lb.Classification(
        class_type=lb.Classification.Type.TEXT,
        scope=lb.Classification.Scope.INDEX,  # Remove this line or set scope to "GLOBAL" if importing global text annotations
        instructions="text_convo"),
      lb.Classification(
        class_type=lb.Classification.Type.CHECKLIST,
        scope=lb.Classification.Scope.INDEX,  # Remove this line or set scope to "GLOBAL" if importing global checklist annotations
        instructions="checklist_convo",
        options=[
          lb.Option(value="first_checklist_answer"),
          lb.Option(value="second_checklist_answer")
        ]
      ),
      lb.Classification(
        class_type=lb.Classification.Type.RADIO,
        instructions="radio_convo",
        scope=lb.Classification.Scope.INDEX, # Remove this line or set scope to "GLOBAL" if importing global radio  annotations
        options=[
          lb.Option(value="first_radio_answer"),
          lb.Option(value="second_radio_answer")
        ]
      )
    ]
  )

  ontology = client.create_ontology("Ontology Conversation Annotations", ontology_builder.asdict())
  ```
</CodeGroup>

### Step 3: Set Up a Labeling Project

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Create Labelbox project
  project = client.create_project(name="conversational_project",
                                      media_type=lb.MediaType.Conversational)

  # Setup your ontology

  project.connect_ontology(ontology) # Connect your ontology
  ```
</CodeGroup>

### Step 4: Send Data Rows to Project

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Setup Batches and Ontology

  # Create a batch to send to your MAL project
  batch = project.create_batch(
    "first-batch-convo-demo", # Each batch in a project must have a unique name
    global_keys=[global_key], # a list of global keys, data row ids or global keys
    priority=5 # priority between 1(highest) - 5(lowest)
  )

  print("Batch: ", batch)
  ```
</CodeGroup>

### Step 5: Create annotation payloads

For help understanding annotation payloads, see [overview](#overview). To declare payloads, you can use Python annotation types (*preferred*) or NDJSON objects.

These examples demonstrate each format and how to compose annotations into labels attached to data rows.

<CodeGroup>
  ```python Python Annotation Payload theme={"theme":"css-variables"}
  label = []
  label.append(
    lb_types.Label(
  		data={"global_key" : global_key },
      annotations=[
        ner_annotation,
        text_annotation,
        checklist_annotation,
        radio_annotation,
        ner_source,
        ner_target,
        ner_relationship
      ]
    )
  )
  ```

  ```python NDJSON Payload theme={"theme":"css-variables"}
  label_ndjson = []
  for annotations in [
      ner_annotation_ndjson,
      text_annotation_ndjson,
      checklist_annotation_ndjson,
      radio_annotation_ndjson,
      ner_source_ndjson,
      ner_target_ndjson,
      ner_relationship_annotation_ndjson,
      ]:
    annotations.update({
        "dataRow": {
            "globalKey": global_key
        }
    })
    label_ndjson.append(annotations)
  ```
</CodeGroup>

### Step 6: Import annotation payload

For prelabeled (model-assisted labeling) scenarios, pass your payload as the value of the `predictions` parameter. For ground truths, pass the payload to the `labels` parameter.

<Warning>
  ### Warning

  Relationship annotations are not supported for ground truth import jobs.
</Warning>

#### Option A: Upload as [prelabels (model assisted labeling)](/horizon/guides/model-assisted-labeling)

This option is helpful for speeding up the initial labeling process and reducing the manual labeling workload for high-volume datasets.

<CodeGroup>
  ```python MAL import theme={"theme":"css-variables"}
  # Upload MAL label for this data row in project
  upload_job = lb.MALPredictionImport.create_from_objects(
      client = client,
      project_id = project.uid,
      name="mal_job"+str(uuid.uuid4()),
      predictions=label
  )

  print(f"Errors: {upload_job.errors}", )
  print(f"Status of uploads: {upload_job.statuses}")
  ```
</CodeGroup>

#### Option B: Upload to a labeling project as [ground truth](/horizon/guides/import-ground-truth)

This option is helpful for loading high-confidence labels from another platform or previous projects that just need review rather than manual labeling effort.

<CodeGroup>
  ```python Label import theme={"theme":"css-variables"}
  # Upload label for this data row in project
  upload_job = lb.LabelImport.create_from_objects(
      client = client,
      project_id = project.uid,
      name="label_import_job"+str(uuid.uuid4()),
      labels=label
  )

  print(f"Errors: {upload_job.errors}", )
  print(f"Status of uploads: {upload_job.statuses}")
  ```
</CodeGroup>


# Import conversational text data
Source: https://docs.labelbox.com/horizon/developer-guides/import-conversational-text-data

How to import conversational text data and sample import formats.

## Specifications

File format: JSON in [our conversation format](#sample-conversation-json) Encoding: UTF-8 (Note: The Editor does not process any special character sequences like HTML Entities, Unicode Escape Sequence, or colon emoji aliases.)

Import methods:

* IAM Delegated Access
* Signed URLs (`https` URLs only)

When importing conversation or thread data to Labelbox, your JSON file should include the following information for each data row.

| Parameter | Required | Description |
| - | - | - |
| `row_data` | Yes | `https` path to a cloud-hosted conversational text JSON file. See the section below for details on our conversation format. For IAM Delegated Access, this URL must be in [virtual-hosted-style format](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#virtual-hosted-style-access). For older regions, your S3 bucket may be in `https://<bucket-name>.s3.<region>.amazonaws.com/<key>` format. If your object URLs are formatted this way, make sure they are in the virtual-hosted-style format before importing. |
| `global_key` | No | Unique user-generated file name or ID for the file. [Global keys](/horizon/developer-guides/data-row-global-keys) are enforced to be unique in your org. Data rows will not be imported if their global keys are duplicated to existing data rows. |
| `media_type` | No | `"CONVERSATIONAL"` (optional media type to provide better validation and error messaging) |
| `metadata_fields` | No | See [Metadata](/horizon/guides/datarow-metadata). |

## Import format

<CodeGroup>
  ```json json theme={"theme":"css-variables"}
  [
      {
        "row_data": "https://storage.googleapis.com/labelbox-datasets/conversational-sample-data/pairwise_shopping_1.json",
        "global_key": "global_key_1"
      },
      {
          "row_data": "https://storage.googleapis.com/labelbox-datasets/conversational-sample-data/pairwise_shopping_2.json",
          "global_key": "global_key_2"
      },
      {
          "row_data": "https://storage.googleapis.com/labelbox-datasets/conversational-sample-data/pairwise_shopping_3.json",
          "global_key": "global_key_3"
      }
  ]
  ```
</CodeGroup>

## Python example

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Generate dummy global keys
  global_key_1 = str(uuid.uuid4())
  global_key_2 = str(uuid.uuid4())
  global_key_3 = str(uuid.uuid4())

  # Create a dataset
  dataset = client.create_dataset(
      name="pairwise_demo_"+str(uuid.uuid4()),
      iam_integration=None
  )
  # Upload data rows
  task = dataset.create_data_rows([
      {
        "row_data": "https://storage.googleapis.com/labelbox-datasets/conversational-sample-data/pairwise_shopping_1.json",
        "global_key": global_key_1
      },
      {
          "row_data": "https://storage.googleapis.com/labelbox-datasets/conversational-sample-data/pairwise_shopping_2.json",
          "global_key": global_key_2
      },
      {
          "row_data": "https://storage.googleapis.com/labelbox-datasets/conversational-sample-data/pairwise_shopping_3.json",
          "global_key": global_key_3
      }
    ])
  task.wait_till_done()
  print("Errors:",task.errors)
  print("Failed data rows:", task.failed_data_rows)
  ```
</CodeGroup>

## Conversation JSON

<Info>
  ### Max conversation size

  For conversations, we support up to:

  * 250 messages.
  * 10000 character limit per message
</Info>

| Parameter | Required | Description |
| - | - | - |
| `type` | Yes | This should always be populated with `application/vnd.labelbox.conversational` |
| `version` | Yes | This should be populated with `1` |
| `messages` | Yes | This is the array of [Message Objects](/horizon/developer-guides/import-conversational-text-data#message-object) that makes up the conversation. Please see below for the fields that must be included for each message. The current maximum is 250 messages in a conversation |

### Message object

| Parameter | Required | Description |
| - | - | - |
| `messageId` | Yes | This is the external ID for the message. This must be unique within the conversation file and will be returned on the exports to help match annotations to message. |
| `timestampUsec` | No | The unix epoch timestamp of the message. |
| `content` | Yes | This field contains the text string of the conversation. This must be under 10000 characters. |
| `user` | Yes | This field is used to describe the person that is speaking. The user defined by two fields: userId and name. The name field is used to show the user name in the editor. `{ "userId": "ID", "name": "Name" }` |
| `align` | No | This field Controls the alignment and indentation of the text string in the editor. If left blank, we will default to `left`. Options:`left``0-left-indent``1-left-indent``2-left-indent``3-left-indent``4-left-indent``5-left-indent``right``0-right-indent``1-right-indent``2-right-indent``3-right-indent``4-right-indent``5-right-indent` |
| `canLabel` | Yes | This field determines if the text string can be annotated as a part of the editor workflow. If set to `true`, the editor will show a white background for the text string and the user will be able to annotated entities on the content string. If set to `false`, the editor will show a grey background for the text and the user will **not** be able to annotate the content string. |

## Sample Conversation JSON

<Info>
  ### Note

  You can't upload the following file from the web interface directly. You must use an import file as described in [Import format](/horizon/developer-guides/import-conversational-text-data#import-format)
</Info>

<CodeGroup>
  ```json Sample conversation json expandable theme={"theme":"css-variables"}
  {
    "type": "application/vnd.labelbox.conversational",
    "version": 1,
    "messages": [
      {
        "messageId": "message-0",
        "timestampUsec": 1530718491,
        "content": "I love iphone! i just bought new iphone! 🥰 📲",
        "user": {
          "userId": "Bot 002",
          "name": "Bot"
        },
        "align": "left",
        "canLabel": false
      },
      {
        "messageId": "message-1",
        "timestampUsec": 1530718503,
        "content": "Thats good for you, i'm not very into new tech",
        "user": {
          "userId": "User 00686",
          "name": "User"
        },
        "align": "right",
        "canLabel": true
      },
      {
        "messageId": "message-2",
        "timestampUsec": 1530718516,
        "content": "I am a college student and i am a college student 👩🏻‍🏫",
        "user": {
          "userId": "Bot 002",
          "name": "Bot"
        },
        "align": "left",
        "canLabel": false
      },
      {
        "messageId": "message-3",
        "timestampUsec": 1530718528,
        "content": "I am go to gym and live on donations 🏋🏿‍♀️",
        "user": {
          "userId": "User 00686",
          "name": "User"
        },
        "align": "right",
        "canLabel": true
      },
      {
        "messageId": "message-4",
        "timestampUsec": 1530718540,
        "content": "I am a vegan and i am in the midwest 🌱",
        "user": {
          "userId": "Bot 002",
          "name": "Bot"
        },
        "align": "left",
        "canLabel": false
      },
      {
        "messageId": "message-5",
        "timestampUsec": 1530718553,
        "content": "So vegan... i have dogs maybe i should told then that they may eat cheap salads insted of meat 🐩",
        "user": {
          "userId": "User 00686",
          "name": "User"
        },
        "align": "right",
        "canLabel": true
      },
      {
        "messageId": "message-6",
        "timestampUsec": 1530718565,
        "content": "I would not mind having them in the office that would be hard for me",
        "user": {
          "userId": "Bot 002",
          "name": "Bot"
        },
        "align": "left",
        "canLabel": false
      },
      {
        "messageId": "message-7",
        "timestampUsec": 1530718578,
        "content": "Dogs or vegan in office?",
        "user": {
          "userId": "User 00686",
          "name": "User"
        },
        "align": "right",
        "canLabel": true
      },
      {
        "messageId": "message-8",
        "timestampUsec": 1530718590,
        "content": "I am a vegetarian so i am vegan",
        "user": {
          "userId": "Bot 002",
          "name": "Bot"
        },
        "align": "left",
        "canLabel": false
      },
      {
        "messageId": "message-9",
        "timestampUsec": 1530718603,
        "content": "Strange answer",
        "user": {
          "userId": "User 00686",
          "name": "User"
        },
        "align": "right",
        "canLabel": true
      },
      {
        "messageId": "message-10",
        "timestampUsec": 1530718615,
        "content": "I guess i just do not know what to ps4 experience with.",
        "user": {
          "userId": "Bot 002",
          "name": "Bot"
        },
        "align": "left",
        "canLabel": false
      }
    ]
  }
  ```
</CodeGroup>


# Import document annotations
Source: https://docs.labelbox.com/horizon/developer-guides/import-document-annotations

Developer guide for importing annotations on document (PDF) data and sample import formats.

<CardGroup>
  <Card title="Open In Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/annotation_import/pdf.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/annotation_import/pdf.ipynb" />
</CardGroup>

## Overview

To import annotations in Labelbox, you need to create an annotations payload. In this section, we provide this payload for every supported annotation type.

### Annotation payload types

Labelbox supports two formats for the annotations payload:

* Python annotation types (recommended)

  * Provides a seamless transition between third-party platforms, machine learning pipelines, and Labelbox.
  * Allows you to build annotations locally with local file paths, numpy arrays, or URLs
  * Easily convert Python Annotation Type format to NDJSON format to quickly import annotations to Labelbox
  * Supports one-level nested classification (radio, checklist, or free-form text) under a tool or classification annotation.

* JSON

  * Skips formatting annotation payload in the Labelbox Python annotation type
  * Supports any levels of nested classification (radio, checklist, or free-form text) under a tool or classification annotation.

### Label import types

Labelbox additionally supports two types of label imports:

* [Model-assisted labeling (MAL)](/horizon/guides/model-assisted-labeling)
  * This workflow allows you to import computer-generated predictions (or simply annotations created outside of Labelbox) as pre-labels on an asset.
* [Ground truth](https://www.google.com/url?q=https%3A%2F%2Fdocs.labelbox.com%2Fdocs%2Fimport-ground-truth)
  * This workflow functionality allows you to bulk import your ground truth annotations from an external or third-party labeling system into Labelbox *Annotate*. Using the label import API to import external data is a useful way to consolidate and migrate all annotations into Labelbox as a single source of truth.

## Supported annotations

The following annotations are supported for a document data row:

* Radio
* Checklist
* Free-form text
* Bounding box
* Entity
* Relationship

<Info>
  ### Annotations are global and page based

  Classification annotations are only supported globally while page-specific tool annotations are supported at a per page basis.
</Info>

## Classifications

### Radio (single-choice, global)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  radio_annotation = lb_types.ClassificationAnnotation(
      name="radio_question",
      value=lb_types.Radio(answer =
          lb_types.ClassificationAnswer(name = "first_radio_answer")
      )
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  radio_annotation_ndjson = {
    "name": "radio_question",
    "answer": {"name": "first_radio_answer"}
  }
  ```
</CodeGroup>

### Checklist (multi-choice, global)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  checklist_annotation = lb_types.ClassificationAnnotation(
      name="checklist_question",
      value=lb_types.Checklist(answer = [
          lb_types.ClassificationAnswer(name = "first_checklist_answer"),
          lb_types.ClassificationAnswer(name = "second_checklist_answer")
      ])
    )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  checklist_annotation_ndjson = {
    "name": "checklist_question",
    "answer": [
      {"name": "first_checklist_answer"},
      {"name": "second_checklist_answer"}
    ]
  }
  ```
</CodeGroup>

### Free-form text (global)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  text_annotation = lb_types.ClassificationAnnotation(
    name="free_text",  # must match your ontology feature"s name
    value=lb_types.Text(answer="sample text")
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  text_annotation_ndjson = {
    "name": "free_text",
    "answer": "sample text"
  }
  ```
</CodeGroup>

### Nested Classifications (global)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  nested_checklist_annotation = lb_types.ClassificationAnnotation(
    name="nested_checklist_question",
    value=lb_types.Checklist(
      answer=[lb_types.ClassificationAnswer(
        name="first_checklist_answer",
        classifications=[
          lb_types.ClassificationAnnotation(
            name="sub_checklist_question",
            value=lb_types.Checklist(
              answer=[lb_types.ClassificationAnswer(
              name="first_sub_checklist_answer"
            )]
          ))
        ]
      )]
    )
  )

  nested_radio_annotation = lb_types.ClassificationAnnotation(
    name="nested_radio_question",
    value=lb_types.Radio(
      answer=lb_types.ClassificationAnswer(
        name="first_radio_answer",
        classifications=[
          lb_types.ClassificationAnnotation(
            name="sub_radio_question",
            value=lb_types.Radio(
              answer=lb_types.ClassificationAnswer(
                name="first_sub_radio_answer"
              )
            )
          )
        ]
      )
    )
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  nested_checklist_annotation_ndjson = {
    "name": "nested_checklist_question",
    "answer": [{
        "name": "first_checklist_answer",
        "classifications" : [
          {
            "name": "sub_checklist_question",
            "answer": {"name": "first_sub_checklist_answer"}
          }
        ]
    }]
  }

  nested_radio_annotation_ndjson = {
    "name": "nested_radio_question",
    "answer": {
        "name": "first_radio_answer",
        "classifications": [{
            "name":"sub_radio_question",
            "answer": { "name" : "first_sub_radio_answer"}
          }]
      }
  }
  ```
</CodeGroup>

## Tools

### Bounding box (page specific)

When importing a bounding box annotation, you need to specify its `DocumentRectangle` value, which defines its precise area on a document page. You can choose from the following `RectangleUnit` options, which determine the measurement unit used to define the dimensions and coordinates of the bounding box:

* INCHES
* PIXELS
* POINTS
* PERCENT

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  bbox_annotation = lb_types.ObjectAnnotation(
      name="bounding_box",  # must match your ontology feature's name
      value=lb_types.DocumentRectangle(
          start=lb_types.Point(x=102.771, y=135.3),  # x = left, y = top
          end=lb_types.Point(x=518.571, y=245.143),  # x = left + width , y = top + height
          page=0,
          unit=lb_types.RectangleUnit.POINTS
          )
      )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  bbox_annotation_ndjson = {
    "name": "bounding_box",
    "bbox": {
              "top": 135.3,
              "left": 102.771,
              "height": 109.843,
              "width": 415.8
        },
    "page": 0,
    "unit": "POINTS"
  }
  ```
</CodeGroup>

### Entity (page specific)

`textSelections` is the payload required for each entity annotation. Each`textSelections` item in the list requires the following fields:

* The`groupId` associated with a group of words.
* A list of `tokenIds` for each word in the group of words.
* The `page` of the document (1-indexed).

Both `tokenIds` and `groupdId` can extracted from the text layer URL attached to the data row. For more information on text layers, visit our [import document data guide](/horizon/developer-guides/import-document-data).

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  entities_annotations = lb_types.ObjectAnnotation(
      name="named_entity",
      value= lb_types.DocumentEntity(
          name="named_entity",
          textSelections=[
              lb_types.DocumentTextSelection(
                  token_ids=[],
                  group_id="",
                  page=1
              )
          ]
      )
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  entities_annotations_ndjson = {
      "name": "named_entity",
      "textSelections": [
          {
              "tokenIds": [
                  "<UUID>", ## ids associated with each word in a group
              ],
              "groupId": "<UUID>", ## id associated with a group of words
              "page": 1,
          }
      ]
  }
  ```
</CodeGroup>

### Tool with nested classifications (page specific)

<CodeGroup>
  ```python Bounding Box Example theme={"theme":"css-variables"}
  bbox_with_radio_subclass_annotation = lb_types.ObjectAnnotation(
      name="bbox_with_radio_subclass",
      value=lb_types.DocumentRectangle(
          start=lb_types.Point(x=317.271, y=226.757), # x = left, y = top
          end=lb_types.Point(x=566.657, y=420.986), # x = left + width , y = top + height
          unit=lb_types.RectangleUnit.POINTS,
          page=1
      ),
      classifications=[
      	lb_types.ClassificationAnnotation(
          	name="sub_radio_question",
        		value=lb_types.Radio(
            answer=lb_types.ClassificationAnswer(
              name="first_sub_radio_answer",
              classifications=[
                lb_types.ClassificationAnnotation(
                  name="second_sub_radio_question",
                  value=lb_types.Radio(
                    answer=lb_types.ClassificationAnswer(
                      name="second_sub_radio_answer"
                    )
                  )
                )
              ]
            )
            )
          )
      ]
  )
  ```

  ```python Entity Example theme={"theme":"css-variables"}
  ner_with_checklist_subclass_annotation = lb_types.ObjectAnnotation(
    name="ner_with_checklist_subclass",
    value=lb_types.DocumentEntity(
      name="ner_with_checklist_subclass",
      text_selections=[
        lb_types.DocumentTextSelection(
          token_ids=[],
          group_id="",
          page=1
        )
      ]
    ),
    classifications=[
      lb_types.ClassificationAnnotation(
        name="sub_checklist_question",
        value=lb_types.Checklist(
        answer=[lb_types.ClassificationAnswer(name="first_sub_checklist_answer")]
        )
      )
    ]
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  bbox_with_radio_subclass_annotation_ndjson = {
    "name": "bbox_with_radio_subclass",
    "classifications": [
      {
        "name": "sub_radio_question",
        "answer": {
            "name": "first_sub_radio_answer",
            "classifications": [
                {
                    "name": "second_sub_radio_question",
                    "answer": {
                        "name": "second_sub_radio_answer"}
                 }
              ]
          }
      }
    ],
    "bbox": {
          "top": 226.757,
          "left": 317.271,
          "height": 194.229,
          "width": 249.386
      },
    "page": 1,
    "unit": "POINTS"
  }
  ```
</CodeGroup>

## Relationships (page specific)

<CodeGroup>
  ```python Bounding Box Example theme={"theme":"css-variables"}
  bbox_source = lb_types.ObjectAnnotation(
      name="bounding_box",
      value=lb_types.DocumentRectangle(
          start=lb_types.Point(x=188.257, y=68.875), # x = left, y = top 
          end=lb_types.Point(x=270.907, y=149.556), # x = left + width , y = top + height
          unit=lb_types.RectangleUnit.POINTS,
          page=1
      ),
  )

  bbox_target = lb_types.ObjectAnnotation(
      name="bounding_box",
      value=lb_types.DocumentRectangle(
          start=lb_types.Point(x=96.424, y=66.251),
          end=lb_types.Point(x=179.074, y=146.932),
          unit=lb_types.RectangleUnit.POINTS,
          page=1
      ),
  )

  bbox_relationship = lb_types.RelationshipAnnotation(
      name="relationship",
      value=lb_types.Relationship(
          source=bbox_source, # Alternatively, specify a source_ontology_name
          target=bbox_target,
          type=lb_types.Relationship.Type.UNIDIRECTIONAL,
      ))
  ```

  ```python Entity Example theme={"theme":"css-variables"}
  entity_source = lb_types.ObjectAnnotation(
      name="named_entity",
      value= lb_types.DocumentEntity(
          name="named_entity",
          textSelections=[
              lb_types.DocumentTextSelection(
                  token_ids=[],
                  group_id="",
                  page=1
              )
          ]
      )
  )

  entity_target = lb_types.ObjectAnnotation(
      name="named_entity",
      value=lb_types.DocumentEntity(
        name="named_entity",
        textSelections=[
          lb_types.DocumentTextSelection(
            token_ids=[],
            group_id="",
            page=1
          )
        ]
      )
  )

  entity_relationship = lb_types.RelationshipAnnotation(
      name="relationship",
      value=lb_types.Relationship(
          source=entity_source, # Alternatively, specify a source_ontology_name
          target=entity_target,
          type=lb_types.Relationship.Type.UNIDIRECTIONAL,
      ))
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  uuid_source = str(uuid.uuid4())
  uuid_target = str(uuid.uuid4())

  entity_source_ndjson = {
    "name": "named_entity",
    "uuid": uuid_source,
    "textSelections": [
      {
        "tokenIds": [
          ""
        ],
        "groupId": "",
        "page": 1
      }
    ]

  }

  entity_target_ndjson = {
    "name": "named_entity",
    "uuid": uuid_target,
    "textSelections": [
      {
        "tokenIds": [
          ""
        ],
        "groupId": "",
        "page": 1
      }
    ]
  }
  ner_relationship_annotation_ndjson = {
      "name": "relationship",
      "relationship": {
        "source": uuid_source, # UUID reference to the source annotation.
        "target": uuid_target, # UUID reference to the target annotation
        "type": "unidirectional"
      }
  }
  ```
</CodeGroup>

### Read-only relationships

<Warning>
  ### This is a [beta](/horizon/guides/product-release-phases#beta) feature
</Warning>

Admins can mark relationships as read\_only to prevent other roles from deleting them:

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  bbox_source = lb_types.ObjectAnnotation(
      name="bounding_box",
      value=lb_types.DocumentRectangle(
          start=lb_types.Point(x=188.257, y=68.875), # x = left, y = top 
          end=lb_types.Point(x=270.907, y=149.556), # x = left + width , y = top + height
          unit=lb_types.RectangleUnit.POINTS,
          page=1
      ),
  )

  bbox_target = lb_types.ObjectAnnotation(
      name="bounding_box",
      value=lb_types.DocumentRectangle(
          start=lb_types.Point(x=96.424, y=66.251),
          end=lb_types.Point(x=179.074, y=146.932),
          unit=lb_types.RectangleUnit.POINTS,
          page=1
      ),
  )

  bbox_relationship = lb_types.RelationshipAnnotation(
      name="relationship",
      value=lb_types.Relationship(
          source=bbox_source, # Alternatively, specify a source_ontology_name
          target=bbox_target,
          type=lb_types.Relationship.Type.UNIDIRECTIONAL,
          readonly=True,
      ))
  ```
</CodeGroup>

### Relationships between classifications and objects

<Warning>
  ### This is a [beta](/horizon/guides/product-release-phases#beta) feature
</Warning>

<CodeGroup>
  ```python Linking multiple objects to classifications expandable theme={"theme":"css-variables"}
  radio_annotation = lb_types.ClassificationAnnotation(
      name="radio_pdf",
      value=lb_types.Radio(
          answer=lb_types.ClassificationAnswer(name="second_radio_answer")
      ),
  )

  checklist_annotation = lb_types.ClassificationAnnotation(
      name="checklist_pdf",  # must match your ontology feature's name
      value=lb_types.Checklist(
          answer=[
              lb_types.ClassificationAnswer(name="first_checklist_answer"),
              lb_types.ClassificationAnswer(name="second_checklist_answer"),
          ]
      ),
  )

  text_annotation = lb_types.ClassificationAnnotation(
      name="text_pdf", value=lb_types.Text(answer="sample text")
  )

  bbox_annotation = lb_types.ObjectAnnotation(
      name="bounding_box",
      value=lb_types.DocumentRectangle(
          start=lb_types.Point(x=0.5, y=0.5),  #  x = left, y = top
          end=lb_types.Point(x=0.7, y=0.7),  # x= left + width , y = top + height
          page=1,
          unit=RectangleUnit.PERCENT,
      ),
  )

  bbox_annotation_2 = lb_types.ObjectAnnotation(
      name="bounding_box",
      value=lb_types.DocumentRectangle(
          start=lb_types.Point(x=0.5, y=0.5),
          end=lb_types.Point(x=0.7, y=0.7),
          page=2,
          unit=RectangleUnit.PERCENT,
      ),
  )

  entity_annotation = lb_types.ObjectAnnotation(
      name="named_entity",
      value=lb_types.DocumentEntity(
          name="named_entity",
          textSelections=[
              lb_types.DocumentTextSelection(
                  token_ids=["9763b68b-9ea7-472d-af40-20ab69fbee71"],
                  group_id="ee108a07-059a-4302-910f-b16eb148f123",
                  page=1,
              )
          ],
      ),
  )

  text_bbox_relationship = lb_types.RelationshipAnnotation(
      name="relationship",
      value=lb_types.Relationship(
          source=text_annotation, # Alternatively, specify a source_ontology_name
          target=bbox_annotation,
          type=lb_types.Relationship.Type.UNIDIRECTIONAL,
      ),
  )

  radio_bbox_relationship = lb_types.RelationshipAnnotation(
      name="relationship",
      value=lb_types.Relationship(
          source=radio_annotation, # Alternatively, specify a source_ontology_name
          target=bbox_annotation,
          type=lb_types.Relationship.Type.UNIDIRECTIONAL,
      ),
  )

  checklist_bbox_relationship = lb_types.RelationshipAnnotation(
      name="relationship",
      value=lb_types.Relationship(
          source=checklist_annotation, # Alternatively, specify a source_ontology_name
          target=bbox_annotation,
          type=lb_types.Relationship.Type.UNIDIRECTIONAL,
      ),
  )

  text_entity_relationship = lb_types.RelationshipAnnotation(
      name="relationship",
      value=lb_types.Relationship(
          source=text_annotation, # Alternatively, specify a source_ontology_name
          target=entity_annotation,
          type=lb_types.Relationship.Type.UNIDIRECTIONAL,
      ),
  )

  bbox_bbox2_relationship = lb_types.RelationshipAnnotation(
      name="relationship",
      value=lb_types.Relationship(
          source=bbox_annotation, # Alternatively, specify a source_ontology_name
          target=bbox_annotation_2,
          type=lb_types.Relationship.Type.UNIDIRECTIONAL,
      ),
  )
  ```

  ```python NDJSON example theme={"theme":"css-variables"}
  uuid_radio = str(uuid.uuid4())
  uuid_checklist = str(uuid.uuid4())
  uuid_text = str(uuid.uuid4())
  uuid_bbox_1 = str(uuid.uuid4())
  uuid_bbox_2 = str(uuid.uuid4())
  uuid_entity = str(uuid.uuid4())

  # Radio annotation
  radio_annotation_ndjson = {
      "name": "radio_pdf",
      "radio": {
          "answer": {
              "name": "second_radio_answer"
          }
      }
  }

  # Checklist annotation
  checklist_annotation_ndjson = {
      "name": "checklist_pdf",
      "checklist": {
          "answer": [
              {"name": "first_checklist_answer"},
              {"name": "second_checklist_answer"}
          ]
      }
  }

  # Text annotation
  text_annotation_ndjson = {
      "name": "text_pdf",
      "text": {
          "answer": "sample text"
      }
  }

  # Bounding box annotation 1
  bbox_annotation_ndjson = {
      "name": "bounding_box",
      "documentRectangle": {
          "start": {"x": 0.5, "y": 0.5},
          "end": {"x": 0.7, "y": 0.7},
          "page": 1,
          "unit": "PERCENT"
      }
  }

  # Bounding box annotation 2
  bbox_annotation_2_ndjson = {
      "name": "bounding_box",
      "documentRectangle": {
          "start": {"x": 0.5, "y": 0.5},
          "end": {"x": 0.7, "y": 0.7},
          "page": 2,
          "unit": "PERCENT"
      }
  }

  # Named entity annotation
  entity_annotation_ndjson = {
      "name": "named_entity",
      "documentEntity": {
          "name": "named_entity",
          "textSelections": [
              {
                  "tokenIds": ["9763b68b-9ea7-472d-af40-20ab69fbee71"],
                  "groupId": "ee108a07-059a-4302-910f-b16eb148f123",
                  "page": 1
              }
          ]
      }
  }

  # Relationships
  text_bbox_relationship_ndjson = {
      "name": "relationship",
      "relationship": {
          "source": uuid_text,
          "target": uuid_bbox_1,
          "type": "unidirectional"
      }
  }

  radio_bbox_relationship_ndjson = {
      "name": "relationship",
      "relationship": {
          "source": uuid_radio,
          "target": uuid_bbox_1,
          "type": "unidirectional"
      }
  }

  checklist_bbox_relationship_ndjson = {
      "name": "relationship",
      "relationship": {
          "source": uuid_checklist,
          "target": uuid_bbox_1,
          "type": "unidirectional"
      }
  }

  text_entity_relationship_ndjson = {
      "name": "relationship",
      "relationship": {
          "source": uuid_text,
          "target": uuid_entity,
          "type": "unidirectional"
      }
  }

  bbox_bbox2_relationship_ndjson = {
      "name": "relationship",
      "relationship": {
          "source": uuid_bbox_1,
          "target": uuid_bbox_2,
          "type": "unidirectional"
      }
  }
  ```
</CodeGroup>

## Creating text entity annotation

For importing entity annotations, you can use your own `text_layer_url` or a Labelbox-generated `text_layer_url`.

You can get the Labelbox-generated `text_layer_url` by exporting the data row. The below code snippet demonstrates this process by exporting from a data row.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Export data row
  task = lb.DataRow.export(client=client,global_keys=[global_key])
  task.wait_till_done()

  if task.has_result():
    stream = task.get_buffered_stream()

    text_layer = ""
    for output in stream:
        output_json = output.json
        text_layer = output_json['media_attributes']['text_layer_url']
    print(text_layer)
  ```
</CodeGroup>

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  import requests
  import json

  # Helper method for updating text selections
  def update_text_selections(annotation, group_id, list_tokens, page):
    return annotation.update({
      "textSelections": [
        {
          "groupId": group_id,
          "tokenIds": list_tokens,
          "page": page
        }
      ]
    })

  # Fetch the content of the text layer
  res = requests.get(text_layer)

  # Phrases that we want to annotation obtained from the text layer url
  content_phrases = ["Metal-insulator (MI) transitions have been one of the" ,
                     "T. Sasaki, N. Yoneyama, and N. Kobayashi",,
                     "Organic charge transfer salts based on the donor",
                     "the experimental investigations on this issue have not"]

  # Parse the text layer
  text_selections = []
  text_selections_ner = []
  text_selections_source = []
  text_selections_target = []

  for obj in json.loads(res.text):
    for group in obj["groups"]:
      if group["content"] == content_phrases[0]:
        list_tokens = [x["id"] for x in group["tokens"]]
        # build text selections for Python Annotation Types
        document_text_selection = lb_types.DocumentTextSelection(groupId=group["id"], tokenIds=list_tokens, page=1)
        text_selections.append(document_text_selection)
        # build text selection for the NDJson annotations
        update_text_selections(annotation=entities_annotations_ndjson,
                               group_id=group["id"], # id representing group of words
                               list_tokens=list_tokens, # ids representing individual words from the group
                               page=1)
      if group["content"] == content_phrases[1]:
        list_tokens_2 = [x["id"] for x in group["tokens"]]
        # build text selections for Python Annotation Types
        ner_text_selection = lb_types.DocumentTextSelection(groupId=group["id"], tokenIds=list_tokens_2, page=1)
        text_selections_ner.append(ner_text_selection)
        # build text selection for the NDJson annotations
        update_text_selections(annotation=ner_with_checklist_subclass_annotation_ndjson,
                               group_id=group["id"], # id representing group of words
                               list_tokens=list_tokens_2, # ids representing individual words from the group
                               page=1)
      if group["content"] == content_phrases[2]:
        relationship_source = [x["id"] for x in group["tokens"]]
        # build text selections for Python Annotation Types
        text_selection_entity_source = lb_types.DocumentTextSelection(groupId=group["id"], tokenIds=relationship_source, page=1)
        text_selections_source.append(text_selection_entity_source)
        # build text selection for the NDJson annotations
        update_text_selections(annotation=entity_source_ndjson,
                               group_id=group["id"], # id representing group of words
                               list_tokens=relationship_source, # ids representing individual words from the group
                               page=1)
      if group["content"] == content_phrases[3]:
          relationship_target = [x["id"] for x in group["tokens"]]
          # build text selections for Python Annotation Types
          text_selection_entity_target =  lb_types.DocumentTextSelection(group_id=group["id"], tokenIds=relationship_target, page=1)
          text_selections_target.append(text_selection_entity_target)
          # build text selections forthe NDJson annotations
          update_text_selections(annotation=entity_target_ndjson,
                                 group_id=group["id"], # id representing group of words
                                 list_tokens=relationship_target, # ids representing individual words from the group
                                 page=1)

  ```
</CodeGroup>

Re-write the Python annotations to include text selections (only required for Python annotation types)

<CodeGroup>
  ```python theme={"theme":"css-variables"}
  # re-write the entity annotation with text selections 
  entities_annotation_document_entity = lb_types.DocumentEntity(name="named_entity", textSelections = text_selections)
  entities_annotation = lb_types.ObjectAnnotation(name="named_entity",value=entities_annotation_document_entity)

  # re-write the entity annotation + subclassification with text selections 
  classifications = [
      lb_types.ClassificationAnnotation(
        name="sub_checklist_question",
        value=lb_types.Checklist(
        answer=[lb_types.ClassificationAnswer(name="first_sub_checklist_answer")]
        )
      )
    ]
  ner_annotation_with_subclass = lb_types.DocumentEntity(name="ner_with_checklist_subclass", textSelections= text_selections_ner)
  ner_with_checklist_subclass_annotation = lb_types.ObjectAnnotation(name="ner_with_checklist_subclass", 
                                                                     value=ner_annotation_with_subclass, 
                                                                     classifications=classifications)

  # re-write the entity source and target annotations withe text selectios
  entity_source_doc = lb_types.DocumentEntity(name="named_entity", text_selections= text_selections_source)
  entity_source = lb_types.ObjectAnnotation(name="named_entity", value=entity_source_doc)

  entity_target_doc = lb_types.DocumentEntity(name="named_entity",  text_selections=text_selections_target)
  entity_target = lb_types.ObjectAnnotation(name="named_entity", value=entity_target_doc)

  # re-write the entity relationship with the re-created entities 
  entity_relationship = lb_types.RelationshipAnnotation(
      name="relationship",
      value=lb_types.Relationship(
          source=entity_source,
          target=entity_target,
          type=lb_types.Relationship.Type.UNIDIRECTIONAL,
      ))


  print(f"entities_annotations_ndjson={entities_annotations_ndjson}")
  print(f"entities_annotation={entities_annotation}")
  print(f"nested_entities_annotation_ndjson={ner_with_checklist_subclass_annotation_ndjson}")
  print(f"nested_entities_annotation={ner_with_checklist_subclass_annotation}")
  print(f"entity_source_ndjson={entity_source_ndjson}")
  print(f"entity_target_ndjson={entity_target_ndjson}")
  print(f"entity_source={entity_source}")
  print(f"entity_target={entity_target}")
  ```
</CodeGroup>

## Example: Import pre-labels or ground truths

The steps to import annotations as pre-labels (machine-assisted learning) are similar to those to import annotations as ground truth labels. However, they vary slightly, and we will describe the differences for each scenario.

### Before you start

The below imports are needed to use the code examples in this section.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  import uuid
  import json
  import labelbox as lb
  import labelbox.types as lb_types
  ```
</CodeGroup>

Replace the value of `API_KEY` with a valid [API key](/horizon/developer-guides/create-api-key) to connect to the Labelbox client.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  API_KEY = None
  client = lb.Client(API_KEY)
  ```
</CodeGroup>

### Step 1: Import data rows

Data rows must first be uploaded to **Catalog** to attach annotations.

This example shows how to create a data row in **Catalog** by attaching it to a [dataset](/horizon/developer-guides/dataset) .

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  global_key = "0801.3483.pdf"
  img_url = {
      "row_data": {
        "pdf_url": "https://storage.googleapis.com/labelbox-datasets/arxiv-pdf/data/99-word-token-pdfs/0801.3483.pdf",
      },
      "global_key": global_key
  }

  dataset = client.create_dataset(name="pdf_demo_dataset")
  task = dataset.create_data_rows([img_url])
  task.wait_till_done()
  print(f"Failed data rows: {task.failed_data_rows}")
  print(f"Errors: {task.errors}")

  ```
</CodeGroup>

### Step 2: Set up ontology

Your project ontology should support the tools and classifications required by your annotations. To ensure accurate schema feature mapping, the value used as the `name` parameter should match the value of the `name` field in your annotation.

For example, when we created an annotation above, we provided a name`annotation_name`. Now, when we set up our ontology, we must ensure that the name of our bounding box tool is also `anotations_name`. The same alignment must hold true for the other tools and classifications we create in our ontology.

This example shows how to create an ontology containing all supported [annotation types](#supported-annotations) .

<CodeGroup>
  ```python Python expandable theme={"theme":"css-variables"}
  ontology_builder = lb.OntologyBuilder(
    classifications=[ # List of Classification objects
      lb.Classification(
        class_type=lb.Classification.Type.RADIO,
        name="radio_question",
        scope = lb.Classification.Scope.GLOBAL,
        options=[
          lb.Option(value="first_radio_answer"),
          lb.Option(value="second_radio_answer")
        ]
      ),
      lb.Classification(
        class_type=lb.Classification.Type.CHECKLIST,
        name="checklist_question",
        scope = lb.Classification.Scope.GLOBAL,
        options=[
          lb.Option(value="first_checklist_answer"),
          lb.Option(value="second_checklist_answer")
        ]
      ),
      lb.Classification(
        class_type=lb.Classification.Type.TEXT,
        name="free_text",
        scope = lb.Classification.Scope.GLOBAL
      ),
      lb.Classification(
          class_type=lb.Classification.Type.RADIO,
          name="nested_radio_question",
          scope = lb.Classification.Scope.GLOBAL,
          options=[
              lb.Option("first_radio_answer",
                  options=[
                      lb.Classification(
                          class_type=lb.Classification.Type.RADIO,
                          name="sub_radio_question",
                          options=[lb.Option("first_sub_radio_answer")]
                      )
                  ])
            ]
      ),
      lb.Classification(
        class_type=lb.Classification.Type.CHECKLIST,
        name="nested_checklist_question",
        scope = lb.Classification.Scope.GLOBAL,
        options=[
            lb.Option("first_checklist_answer",
              options=[
                lb.Classification(
                    class_type=lb.Classification.Type.CHECKLIST,
                    name="sub_checklist_question",
                    options=[lb.Option("first_sub_checklist_answer")]
                )
            ])
        ]
      ),
    ],
    tools=[ # List of Tool objects
      lb.Tool( tool=lb.Tool.Type.BBOX,name="bounding_box"),
      lb.Tool(tool=lb.Tool.Type.NER, name="named_entity"),
      lb.Tool(tool=lb.Tool.Type.RELATIONSHIP,name="relationship"),
      lb.Tool(tool=lb.Tool.Type.NER,
              name="ner_with_checklist_subclass",
              classifications=[
                lb.Classification(
                  class_type=lb.Classification.Type.CHECKLIST,
                  name="sub_checklist_question",
                  options=[
                    lb.Option(value="first_sub_checklist_answer")
                  ]
                )
            ]),
      lb.Tool( tool=lb.Tool.Type.BBOX,
              name="bbox_with_radio_subclass",
              classifications=[
                lb.Classification(
                    class_type=lb.Classification.Type.RADIO,
                    name="sub_radio_question",
                    options=[
                      lb.Option(
                        value="first_sub_radio_answer" ,
                        options=[
                          lb.Classification(
                            class_type=lb.Classification.Type.RADIO,
                            name="second_sub_radio_question",
                            options=[lb.Option("second_sub_radio_answer")]
                          )]
                      )]
                  )]
        )]
  )

  ontology = client.create_ontology("Document Annotation Import Demo",
  ontology_builder.asdict(),
  media_type=lb.MediaType.Document)

  ```
</CodeGroup>

### Step 3: Set Up a Labeling Project

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Create a Labelbox project
  project = client.create_project(name="PDF_annotation_demo",
                                      queue_mode=QueueMode.Batch,
                                      media_type=lb.MediaType.Document)
  project.connect_ontology(ontology)
  ```
</CodeGroup>

### Step 4: Send Data Rows to Project

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  project.create_batch(
    "PDF_annotation_batch", # Each batch in a project must have a unique name
    global_keys=[global_key] , # a list of global keys, data rows, or data row ids
    priority=5 # priority between 1(highest) - 5(lowest)
  )
  ```
</CodeGroup>

### Step 5: Create annotation payloads

For help understanding annotation payloads, see [overview](#overview). To declare payloads, you can use Python annotation types (*preferred*) or NDJSON objects.

These examples demonstrate each format and how to compose annotations into labels attached to data rows.

<CodeGroup>
  ```python Python annotation payload theme={"theme":"css-variables"}
  # create a Label
  labels = []

  labels.append(
      lb_types.Label(
          data={"global_key" : global_key },
          annotations = [
              entities_annotation,
              checklist_annotation,
              nested_checklist_annotation, 
              text_annotation,
              radio_annotation,
              nested_radio_annotation,
              bbox_annotation,
              bbox_with_radio_subclass_annotation,
              ner_with_checklist_subclass_annotation,
              entity_source, 
              entity_target, 
              entity_relationship,# Only supported for MAL imports 
              bbox_source,
              bbox_target,
              bbox_relationship  # Only supported for MAL imports 
          ]
    )
  )
  ```

  ```python NDJSON payload theme={"theme":"css-variables"}
  label_ndjson = []
  for annot in [
      entities_annotations_ndjson,
      checklist_annotation_ndjson,
      nested_checklist_annotation_ndjson,
      text_annotation_ndjson,
      radio_annotation_ndjson,
      nested_radio_annotation_ndjson,
      bbox_annotation_ndjson,
      bbox_with_radio_subclass_annotation_ndjson,
      ner_with_checklist_subclass_annotation_ndjson,
      entity_source_ndjson,
      entity_target_ndjson,
      ner_relationship_annotation_ndjson, # Only supported for MAL imports
      bbox_source_ndjson,
      bbox_target_ndjson,
      bbox_relationship_annotation_ndjson # Only supported for MAL imports
    ]:
    annot.update({
        "dataRow": {"globalKey": global_key},
    })
    label_ndjson.append(annot)
  ```
</CodeGroup>

### Step 6: Import annotation payload

For prelabeled (model-assisted labeling) scenarios, pass your payload as the value of the `predictions` parameter. For ground truths, pass the payload to the `labels` parameter.

<Warning>
  ### Warning

  Relationship annotations are not supported for ground truth import jobs.
</Warning>

#### Option A: Upload as [prelabels (model assisted labeling)](/horizon/guides/model-assisted-labeling)

This option is helpful for speeding up the initial labeling process and reducing the manual labeling workload for high-volume datasets.

<CodeGroup>
  ```python MAL import theme={"theme":"css-variables"}
  # Upload MAL label for this data row in project
  upload_job = lb.MALPredictionImport.create_from_objects(
      client = client,
      project_id = project.uid,
      name="mal_job"+str(uuid.uuid4()),
      predictions=label
  )

  print(f"Errors: {upload_job.errors}", )
  print(f"Status of uploads: {upload_job.statuses}")
  ```
</CodeGroup>

#### Option B: Upload to a labeling project as [ground truth](/horizon/guides/import-ground-truth)

This option is helpful for loading high-confidence labels from another platform or previous projects that just need review rather than manual labeling effort.

<CodeGroup>
  ```python Label import theme={"theme":"css-variables"}
  # Upload label for this data row in project
  upload_job = lb.LabelImport.create_from_objects(
      client = client,
      project_id = project.uid,
      name="label_import_job"+str(uuid.uuid4()),
      labels=label
  )

  print(f"Errors: {upload_job.errors}", )
  print(f"Status of uploads: {upload_job.statuses}")
  ```
</CodeGroup>


# Import document data
Source: https://docs.labelbox.com/horizon/developer-guides/import-document-data

How to import document data and sample import formats.

## Specifications

Format: PDF Recommended size: 100 pages or fewer Import methods:

* IAM Delegated Access
* Signed URLs (`https` URLs only)

When importing document data to Labelbox, you are no longer required to provide an OCR extract in the form of a JSON file. Labelbox generates text layers automatically during PDF import using Google Document AI if the data row doesn't include a text layer. The JSON file generated will be your text layer, rendered on top of your PDF in the Document Editor.

<Info>
  ### Text layer limit

  Previously generated PDF documents without text layers can't be retroactively filled with the text layer generated by Labelbox.
</Info>

Google Document AI has the following limitations:

* The document must have no more than 15 pages
* The file size should not exceed 20 MB.

Additionally, Google Document AI optimizes documents before OCR processing. This optimization might include rotating images or pages to ensure text appears horizontally. Consequently, token coordinates are calculated based on the rotated/optimized images, resulting in potential discrepancies with the original PDF document.

For example, the document is rotated 90 degrees before processing in a landscape-oriented PDF. As a result, all tokens in the text layer are also rotated by 90 degrees.

<Warning>
  ### Image encoding

  If your PDF files contain images, use JPEG encoding and RGB colorspace for color images.
</Warning>

## Text Layer Validation Schema

If you want to upload your own text layer, the textLayer JSON file must adhere to the following JSON schema.

<CodeGroup>
  ```json Text Layer Validation Schema expandable theme={"theme":"css-variables"}
  {
    "type": "array",
    "items": {
      "$ref": "#/$defs/page"
    },
    "$defs": {
      "page": {
        "type": "object",
        "properties": {
          "width": {
            "type": "number"
          },
          "height": {
            "type": "number"
          },
          "number": {
            "type": "number"
          },
          "units": {
            "enum": ["POINTS", "PERCENT"]
          },
          "groups": {
            "type": "array",
            "items": {
              "$ref": "#/$defs/group"
            }
          }
        },
        "required": ["number", "units", "groups"]
      },
      "group": {
        "type": "object",
        "properties": {
          "id": {
            "type": "string"
          },
          "content": {
            "type": "string"
          },
          "geometry": {
            "$ref": "#/$defs/geometry"
          },
          "tokens": {
            "type": "array",
            "items": {
              "$ref": "#/$defs/token"
            }
          }
        },
        "required": ["id", "content", "geometry", "tokens"]
      },
      "geometry": {
        "type": "object",
        "properties": {
          "left": {
            "type": "number"
          },
          "top": {
            "type": "number"
          },
          "width": {
            "type": "number"
          },
          "height": {
            "type": "number"
          }
        },
        "required": ["left", "top", "width", "height"]
      },
      "token": {
        "type": "object",
        "properties": {
          "id": {
            "type": "string"
          },
          "content": {
            "type": "string"
          },
          "geometry": {
            "$ref": "#/$defs/geometry"
          }
        },
        "required": ["id", "geometry", "content"]
      }
    }
  }
  ```

  ```json Document AI example expandable theme={"theme":"css-variables"}
  [
      {
          "width": 1601,
          "height": 2498,
          "number": 1,
          "units": "PERCENT",
          "groups": [
              {
                  "id": "b4f4e1da-4088-44b3-a578-22f88ce9e166",
                  "content": "ΑΝ",
                  "geometry": {
                      "left": 0.4846970736980438,
                      "top": 0.17333866655826569,
                      "width": 0.028107434511184692,
                      "height": 0.007205769419670105
                  },
                  "tokens": [
                      {
                          "id": "0791adf8-80e9-4d3d-9d37-b3ad42dd061e",
                          "content": "ΑΝ",
                          "geometry": {
                              "left": 0.4846970736980438,
                              "top": 0.17333866655826569,
                              "width": 0.028107434511184692,
                              "height": 0.007205769419670105
                          }
                      }
                  ]
              },
              {
                  "id": "6f7c7e45-b9e8-4845-8c4e-63915e4a2e3d",
                  "content": "ESSAY",
                  "geometry": {
                      "left": 0.43410369753837585,
                      "top": 0.22377902269363403,
                      "width": 0.1274203360080719,
                      "height": 0.014811843633651733
                  },
                  "tokens": [
                      {
                          "id": "5adfb30a-235a-41c0-9902-a930cad660fc",
                          "content": "ESSAY",
                          "geometry": {
                              "left": 0.43410369753837585,
                              "top": 0.22377902269363403,
                              "width": 0.1274203360080719,
                              "height": 0.014811843633651733
                          }
                      }
                  ]
              },
              {
                  "id": "beec4d6e-cc9f-45e8-858c-c6c3f7c3ae49",
                  "content": "ON THE",
                  "geometry": {
                      "left": 0.46283572912216187,
                      "top": 0.2810248136520386,
                      "width": 0.07307934761047363,
                      "height": 0.006805449724197388
                  },
                  "tokens": [
                      {
                          "id": "a9456ab4-bfd0-49f2-a10b-c0ad2fa2fbb4",
                          "content": "ON",
                          "geometry": {
                              "left": 0.46283572912216187,
                              "top": 0.2810248136520386,
                              "width": 0.0237351655960083,
                              "height": 0.006405144929885864
                          }
                      },
                      {
                          "id": "fbdb497f-8241-406c-a06e-87027fb9b0b2",
                          "content": "THE",
                          "geometry": {
                              "left": 0.4971892535686493,
                              "top": 0.2810248136520386,
                              "width": 0.038725823163986206,
                              "height": 0.006405144929885864
                          }
                      }
                  ]
              },
              {
                  "id": "4caeda09-c92e-4c47-bb7a-3aea91a7dc29",
                  "content": "PRINCIPLE OF POPULATION,",
                  "geometry": {
                      "left": 0.2841973900794983,
                      "top": 0.31545236706733704,
                      "width": 0.425983726978302,
                      "height": 0.013210564851760864
                  },
                  "tokens": [
                      {
                          "id": "493ef02d-8c10-45cb-a889-10abc907a30c",
                          "content": "PRINCIPLE",
                          "geometry": {
                              "left": 0.2841973900794983,
                              "top": 0.31545236706733704,
                              "width": 0.15990003943443298,
                              "height": 0.013210564851760864
                          }
                      },
                      {
                          "id": "23ae78f7-9c04-43cb-b7a0-180806ec4472",
                          "content": "OF",
                          "geometry": {
                              "left": 0.4609619081020355,
                              "top": 0.31545236706733704,
                              "width": 0.036851972341537476,
                              "height": 0.013210564851760864
                          }
                      },
                      {
                          "id": "33462d2b-af22-456e-bd3b-5ed03ca091f3",
                          "content": "POPULATION",
                          "geometry": {
                              "left": 0.5115552544593811,
                              "top": 0.31545236706733704,
                              "width": 0.19300436973571777,
                              "height": 0.013210564851760864
                          }
                      },
                      {
                          "id": "f989c150-075b-4ea8-ac30-3cb8bd5697a7",
                          "content": ",",
                          "geometry": {
                              "left": 0.7033104300498962,
                              "top": 0.31545236706733704,
                              "width": 0.006870687007904053,
                              "height": 0.013210564851760864
                          }
                      }
                  ]
              },
              {
                  "id": "be9c8936-10b8-4013-8259-b34d309b1ce9",
                  "content": "AS IT AFFECTS",
                  "geometry": {
                      "left": 0.42910680174827576,
                      "top": 0.3570856750011444,
                      "width": 0.13991257548332214,
                      "height": 0.006805449724197388
                  },
                  "tokens": [
                      {
                          "id": "3af460fb-30cf-4988-82a3-4bdd6814d37e",
                          "content": "AS",
                          "geometry": {
                              "left": 0.42910680174827576,
                              "top": 0.3570856750011444,
                              "width": 0.02123674750328064,
                              "height": 0.006805449724197388
                          }
                      },
                      {
                          "id": "56b8b804-d802-48ca-af12-9fca0f16fe22",
                          "content": "IT",
                          "geometry": {
                              "left": 0.4603372812271118,
                              "top": 0.3570856750011444,
                              "width": 0.018113672733306885,
                              "height": 0.006805449724197388
                          }
                      },
                      {
                          "id": "29c69737-a6ff-48c6-b09d-fa6b04739c91",
                          "content": "AFFECTS",
                          "geometry": {
                              "left": 0.4890693426132202,
                              "top": 0.3570856750011444,
                              "width": 0.07995003461837769,
                              "height": 0.006805449724197388
                          }
                      }
                  ]
              },
              {
                  "id": "e0334ded-e13b-4b56-a53e-e8671eae6871",
                  "content": "THE FUTURE IMPROVEMENT OF SOCIETY.",
                  "geometry": {
                      "left": 0.21549031138420105,
                      "top": 0.3995196223258972,
                      "width": 0.5671455562114716,
                      "height": 0.010008007287979126
                  },
                  "tokens": [
                      {
                          "id": "fbf57954-5e06-4a78-8d62-f012d18683d7",
                          "content": "THE",
                          "geometry": {
                              "left": 0.21549031138420105,
                              "top": 0.3995196223258972,
                              "width": 0.05496564507484436,
                              "height": 0.010008007287979126
                          }
                      },
                      {
                          "id": "43be97d3-1179-4ea9-b07d-aa0e369f770f",
                          "content": "FUTURE",
                          "geometry": {
                              "left": 0.2841973900794983,
                              "top": 0.3995196223258972,
                              "width": 0.1043097972869873,
                              "height": 0.010008007287979126
                          }
                      },
                      {
                          "id": "57d3e552-8fae-466a-a090-5c3f1245c803",
                          "content": "IMPROVEMENT",
                          "geometry": {
                              "left": 0.4022485911846161,
                              "top": 0.3995196223258972,
                              "width": 0.20112428069114685,
                              "height": 0.010008007287979126
                          }
                      },
                      {
                          "id": "b2f2323b-6e5b-41ab-80c8-4c4dc3d7d1d5",
                          "content": "OF",
                          "geometry": {
                              "left": 0.6171143054962158,
                              "top": 0.3995196223258972,
                              "width": 0.031230449676513672,
                              "height": 0.010008007287979126
                          }
                      },
                      {
                          "id": "1284a561-2359-4658-9a23-1957ea9c34e0",
                          "content": "SOCIETY",
                          "geometry": {
                              "left": 0.6627107858657837,
                              "top": 0.3995196223258972,
                              "width": 0.11242973804473877,
                              "height": 0.010008007287979126
                          }
                      },
                      {
                          "id": "1e91b8d6-668f-4d25-bf30-75c4478ae33e",
                          "content": ".",
                          "geometry": {
                              "left": 0.7763897776603699,
                              "top": 0.3995196223258972,
                              "width": 0.006246089935302734,
                              "height": 0.010008007287979126
                          }
                      }
                  ]
              },
              {
                  "id": "027c0a96-c979-47de-8fca-44d0b07c3ac3",
                  "content": "WITH REMARKS",
                  "geometry": {
                      "left": 0.42410993576049805,
                      "top": 0.44515612721443176,
                      "width": 0.14990627765655518,
                      "height": 0.0064051151275634766
                  },
                  "tokens": [
                      {
                          "id": "60143ecf-ae4f-4635-a1e6-01fca29d286c",
                          "content": "WITH",
                          "geometry": {
                              "left": 0.42410993576049805,
                              "top": 0.44515612721443176,
                              "width": 0.05059337615966797,
                              "height": 0.0064051151275634766
                          }
                      },
                      {
                          "id": "daa24689-add8-4de3-9b58-36d6ea8b871f",
                          "content": "REMARKS",
                          "geometry": {
                              "left": 0.48657089471817017,
                              "top": 0.44515612721443176,
                              "width": 0.08744531869888306,
                              "height": 0.0064051151275634766
                          }
                      }
                  ]
              },
              {
                  "id": "6f9dac4a-4f7e-4409-bbaa-205817a45880",
                  "content": "ON THE SPECULATIONS OF MR. GODWIN,",
                  "geometry": {
                      "left": 0.2804497182369232,
                      "top": 0.47718173265457153,
                      "width": 0.43847593665122986,
                      "height": 0.011208981275558472
                  },
                  "tokens": [
                      {
                          "id": "b4828d24-46a6-4530-ae0f-dc6290f5d34d",
                          "content": "ON",
                          "geometry": {
                              "left": 0.2804497182369232,
                              "top": 0.47758206725120544,
                              "width": 0.027482837438583374,
                              "height": 0.009207367897033691
                          }
                      },
                      {
                          "id": "472a19ce-d3b9-4fd2-8d2f-645a1b1ce6da",
                          "content": "THE",
                          "geometry": {
                              "left": 0.31980013847351074,
                              "top": 0.47758206725120544,
                              "width": 0.042473435401916504,
                              "height": 0.009207367897033691
                          }
                      },
                      {
                          "id": "ce03974a-0d54-43b0-8ded-78f67732e070",
                          "content": "SPECULATIONS",
                          "geometry": {
                              "left": 0.37289193272590637,
                              "top": 0.47758206725120544,
                              "width": 0.15427860617637634,
                              "height": 0.010008007287979126
                          }
                      },
                      {
                          "id": "0d9584f4-c1cd-4acf-b8b1-2bb2a812f344",
                          "content": "OF",
                          "geometry": {
                              "left": 0.5377888679504395,
                              "top": 0.4783827066421509,
                              "width": 0.02560901641845703,
                              "height": 0.009207367897033691
                          }
                      },
                      {
                          "id": "6d44ba8a-f037-4762-83e7-864d3d91293b",
                          "content": "MR",
                          "geometry": {
                              "left": 0.5740162134170532,
                              "top": 0.4783827066421509,
                              "width": 0.031855106353759766,
                              "height": 0.009207367897033691
                          }
                      },
                      {
                          "id": "6aa31fc2-ca8f-49b8-b640-75d1ba9c14b5",
                          "content": ".",
                          "geometry": {
                              "left": 0.6071205735206604,
                              "top": 0.4787830412387848,
                              "width": 0.004372239112854004,
                              "height": 0.009207338094711304
                          }
                      },
                      {
                          "id": "90d91952-9233-47f2-aefd-ec4a446a51f0",
                          "content": "GODWIN",
                          "geometry": {
                              "left": 0.6233603954315186,
                              "top": 0.4783827066421509,
                              "width": 0.0855715274810791,
                              "height": 0.010008007287979126
                          }
                      },
                      {
                          "id": "a5719dea-8694-4ed6-97ad-eea86a92c0ea",
                          "content": ",",
                          "geometry": {
                              "left": 0.712054967880249,
                              "top": 0.4791833460330963,
                              "width": 0.006870687007904053,
                              "height": 0.009207367897033691
                          }
                      }
                  ]
              },
              {
                  "id": "8b5a88c6-c39e-4f8d-88f2-30eaa7c3f935",
                  "content": "M. CONDORCET,",
                  "geometry": {
                      "left": 0.41474080085754395,
                      "top": 0.5148118734359741,
                      "width": 0.1698938012123108,
                      "height": 0.01080864667892456
                  },
                  "tokens": [
                      {
                          "id": "63b99375-0adc-461b-9539-0d907eebac3b",
                          "content": "M.",
                          "geometry": {
                              "left": 0.41474080085754395,
                              "top": 0.5152121782302856,
                              "width": 0.02435976266860962,
                              "height": 0.009207367897033691
                          }
                      },
                      {
                          "id": "2a2f1ef8-6018-41f7-9b06-9f957313251b",
                          "content": "CONDORCET",
                          "geometry": {
                              "left": 0.4509681463241577,
                              "top": 0.5152121782302856,
                              "width": 0.12679576873779297,
                              "height": 0.010408341884613037
                          }
                      },
                      {
                          "id": "0623b165-4bb6-4ac7-b115-5989dab0d188",
                          "content": ",",
                          "geometry": {
                              "left": 0.5771392583847046,
                              "top": 0.516413152217865,
                              "width": 0.0074953436851501465,
                              "height": 0.008807003498077393
                          }
                      }
                  ]
              },
              {
                  "id": "4f5337fe-082a-4937-a08f-8c3304b84c7a",
                  "content": "AND OTHER WRITERS.",
                  "geometry": {
                      "left": 0.3791380524635315,
                      "top": 0.5536429286003113,
                      "width": 0.23860085010528564,
                      "height": 0.0072057247161865234
                  },
                  "tokens": [
                      {
                          "id": "ba603d0c-b146-4d01-9e1f-ad47ce600c29",
                          "content": "AND",
                          "geometry": {
                              "left": 0.3791380524635315,
                              "top": 0.5536429286003113,
                              "width": 0.04434725642204285,
                              "height": 0.0072057247161865234
                          }
                      },
                      {
                          "id": "3bdbfa39-0418-4758-a855-1104a8d8fb02",
                          "content": "OTHER",
                          "geometry": {
                              "left": 0.4347282946109772,
                              "top": 0.5536429286003113,
                              "width": 0.07245472073554993,
                              "height": 0.0072057247161865234
                          }
                      },
                      {
                          "id": "2a729df2-5199-45dd-bdf6-4787b95efe65",
                          "content": "WRITERS",
                          "geometry": {
                              "left": 0.5165521502494812,
                              "top": 0.5536429286003113,
                              "width": 0.09369146823883057,
                              "height": 0.0072057247161865234
                          }
                      },
                      {
                          "id": "40924cb4-fc4f-45a2-b4f7-1c31368f243b",
                          "content": ".",
                          "geometry": {
                              "left": 0.612742006778717,
                              "top": 0.5536429286003113,
                              "width": 0.004996895790100098,
                              "height": 0.0072057247161865234
                          }
                      }
                  ]
              },
              {
                  "id": "75a050e5-46d5-4773-9909-0bc2309892da",
                  "content": "LONDON:",
                  "geometry": {
                      "left": 0.45284196734428406,
                      "top": 0.6016813516616821,
                      "width": 0.09306684136390686,
                      "height": 0.00840669870376587
                  },
                  "tokens": [
                      {
                          "id": "cea98dbb-45d8-4562-8dee-a965cee86db7",
                          "content": "LONDON",
                          "geometry": {
                              "left": 0.45284196734428406,
                              "top": 0.6016813516616821,
                              "width": 0.08432230353355408,
                              "height": 0.00840669870376587
                          }
                      },
                      {
                          "id": "92b73d06-bae1-4a32-9970-e6cbc99ee74f",
                          "content": ":",
                          "geometry": {
                              "left": 0.5402873158454895,
                              "top": 0.6024819612503052,
                              "width": 0.005621492862701416,
                              "height": 0.007606089115142822
                          }
                      }
                  ]
              },
              {
                  "id": "c31fd2a2-f11e-4b8e-8b84-8592bd65b235",
                  "content": "PRINTED FOR J. JOHNSON, IN ST. PAUL'S",
                  "geometry": {
                      "left": 0.30605870485305786,
                      "top": 0.6381104588508606,
                      "width": 0.38538414239883423,
                      "height": 0.008406758308410645
                  },
                  "tokens": [
                      {
                          "id": "920b4d6d-150a-4e53-b380-315190fc3c96",
                          "content": "PRINTED",
                          "geometry": {
                              "left": 0.30605870485305786,
                              "top": 0.6381104588508606,
                              "width": 0.08119925856590271,
                              "height": 0.00960773229598999
                          }
                      },
                      {
                          "id": "356a78e0-ebd7-44a0-8f04-015d3fe3527b",
                          "content": "FOR",
                          "geometry": {
                              "left": 0.3978763222694397,
                              "top": 0.6377101540565491,
                              "width": 0.03560274839401245,
                              "height": 0.010008037090301514
                          }
                      },
                      {
                          "id": "b2ddf115-27c1-4078-b96c-a80464d54c2a",
                          "content": "J.",
                          "geometry": {
                              "left": 0.4409743845462799,
                              "top": 0.6377101540565491,
                              "width": 0.012492209672927856,
                              "height": 0.009607672691345215
                          }
                      },
                      {
                          "id": "d25272ac-5ae1-4510-9778-1f8c72613960",
                          "content": "JOHNSON",
                          "geometry": {
                              "left": 0.46283572912216187,
                              "top": 0.6377101540565491,
                              "width": 0.08432227373123169,
                              "height": 0.009607672691345215
                          }
                      },
                      {
                          "id": "79866cbf-139e-4b14-805b-6e8f8ea1c8cb",
                          "content": ",",
                          "geometry": {
                              "left": 0.5496564507484436,
                              "top": 0.6373098492622375,
                              "width": 0.004372298717498779,
                              "height": 0.009607672691345215
                          }
                      },
                      {
                          "id": "dcddbbd2-b954-48ff-91b2-a323310724a2",
                          "content": "IN",
                          "geometry": {
                              "left": 0.5652716755867004,
                              "top": 0.6373098492622375,
                              "width": 0.01873832941055298,
                              "height": 0.009607672691345215
                          }
                      },
                      {
                          "id": "46dbb8a6-dc00-4a7b-8703-aec10245d262",
                          "content": "ST",
                          "geometry": {
                              "left": 0.5946283340454102,
                              "top": 0.6373098492622375,
                              "width": 0.019362926483154297,
                              "height": 0.009607672691345215
                          }
                      },
                      {
                          "id": "e6ef7add-c8c1-4f18-8bfc-55cc3a810449",
                          "content": ".",
                          "geometry": {
                              "left": 0.6164897084236145,
                              "top": 0.6373098492622375,
                              "width": 0.004372239112854004,
                              "height": 0.009607672691345215
                          }
                      },
                      {
                          "id": "4a743990-3683-4017-9c62-4e6cae7513a2",
                          "content": "PAUL'S",
                          "geometry": {
                              "left": 0.6308557391166687,
                              "top": 0.636909544467926,
                              "width": 0.06058710813522339,
                              "height": 0.009607672691345215
                          }
                      }
                  ]
              },
              {
                  "id": "47f38d95-c06c-47e3-8c4e-e7db108ad5c8",
                  "content": "CHURCH-YARD",
                  "geometry": {
                      "left": 0.4303560256958008,
                      "top": 0.6717373728752136,
                      "width": 0.13616490364074707,
                      "height": 0.006004810333251953
                  },
                  "tokens": [
                      {
                          "id": "9805d04e-d122-48cc-8ed3-ea49ec5fd231",
                          "content": "CHURCH",
                          "geometry": {
                              "left": 0.4303560256958008,
                              "top": 0.6717373728752136,
                              "width": 0.076202392578125,
                              "height": 0.006004810333251953
                          }
                      },
                      {
                          "id": "6949f56c-30db-484c-99ad-8c3f15d4c7f2",
                          "content": "-",
                          "geometry": {
                              "left": 0.5103060603141785,
                              "top": 0.6717373728752136,
                              "width": 0.008119940757751465,
                              "height": 0.006004810333251953
                          }
                      },
                      {
                          "id": "b17eee41-1697-4065-ae24-757f35a79178",
                          "content": "YARD",
                          "geometry": {
                              "left": 0.5190505981445312,
                              "top": 0.6717373728752136,
                              "width": 0.0474703311920166,
                              "height": 0.006004810333251953
                          }
                      }
                  ]
              },
              {
                  "id": "87f29527-2a25-4ec9-b946-e2b3005758c2",
                  "content": "1798.",
                  "geometry": {
                      "left": 0.46221113204956055,
                      "top": 0.7233787178993225,
                      "width": 0.07432854175567627,
                      "height": 0.014011204242706299
                  },
                  "tokens": [
                      {
                          "id": "e6ee28b1-65a5-4926-938d-6548045bb9b2",
                          "content": "1798",
                          "geometry": {
                              "left": 0.46221113204956055,
                              "top": 0.723779022693634,
                              "width": 0.06433475017547607,
                              "height": 0.013610899448394775
                          }
                      },
                      {
                          "id": "e575c04a-33d5-4332-bfb0-9f208632108b",
                          "content": ".",
                          "geometry": {
                              "left": 0.5302935838699341,
                              "top": 0.723779022693634,
                              "width": 0.006246089935302734,
                              "height": 0.013210594654083252
                          }
                      }
                  ]
              }
          ]
      }
  ]
  ```
</CodeGroup>

## Parameters

| Parameter | Required | Description |
| - | - | - |
| `row_data` | Yes | A dictionary of`{ "pdf_url": str, "text_layer_url": str }` For IAM Delegated Access, this URL must be in [virtual-hosted-style format](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#virtual-hosted-style-access). |
| `row_data['pdf_url']` | Yes | `https` path to a cloud-hosted PDF. It must be specified within `row_data` dictionary. |
| `row_data['text_layer_url']` | No | `https` path to a cloud-hosted JSON extract of the PDF. |
| `global_key` | No | Unique user-generated file name or ID for the file. [Global keys](/horizon/developer-guides/data-row-global-keys) are enforced to be unique in your org. Data rows will not be imported if its global keys are duplicated to existing data rows. |
| `media_type` | No | `"PDF"` (optional media type to provide better validation and error messaging) |
| `metadata_fields` | No | See [Metadata](/horizon/guides/datarow-metadata). |
| `attachments` | No | See [Attachments](/horizon/guides/label-data) and [Asset overlays](/horizon/guides/label-data). |

## Import format

<CodeGroup>
  ```json Delegated Access URL theme={"theme":"css-variables"}
  [
    {
      "row_data": {
        "pdf_url": "https://lb-test-data.s3.us-west-1.amazonaws.com/document-samples/0801.3483.pdf",
        // You don't need to provide a text_layer_url. Labelbox automatically generates a text layer when importing an asset without one.
        "text_layer_url": "https://lb-test-data.s3.us-west-1.amazonaws.com/document-samples/0801.3483-lb-textlayer.json"
      },
      "global_key": "https://lb-test-data.s3.us-west-1.amazonaws.com/document-samples/0801.3483.pdf",
      "media_type": "PDF",
      "metadata_fields": [{"schema_id": "cko8s9r5v0001h2dk9elqdidh", "value": "tag_string"}],
      "attachments": [{"type": "HTML", "value": "https://www.wikipedia.org/" }]
    },
    {
      "row_data": {
        "pdf_url": "https://lb-test-data.s3.us-west-1.amazonaws.com/document-samples/0803.1972.pdf",
         // You don't need to provide a text_layer_url. Labelbox automatically generates a text layer when importing an asset without one.
        "text_layer_url": "https://lb-test-data.s3.us-west-1.amazonaws.com/document-samples/0803.1972-lb-textlayer.json"
      },
      "global_key": "https://lb-test-data.s3.us-west-1.amazonaws.com/document-samples/0803.1972.pdf",
      "media_type": "PDF",
      "metadata_fields": [{"name": "<metadata_field_name>", "value": "tag_string"}],
      "attachments": [{"type": "TEXT_URL", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt"}]
    }
  ]
  ```

  ```json Standard URL theme={"theme":"css-variables"}
  [
    {
      "row_data": {
        "pdf_url": "https://storage.googleapis.com/labelbox-datasets/arxiv-pdf/data/99-word-token-pdfs/0801.3483.pdf",
        // You don't need to provide a text_layer_url. Labelbox automatically generates a text layer when importing an asset without one.
        "text_layer_url": "https://storage.googleapis.com/labelbox-datasets/arxiv-pdf/data/99-word-token-pdfs/0801.3483-lb-textlayer.json"
      },
      "global_key": "https://storage.googleapis.com/labelbox-datasets/arxiv-pdf/data/99-word-token-pdfs/0801.3483.pdf",
      "media_type": "PDF",
      "metadata_fields": [{"schema_id": "cko8s9r5v0001h2dk9elqdidh", "value": "tag_string"}],
      "attachments": [{"type": "HTML", "value": "https://www.wikipedia.org/" }]
    }
  ]
  ```
</CodeGroup>

## Python example

<CodeGroup>
  ```python import example with custom text layer theme={"theme":"css-variables"}
  from labelbox import Client
  from uuid import uuid4 ## to generate unique IDs
  import datetime

  client = Client(api_key="<YOUR_API_KEY>")

  dataset = client.create_dataset(name="Bulk import example - Documents")

  assets = [
    {
      "row_data": {
        "pdf_url": "https://storage.googleapis.com/labelbox-datasets/arxiv-pdf/data/99-word-token-pdfs/0801.3483.pdf",
  		# You don't need to provide a text_layer_url. Labelbox automatically generates a text layer when importing an asset without one.
        "text_layer_url": "https://storage.googleapis.com/labelbox-datasets/arxiv-pdf/data/99-word-token-pdfs/0801.3483-lb-textlayer.json"
      },
      "global_key": "https://storage.googleapis.com/labelbox-datasets/arxiv-pdf/data/99-word-token-pdfs/0801.3483.pdf",
      "media_type": "PDF",
      "metadata_fields": [{"name": "<metadata_field_name>", "value": "tag_string"}],
      "attachments": [{"type": "HTML", "value": "https://www.wikipedia.org/" }]
    }
  ]

  task = dataset.create_data_rows(assets)
  task.wait_till_done()
  print(task.errors)
  ```

  ```python local files theme={"theme":"css-variables"}
  local_file_paths = ['path/to/local/file1', 'path/to/local/file1'] # limit: 15k files

  new_dataset = client.create_dataset(name = "Local files upload")

  try:
      task = new_dataset.create_data_rows(local_file_paths)
      task.wait_till_done()
  except Exception as err:
      print(f'Error while creating labelbox dataset -  Error: {err}')
  ```
</CodeGroup>

### Verify files are processed

<Warning>
  ### File processing can take up to 20 mins

  Since PDFs and OCR'ed files can be very large, the conversion can sometimes take up to 20 minutes to perform a data upload.
</Warning>

By checking the Media Attributes section, you can verify whether a file conversion using a custom or Labelbox-generated text layer is complete.

* If `Is text layer valid = true`, the file was successfully processed.


# Import geospatial annotations
Source: https://docs.labelbox.com/horizon/developer-guides/import-geospatial-annotations

Developer guide for importing annotations on geospatial data and sample import formats.

<CardGroup>
  <Card title="Open In Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/annotation_import/tiled.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/annotation_import/tiled.ipynb" />
</CardGroup>

## Overview

To import annotations in Labelbox, you need to create an annotations payload. In this section, we provide this payload for every supported annotation type.

### Annotation payload types

Labelbox supports two formats for the annotations payload:

* Python annotation types (recommended)

  * Provides a seamless transition between third-party platforms, machine learning pipelines, and Labelbox.
  * Allows you to build annotations locally with local file paths, numpy arrays, or URLs
  * Easily convert Python Annotation Type format to NDJSON format to quickly import annotations to Labelbox
  * Supports one-level nested classification (radio, checklist, or free-form text) under a tool or classification annotation.

* JSON

  * Skips formatting annotation payload in the Labelbox Python annotation type
  * Supports any levels of nested classification (radio, checklist, or free-form text) under a tool or classification annotation.

### Label import types

Labelbox additionally supports two types of label imports:

* [Model-assisted labeling (MAL)](/horizon/guides/model-assisted-labeling)
  * This workflow allows you to import computer-generated predictions (or simply annotations created outside of Labelbox) as pre-labels on an asset.
* [Ground truth](https://www.google.com/url?q=https%3A%2F%2Fdocs.labelbox.com%2Fdocs%2Fimport-ground-truth)
  * This workflow functionality allows you to bulk import your ground truth annotations from an external or third-party labeling system into Labelbox *Annotate*. Using the label import API to import external data is a useful way to consolidate and migrate all annotations into Labelbox as a single source of truth.

## Supported annotations

The following annotations are supported for an geospatial data row:

* Radio
* Checklist
* Free-form text
* Point
* Polyline
* Polygon
* Bounding Box

## Classifications

### Radio (single choice)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  radio_annotation = lb_types.ClassificationAnnotation(
      name="radio_question",
      value=lb_types.Radio(answer =
          lb_types.ClassificationAnswer(name = "first_radio_answer")
      )
  )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  radio_annotation_ndjson = {
    "name": "radio_question",
    "answer": {"name": "first_radio_answer"}
  }
  ```
</CodeGroup>

### Checklist (multiple choice)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  checklist_annotation = lb_types.ClassificationAnnotation(
      name="checklist_question",
      value=lb_types.Checklist(answer = [
          lb_types.ClassificationAnswer(name = "first_checklist_answer"),
          lb_types.ClassificationAnswer(name = "second_checklist_answer"),
          lb_types.ClassificationAnswer(name = "third_checklist_answer")
      ])
    )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  checklist_annotation_ndjson = {
    "name": "checklist_question",
    "answer": [
      {"name": "first_checklist_answer"},
      {"name": "second_checklist_answer"},
      {"name": "third_checklist_answer"},
    ]
  }
  ```
</CodeGroup>

### Free-form text

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  text_annotation = lb_types.ClassificationAnnotation(
      name = "free_text",
      value = lb_types.Text(answer="sample text")
  )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  text_annotation_ndjson = {
    "name": "free_text",
    "answer": "sample text",
  }
  ```
</CodeGroup>

### Nested classifications

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  nested_radio_annotation = lb_types.ClassificationAnnotation(
    name="nested_radio_question",
    value=lb_types.Radio(
      answer=lb_types.ClassificationAnswer(
        name="first_radio_answer",
        classifications=[
          lb_types.ClassificationAnnotation(
            name="sub_radio_question",
            value=lb_types.Radio(
              answer=lb_types.ClassificationAnswer(
                name="first_sub_radio_answer"
              )
            )
          )
        ]
      )
    )
  )

  nested_checklist_annotation = lb_types.ClassificationAnnotation(
    name="nested_checklist_question",
    value=lb_types.Checklist(
      answer=[lb_types.ClassificationAnswer(
        name="first_checklist_answer",
        classifications=[
          lb_types.ClassificationAnnotation(
            name="sub_checklist_question",
            value=lb_types.Checklist(
              answer=[lb_types.ClassificationAnswer(
              name="first_sub_checklist_answer"
            )]
          ))
        ]
      )]
    )
  )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  nested_radio_annotation_ndjson= {
    "name": "nested_radio_question",
    "answer": {
        "name": "first_radio_answer",
        "classifications": [
          {
            "name": "sub_radio_question",
            "answer": {"name": "first_sub_radio_answer"}
          }
        ]
      }
  }

  nested_checklist_annotation_ndjson = {
    "name": "nested_checklist_question",
    "answer": [{
        "name": "first_checklist_answer",
        "classifications" : [
          {
            "name": "sub_checklist_question",
            "answer": {"name": "first_sub_checklist_answer"}
          }
        ]
    }]
  }
  ```
</CodeGroup>

### Point

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  point_annotation = lb_types.ObjectAnnotation(
    name = "point_geo",
    value = lb_types.Point(x=-99.20647859573366, y=19.40018029091072),
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  point_annotation_ndjson = {
      "name": "point_geo",
      "point": {
           "x": -99.20647859573366,
           "y": 19.40018029091072
       }
  }
  ```
</CodeGroup>

### Polyline

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  # Coordinates in the desired EPSG coordinate system
  coords = [
              [
                  -99.20842051506044,
                  19.40032196622975
              ],
              [
                  -99.20809864997865,
                  19.39758963475322
              ],
              [
                  -99.20758366584778,
                  19.39776167179227
              ],
              [
                  -99.20728325843811,
                  19.3973265189299
              ]
          ]

  line_points = []
  line_points_ndjson = []

  for sub in coords: 
    line_points.append(lb_types.Point(x=sub[0], y=sub[1]))
    line_points_ndjson.append({"x":sub[0], "y":sub[1]})

  # Python Annotation 
  polyline_annotation = lb_types.ObjectAnnotation(
    name = "polyline_geo",
    value = lb_types.Line(points=line_points),
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  # Coordinates in the desired EPSG coordinate system
  coords = [
    [
      -99.20842051506044,
      19.40032196622975
    ],
    [
      -99.20809864997865,
      19.39758963475322
    ],
    [
      -99.20758366584778,
      19.39776167179227
    ],
    [
      -99.20728325843811,
      19.3973265189299
    ]
  ]

  line_points = []
  line_points_ndjson = []

  for sub in coords: 
    line_points.append(lb_types.Point(x=sub[0], y=sub[1]))
    line_points_ndjson.append({"x":sub[0], "y":sub[1]})

  # NDJSON 
  polyline_annotation_ndjson = {
    "name": "polyline_geo",
    "line": line_points_ndjson
  }
  ```
</CodeGroup>

### Polygon

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  polygon_points = []

  for sub in coords_polygon: 
    polygon_points.append(lb_types.Point(x=sub[0], y=sub[1]))


  polygon_annotation = lb_types.ObjectAnnotation(
    name = "polygon_geo",
    value = lb_types.Polygon(points=polygon_points),
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  # Coordinates in the desired EPSG coordinate system

  polygon_points_ndjson = []

  for sub in coords_polygon:
    polygon_points_ndjson.append({"x":sub[0], "y":sub[1]})

  polygon_annotation_ndjson = {
    "name": "polygon_geo",
    "polygon": polygon_points_ndjson
  }
  ```

  ```python coords_polygon expandable theme={"theme":"css-variables"}
  coords_polygon = [
      [
          -99.21042680740356,
          19.40036244486966
      ],
      [
          -99.2104160785675,
          19.40017017124035
      ],
      [
          -99.2103409767151,
          19.400008256428897
      ],
      [
          -99.21014785766603,
          19.400008256428897
      ],
      [
          -99.21019077301027,
          19.39983622176518
      ],
      [
          -99.21022295951845,
          19.399674306621385
      ],
      [
          -99.21029806137086,
          19.39951239131646
      ],
      [
          -99.2102873325348,
          19.399340356128437
      ],
      [
          -99.21025514602663,
          19.399117722085677
      ],
      [
          -99.21024441719057,
          19.39892544698541
      ],
      [
          -99.2102336883545,
          19.39874329141769
      ],
      [
          -99.21021223068239,
          19.398561135646027
      ],
      [
          -99.21018004417421,
          19.398399219233365
      ],
      [
          -99.21011567115785,
          19.39822718286836
      ],
      [
          -99.20992255210878,
          19.398136104719125
      ],
      [
          -99.20974016189577,
          19.398085505725305
      ],
      [
          -99.20957922935487,
          19.398004547302467
      ],
      [
          -99.20939683914186,
          19.39792358883935
      ],
      [
          -99.20918226242067,
          19.39786286996558
      ],
      [
          -99.20899987220764,
          19.397822390703805
      ],
      [
          -99.20891404151918,
          19.397994427496787
      ],
      [
          -99.20890331268312,
          19.398176583902874
      ],
      [
          -99.20889258384706,
          19.398368859888045
      ],
      [
          -99.20889258384706,
          19.398540896103246
      ],
      [
          -99.20890331268312,
          19.39872305189756
      ],
      [
          -99.20889258384706,
          19.39890520748796
      ],
      [
          -99.20889258384706,
          19.39907724313608
      ],
      [
          -99.20889258384706,
          19.399259398329956
      ],
      [
          -99.20890331268312,
          19.399431433603585
      ],
      [
          -99.20890331268312,
          19.39961358840092
      ],
      [
          -99.20890331268312,
          19.399785623300048
      ],
      [
          -99.20897841453552,
          19.399937418648214
      ],
      [
          -99.20919299125673,
          19.399937418648214
      ],
      [
          -99.2093861103058,
          19.39991717927664
      ],
      [
          -99.20956850051881,
          19.39996777770086
      ],
      [
          -99.20961141586305,
          19.40013981222548
      ],
      [
          -99.20963287353517,
          19.40032196622975
      ],
      [
          -99.20978307724,
          19.4004130431554
      ],
      [
          -99.20996546745302,
          19.40039280384301
      ],
      [
          -99.21019077301027,
          19.400372564528084
      ],
      [
          -99.21042680740356,
          19.40036244486966
      ]

  ]
  ```
</CodeGroup>

### Bounding Box

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  bbox_top_left = lb_types.Point(x= -99.20746564865112, y=19.39799442829336)
  bbox_bottom_right = lb_types.Point(x=-99.20568466186523, y=19.39925939999194)

  # Python Annotation

  bbox_annotation = lb_types.ObjectAnnotation(
  name = "bbox_geo",
  value = lb_types.Rectangle(start=bbox_top_left, end=bbox_bottom_right)
  )

  ```

  ```python NDJSON theme={"theme":"css-variables"}
  # Coordinates in the desired EPSG coordinate system
  coord_object =  {
    "coordinates" : [[
      [
        -99.20746564865112,
        19.39799442829336
      ],
      [
        -99.20746564865112,
        19.39925939999194
      ],
      [
        -99.20568466186523,
        19.39925939999194
      ],
      [
        -99.20568466186523,
        19.39799442829336
      ],
      [
        -99.20746564865112,
        19.39799442829336
      ]
    ]]
  }

  # NDJSON
  bbox_annotation_ndjson = {
    "name" : "bbox_geo",
    "bbox" : {
      'top': coord_object["coordinates"][0][1][1],
      'left': coord_object["coordinates"][0][1][0],
      'height': coord_object["coordinates"][0][3][1] - coord_object["coordinates"][0][1][1],
      'width': coord_object["coordinates"][0][3][0] - coord_object["coordinates"][0][1][0]
    }
  }
  ```
</CodeGroup>

### Mask to bounding box

Additionally, you create a bounding box with a segmentation mask using the code below.

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  # Let's create another polygon annotation with Python annotation tools that draws the image using cv2 libraries

  hsv = cv2.cvtColor(tiled*image_data.value, cv2.COLOR_RGB2HSV)
  mask = cv2.inRange(hsv, (25, 50, 25), (100, 150, 255))
  kernel = np.ones((15, 20), np.uint8)
  mask = cv2.erode(mask, kernel)
  mask = cv2.dilate(mask, kernel)
  mask_annotation = lb_types.MaskData.from_2D_arr(mask)
  mask_data = lb_types.Mask(mask=mask_annotation, color=[255, 255, 255])
  h, w, * = tiled_image_data.value.shape
  pixel_bounds = lb_types.TiledBounds(epsg=lb_types.EPSG.SIMPLEPIXEL,
  bounds=[lb_types.Point(x=0, y=0),
  lb_types.Point(x=w, y=h)])
  transformer = lb_types.EPSGTransformer.create_pixel_to_geo_transformer(
  src_epsg=pixel_bounds.epsg,
  pixel_bounds=pixel_bounds,
  geo_bounds=tiled_image_data.tile_bounds,
  zoom=20)
  pixel_polygons = mask_data.shapely.simplify(3)
  list_of_polygons = [transformer(lb_types.Polygon.from_shapely(p)) for p in pixel_polygons.geoms]
  polygon_annotation_two = lb_types.ObjectAnnotation(value=list_of_polygons[0], name="polygon_geo_2")

  ```
</CodeGroup>

### Tool with nested classification

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  tool_with_radio_subclass_annotation = lb_types.ObjectAnnotation(
      name=# Feature name,
      value=# Add tool annotation (lb_types."tool"),
      classifications=[
          lb_types.ClassificationAnnotation(
              name="sub_radio_question",
              value=lb_types.Radio(answer=lb_types.ClassificationAnswer(
                  name="first_sub_radio_answer")))
      ])
  ```

  ```python Bounding Box Example theme={"theme":"css-variables"}
  bbox_with_radio_subclass_annotation = lb_types.ObjectAnnotation(
      name="bbox_with_radio_subclass",
      value=lb_types.Rectangle(
          start=lb_types.Point(x=541, y=933),  # x = left, y = top
          end=lb_types.Point(x=871, y=1124),  # x= left + width , y = top + height
      ),
      classifications=[
          lb_types.ClassificationAnnotation(
              name="sub_radio_question",
              value=lb_types.Radio(answer=lb_types.ClassificationAnswer(
                  name="first_sub_radio_answer")))
      ])
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  bbox_with_radio_subclass_ndjson = {
      "name": "bbox_with_radio_subclass",
      "classifications": [{
          "name": "sub_radio_question",
          "answer": {
              "name": "first_sub_radio_answer"
          }
      }],
      "bbox": {
          "top": 933,
          "left": 541,
          "height": 191,
          "width": 330
      }
  }
  ```
</CodeGroup>

## Example: Import pre-labels or ground truths

The steps to import annotations as pre-labels (machine-assisted learning) are similar to those to import annotations as ground truth labels. However, they vary slightly, and we will describe the differences for each scenario.

### Before you start

The below imports are needed to use the code examples in this section.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  import uuid
  import numpy as np
  import cv2
  import labelbox as lb
  import labelbox.types as lb_types
  ```
</CodeGroup>

Replace the value of `API_KEY` with a valid [API key](/horizon/developer-guides/create-api-key) to connect to the Labelbox client.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  API_KEY = None
  client = lb.Client(API_KEY)
  ```
</CodeGroup>

### Step 1: Import data rows

Data rows must first be uploaded to **Catalog** to attach annotations.

This example shows how to create a data row in **Catalog** by attaching it to a [dataset](/horizon/developer-guides/dataset) .

<CodeGroup>
  ```python theme={"theme":"css-variables"}
  top_left_bound = lb_types.Point(x=-99.21052827588443, y=19.400498983095076)
  bottom_right_bound = lb_types.Point(x=-99.20534818927473, y=19.39533555271248)

  epsg = lb_types.EPSG.EPSG4326
  bounds = lb_types.TiledBounds(epsg=epsg, bounds=[top_left_bound, bottom_right_bound])
  global_key = "mexico_city"

  tile_layer = lb_types.TileLayer(
      url="https://s3-us-west-1.amazonaws.com/lb-tiler-layers/mexico_city/{z}/{x}/{y}.png"
  )

  tiled_image_data = lb_types.TiledImageData(tile_layer=tile_layer,
                                    tile_bounds=bounds,
                                    zoom_levels=[17, 23])

  asset = {
      "row_data": tiled_image_data.asdict(),
      "global_key": global_key,
      "media_type": "TMS_GEO"
  }

  dataset = client.create_dataset(name="geo_demo_dataset")
  task= dataset.create_data_rows([asset])
  print("Errors:",task.errors)
  print("Failed data rows:", task.failed_data_rows)
  ```
</CodeGroup>

### Step 2: Set up ontology

Your project ontology should support the tools and classifications required by your annotations. To ensure accurate schema feature mapping, the value used as the `name` parameter should match the value of the `name` field in your annotation.

For example, when we created an annotation above, we provided a name`annotation_name`. Now, when we set up our ontology, we must ensure that the name of our bounding box tool is also `anotations_name`. The same alignment must hold true for the other tools and classifications we create in our ontology.

This example shows how to create an ontology containing all supported [annotation types](#supported-annotations) .

<CodeGroup>
  ```python Python expandable theme={"theme":"css-variables"}
  ontology_builder = lb.OntologyBuilder(
      tools=[
          lb.Tool(tool=lb.Tool.Type.POINT, name="point_geo"),
          lb.Tool(tool=lb.Tool.Type.LINE, name="polyline_geo"),
          lb.Tool(tool=lb.Tool.Type.POLYGON, name="polygon_geo"),
          lb.Tool(tool=lb.Tool.Type.POLYGON, name="polygon_geo_2"),
          lb.Tool(tool=lb.Tool.Type.BBOX, name="bbox_geo"),
          lb.Tool(
            tool=lb.Tool.Type.BBOX,
            name="bbox_checklist_geo",
            classifications=[
                  lb.Classification(
                      class_type=lb.Classification.Type.CHECKLIST,
                      name="checklist_class_name",
                      options=[
                        lb.Option(value="first_checklist_answer")
                      ]
                  ),
              ]
            ),
          lb.Tool(
            tool=lb.Tool.Type.BBOX,
            name="bbox_text_geo",
            classifications=[
                  lb.Classification(
                      class_type=lb.Classification.Type.TEXT,
                      name="free_text_geo"
                  ),
              ]
            )
        ],
        classifications = [
            lb.Classification(
                class_type=lb.Classification.Type.CHECKLIST,
                name="checklist_question_geo",
                options=[
                    lb.Option(value="first_checklist_answer"),
                    lb.Option(value="second_checklist_answer"),
                    lb.Option(value="third_checklist_answer")
                ]
            ),
            lb.Classification(
                class_type=lb.Classification.Type.RADIO,
                name="radio_question_geo",
                options=[
                    lb.Option(value="first_radio_answer")
                ]
            ),

        lb.Classification(
          class_type=lb.Classification.Type.RADIO,
          name="nested_radio_question",
          options=[
            lb.Option(value="first_radio_answer",
              options=[
                  lb.Classification(
                    class_type=lb.Classification.Type.RADIO,
                    name="sub_radio_question",
                    options=[
                      lb.Option(value="first_sub_radio_answer")
                    ]
                ),
              ]
            ),
          ],
        ),
        lb.Classification(class_type=lb.Classification.Type.TEXT,
                          name="free_text"),
        lb.Classification(
          class_type=lb.Classification.Type.CHECKLIST,
          name="nested_checklist_question",
          options=[
              lb.Option("first_checklist_answer",
                options=[
                  lb.Classification(
                      class_type=lb.Classification.Type.CHECKLIST,
                      name="sub_checklist_question",
                      options=[lb.Option("first_sub_checklist_answer")]
                  )
              ]
            )
          ]
      )
    ]
  )

  ontology = client.create_ontology("Ontology Geospatial Annotations",
  ontology_builder.asdict(),
  media_type=lb.MediaType.Geospatial_Tile)
  ```
</CodeGroup>

### Step 3: Set Up a Labeling Project

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Create a project
  project = client.create_project(
    name="Geospatial Project Demo",
    media_type=lb.MediaType.Geospatial_Tile
  )

  # Connect the ontology

  project.connect_ontology(ontology)
  ```
</CodeGroup>

### Step 4: Send Data Rows to Project

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Create a batch to send to your MAL project
  batch = project.create_batch(
    "first-batch-geo-demo", # Each batch in a project must have a unique name
    global_keys=[global_key], # Paginated collection of data row objects, list of data row ids or global keys
    priority=5 # priority between 1(Highest) - 5(lowest)
  )

  print("Batch: ", batch)
  ```
</CodeGroup>

### Step 5: Create annotation payloads

For help understanding annotation payloads, see [overview](#overview). To declare payloads, you can use Python annotation types (*preferred*) or NDJSON objects. For annotations that you want to import as ground truth labels, you can also specify [benchmarks](/horizon/guides/benchmark) using the `is_benchmark_reference` flag.

These examples demonstrate each format and how to compose annotations into labels attached to data rows.

<CodeGroup>
  ```python Python annotation payload theme={"theme":"css-variables"}
  labels =[]
  labels.append(
      lb_types.Label(
          data={
              "global_key": global_key,
              "tile_layer": tile_layer,
              "tile_bounds":bounds,
              "zoom_levels": [12, 20]
          },
          annotations = [
              point_annotation,
              polyline_annotation,
              polygon_annotation,
              bbox_annotation,
              radio_annotation,
              bbox_with_checklist_subclass,
              bbox_with_free_text_subclass,
              checklist_annotation,
              polygon_annotation_two,
              nested_checklist_annotation,
              nested_radio_annotation,
              text_annotation
          ],
          # Optional: set the label as a benchmark
          # Only supported for groud truth imports
          is_benchmark_reference = True
      )
  )
  ```

  ```python NDJSON payload theme={"theme":"css-variables"}
  label_ndjson = []

  for annotations in [point_annotation_ndjson,
                      polyline_annotation_ndjson,
                      polygon_annotation_ndjson,
                      bbox_annotation_ndjson,
                      radio_annotation_ndjson,
                      bbox_with_checklist_subclass_ndjson,
                      bbox_with_free_text_subclass_ndjson,
                      checklist_annotation_ndjson,
                      nested_checklist_annotation_ndjson,
                      nested_radio_annotation_ndjson,
                      text_annotation_ndjson
                      ]:
    annotations.update({
        'dataRow': {
            'globalKey': global_key
        }
    })
    label_ndjson.append(annotations)
  ```
</CodeGroup>

### Step 6: Import annotation payload

For prelabeled (model-assisted labeling) scenarios, pass your payload as the value of the `predictions` parameter. For ground truths, pass the payload to the `labels` parameter.

#### Option A: Upload as [prelabels (model assisted labeling)](/horizon/guides/model-assisted-labeling)

This option is helpful for speeding up the initial labeling process and reducing the manual labeling workload for high-volume datasets.

<CodeGroup>
  ```python MAL import theme={"theme":"css-variables"}
  # Upload MAL label for this data row in project
  upload_job = lb.MALPredictionImport.create_from_objects(
      client = client,
      project_id = project.uid,
      name="mal_job"+str(uuid.uuid4()),
      predictions=label
  )

  print(f"Errors: {upload_job.errors}", )
  print(f"Status of uploads: {upload_job.statuses}")
  ```
</CodeGroup>

#### Option B: Upload to a labeling project as [ground truth](/horizon/guides/import-ground-truth)

This option is helpful for loading high-confidence labels from another platform or previous projects that just need review rather than manual labeling effort.

<CodeGroup>
  ```python Label import theme={"theme":"css-variables"}
  # Upload label for this data row in project
  upload_job = lb.LabelImport.create_from_objects(
      client = client,
      project_id = project.uid,
      name="label_import_job"+str(uuid.uuid4()),
      labels=label
  )

  print(f"Errors: {upload_job.errors}", )
  print(f"Status of uploads: {upload_job.statuses}")
  ```
</CodeGroup>


# Import geospatial data
Source: https://docs.labelbox.com/horizon/developer-guides/import-geospatial-data

How to import geospatial data and sample import formats.

## Specifications

Supported file formats:

* [Slippy maps:](https://developers.planet.com/docs/planetschool/xyz-tiles-and-slippy-maps/) `url/zoom-level/x-coordinate/y-coordinate.png`
* Deep zoom servers for pathology use case, such as [Openslide](https://openslide.org/api/python/) and [tiffslide](https://github.com/bayer-science-for-a-better-life/tiffslide)
* GeoTIFF: Standard or Cloud-optimized
* NITF: 2.0 or 2.1 format

Import methods:

* Slippy map URL
* IAM Delegated Access (NITF, GeoTIFF, or COG files only)
* Signed URLs (`https` URLs only)

<Danger>
  **IAM Delegated Access not supported with slippy maps**

  IAM Delegated Access is not supported with slippy maps due to the nature of how slippy maps work. We strongly encourage you to use cloud-optimized GeoTIFFs instead of slippy maps.

  To use IAM Delegated Access with geospatial/tiled imagery, please use NITF, GeoTIFF, or COG files.
</Danger>

## Slippy maps

### Definitions

When importing slippy map data, make sure your import file contains this information and follows this format.

| Parameter | Required | Description |
| - | - | - |
| `row_data` | Yes | A dictionary of`{ "tile_layer_url": str, "min_zoom": int, "max_zoom": int, "bounds": \[[float, float],[float, ffloat]], "epsg": "EPSG3857" ... }` For IAM Delegated Access, this URL must be in [virtual-hosted-style format](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#virtual-hosted-style-access). |
| `row_data['version']` | No | Add version to your import to distinguish it from the legacy format. If `version: 2` is present, Labelbox will read all coordinates as `[long, lat]` or `[x, y]` depending on the data format. If `version: 1` is present, Labelbox will read all coordinates as `[lat, long]` or `[y, x]` depending on the data format. If no version is provided, the default value for Slippy map files using an espg coordinate system and COG files is `version: 1` and the default value for cartesian and simple coordinate system is `version: 2` |
| `row_data['tile_layer_url']` | Yes | URL where map data is hosted. Must be in the following format: `<https://URL/{z}/{x}/{y}.png`> Slippy map tiles in JPG or PNG. |
| `row_data['min_zoom']` | No | Minimum map zoom level down to which this layer will be displayed (inclusive). If not provided, users will potentially be able to zoom beyond where tiles are available. |
| `row_data['max_zoom']` | No | Maximum map zoom level up to which this layer will be displayed (inclusive). If not provided, users will potentially be able to zoom beyond where tiles are available. |
| `row_data['max_native_zoom']` | No | Maximum zoom number the tile source has available (`maxNativeZoom` & `maxZoom`). If specified, the tiles on all zoom levels higher than `maxNativeZoom` will be auto-scaled. |
| `row_data['bounds']` | Yes | The bounds where the projection is valid. For the Simple coordinate system, bounds are in x,y. For the EPSG coordinate system, bounds are in lat,long. |
| `row_data['epsg']` | Yes | Structured dataset of coordinate reference systems and coordinate transformations. Can be `EPSG3857`, `EPSG3395`, `EPSG4326`, `Simple`, or `Cartesian`. If using `Cartesian`, you must pass in the parameters for the pixel dimensions in the `dimensions` field. |
| `row_data['tile_size']` | No | Specifies the size of the individual slippy map tiles. Common sizes are 256, 512, 1024. Note: Currently, only square tiles are supported which is why `tileSize` is a single number that indicates the width and height of the tile. |
| `row_data['alternative_layers']` | No | Additional tile layers. Only `alternativeLayers.tileLayerUrl` and `alternativeLayers.name` are required. If no values for optional parameters are given, those values will default to the top-level keys. |
| `row_data['dimensions']` | No | This is required if using the Cartesian coordinate system. This represents the pixel height and width of the entire image. The format is: `"dimensions": { "width": <Pixel width>, "height": <Pixel height> }` |
| `global_key` | No | Unique user-generated file name or ID for the file. [Global keys](/horizon/developer-guides/data-row-global-keys) are enforced to be unique in your org. Data rows will not be imported if its global keys are duplicated to existing data rows. |
| `media_type` | No | `"TMS_GEO"` (optional media type to provide better validation and error messaging) |
| `metadata_fields` | No | See [Metadata](/horizon/guides/datarow-metadata). |
| `attachments` | No | See [Attachments](/horizon/guides/label-data) and [Asset overlays](/horizon/guides/label-data). |

### Import format

<CodeGroup>
  ```json json theme={"theme":"css-variables"}
  [
    {
      "row_data":{
        "tile_layer_url": "https://s3-us-west-1.amazonaws.com/lb-tiler-layers/mexico_city/{z}/{x}/{y}.png",
        "bounds": [
          [
            19.405662413477728,
            -99.21052827588443
          ],
          [
            19.400498983095076,
            -99.20534818927473
          ]
        ],
        "min_zoom": 12,
        "max_zoom": 20,
        "epsg": "EPSG4326",
        "alternative_layers": [
          {
            "tile_layer_url": "https://api.mapbox.com/styles/v1/mapbox/satellite-streets-v11/tiles/{z}/{x}/{y}?access_token=pk.eyJ1IjoibWFwYm94IiwiYSI6ImNpejY4NXVycTA2emYycXBndHRqcmZ3N3gifQ.rJcFIG214AriISLbB6B5aw",
            "name": "Satellite"
          },
          {
            "tile_layer_url": "https://api.mapbox.com/styles/v1/mapbox/navigation-guidance-night-v4/tiles/{z}/{x}/{y}?access_token=pk.eyJ1IjoibWFwYm94IiwiYSI6ImNpejY4NXVycTA2emYycXBndHRqcmZ3N3gifQ.rJcFIG214AriISLbB6B5aw",
            "name": "Guidance"
          }
        ]
      },
      "global_key": "https://s3-us-west-1.amazonaws.com/lb-tiler-layers/mexico_city/{z}/{x}/{y}.png",
      "media_type": "TMS_GEO",
      "metadata_fields": [{"name": "<metadata_field_name>", "value": "tag_string"}],
      "attachments": [{"type": "TEXT_URL", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt"}]
    }
  ]
  ```

  ```json Additional layers theme={"theme":"css-variables"}
  [
    {
      "row_data": {
        "tile_layer_url": "https://labelbox.s3-us-west-2.amazonaws.com/tiler/drone-map/rgb/{z}/{x}/{y}.png",
        "bounds": [
          [17.983835501492813,74.41417694091798],
          [17.996825561127697, 74.44644927978517]
        ],
        "min_zoom": 8,
        "max_zoom": 22,
        "max_native_zoom": 22,
        "epsg": "EPSG3857",
        "version": 2,
        "alternative_layers": [
          {
            "tileLayer_url": "https://labelbox.s3-us-west-2.amazonaws.com/tiler/drone-map/ndvi/{z}/{x}/{y}.png",
            "name": "NDVI"
          },
          {
            "tileLayer_url": "https://labelbox.s3-us-west-2.amazonaws.com/tiler/drone-map/dtm/{z}/{x}/{y}.png",
            "name": "DTM"
          }
        ]
      },
      "global_key": "https://labelbox.s3-us-west-2.amazonaws.com/tiler/drone-map/rgb/{z}/{x}/{y}.png",
      "media_type": "TMS_GEO",
      "metadata_fields": [{"schema_id": "cko8s9r5v0001h2dk9elqdidh", "value": "tag_string"}],
      "attachments": [{"type": "TEXT_URL", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt"}]
    }
  ]
  ```

  ```json Map sections theme={"theme":"css-variables"}
  [
    {
      "row_data": {
        "tile_layer_url": "https://public-tiles.dronedeploy.com/1499994155_DANIELOPENPIPELINE_ortho_qfs/{z}/{x}/{y}.png?Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9wdWJsaWMtdGlsZXMuZHJvbmVkZXBsb3kuY29tLzE0OTk5OTQxNTVfREFOSUVMT1BFTlBJUEVMSU5FX29ydGhvX3Fmcy8qIiwiQ29uZGl0aW9uIjp7IkRhdGVMZXNzVGhhbiI6eyJBV1M6RXBvY2hUaW1lIjoyMTQ1OTE0MTE4fX19XX0_&Signature=O~50rrGXdEC6Hi8jPJ3dbT~UtBd7Cw6iQPTxdJ8LU2IaoxeP22R3JpKPkLN3T3~Lcw3CyX7uft2Baj0MH93qUoCYyN~~jNX3OMkYV2jbrHDezf6zQRHAabXX-L2bL-JEGfFL6z3DWccOFeCH56CuhgC29k5CJx7I34P-LQJdnAUsA-KaqKH1IyYsHStRIfmMzdXNAWU58FTfqVljq9SbKXxfgdr2SZ~7VgLaZ8IhA0WnlKUo-JgqTd~jYa5mGCpR8351IMK0aMuY4Mld4SOXssQ-rOtlZtypvo8FDp474TlGIEGz5PHxGOPsqLPF19hEYTgoPqsUj8QEuiTfg-cmsg__&Key-Pair-Id=APKAJXGC45PGQXCMCXSA",
        "bounds": [
          [
            37.87488726890353,
            -122.32488870620728
          ],
          [
            37.87280390440759,
            -122.32154130935669
          ]
        ],
        "min_zoom": 10,
        "max_zoom": 23,
        "epsg": "EPSG4326",
        "version": 2
      }
    },
    {
      "row_data": {
        "tile_layer_url": "https://public-tiles.dronedeploy.com/1499994155_DANIELOPENPIPELINE_ortho_qfs/{z}/{x}/{y}.png?Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9wdWJsaWMtdGlsZXMuZHJvbmVkZXBsb3kuY29tLzE0OTk5OTQxNTVfREFOSUVMT1BFTlBJUEVMSU5FX29ydGhvX3Fmcy8qIiwiQ29uZGl0aW9uIjp7IkRhdGVMZXNzVGhhbiI6eyJBV1M6RXBvY2hUaW1lIjoyMTQ1OTE0MTE4fX19XX0_&Signature=O~50rrGXdEC6Hi8jPJ3dbT~UtBd7Cw6iQPTxdJ8LU2IaoxeP22R3JpKPkLN3T3~Lcw3CyX7uft2Baj0MH93qUoCYyN~~jNX3OMkYV2jbrHDezf6zQRHAabXX-L2bL-JEGfFL6z3DWccOFeCH56CuhgC29k5CJx7I34P-LQJdnAUsA-KaqKH1IyYsHStRIfmMzdXNAWU58FTfqVljq9SbKXxfgdr2SZ~7VgLaZ8IhA0WnlKUo-JgqTd~jYa5mGCpR8351IMK0aMuY4Mld4SOXssQ-rOtlZtypvo8FDp474TlGIEGz5PHxGOPsqLPF19hEYTgoPqsUj8QEuiTfg-cmsg__&Key-Pair-Id=APKAJXGC45PGQXCMCXSA",
        "bounds": [
          [
            37.8749042065848,
            -122.32149839401245
          ],
          [
            37.87278696624341,
            -122.31810808181763
          ]
        ],
        "min_zoom": 10,
        "max_zoom": 23,
        "epsg": "EPSG4326",
        "version": 2
      }

    },
    {
      "row_data": {
        "tile_layer_url": "https://public-tiles.dronedeploy.com/1499994155_DANIELOPENPIPELINE_ortho_qfs/{z}/{x}/{y}.png?Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9wdWJsaWMtdGlsZXMuZHJvbmVkZXBsb3kuY29tLzE0OTk5OTQxNTVfREFOSUVMT1BFTlBJUEVMSU5FX29ydGhvX3Fmcy8qIiwiQ29uZGl0aW9uIjp7IkRhdGVMZXNzVGhhbiI6eyJBV1M6RXBvY2hUaW1lIjoyMTQ1OTE0MTE4fX19XX0_&Signature=O~50rrGXdEC6Hi8jPJ3dbT~UtBd7Cw6iQPTxdJ8LU2IaoxeP22R3JpKPkLN3T3~Lcw3CyX7uft2Baj0MH93qUoCYyN~~jNX3OMkYV2jbrHDezf6zQRHAabXX-L2bL-JEGfFL6z3DWccOFeCH56CuhgC29k5CJx7I34P-LQJdnAUsA-KaqKH1IyYsHStRIfmMzdXNAWU58FTfqVljq9SbKXxfgdr2SZ~7VgLaZ8IhA0WnlKUo-JgqTd~jYa5mGCpR8351IMK0aMuY4Mld4SOXssQ-rOtlZtypvo8FDp474TlGIEGz5PHxGOPsqLPF19hEYTgoPqsUj8QEuiTfg-cmsg__&Key-Pair-Id=APKAJXGC45PGQXCMCXSA",
        "bounds": [
          [
            37.8749042065848,
            -122.31806516647339
          ],
          [
            37.87278696624341,
            -122.31529712677002
          ]
        ],
        "min_zoom": 10,
        "max_zoom": 23,
        "epsg": "EPSG4326",
        "version": 2
      }
    }
  ]
  ```

  ```json Cartesian Coordinates Examples theme={"theme":"css-variables"}
  [
    {
      "row_data": {
        "tile_layer_url": "https://storage.googleapis.com/labelbox-developer-testing-assets/cartesian/sample_5184x2000_maxZoom_3/{z}/{x}/{y}.png",
        "bounds": [[0,0],[500,500]],
        "epsg": "Cartesian",
        "min_zoom": 0,
        "max_zoom": 3,
        "dimensions": {
          "width": 5184,
          "height": 2000
        },
      }
    },
    {
      "row_data":{
        "tile_layer_url": "https://storage.googleapis.com/labelbox-developer-testing-assets/cartesian/sample_5184x2000_maxZoom_3/{z}/{x}/{y}.png",
        "bounds": [[500, 500], [1000,1000]],
        "epsg": "Cartesian",
        "min_zoom": 0,
        "max_zoom": 3,
        "dimensions": {
          "width": 5184,
          "height": 2000
        },
      }
    },
    {
      "row_data":{
        "tile_layer_url": "https://storage.googleapis.com/labelbox-developer-testing-assets/cartesian/sample_5184x2000_maxZoom_3/{z}/{x}/{y}.png",
        "bounds": [[0, 0],[5184,2000]],
        "epsg": "Cartesian",
        "min_zoom": 0,
        "max_zoom": 3,
        "dimensions": {
          "width": 5184,
          "height": 2000
        },
      }
    }
  ]
  ```
</CodeGroup>

## Deep Zoom Servers (ex. Pathology Slides)

### Definitions

When using a Deep Zoom server endpoint (like OpenSlide or tiffslides), please use the following format.

| Parameter | Required | Description |
| - | - | - |
| `row_data` | Yes | A dictionary of`{ "tile_layer_url": str, "is_deep_zoom": bool, "min_zoom": int, "max_zoom": int, "bounds": [[float, float],[float, float]], "epsg": "EPSG3857" ... }` For IAM Delegated Access, this URL must be in [virtual-hosted-style format](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#virtual-hosted-style-access). |
| `row_data['version']` | No | Add version to your import to distinguish it from the legacy format. If `version: 2` is present, Labelbox will read all coordinates as `[long, lat]` or `[x, y]` depending on the data format. If `version: 1` is present, Labelbox will read all coordinates as `[lat, long]` or `[y, x]` depending on the data format. If no version is provided, the default value for DeepZoom files is `version: 2` |
| `row_data['tile_layer_url']` | Yes | For DeepZoom servers, the URL should be the API to return the tile with parameters for \{x}, \{y}, and \{z}. If using a Deep Zoom server, you must set up your coordinate system as 'Cartesian' |
| `row_data['is_deep_zoom']` | No | This must be set to "true" if using a Deep Zoom server |
| `row_data['min_zoom']` | No | Minimum map zoom level down to which this layer will be displayed (inclusive). If not populated, we will default to 0. |
| `row_data['max_zoom']` | No | Maximum map zoom level up to which this layer will be displayed (inclusive). If not populated, we will calculate this from your data. |
| `row_data['max_native_zoom']` | No | Maximum zoom number the tile source has available (`maxNativeZoom` & `maxZoom`). If specified, the tiles on all zoom levels higher than `maxNativeZoom` will be auto-scaled. |
| `row_data['bounds']` | Yes | The bounds where the projection is valid. For the Simple coordinate system, bounds are in x,y. For the EPSG coordinate system, bounds are in lat,long. To annotate the entire asset, the bounds should match up with your values in `dimensions` |
| `row_data['epsg']` | Yes | This should be set to `Cartesian` for Deep Zoom use cases. |
| `row_data['alternative_layers']` | No | Additional tile layers. Only `alternativeLayers.tileLayerUrl` and `alternativeLayers.name` are required. If no values for optional parameters are given, those values will default to the top-level keys. |
| `row_data['dimensions']` | No | This is required if using the Cartesian coordinate system. This represents the pixel height and width of the entire image. The format is: `"dimensions": { "width": <Pixel width>, "height": <Pixel height> }` |
| `global_key` | No | Unique user-generated file name or ID for the file. [Global keys](/horizon/developer-guides/data-row-global-keys) are enforced to be unique in your org. Data rows will not be imported if its global keys are duplicated to existing data rows. |
| `media_type` | No | `"TMS_GEO"` (optional media type to provide better validation and error messaging) |
| `metadata_fields` | No | See [Metadata](/horizon/guides/datarow-metadata). |
| `attachments` | No | See [Attachments](/horizon/guides/label-data) and [Asset overlays](/horizon/guides/label-data). |

### Import format

<CodeGroup>
  ```json Pathology Deep Zoom Exampe theme={"theme":"css-variables"}
  [
      {
          "row_data":{
              "tile_layer_url": "https://storage.googleapis.com/labelbox-datasets/openslides/CMU-1/slide_files/{z}/{x}_{y}.png",
              "is_deep_zoom": true,
              "epsg": "Cartesian",
              "min_zoom": 8,
              "max_xoom": 16,
              "dimensions": {
                  "height": 32914,
                  "width": 46000
              },
              "bounds": [[0,0], [46000,32914]]
          },
          "global_key": "https://storage.googleapis.com/labelbox-datasets/openslides/CMU-1/slide_files/{z}/{x}_{y}.png",
          "media_type": "TMS_GEO",
          "metadata_fields": [{"name": "<metadata_field_name>", "value": "tag_string"}],
          "attachments": [{"type": "TEXT_URL", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt"}]
      }
  ]
  ```
</CodeGroup>

## GeoTIFF (standard or cloud-optimized) and NITF specifications

### Definitions

When you upload a NITF file, Labelbox will convert it to a COG file and upload the asset. If it is already a COG file, Labelbox will skip the conversion and upload the asset natively. Since COG files can have many extension names, Labelbox will check the header of the file to verify that it is a COG file.

| Item | Supported Options | Comments |
| - | - | - |
| Coordinate Systems | `EPSG3857`, `EPSG3395`, or `EPSG4326` | If your COG files are not one of these supported coordinate systems, it must be converted prior to upload into Labelbox. We require that the files are geo-referenced in order to render on our map |
| Coordinate System Type | Projected CRS | Our editor only supports a Projected CRS (coordinate reference system). If you are using a geographic coordinate reference system, please convert it. |
| Bands | We support up to 4 bands: - Red - Green - Blue - Alpha Internal color-interpretation must be set on NITF files. | If your COG files have a band outside of these four support bands, the tiles will not be able to render in our editor. The alpha band must also be an "explicit alpha" meaning that the fourth band must be grouped under the alpha band and not in the "unknown" band |
| Alternative layers | | Additional GeoTIFF/NITF layer, specified as a URL/path to the file. |
| NITF Version | If you are using NITF files, they must be in NITF 2.0 or 2.1 format. | |
| File Size | up to 1 GB recommended | If you require larger files - please convert the files to cloud-optimized GeoTIFF and upload |

### Import format

<CodeGroup>
  ```json Standard URL - GeoTIFF theme={"theme":"css-variables"}
  [
      {
          "tileLayerUrl": "https://storage.googleapis.com/jakub-test/pre_DSM_hill_camera_4326.cog.tif",
          "alternativeLayers": [
            {
              "tileLayerUrl": "https://storage.googleapis.com/jakub-test/pre_DSM_hill_camera_4326_red.cog.tif",
              "name": "Red Lee Hill"
            }
          ],
          "bounds": [
            [40.05578109295705, -105.32837712340128],
            [40.06485226245013, -105.2813925407782]
          ]
      },
      {
          "tileLayerUrl": "https://storage.googleapis.com/jakub-test/pre_DSM_hill_camera_4326.cog.tif",
          "alternativeLayers": [
            {
              "tileLayerUrl": "https://storage.googleapis.com/jakub-test/pre_DSM_hill_camera_4326_red.cog.tif",
              "name": "Red Lee Hill"
            }
          ],
          "bounds": [
            [40.06485226245013, -105.32837712340128],
            [40.07392343194322, -105.2813925407782]
          ]
      }
  ]
  ```

  ```json Delegated Access URL - GeoTIFF theme={"theme":"css-variables"}
  [
    {
      "global_key": "https://lb-test-data.s3.us-west-1.amazonaws.com/geospatial-samples/sample-geotiff.tif",
      "row_data": {
        "tile_layer_url": "https://lb-test-data.s3.us-west-1.amazonaws.com/geospatial-samples/sample-geotiff.tif",
      },
      "media_type": "TMS_GEO",
      "metadata_fields": [{"schema_id": "cko8s9r5v0001h2dk9elqdidh", "value": "tag_string"}],
      "attachments": [{"type": "TEXT_URL", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt"}]
    }
  ]
  ```
</CodeGroup>

### Verify files are processed

<Warning>
  ### File processing can take up to 20 mins

  Since GeoTIFF or NITF files can be very large, the conversion can sometimes take up to 20 minutes to perform a data upload.
</Warning>

You can verify whether a file conversion is complete by checking the Media Attributes section.

If the Mime type shows as `application/x-tms-geo`, the file was successfully processed.

If the Mime type still shows as `application/json`, the file pre-processing has not yet been completed.

## Python example

For geospatial and custom-defined Data Rows, you have to specify the Data Row in a JSON format. Here's an example with alternative layers, bounds, attachments (text, image, and google maps widget).

<CodeGroup>
  ```python import example theme={"theme":"css-variables"}
  from labelbox import Client
  from uuid import uuid4 ## to generate unique IDs
  import datetime

  client = Client(api_key="<YOUR_API_KEY>")

  dataset = client.create_dataset(name="Bulk import example - Geospatial")

  assets = [
    {
      "global_key": "https://lb-test-data.s3.us-west-1.amazonaws.com/geospatial-samples/sample-geotiff.tif",
      "row_data": {
        "tile_layer_url": "https://lb-test-data.s3.us-west-1.amazonaws.com/geospatial-samples/sample-geotiff.tif",
      },
      "media_type": "TMS_GEO",
      "metadata_fields": [{"name": "<metadata_field_name>", "value": "tag_string"}],
      "attachments": [{"type": "TEXT_URL", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt"}]
    }
  ]

  task = dataset.create_data_rows(assets)
  task.wait_till_done()
  print(task.errors)
  ```
</CodeGroup>

<br />


# Import HTML annotations
Source: https://docs.labelbox.com/horizon/developer-guides/import-html-annotations

Developer guide for importing annotations on HTML data and sample import formats.

<CardGroup>
  <Card title="Open In Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/annotation_import/html.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/annotation_import/html.ipynb" />
</CardGroup>

## Overview

To import annotations in Labelbox, you need to create an annotations payload. In this section, we provide this payload for every supported annotation type.

### Annotation payload types

Labelbox supports two formats for the annotations payload:

* Python annotation types (recommended)

  * Provides a seamless transition between third-party platforms, machine learning pipelines, and Labelbox.
  * Allows you to build annotations locally with local file paths, numpy arrays, or URLs
  * Easily convert Python Annotation Type format to NDJSON format to quickly import annotations to Labelbox
  * Supports one-level nested classification (radio, checklist, or free-form text) under a tool or classification annotation.

* JSON

  * Skips formatting annotation payload in the Labelbox Python annotation type
  * Supports any levels of nested classification (radio, checklist, or free-form text) under a tool or classification annotation.

### Label import types

Labelbox additionally supports two types of label imports:

* [Model-assisted labeling (MAL)](/horizon/guides/model-assisted-labeling)
  * This workflow allows you to import computer-generated predictions (or simply annotations created outside of Labelbox) as pre-labels on an asset.
* [Ground truth](https://www.google.com/url?q=https%3A%2F%2Fdocs.labelbox.com%2Fdocs%2Fimport-ground-truth)
  * This workflow functionality allows you to bulk import your ground truth annotations from an external or third-party labeling system into Labelbox *Annotate*. Using the label import API to import external data is a useful way to consolidate and migrate all annotations into Labelbox as a single source of truth.

## Supported annotations

The following annotations are supported for a HTML data row:

* Radio
* Checklist
* Free-form text

## Classifications

### Radio (single choice)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  radio_annotation = lb_types.ClassificationAnnotation(
      name="radio_question",
      value=lb_types.Radio(answer =
          lb_types.ClassificationAnswer(name = "first_radio_answer")
      )
  )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  radio_annotation_ndjson = {
    "name": "radio_question",
    "answer": {"name": "first_radio_answer"}
  }
  ```
</CodeGroup>

### Checklist (multiple choice)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  checklist_annotation = lb_types.ClassificationAnnotation(
      name="checklist_question",
      value=lb_types.Checklist(answer = [
          lb_types.ClassificationAnswer(name = "first_checklist_answer"),
          lb_types.ClassificationAnswer(name = "second_checklist_answer"),
          lb_types.ClassificationAnswer(name = "third_checklist_answer")
      ])
    )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  checklist_annotation_ndjson = {
    "name": "checklist_question",
    "answer": [
      {"name": "first_checklist_answer"},
      {"name": "second_checklist_answer"},
      {"name": "third_checklist_answer"},
    ]
  }
  ```
</CodeGroup>

### Free-form text

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  text_annotation = lb_types.ClassificationAnnotation(
      name = "free_text",
      value = lb_types.Text(answer="sample text")
  )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  text_annotation_ndjson = {
    "name": "free_text",
    "answer": "sample text",
  }
  ```
</CodeGroup>

### Nested classifications

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  nested_radio_annotation = lb_types.ClassificationAnnotation(
    name="nested_radio_question",
    value=lb_types.Radio(
      answer=lb_types.ClassificationAnswer(
        name="first_radio_answer",
        classifications=[
          lb_types.ClassificationAnnotation(
            name="sub_radio_question",
            value=lb_types.Radio(
              answer=lb_types.ClassificationAnswer(
                name="first_sub_radio_answer"
              )
            )
          )
        ]
      )
    )
  )

  nested_checklist_annotation = lb_types.ClassificationAnnotation(
    name="nested_checklist_question",
    value=lb_types.Checklist(
      answer=[lb_types.ClassificationAnswer(
        name="first_checklist_answer",
        classifications=[
          lb_types.ClassificationAnnotation(
            name="sub_checklist_question",
            value=lb_types.Checklist(
              answer=[lb_types.ClassificationAnswer(
              name="first_sub_checklist_answer"
            )]
          ))
        ]
      )]
    )
  )

  ```

  ```json NDJSON theme={"theme":"css-variables"}
  nested_radio_annotation_ndjson= {
    "name": "nested_radio_question",
    "answer": {
        "name": "first_radio_answer",
        "classifications": [
          {
            "name": "sub_radio_question",
            "answer": {"name": "first_sub_radio_answer"}
          }
        ]
      }
  }

  nested_checklist_annotation_ndjson = {
    "name": "nested_checklist_question",
    "answer": [{
        "name": "first_checklist_answer",
        "classifications" : [
          {
            "name": "sub_checklist_question",
            "answer": {"name": "first_sub_checklist_answer"}
          }
        ]
    }]
  }
  ```
</CodeGroup>

## Example: Import pre-labels or ground truths

The steps to import annotations as pre-labels (machine-assisted learning) are similar to those to import annotations as ground truth labels. However, they vary slightly, and we will describe the differences for each scenario.

### Before you start

The below imports are needed to use the code examples in this section.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  import labelbox as lb
  import uuid
  import labelbox.types as lb_types
  ```
</CodeGroup>

Replace the value of `API_KEY` with a valid [API key](/horizon/developer-guides/create-api-key) to connect to the Labelbox client.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  API_KEY = None
  client = lb.Client(API_KEY)
  ```
</CodeGroup>

### Step 1: Import data rows

Data rows must first be uploaded to **Catalog** to attach annotations.

This example shows how to create a data row in **Catalog** by attaching it to a [dataset](/horizon/developer-guides/dataset) .

<CodeGroup>
  ```python theme={"theme":"css-variables"}
  global_key = "sample_html_1.html"

  asset = {
      "row_data": "https://storage.googleapis.com/labelbox-datasets/html_sample_data/sample_html_1.html",
      "global_key": global_key
  }

  dataset = client.create_dataset(
      name="html_annotation_import_demo_dataset", 
      iam_integration=None # Removing this argument will default to the organziation's default iam integration
  ) 
  task = dataset.create_data_rows([asset])
  task.wait_till_done()
  print("Errors:", task.errors)
  print("Failed data rows: ", task.failed_data_rows)
  ```
</CodeGroup>

### Step 2: Set up ontology

Your project ontology should support the tools and classifications required by your annotations. To ensure accurate schema feature mapping, the value used as the `name` parameter should match the value of the `name` field in your annotation.

For example, when we created an annotation above, we provided a name`annotation_name`. Now, when we set up our ontology, we must ensure that the name of our bounding box tool is also `anotations_name`. The same alignment must hold true for the other tools and classifications we create in our ontology.

This example shows how to create an ontology containing all supported [annotation types](#supported-annotations) .

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  ontology_builder = lb.OntologyBuilder(
    classifications=[
      lb.Classification(
        class_type=lb.Classification.Type.TEXT,
        name="text_html"),
      lb.Classification(
        class_type=lb.Classification.Type.CHECKLIST,
        name="checklist_html",
        options=[
          lb.Option(value="first_checklist_answer"),
          lb.Option(value="second_checklist_answer")
        ]
      ),
      lb.Classification(
        class_type=lb.Classification.Type.RADIO,
        name="radio_html",
        options=[
          lb.Option(value="first_radio_answer"),
          lb.Option(value="second_radio_answer")
        ]
      )
    ]
  )

  ontology = client.create_ontology("Ontology HTML Annotations", ontology_builder.asdict(), media_type=lb.MediaType.Html)
  ```
</CodeGroup>

### Step 3: Set Up a Labeling Project

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  project = client.create_project(name="html_project",
                                      media_type=lb.MediaType.Html)

  # Setup your ontology

  project.connect_ontology(ontology)

  ```
</CodeGroup>

### Step 4: Send Data Rows to Project

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  batch = project.create_batch(
    "first-batch-html-demo", # Each batch in a project must have a unique name
    global_keys=[global_key], # Paginated collection of data row objects, list of data row ids or global keys
    priority=5 # priority between 1(highest) - 5(lowest)
  )

  print("Batch: ", batch)
  ```
</CodeGroup>

### Step 5: Create annotation payloads

For help understanding annotation payloads, see [overview](#overview). To declare payloads, you can use Python annotation types (*preferred*) or NDJSON objects.

These examples demonstrate each format and how to compose annotations into labels attached to data rows.

<CodeGroup>
  ```python Python Annotation Payload theme={"theme":"css-variables"}
  label = []
  label.append(
    lb_types.Label(
      data={"global_key" : global_key },
      annotations=[
        text_annotation,
        checklist_annotation,
        radio_annotation
      ]
    )
  )
  ```

  ```python NDJSON Payload theme={"theme":"css-variables"}
  label_ndjson = []
  for annotations in [text_annotation_ndjson,
                      checklist_annotation_ndjson,
                      radio_annotation_ndjson]:
    annotations.update({
        'dataRow': {
            'globalKey': global_key
        }
    })
    label_ndjson.append(annotations)
  ```
</CodeGroup>

### Step 6: Import annotation payload

For prelabeled (model-assisted labeling) scenarios, pass your payload as the value of the `predictions` parameter. For ground truths, pass the payload to the `labels` parameter.

#### Option A: Upload as [prelabels (model assisted labeling)](/horizon/guides/model-assisted-labeling)

This option is helpful for speeding up the initial labeling process and reducing the manual labeling workload for high-volume datasets.

<CodeGroup>
  ```python MAL import theme={"theme":"css-variables"}
  # Upload MAL label for this data row in project
  upload_job = lb.MALPredictionImport.create_from_objects(
      client = client,
      project_id = project.uid,
      name="mal_job"+str(uuid.uuid4()),
      predictions=label
  )

  print(f"Errors: {upload_job.errors}", )
  print(f"Status of uploads: {upload_job.statuses}")
  ```
</CodeGroup>

#### Option B: Upload to a labeling project as [ground truth](/horizon/guides/import-ground-truth)

This option is helpful for loading high-confidence labels from another platform or previous projects that just need review rather than manual labeling effort.

<CodeGroup>
  ```python Label import theme={"theme":"css-variables"}
  # Upload label for this data row in project
  upload_job = lb.LabelImport.create_from_objects(
      client = client,
      project_id = project.uid,
      name="label_import_job"+str(uuid.uuid4()),
      labels=label
  )

  print(f"Errors: {upload_job.errors}", )
  print(f"Status of uploads: {upload_job.statuses}")
  ```
</CodeGroup>


# Import HTML data
Source: https://docs.labelbox.com/horizon/developer-guides/import-html-data

How to import HTML data and sample import formats.

<Info>
  ### Supported file formats and import methods

  Format: HTML Import methods:

  * IAM Delegated Access
  * Signed URLs (`https` URLs only)
</Info>

## Parameters

Import methods:

* IAM Delegated Access
* Signed URLs (`https` URLs only)

| Parameter | Required | Description |
| - | - | - |
| `row_data` | Yes | `https` path to an HTML file. For IAM Delegated Access, this URL must be in [virtual-hosted-style format](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#virtual-hosted-style-access). For older regions, your S3 bucket may be in the`https://<bucket-name>.s3.<region>.amazonaws.com/<key>` format. If your object URLs are formatted this way, make sure they are in the virtual-hosted-style format before importing. |
| `global_key` | No | Unique user-generated file name or ID for the file. [Global keys](/horizon/developer-guides/data-row-global-keys) are enforced to be unique in your org. Data rows will not be imported if its global keys are duplicated to existing data rows. |
| `media_type` | No | `"HTML"` (optional media type to provide better validation and error messaging) |
| `metadata_fields` | No | See [Metadata](/horizon/guides/datarow-metadata). |
| `attachments` | No | See [Attachments](/horizon/guides/label-data) and [Asset overlays](/horizon/guides/label-data), |

## Import format

<CodeGroup>
  ```json Delegated Access URL theme={"theme":"css-variables"}
  [
    {
      "row_data": "https://lb-test-data.s3.us-west-1.amazonaws.com/sample_html_1.html",
      "global_key": "https://lb-test-data.s3.us-west-1.amazonaws.com/sample_html_1.html",
      "metadata_fields": [{"name": "<metadata_field_name>", "value": "tag_string"}],
      "attachments": [{"type": "HTML", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/windy.html" }]
    },
    {
      "row_data": "https://lb-test-data.s3.us-west-1.amazonaws.com/sample_html_2.html",
      "global_key": "https://lb-test-data.s3.us-west-1.amazonaws.com/sample_html_2.html",
      "metadata_fields": [{"name": "<metadata_field_name>", "value": "tag_string"}],
      "attachments": [{"type": "TEXT_URL", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt"}]
    }
  ]
  ```

  ```json Standard URL theme={"theme":"css-variables"}
  [
    {
      "row_data": "https://storage.googleapis.com/labelbox-datasets/html_sample_data/sample_html_1.html",
      "global_key": "https://storage.googleapis.com/labelbox-datasets/html_sample_data/sample_html_1.html",
      "metadata_fields": [{"schema_id": "cko8s9r5v0001h2dk9elqdidh", "value": "tag_string"}],
      "attachments": [{"type": "HTML", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/windy.html" }]
    },
    {
      "row_data": "https://storage.googleapis.com/labelbox-datasets/html_sample_data/sample_html_2.html",
      "global_key": "https://storage.googleapis.com/labelbox-datasets/html_sample_data/sample_html_2.html",
      "metadata_fields": [{"schema_id": "cko8s9r5v0001h2dk9elqdidh", "value": "tag_string"}],
      "attachments": [{"type": "TEXT_URL", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt"}]
    }
  ]
  ```

  ```html HTML example theme={"theme":"css-variables"}
  <html>
    <head>
      <title>HTML File Example</title>
    </head>
  <body bgcolor="ffffff">

  <center><img src="https://labelbox.com/static/images/logo-v4.svg" align="bottom">

  <hr>

  <h1>Get to production AI faster</h1>

  <p>Save time by creating and managing your training data, people, and processes in a single place — so you can focus on building the next big thing.</p>

  <p><a href="https://labelbox.com/sales">Get a demo</a> or <a href="https://app.labelbox.com/">start for free.</a>
  </center>
  <hr>

  </body>

  </html>
  ```
</CodeGroup>

## Python example

<CodeGroup>
  ```python bulk import example theme={"theme":"css-variables"}
  from labelbox import Client
  from uuid import uuid4 ## to generate unique IDs
  import datetime

  client = Client(api_key="<YOUR_API_KEY>")

  dataset = client.create_dataset(name="Bulk import example - HTML")

  assets = [
    {
      "row_data": "https://storage.googleapis.com/labelbox-datasets/html_sample_data/sample_html_1.html",
      "global_key": "https://storage.googleapis.com/labelbox-datasets/html_sample_data/sample_html_1.html",
      "metadata_fields": [{"name": "<metadata_field_name>", "value": "tag_string"}],
      "attachments": [{"type": "HTML", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/windy.html" }]
    },
    {
      "row_data": "https://storage.googleapis.com/labelbox-datasets/html_sample_data/sample_html_2.html",
      "global_key": "https://storage.googleapis.com/labelbox-datasets/html_sample_data/sample_html_2.html",
      "metadata_fields": [{"name": "<metadata_field_name>", "value": "tag_string"}],
      "attachments": [{"type": "TEXT_URL", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt"}]
    }
  ]

  task = dataset.create_data_rows(assets)
  task.wait_till_done()
  print(task.errors)
  ```

  ```python local files theme={"theme":"css-variables"}
  local_file_paths = ['path/to/local/file1', 'path/to/local/file1'] # limit: 15k files

  new_dataset = client.create_dataset(name = "Local files upload")

  try:
      task = new_dataset.create_data_rows(local_file_paths)
      task.wait_till_done()
  except Exception as err:
      print(f'Error while creating labelbox dataset -  Error: {err}')
  ```
</CodeGroup>


# Import image annotations
Source: https://docs.labelbox.com/horizon/developer-guides/import-image-annotations

Developer guide for importing annotations on image data and sample import formats.

<CardGroup>
  <Card title="Open In Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/annotation_import/image.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/annotation_import/image.ipynb" />
</CardGroup>

## Overview

To import annotations in Labelbox, you need to create an annotations payload. In this section, we provide this payload for every supported annotation type.

### Annotation payload types

Labelbox supports two formats for the annotations payload:

* Python annotation types (recommended)
  * Provides a seamless transition between third-party platforms, machine learning pipelines, and Labelbox.
  * Allows you to build annotations locally with local file paths, numpy arrays, or URLs
  * Easily convert Python Annotation Type format to NDJSON format to quickly import annotations to Labelbox
  * Supports one-level nested classification (radio, checklist, or free-form text) under a tool or classification annotation.
* JSON
  * Skips formatting annotation payload in the Labelbox Python annotation type
  * Supports any levels of nested classification (radio, checklist, or free-form text) under a tool or classification annotation.

### Label import types

Labelbox additionally supports two types of label imports:

* [Model-assisted labeling (MAL)](/horizon/guides/model-assisted-labeling)
  * This workflow allows you to import computer-generated predictions (or simply annotations created outside of Labelbox) as pre-labels on an asset.
* [Ground truth](https://www.google.com/url?q=https%3A%2F%2Fdocs.labelbox.com%2Fdocs%2Fimport-ground-truth)
  * This workflow functionality allows you to bulk import your ground truth annotations from an external or third-party labeling system into Labelbox *Annotate*. Using the label import API to import external data is a useful way to consolidate and migrate all annotations into Labelbox as a single source of truth.

## Supported annotations

The following annotations are supported for an image data row:

* Radio
* Checklist
* Free-form text
* Bounding box
* Point
* Polyline
* Polygon
* Segmentation masks
* Relationships

## Classifications

### Radio (single choice)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  radio_annotation = lb_types.ClassificationAnnotation(
      name="radio_question",
      value=lb_types.Radio(answer =
          lb_types.ClassificationAnswer(name = "first_radio_answer")
      )
  )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  radio_annotation_ndjson = {
    "name": "radio_question",
    "answer": {"name": "first_radio_answer"}
  }
  ```
</CodeGroup>

### Checklist (multiple choice)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  checklist_annotation = lb_types.ClassificationAnnotation(
      name="checklist_question",
      value=lb_types.Checklist(answer = [
          lb_types.ClassificationAnswer(name = "first_checklist_answer"),
          lb_types.ClassificationAnswer(name = "second_checklist_answer"),
          lb_types.ClassificationAnswer(name = "third_checklist_answer")
      ])
    )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  checklist_annotation_ndjson = {
    "name": "checklist_question",
    "answer": [
      {"name": "first_checklist_answer"},
      {"name": "second_checklist_answer"},
      {"name": "third_checklist_answer"},
    ]
  }
  ```
</CodeGroup>

### Free-form text

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  text_annotation = lb_types.ClassificationAnnotation(
      name = "free_text",
      value = lb_types.Text(answer="sample text")
  )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  text_annotation_ndjson = {
    "name": "free_text",
    "answer": "sample text",
  }
  ```
</CodeGroup>

### Nested classifications

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  nested_radio_annotation = lb_types.ClassificationAnnotation(
    name="nested_radio_question",
    value=lb_types.Radio(
      answer=lb_types.ClassificationAnswer(
        name="first_radio_answer",
        classifications=[
          lb_types.ClassificationAnnotation(
            name="sub_radio_question",
            value=lb_types.Radio(
              answer=lb_types.ClassificationAnswer(
                name="first_sub_radio_answer"
              )
            )
          )
        ]
      )
    )
  )

  nested_checklist_annotation = lb_types.ClassificationAnnotation(
    name="nested_checklist_question",
    value=lb_types.Checklist(
      answer=[lb_types.ClassificationAnswer(
        name="first_checklist_answer",
        classifications=[
          lb_types.ClassificationAnnotation(
            name="sub_checklist_question",
            value=lb_types.Checklist(
              answer=[lb_types.ClassificationAnswer(
              name="first_sub_checklist_answer"
            )]
          ))
        ]
      )]
    )
  )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  nested_radio_annotation_ndjson= {
    "name": "nested_radio_question",
    "answer": {
        "name": "first_radio_answer",
        "classifications": [
          {
            "name":"sub_radio_question",
            "answer": { "name" : "first_sub_radio_answer"}
          }
        ]
      }
  }

  nested_checklist_annotation_ndjson = {
    "name": "nested_checklist_question",
    "answer": [{
        "name": "first_checklist_answer",
        "classifications" : [
          {
            "name": "sub_checklist_question",
            "answer": {"name": "first_sub_checklist_answer"}
          }
        ]
    }]
  }
  ```
</CodeGroup>

## Tools

### Bounding Box

<CodeGroup>
  ```python Python Annotation theme={"theme":"css-variables"}
  bbox_annotation = lb_types.ObjectAnnotation(
      name="bounding_box",  # must match your ontology feature's name
      value=lb_types.Rectangle(
          start=lb_types.Point(x=1690, y=977),  # x = left, y = top
          end=lb_types.Point(x=1915, y=1307),  # x= left + width , y = top + height
      ))
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  bbox_annotation_ndjson = {
      "name": "bounding_box",
      "bbox": {
          "top": 977,
          "left": 1690,
          "height": 330,
          "width": 225
      }
  }
  ```
</CodeGroup>

### Segmentation mask

The following examples show how to import composite masks or a specific annotation instance from a composite mask by using the instance's unique `color_rgb` value.

<Warning>
  ### Size limit

  Mask imports are limited to 9,000 x 9,000 pixels. Larger masks can't be imported.
</Warning>

<CodeGroup>
  ```python Composite masks theme={"theme":"css-variables"}
  # First we need to extract all the unique colors from the composite mask
  def extract_rgb_colors_from_url(image_url):
      response = requests.get(image_url)
      img = Image.open(BytesIO(response.content))

      colors = set()
      for x in range(img.width):
          for y in range(img.height):
              pixel = img.getpixel((x, y))
              if pixel[:3] != (0,0,0):
                  colors.add(pixel[:3])  # Get only the RGB values

      return colors
  ```

  ```python Specific annotations theme={"theme":"css-variables"}
  def extract_single_rgb_color(image_url, target_color_rgb):
      response = requests.get(image_url)
      img = Image.open(BytesIO(response.content))

      # Check if the target color exists in the image
      for x in range(img.width):
          for y in range(img.height):
              pixel = img.getpixel((x, y))
              if pixel[:3] == target_color_rgb:
                  return target_color_rgb

      raise ValueError(f"Target color {target_color_rgb} not found in the image.")
  ```
</CodeGroup>

<CodeGroup>
  ```python Python Annotation theme={"theme":"css-variables"}
  cp_mask_url = "https://storage.googleapis.com/labelbox-datasets/image_sample_data/composite_mask.png"
  colors = extract_rgb_colors_from_url(cp_mask_url)
  mask_data = lb.types.MaskData(url=cp_mask_url)
  # These are the colors that will be associated with the mask_with_text_subclass tool
  rgb_colors_for_mask_with_text_subclass_tool = [(73, 39, 85), (111, 87, 176), (23, 169, 254)]

  cp_mask = []
  for color in colors:
    # We are assigning the color related to the mask_with_text_subclass tool by identifying the unique RGB colors
    if color in rgb_colors_for_mask_with_text_subclass_tool:
      cp_mask.append(
        lb_types.ObjectAnnotation(
          name = "mask_with_text_subclass", # must match your ontology feature"s name
          value=lb_types.Mask(
            mask=mask_data,
            color=color),
          classifications=[
            lb_types.ClassificationAnnotation(
              name="sub_free_text",
              value=lb_types.Text(answer="free text answer sample")
            )]
        )
    )
    else:
       # Create ObjectAnnotation for other masks
      cp_mask.append(
        lb_types.ObjectAnnotation(
          name="mask",
          value=lb_types.Mask(
              mask=mask_data,
              color=color
          )
      )
    )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  # NDJSON using bytes array
  cp_mask_url = "https://storage.googleapis.com/labelbox-datasets/image_sample_data/composite_mask.png"
  colors = extract_rgb_colors_from_url(cp_mask_url)
  cp_mask_ndjson = []
  #Using bytes array.
  response = requests.get(cp_mask_url)
  im_bytes = base64.b64encode(response.content).decode('utf-8')
  for color in colors:
    if color in rgb_colors_for_mask_with_text_subclass_tool:
      cp_mask_ndjson.append({
        "name": "mask_with_text_subclass",
        "mask": {"imBytes": im_bytes,
        "colorRGB": color },
        "classifications":[{
          "name": "sub_free_text",
          "answer": "free text answer"
        }]
      }
    )
    else:
      cp_mask_ndjson.append({
            "name": "mask",
            "classifications": [],
            "mask": {
              "imBytes": im_bytes,
              "colorRGB": color
        }
      }
    )
  ```
</CodeGroup>

### Point

<CodeGroup>
  ```python Python Annotation theme={"theme":"css-variables"}
  point_annotation = lb_types.ObjectAnnotation(
      name="point",  # must match your ontology feature's name
      value=lb_types.Point(x=1166.606, y=1441.768),
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  point_annotation_ndjson = {
      "name": "point",
      "classifications": [],
      "point": {
          "x": 1166.606,
          "y": 1441.768
      }
  }
  ```
</CodeGroup>

### Polyline

<CodeGroup>
  ```python Python Annotation theme={"theme":"css-variables"}
  polyline_annotation = lb_types.ObjectAnnotation(
      name="polyline",  # must match your ontology feature's name
      value=lb_types.Line(  # Coordinates for the keypoints in your polyline
          points=[
              lb_types.Point(x=2534.353, y=249.471),
              lb_types.Point(x=2429.492, y=182.092),
              lb_types.Point(x=2294.322, y=221.962),
              lb_types.Point(x=2224.491, y=180.463),
              lb_types.Point(x=2136.123, y=204.716),
              lb_types.Point(x=1712.247, y=173.949),
              lb_types.Point(x=1703.838, y=84.438),
              lb_types.Point(x=1579.772, y=82.61),
              lb_types.Point(x=1583.442, y=167.552),
              lb_types.Point(x=1478.869, y=164.903),
              lb_types.Point(x=1418.941, y=318.149),
              lb_types.Point(x=1243.128, y=400.815),
              lb_types.Point(x=1022.067, y=319.007),
              lb_types.Point(x=892.367, y=379.216),
              lb_types.Point(x=670.273, y=364.408),
              lb_types.Point(x=613.114, y=288.16),
              lb_types.Point(x=377.559, y=238.251),
              lb_types.Point(x=368.087, y=185.064),
              lb_types.Point(x=246.557, y=167.286),
              lb_types.Point(x=236.648, y=285.61),
              lb_types.Point(x=90.929, y=326.412)
          ]),
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  polyline_annotation_ndjson = {
    "name": "polyline",
    "classifications": [],
    "line": [
      {"x": 2534.353, "y": 249.471},
      {"x": 2429.492, "y": 182.092},
      {"x": 2294.322, "y": 221.962},
      {"x": 2224.491, "y": 180.463},
      {"x": 2136.123, "y": 204.716},
      {"x": 1712.247, "y": 173.949},
      {"x": 1703.838, "y": 84.438},
      {"x": 1579.772, "y": 82.61},
      {"x": 1583.442, "y": 167.552},
      {"x": 1478.869, "y": 164.903},
      {"x": 1418.941, "y": 318.149},
      {"x": 1243.128, "y": 400.815},
      {"x": 1022.067, "y": 319.007},
      {"x": 892.367, "y": 379.216},
      {"x": 670.273, "y": 364.408},
      {"x": 613.114, "y": 288.16},
      {"x": 377.559, "y": 238.251},
      {"x": 368.087, "y": 185.064},
      {"x": 246.557, "y": 167.286},
      {"x": 236.648, "y": 285.61},
      {"x": 90.929, "y": 326.412}
    ]
  }
  ```
</CodeGroup>

### Polygon

<CodeGroup>
  ```python Python Annotation theme={"theme":"css-variables"}
  polygon_annotation = lb_types.ObjectAnnotation(
      name="polygon",  # must match your ontology feature"s name
      value=lb_types.Polygon(  # Coordinates for the vertices of your polygon
          points=[
              lb_types.Point(x=1489.581, y=183.934),
              lb_types.Point(x=2278.306, y=256.885),
              lb_types.Point(x=2428.197, y=200.437),
              lb_types.Point(x=2560.0, y=335.419),
              lb_types.Point(x=2557.386, y=503.165),
              lb_types.Point(x=2320.596, y=503.103),
              lb_types.Point(x=2156.083, y=628.943),
              lb_types.Point(x=2161.111, y=785.519),
              lb_types.Point(x=2002.115, y=894.647),
              lb_types.Point(x=1838.456, y=877.874),
              lb_types.Point(x=1436.53, y=874.636),
              lb_types.Point(x=1411.403, y=758.579),
              lb_types.Point(x=1353.853, y=751.74),
              lb_types.Point(x=1345.264, y=453.461),
              lb_types.Point(x=1426.011, y=421.129),
          ]),
  )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  polygon_annotation_ndjson = {
      "name":
          "polygon",
      "polygon": [
          {
              "x": 1489.581,
              "y": 183.934
          },
          {
              "x": 2278.306,
              "y": 256.885
          },
          {
              "x": 2428.197,
              "y": 200.437
          },
          {
              "x": 2560.0,
              "y": 335.419
          },
          {
              "x": 2557.386,
              "y": 503.165
          },
          {
              "x": 2320.596,
              "y": 503.103
          },
          {
              "x": 2156.083,
              "y": 628.943
          },
          {
              "x": 2161.111,
              "y": 785.519
          },
          {
              "x": 2002.115,
              "y": 894.647
          },
          {
              "x": 1838.456,
              "y": 877.874
          },
          {
              "x": 1436.53,
              "y": 874.636
          },
          {
              "x": 1411.403,
              "y": 758.579
          },
          {
              "x": 1353.853,
              "y": 751.74
          },
          {
              "x": 1345.264,
              "y": 453.461
          },
          {
              "x": 1426.011,
              "y": 421.129
          },
          {
              "x": 1489.581,
              "y": 183.934
          },
      ],
  }
  ```
</CodeGroup>

### Tool with nested classification

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  tool_with_radio_subclass_annotation = lb_types.ObjectAnnotation(
      name=# Feature name,
      value=# Add tool annotation (lb_types."tool"),
      classifications=[
          lb_types.ClassificationAnnotation(
              name="sub_radio_question",
              value=lb_types.Radio(answer=lb_types.ClassificationAnswer(
                  name="first_sub_radio_answer")))
      ])
  ```

  ```python Bounding Box Example theme={"theme":"css-variables"}
  bbox_with_radio_subclass_annotation = lb_types.ObjectAnnotation(
      name="bbox_with_radio_subclass",
      value=lb_types.Rectangle(
          start=lb_types.Point(x=541, y=933),  # x = left, y = top
          end=lb_types.Point(x=871, y=1124),  # x= left + width , y = top + height
      ),
      classifications=[
          lb_types.ClassificationAnnotation(
              name="sub_radio_question",
              value=lb_types.Radio(answer=lb_types.ClassificationAnswer(
                  name="first_sub_radio_answer")))
      ])
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  bbox_with_radio_subclass_ndjson = {
      "name": "bbox_with_radio_subclass",
      "classifications": [{
          "name": "sub_radio_question",
          "answer": {
              "name": "first_sub_radio_answer"
          }
      }],
      "bbox": {
          "top": 933,
          "left": 541,
          "height": 191,
          "width": 330
      }
  }
  ```
</CodeGroup>

## Relationships

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  relationship = lb_types.RelationshipAnnotation(
      name=# Relationship name,
      value=lb_types.Relationship(
          source= # Source tool. Alternatively, specify a source_ontology_name
          target= # Target tool,
          type=lb_types.Relationship.Type.UNIDIRECTIONAL,
      ))
  ```

  ```python Bounding Box Example theme={"theme":"css-variables"}
  bbox_source = lb_types.ObjectAnnotation(
      name="bounding_box",
      value=lb_types.Rectangle(
          start=lb_types.Point(x=2096, y=1264),
          end=lb_types.Point(x=2240, y=1689),
      ),
  )

  bbox_target = lb_types.ObjectAnnotation(
      name="bounding_box",
      value=lb_types.Rectangle(
          start=lb_types.Point(x=2272, y=1346),
          end=lb_types.Point(x=2416, y=1704),
      ),
  )

  relationship = lb_types.RelationshipAnnotation(
      name="relationship",
      value=lb_types.Relationship(
          source=bbox_source, # Python annotations do not required a UUID reference
          target=bbox_target,
          type=lb_types.Relationship.Type.UNIDIRECTIONAL,
      ))
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  uuid_source = str(uuid.uuid4())
  uuid_target = str(uuid.uuid4())

  bbox_source_ndjson = {
      "uuid": uuid_source,
      "name": "bounding_box",
      "bbox": {
          "top": 1264.0,
          "left": 2096.0,
          "height": 425.0,
          "width": 144.0
      }
  }

  bbox_target_ndjson = {
      "uuid": uuid_target,
      "name": "bounding_box",
      "bbox": {
          "top": 1346.0,
          "left": 2272.0,
          "height": 358.0,
          "width": 144.0
      }
  }

  relationship_ndjson = {
      "name": "relationship",
      "relationship": {
          "source": uuid_source, # Alternatively, specify a source_ontology_name
          "target": uuid_target, # UUID reference to the target annotation
          "type": "unidirectional"
      }
  }
  ```
</CodeGroup>

## Example: Import pre-labels or ground truths

The steps to import annotations as pre-labels (machine-assisted learning) are similar to those to import annotations as ground truth labels. However, they vary slightly, and we will describe the differences for each scenario.

### Before you start

The below imports are needed to use the code examples in this section.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  import uuid
  from PIL import Image
  import requests
  import base64
  import labelbox as lb
  import labelbox.types as lb_types
  from io import BytesIO
  ```
</CodeGroup>

Replace the value of `API_KEY` with a valid [API key](/horizon/developer-guides/create-api-key) to connect to the Labelbox client.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  API_KEY = None
  client = lb.Client(API_KEY)
  ```
</CodeGroup>

### Step 1: Import data rows

Data rows must first be uploaded to **Catalog** to attach annotations.

This example shows how to create a data row in **Catalog** by attaching it to a [dataset](/horizon/developer-guides/dataset) .

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # send a sample image as batch to the project
  global_key = "2560px-Kitano_Street_Kobe01s5s4110.jpeg"

  test_img_url = {
      "row_data":
          "https://storage.googleapis.com/labelbox-datasets/image_sample_data/2560px-Kitano_Street_Kobe01s5s4110.jpeg",
      "global_key":
          global_key
  }

  dataset = client.create_dataset(name="image-demo-dataset")
  task = dataset.create_data_rows([test_img_url])
  task.wait_till_done()

  print(f"Failed data rows: {task.failed_data_rows}")
  print(f"Errors: {task.errors}")

  if task.errors:
      for error in task.errors:
          if 'Duplicate global key' in error['message'] and dataset.row_count == 0:
              # If the global key already  exists in the workspace the dataset will be created empty, so we can delete it.
              print(f"Deleting empty dataset: {dataset}")
              dataset.delete()
  ```
</CodeGroup>

### Step 2: Set up ontology

Your project ontology should support the tools and classifications required by your annotations. To ensure accurate schema feature mapping, the value used as the `name` parameter should match the value of the `name` field in your annotation.

For example, when we created an annotation above, we provided a name`annotation_name`. Now, when we set up our ontology, we must ensure that the name of our bounding box tool is also `anotations_name`. The same alignment must hold true for the other tools and classifications we create in our ontology.

This example shows how to create an ontology containing all supported [annotation types](#supported-annotations) .

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  ontology_builder = lb.OntologyBuilder(
      classifications=[  # list of classification objects
          lb.Classification(class_type=lb.Classification.Type.RADIO,
                            name="radio_question",
                            options=[
                                lb.Option(value="first_radio_answer"),
                                lb.Option(value="second_radio_answer")
                            ]),
          lb.Classification(class_type=lb.Classification.Type.CHECKLIST,
                            name="checklist_question",
                            options=[
                                lb.Option(value="first_checklist_answer"),
                                lb.Option(value="second_checklist_answer")
                            ]),
          lb.Classification(class_type=lb.Classification.Type.TEXT,
                            name="free_text"),
          lb.Classification(
              class_type=lb.Classification.Type.RADIO,
              name="nested_radio_question",
              options=[
                  lb.Option("first_radio_answer",
                            options=[
                                lb.Classification(
                                    class_type=lb.Classification.Type.RADIO,
                                    name="sub_radio_question",
                                    options=[lb.Option("first_sub_radio_answer")])
                            ])
              ]),
          lb.Classification(
              class_type=lb.Classification.Type.CHECKLIST,
              name="nested_checklist_question",
              options=[
                  lb.Option(
                      "first_checklist_answer",
                      options=[
                          lb.Classification(
                              class_type=lb.Classification.Type.CHECKLIST,
                              name="sub_checklist_question",
                              options=[lb.Option("first_sub_checklist_answer")])
                      ])
              ]),
      ],
      tools=[  # List of Tool objects
          lb.Tool(tool=lb.Tool.Type.BBOX, name="bounding_box"),
          lb.Tool(tool=lb.Tool.Type.BBOX,
                  name="bbox_with_radio_subclass",
                  classifications=[
                      lb.Classification(
                          class_type=lb.Classification.Type.RADIO,
                          name="sub_radio_question",
                          options=[lb.Option(value="first_sub_radio_answer")]),
                  ]),
          lb.Tool(tool=lb.Tool.Type.POLYGON, name="polygon"),
          lb.Tool(tool=lb.Tool.Type.RASTER_SEGMENTATION, name="mask"),
        	lb.Tool(tool=lb.Tool.Type.RASTER_SEGMENTATION,
                  name="mask_with_text_subclass",
                  classifications=[
                      lb.Classification(
                          class_type=lb.Classification.Type.TEXT,
                          name="sub_free_text")
                  ]),
          lb.Tool(tool=lb.Tool.Type.POINT, name="point"),
          lb.Tool(tool=lb.Tool.Type.LINE, name="polyline"),
          lb.Tool(tool=lb.Tool.Type.RELATIONSHIP, name="relationship")
      ])

  ontology = client.create_ontology("Image Annotation Import Demo Ontology",
                                    ontology_builder.asdict(),
                                    media_type=lb.MediaType.Image)
  ```
</CodeGroup>

### Step 3: Set Up a Labeling Project

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # create a project and configure the ontology
  project = client.create_project(name="Image Annotation Import Demo",
                                  media_type=lb.MediaType.Image)

  project.connect_ontology(ontology)
  ```
</CodeGroup>

### Step 4: Send Data Rows to Project

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  batch = project.create_batch(
      "image-demo-batch",  # each batch in a project must have a unique name
      global_keys=[global_key], # paginated collection of data row objects, list of data row ids or global keys
      priority=1  # priority between 1(highest) - 5(lowest)
  )

  print(f"Batch: {batch}")
  ```
</CodeGroup>

### Step 5: Create annotation payloads

For help understanding annotation payloads, see [overview](#overview). To declare payloads, you can use Python annotation types (*preferred*) or NDJSON objects. For annotations that you want to import as ground truth labels, you can also specify [benchmarks](/horizon/guides/benchmark) using the `is_benchmark_reference` flag.

These examples demonstrate each format and how to compose annotations into labels attached to data rows.

<CodeGroup>
  ```python Python annotation payload theme={"theme":"css-variables"}
  # create a Label
  label = []
  annotations = [
      radio_annotation,
      nested_radio_annotation,
      checklist_annotation,
      nested_checklist_annotation,
      text_annotation,
      bbox_annotation,
      bbox_with_radio_subclass_annotation,
      polygon_annotation,
      mask_annotation,
    	mask_with_text_subclass_annotation,
      point_annotation,
      polyline_annotation,
      bbox_source,
      bbox_target,
      relationship,
  ] + cp_mask
  label.append(
      lb_types.Label(data={"global_key" : global_key },
                     annotations=annotations
                     # Optional: set the label as a benchmark
                     # Only supported for groud truth imports
                     is_benchmark_reference = True
                    )
  )
  ```

  ```python NDJSON payload theme={"theme":"css-variables"}
  label_ndjson = []
  annotations = [
      radio_annotation_ndjson,
      nested_radio_annotation_ndjson,
      nested_checklist_annotation_ndjson,
      checklist_annotation_ndjson,
      text_annotation_ndjson,
      bbox_annotation_ndjson,
      bbox_with_radio_subclass_ndjson,
      polygon_annotation_ndjson,
      mask_annotation_ndjson,
    	mask_with_text_subclass_ndjson,
      point_annotation_ndjson,
      polyline_annotation_ndjson,
      bbox_source_ndjson,
      bbox_target_ndjson,
      relationship_ndjson,  ## Only supported for MAL imports
  ] + cp_mask_ndjson
  for annotation in annotations:
      annotation.update({
          "dataRow": {
              "globalKey": global_key
          },
      })
      label_ndjson.append(annotation)
  ```
</CodeGroup>

### Step 6: Import annotation payload

For prelabeled (model-assisted labeling) scenarios, pass your payload as the value of the `predictions` parameter. For ground truths, pass the payload to the `labels` parameter.

<Warning>
  ### Warning

  Relationship annotations are not supported for ground truth import jobs.
</Warning>

#### Option A: Upload as [prelabels (model assisted labeling)](/horizon/guides/model-assisted-labeling)

This option is helpful for speeding up the initial labeling process and reducing the manual labeling workload for high-volume datasets.

<CodeGroup>
  ```python MAL import theme={"theme":"css-variables"}
  # Upload MAL label for this data row in project
  upload_job = lb.MALPredictionImport.create_from_objects(
      client = client,
      project_id = project.uid,
      name="mal_job"+str(uuid.uuid4()),
      predictions=label
  )

  print(f"Errors: {upload_job.errors}", )
  print(f"Status of uploads: {upload_job.statuses}")
  ```
</CodeGroup>

#### Option B: Upload to a labeling project as [ground truth](/horizon/guides/import-ground-truth)

This option is helpful for loading high-confidence labels from another platform or previous projects that just need review rather than manual labeling effort.

<CodeGroup>
  ```python Label import theme={"theme":"css-variables"}
  # Upload label for this data row in project
  upload_job = lb.LabelImport.create_from_objects(
      client = client,
      project_id = project.uid,
      name="label_import_job"+str(uuid.uuid4()),
      labels=label
  )

  print(f"Errors: {upload_job.errors}", )
  print(f"Status of uploads: {upload_job.statuses}")
  ```
</CodeGroup>


# Import image data
Source: https://docs.labelbox.com/horizon/developer-guides/import-image-data

How to import image data and sample import formats.

## Specifications

Supported file formats: JPG, PNG, and BMP Import methods:

* IAM Delegated Access
* Signed URLs (`https` URLs only)
* Direct upload of local files (256 MB max file size) Note: Direct upload currently does not support adding additional metadata and attachments, see below Python example.

<Danger>
  **Image size limit**

  Please note that Labelbox does not recommend labeling images larger than 9000px by 9000px. Labeling images of this size may cause annotations to be irretrievable.
</Danger>

When importing cloud-hosted image data to Labelbox, your JSON file must include the following information for each image.

| Parameter | Required | Description |
| - | - | - |
| `row_data` | Yes | `https` path to a cloud-hosted image. For IAM Delegated Access, this URL must be in [virtual-hosted-style format](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#virtual-hosted-style-access). For older regions, your S3 bucket may be in `https://<bucket-name>.s3.<region>.amazonaws.com/<key>` format. If your object URLs are formatted this way, make sure they are in the virtual-hosted-style format before importing. |
| `global_key` | No | Unique user-generated file name or ID for the file. [Global keys](/horizon/developer-guides/data-row-global-keys) are enforced to be unique in your org. Data rows will not be imported if its global keys are duplicated to existing data rows. |
| `media_type` | No | `"IMAGE"` (optional media type to provide better validation and error messaging) |
| `metadata_fields` | No | See [Metadata](/horizon/guides/datarow-metadata). |
| `attachments` | No | See [Attachments](/horizon/guides/label-data) and [Asset overlays](/horizon/guides/label-data). |

## Import format

This import format can be used for both importing data assets via [Uploading a file](/horizon/guides/datasets-datarows##option-2-upload-a-json-file-with-delegated-access-urls), and for importing data assets via Python SDK.

<CodeGroup>
  ```json Delegated Access URL theme={"theme":"css-variables"}
  [
    {
      "row_data": "https://lb-test-data.s3.us-west-1.amazonaws.com/image-samples/sample-image-1.jpg",
      "global_key": "https://lb-test-data.s3.us-west-1.amazonaws.com/image-samples/sample-image-1.jpg",
      "media_type": "IMAGE",
      "metadata_fields": [{"name": "<metadata_field_name>", "value": "tag_string"}],
      "attachments": [{"type": "IMAGE_OVERLAY", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/rgb.jpg" }]
    },
    {
      "row_data": "https://lb-test-data.s3.us-west-1.amazonaws.com/image-samples/sample-image-1.jpg",
      "global_key": "https://lb-test-data.s3.us-west-1.amazonaws.com/image-samples/sample-image-2.jpg",
      "media_type": "IMAGE",
      "metadata_fields": [{"name": "<metadata_field_name>", "value": "tag_string"}],
      "attachments": [{"type": "TEXT_URL", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt"}]
    }
  ]
  ```

  ```json Standard URL theme={"theme":"css-variables"}
  [
    {
      "row_data": "https://storage.googleapis.com/labelbox-datasets/image_sample_data/image-sample-1.jpg",
      "global_key": "https://storage.googleapis.com/labelbox-datasets/image_sample_data/image-sample-1.jpg",
      "media_type": "IMAGE",
      "metadata_fields": [{"schema_id": "cko8s9r5v0001h2dk9elqdidh", "value": "tag_string"}],
      "attachments": [{"type": "IMAGE_OVERLAY", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/rgb.jpg", "name": "RGB" }]
    },
    {
      "row_data": "https://storage.googleapis.com/labelbox-datasets/image_sample_data/image-sample-2.jpg",
      "global_key": "https://storage.googleapis.com/labelbox-datasets/image_sample_data/image-sample-2.jpg",
      "media_type": "IMAGE",
      "metadata_fields": [{"schema_id": "cko8s9r5v0001h2dk9elqdidh", "value": "tag_string"}],
      "attachments": [{"type": "TEXT_URL", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt"}]
    }
  ]
  ```
</CodeGroup>

## Python example

<CodeGroup>
  ```python single data row import theme={"theme":"css-variables"}
  from labelbox import Client
  from uuid import uuid4 ## to generate unique IDs
  import datetime

  client = Client(api_key="<YOUR_API_KEY>")
  metadata_ontology = client.get_data_row_metadata_ontology()

  dataset = client.create_dataset(name="Single import example - Image")

  asset = {
    "row_data": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/basic.jpg",
    "global_key": str(uuid4()),
    "media_type": "IMAGE",
    "metadata_fields": [{"name": "captureDateTime", "value": datetime.datetime.utcnow()},
    										{"name": "tag", "value": "tag_string"},
    										{"name": "split", "value": "train"}],
    "attachments": [{"type": "IMAGE_OVERLAY", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/rgb.jpg"},
                    {"type": "IMAGE_OVERLAY", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/cir.jpg"},
                    {"type": "IMAGE_OVERLAY", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/weeds.jpg"},
                    {"type": "RAW_TEXT", "value": "IOWA, Zone 2232, June 2022 [Text string]"},
                    {"type": "TEXT_URL", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt"},
                    {"type": "IMAGE", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/disease_attachment.jpeg"},
                    {"type": "VIDEO", "value":  "https://storage.googleapis.com/labelbox-sample-datasets/Docs/drone_video.mp4"},
                    {"type": "HTML", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/windy.html"}],
  }

  dataset.create_data_row(**asset)

  # equivalent method if using create_data_rows bulk endpoint
  dataset.create_data_rows([asset])
  ```

  ```python bulk data rows import theme={"theme":"css-variables"}
  from labelbox import Client
  from uuid import uuid4 ## to generate unique IDs
  import datetime

  client = Client(api_key="<YOUR_API_KEY>")

  dataset = client.create_dataset(name="Bulk import example - Image")

  assets = [
    {
      "row_data": "https://lb-test-data.s3.us-west-1.amazonaws.com/image-samples/sample-image-1.jpg",
      "global_key": "https://lb-test-data.s3.us-west-1.amazonaws.com/image-samples/sample-image-1.jpg" + str(uuid4()),
      "media_type": "IMAGE",
      "metadata_fields": [{"name": "tag", "value": "tag_string"}],
      "attachments": [{"type": "IMAGE_OVERLAY", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/rgb.jpg", "name": "RGB" }],
    },
    {
      "row_data": "https://lb-test-data.s3.us-west-1.amazonaws.com/image-samples/sample-image-1.jpg",
      "global_key": "https://lb-test-data.s3.us-west-1.amazonaws.com/image-samples/sample-image-2.jpg-1"+ str(uuid4()),
      "media_type": "IMAGE",
      "metadata_fields": [{"name": "tag", "value": "tag_string"}],
      "attachments": [{"type": "TEXT_URL", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt"}]
    }
  ]

  task = dataset.create_data_rows(assets)
  task.wait_till_done()
  print(task.errors)
  ```

  ```python local files theme={"theme":"css-variables"}
  local_file_paths = ['path/to/local/file1', 'path/to/local/file1'] # limit: 15k files

  new_dataset = client.create_dataset(name = "Local files upload")

  try:
      task = new_dataset.create_data_rows(local_file_paths)
      task.wait_till_done()
  except Exception as err:
      print(f'Error while creating labelbox dataset -  Error: {err}')
  ```
</CodeGroup>


# Import LLM response evaluations annotations
Source: https://docs.labelbox.com/horizon/developer-guides/import-llm-response-evaluation-annotations

Developer guide for importing LLM response evaluations annotations and sample import formats.

<CardGroup>
  <Card title="Open In Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/annotation_import/conversational_LLM.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/annotation_import/conversational_LLM.ipynb" />
</CardGroup>

## Overview

To import annotations in Labelbox, you need to create an annotations payload. In this section, we provide this payload for every supported annotation type.

### Annotation payload types

Labelbox supports two formats for the annotations payload:

* Python annotation types (recommended)

  * Provides a seamless transition between third-party platforms, machine learning pipelines, and Labelbox.
  * Allows you to build annotations locally with local file paths, numpy arrays, or URLs
  * Easily convert Python Annotation Type format to NDJSON format to quickly import annotations to Labelbox
  * Supports one-level nested classification (radio, checklist, or free-form text) under a tool or classification annotation.

* JSON

  * Skips formatting annotation payload in the Labelbox Python annotation type
  * Supports any levels of nested classification (radio, checklist, or free-form text) under a tool or classification annotation.

### Label import types

Labelbox additionally supports two types of label imports:

* [Model-assisted labeling (MAL)](/horizon/guides/model-assisted-labeling)
  * This workflow allows you to import computer-generated predictions (or simply annotations created outside of Labelbox) as pre-labels on an asset.
* [Ground truth](https://www.google.com/url?q=https%3A%2F%2Fdocs.labelbox.com%2Fdocs%2Fimport-ground-truth)
  * This workflow functionality allows you to bulk import your ground truth annotations from an external or third-party labeling system into Labelbox *Annotate*. Using the label import API to import external data is a useful way to consolidate and migrate all annotations into Labelbox as a single source of truth.

## Supported annotations

The following annotations are supported for a LLM human preference data row:

* Radio
* Checklist
* Free-form text
* Entity

<Info>
  ### Annotations can be message or global based

  Classification annotations can be supported both message and global based while tool annotations are supported only message specific. Removing the message\_id key inside a conversation text classification annotation results in the annotation becoming global.
</Info>

## Classifications

### Radio (single-choice)

<CodeGroup>
  ```python Python Annotation theme={"theme":"css-variables"}
  radio_annotation = lb_types.ClassificationAnnotation(
      name="Choose the best response",
      value=lb_types.Radio(answer=lb_types.ClassificationAnswer(
          name="Response B")))
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  radio_annotation_ndjson = {
      "name": "radio_convo",
      "answer": {
          "name": "first_radio_answer"
      }
  }
  ```
</CodeGroup>

### Checklist (multi-choice)

<CodeGroup>
  ```python Python Annotation theme={"theme":"css-variables"}
  checklist_annotation= lb_types.ClassificationAnnotation(
    name="checklist_convo", # must match your ontology feature"s name
    value=lb_types.Checklist(
        answer = [
          lb_types.ClassificationAnswer(
              name = "first_checklist_answer"
          ),
          lb_types.ClassificationAnswer(
              name = "second_checklist_answer"
          )
        ]
      ),
    message_id="message-1" # Message specific annotation
    )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  checklist_annotation_ndjson = {
      "name": "checklist_convo",
      "answers": [
          {"name": "first_checklist_answer"},
          {"name": "second_checklist_answer"}
      ]
  }
  ```
</CodeGroup>

### Free-form text

<CodeGroup>
  ```python Python Annotation theme={"theme":"css-variables"}
  text_annotation = lb_types.ClassificationAnnotation(
      name="Provide a reason for your choice",
      value=lb_types.Text(answer="the answer to the text questions right here")
  )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  text_annotation_ndjson = {
      "name": "text_convo",
      "answer": "the answer to the text questions right here",
  }
  ```
</CodeGroup>

### Nested classification

<CodeGroup>
  ```python Python Annotation theme={"theme":"css-variables"}
  nested_radio_annotation = lb_types.ClassificationAnnotation(
    name="nested_radio_question",
    value=lb_types.Radio(
      answer=lb_types.ClassificationAnswer(
        name="first_radio_answer",
        classifications=[
          lb_types.ClassificationAnnotation(
            name="sub_radio_question",
            value=lb_types.Radio(
              answer=lb_types.ClassificationAnswer(
                name="first_sub_radio_answer"
              )
            )
          )
        ]
      )
    )
  )

  nested_checklist_annotation = lb_types.ClassificationAnnotation(
    name="nested_checklist_question",
    message_id="message-1",
    value=lb_types.Checklist(
      answer=[lb_types.ClassificationAnswer(
        name="first_checklist_answer",
        classifications=[
          lb_types.ClassificationAnnotation(
            name="sub_checklist_question",
            value=lb_types.Checklist(
              answer=[lb_types.ClassificationAnswer(
              name="first_sub_checklist_answer"
            )]
          ))
        ]
      )]
    )
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  nested_radio_annotation_ndjson = {
    "name": "nested_radio_question",
    "answer": {
        "name": "first_radio_answer",
        "classifications": [{
            "name":"sub_radio_question",
            "answer": { "name" : "first_sub_radio_answer"}
          }]
      }
  }

  nested_checklist_annotation_ndjson = {
    "name": "nested_checklist_question",
    "messageId": "message-1",
    "answer": [{
        "name": "first_checklist_answer",
        "classifications" : [
          {
            "name": "sub_checklist_question",
            "answer": {
              "name": "first_sub_checklist_answer",
            }
          }
        ]
    }]
  }
  ```
</CodeGroup>

## Tools

### Entity

<CodeGroup>
  ```python Python Annotation theme={"theme":"css-variables"}
  ner_annotation = lb_types.ObjectAnnotation(
      name="ner",
      value=lb_types.ConversationEntity(
          start=0,
          end=8,
          message_id="message-1"
      )
  )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  ner_annotation_ndjson = {
          "name": "ner",
          "location": {
              "start": 0,
              "end": 8
          },
          "messageId": "message-1"
      }
  ```
</CodeGroup>

### Tool with nested classification

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  tool_with_radio_subclass_annotation = lb_types.ObjectAnnotation(
      name=# Feature name,
      value=# Add tool annotation (lb_types."tool"),
      classifications=[
          lb_types.ClassificationAnnotation(
              name="sub_radio_question",
              value=lb_types.Radio(answer=lb_types.ClassificationAnswer(
                  name="first_sub_radio_answer")))
      ])
  ```

  ```python Entity Example theme={"theme":"css-variables"}
  entity_with_radio_subclass_annotation = lb_types.ObjectAnnotation(
      name="ner",
      value=lb_types.ConversationEntity(
          start=0,
          end=8,
          message_id="message-1"
      	)
  		),
      classifications=[
          lb_types.ClassificationAnnotation(
              name="sub_radio_question",
              value=lb_types.Radio(answer=lb_types.ClassificationAnswer(
                  name="first_sub_radio_answer")))
      ])
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  ner_with_radio_subclass_ndjson = {
      "name": "bbox_with_radio_subclass",
      "classifications": [{
          "name": "sub_radio_question",
          "answer": {
              "name": "first_sub_radio_answer"
          }
      }],
      "ner": {
          "location": {
              "start": 0,
              "end": 8
          },
          "messageId": "message-1"
      }
  }
  ```
</CodeGroup>

## Example: Import pre-labels or ground truths

The steps to import annotations as pre-labels (machine-assisted learning) are similar to those to import annotations as ground truth labels. However, they vary slightly, and we will describe the differences for each scenario.

### Before you start

The below imports are needed to use the code examples in this section.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  import labelbox as lb
  import uuid
  import labelbox.types as lb_types
  ```
</CodeGroup>

### Step 1: Import data rows

Data rows must first be uploaded to **Catalog** to attach annotations.

This example shows how to create a data row in **Catalog** by attaching it to a [dataset](/horizon/developer-guides/dataset) .

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  global_key = "pairwise_shooping_asset"

  # Upload data rows

  convo_data = {
    "row_data": "https://storage.googleapis.com/labelbox-datasets/conversational-sample-data/pairwise_shopping_2.json",
    "global_key": global_key
  }

  # Create a dataset

  dataset = client.create_dataset(name="pairwise_annotation_demo")

  # Create a datarow

  task = dataset.create_data_rows([convo_data])
  task.wait_till_done()
  print("Errors:",task.errors)
  print("Failed data rows:", task.failed_data_rows)
  ```

  ```json Sample pairwise_shopping_2.json theme={"theme":"css-variables"}
  {
    "type": "application/vnd.labelbox.conversational",
    "version": 1,
    "messages": [
      {
        "messageId": "message-0",
        "timestampUsec": 1530718491,
        "content": "Hi! How can I help?",
        "user": {
          "userId": "Bot 002",
          "name": "Bot"
        },
        "align": "left",
        "canLabel": false
      },
      {
        "messageId": "message-1",
        "timestampUsec": 1530718503,
        "content": "I just bought a vacuum cleaner from you guys like a week ago and it's already broken!!",
        "user": {
          "userId": "User 00686",
          "name": "User"
        },
        "align": "right",
        "canLabel": true
      }

    ],
    "modelOutputs": [
      {
        "title": "Response A",
        "content": "I'm really sorry to hear that your vacuum cleaner is not working as expected. We certainly understand how frustrating this can be, especially with a recent purchase. I assure you, we're here to help!\n\n To proceed with resolving this issue, could you please provide some details about the problem you're experiencing with the vacuum cleaner? Additionally, if you could share your order number or receipt details, that would be very helpful. We will definitely look into the available options for you, which may include a replacement or a refund, depending on the specific circumstances and our return policy.\n\n Your satisfaction is our priority, and we'll do our best to resolve this matter promptly and to your satisfaction.",
        "modelConfigName": "GPT-3.5 with temperature 0"
      },
      {
        "title": "Response B",
        "content": "I'm sorry about the issue with your vacuum cleaner. Please send us the order number or receipt details so we can quickly assist you with a replacement. Your satisfaction is important to us!",
        "modelConfigName": "Fine Tuned GPT-3.5 with demo data"
      }
    ]
  }
  ```
</CodeGroup>

### Step 2: Set up ontology

Your project ontology should support the tools and classifications required by your annotations. To ensure accurate schema feature mapping, the value used as the `name` parameter should match the value of the `name` field in your annotation.

For example, when we created an annotation above, we provided a name`annotation_name`. Now, when we set up our ontology, we must ensure that the name of our bounding box tool is also `anotations_name`. The same alignment must hold true for the other tools and classifications we create in our ontology.

This example shows how to create an ontology containing all supported [annotation types](#supported-annotations) .

<CodeGroup>
  ```python Python expandable theme={"theme":"css-variables"}
  ontology_builder = lb.OntologyBuilder(
    tools=[
      lb.Tool(tool=lb.Tool.Type.NER,name="ner"),
    ],
    classifications=[
      lb.Classification(
        class_type=lb.Classification.Type.RADIO,
        scope=lb.Classification.Scope.GLOBAL,
        name="Choose the best response",
        options=[lb.Option(value="Response A"), lb.Option(value="Response B"), lb.Option(value="Tie")]
      ),
      lb.Classification(
        class_type=lb.Classification.Type.TEXT,
        name="Provide a reason for your choice"
      ),
      lb.Classification(
        class_type=lb.Classification.Type.CHECKLIST,
        scope=lb.Classification.Scope.INDEX,
        name="checklist_convo",
        options=[
          lb.Option(value="first_checklist_answer"),
          lb.Option(value="second_checklist_answer")
        ]
      ),
      lb.Classification(
        class_type=lb.Classification.Type.CHECKLIST,
        name="nested_checklist_question",
        scope = lb.Classification.Scope.INDEX,
        options=[
            lb.Option("first_checklist_answer",
              options=[
                lb.Classification(
                    class_type=lb.Classification.Type.CHECKLIST,
                    name="sub_checklist_question",
                    options=[lb.Option("first_sub_checklist_answer")]
                )
            ])
        ]
      ),
      lb.Classification(
          class_type=lb.Classification.Type.RADIO,
          name="nested_radio_question",
          scope = lb.Classification.Scope.GLOBAL,
          options=[
              lb.Option("first_radio_answer",
                  options=[
                      lb.Classification(
                          class_type=lb.Classification.Type.RADIO,
                          name="sub_radio_question",
                          options=[lb.Option("first_sub_radio_answer")]
                      )
                  ])
            ]
      )
    ]
  )

  ontology = client.create_ontology("Pairwise comparison ontology", ontology_builder.asdict(), media_type=lb.MediaType.Conversational)
  ```
</CodeGroup>

### Step 3: Set Up a Labeling Project

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Create Labelbox project
  project = client.create_project(name="Conversational Text Annotation Import Demo (Pairwise comparison)",
                                      media_type=lb.MediaType.Conversational)

  # Setup your ontology
  project.connect_ontology(ontology) # Connect the ontology to your project
  ```
</CodeGroup>

### Step 4: Send Data Rows to Project

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Create a batch to send to your project
  batch = project.create_batch(
    "first-batch-convo-demo", # Each batch in a project must have a unique name
    global_keys=[global_key], # Paginated collection of data row objects, list of data row ids or global keys
    priority=5 # priority between 1(Highest) - 5(lowest)
  )

  print("Batch: ", batch)

  ```
</CodeGroup>

### Step 5: Create annotation payloads

For help understanding annotation payloads, see [overview](#overview). To declare payloads, you can use Python annotation types (*preferred*) or NDJSON objects.

These examples demonstrate each format and how to compose annotations into labels attached to data rows.

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  label = []
  label.append(
    lb_types.Label(
      data={"global_key" : global_key },
      annotations=[
        ner_annotation,
        text_annotation,
        checklist_annotation,
        radio_annotation,
        nested_radio_annotation,
        nested_checklist_annotation
      ]
    )
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  label_ndjson = []
  for annotations in [
      ner_annotation_ndjson,
      text_annotation_ndjson,
      checklist_annotation_ndjson,
      radio_annotation_ndjson,
      nested_checklist_annotation_ndjson,
      nested_radio_annotation_ndjson
      ]:
    annotations.update({
        "dataRow": {
            "globalKey": global_key
        }
    })
    label_ndjson.append(annotations)
  ```
</CodeGroup>

### Step 6: Import annotation payload

For prelabeled (model-assisted labeling) scenarios, pass your payload as the value of the `predictions` parameter. For ground truths, pass the payload to the `labels` parameter.

#### Option A: Upload as [prelabels (model assisted labeling)](/horizon/guides/model-assisted-labeling)

This option is helpful for speeding up the initial labeling process and reducing the manual labeling workload for high-volume datasets.

<CodeGroup>
  ```python MAL import theme={"theme":"css-variables"}
  # Upload MAL label for this data row in project
  upload_job = lb.MALPredictionImport.create_from_objects(
      client = client,
      project_id = project.uid,
      name="mal_job"+str(uuid.uuid4()),
      predictions=label
  )

  print(f"Errors: {upload_job.errors}", )
  print(f"Status of uploads: {upload_job.statuses}")
  ```
</CodeGroup>

#### Option B: Upload to a labeling project as [ground truth](/horizon/guides/import-ground-truth)

This option is helpful for loading high-confidence labels from another platform or previous projects that just need review rather than manual labeling effort.

<CodeGroup>
  ```python Label import theme={"theme":"css-variables"}
  # Upload label for this data row in project
  upload_job = lb.LabelImport.create_from_objects(
      client = client,
      project_id = project.uid,
      name="label_import_job"+str(uuid.uuid4()),
      labels=label
  )

  print(f"Errors: {upload_job.errors}", )
  print(f"Status of uploads: {upload_job.statuses}")
  ```
</CodeGroup>


# Import multimodal chat annotations
Source: https://docs.labelbox.com/horizon/developer-guides/import-multimodal-chat-annotations

Learn how to import multimodal chat evaluation annotations and sample import formats.

<CardGroup>
  <Card title="Open In Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/annotation_import/offline_multimodal_chat_evaluation.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/tree/main/annotation_import/offline_multimodal_chat_evaluation.ipynb" />
</CardGroup>

## Overview

To import annotations in Labelbox, you need to create an annotations payload. In this section, we provide this payload for every supported annotation type.

### Annotation payload types

Labelbox supports two formats for the annotations payload:

* Python annotation types (recommended)
  * Provides a seamless transition between third-party platforms, machine learning pipelines, and Labelbox.
  * Allows you to build annotations locally with local file paths, numpy arrays, or URLs
  * Easily convert Python Annotation Type format to NDJSON format to quickly import annotations to Labelbox
  * Supports one-level nested classification (radio, checklist, or free-form text) under a tool or classification annotation.
* JSON
  * Skips formatting annotation payload in the Labelbox Python annotation type
  * Supports any levels of nested classification (radio, checklist, or free-form text) under a tool or classification annotation.

### Label import types

Labelbox supports two types of label imports:

* [Model-assisted labeling (MAL)](/horizon/guides/model-assisted-labeling)
  * This workflow allows you to import computer-generated predictions (or simply annotations created outside of Labelbox) as pre-labels on an asset.
* [Ground truth](/horizon/guides/import-ground-truth)
  * This workflow functionality allows you to bulk import your ground truth annotations from an external or third-party labeling system into Labelbox *Annotate*. Using the label import API to import external data is a useful way to consolidate and migrate all annotations into Labelbox as a single source of truth.

## Supported annotations

The following annotations are supported for a LLM human preference data row:

* `Tool`
  * Message ranking
  * Single message selection
  * Multiple message selection
* `Classification`
  * Radio (single-choice)
  * Checklist (multi-choice)
  * Free-form text

<Info>
  ### Message and global-based annotations

  Radio and free-form text annotations can be both message and global based. To make a message-based annotation global, remove the `message_id` key inside the annotation.
</Info>

## Tools

### Message ranking

<CodeGroup>
  ```python Python Annotation theme={"theme":"css-variables"}
  message_ranking_annotation = lb_types.MessageEvaluationTaskAnnotation(
      name="Message ranking",
      value=MessageRankingTask(
          parent_message_id="message-0",
          ranked_messages=[
              OrderedMessageInfo(
                  message_id="message-1",
                  model_config_name="model-config-1",
                  order=1,
              ),
              OrderedMessageInfo(
                  message_id="message-2",
                  model_config_name="model-config-2",
                  order=2,
              ),
          ],
      ),
  )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  message_ranking_annotation_ndjson = {
      "name": "model output multi ranking",
      "messageEvaluationTask": {
          "format": "message-ranking",
          "data": {
              "parentMessageId": "message-0",
              "rankedMessages": [
                  {
                      "messageId": "message-1",
                      "modelConfigName": "model-config-1",
                      "order": 2,
                  },
                  {
                      "messageId": "message-2",
                      "modelConfigName": "model-config-2",
                      "order": 1,
                  },
              ],
          },
      },
  }
  ```
</CodeGroup>

### Single message selection

<CodeGroup>
  ```python Python Annotation theme={"theme":"css-variables"}
  single_message_selection_annotation = lb_types.MessageEvaluationTaskAnnotation(
      name="Single message selection",
      value=MessageSingleSelectionTask(
          message_id="message-1",
          parent_message_id="message-0",
          model_config_name="model-config-1",
      ),
  )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  single_message_selection_annotation_ndjson = {
      "name": "Single message selection",
      "messageEvaluationTask": {
          "format": "message-single-selection",
          "data": {
              "messageId": "message-1",
              "parentMessageId": "message-0",
              "modelConfigName": "model-config-1",
          },
      },
  }
  ```
</CodeGroup>

### Multiple message selection

<CodeGroup>
  ```python Python Annotation theme={"theme":"css-variables"}
  multiple_message_selection_annotation = lb_types.MessageEvaluationTaskAnnotation(
      name="Multi message selection",
      value=MessageMultiSelectionTask(
          parent_message_id="message-0",
          selected_messages=[
              MessageInfo(
                  message_id="message-1",
                  model_config_name="model-config-1",
              ),
              MessageInfo(
                  message_id="message-2",
                  model_config_name="model-config-2",
              ),
          ],
      ),
  )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  multiple_message_selection_annotation_ndjson = {
      "name": "Multi message selection",
      "messageEvaluationTask": {
          "format": "message-multi-selection",
          "data": {
              "parentMessageId": "message-0",
              "selectedMessages": [
                  {
                      "messageId": "message-1",
                      "modelConfigName": "model-config-1",
                  },
                  {
                      "messageId": "message-2",
                      "modelConfigName": "model-config-2",
                  },
              ],
          },
      },
  }
  ```
</CodeGroup>

## Classifications

### Radio

<CodeGroup>
  ```python Python Annotation theme={"theme":"css-variables"}
  radio_annotation = lb_types.ClassificationAnnotation(
      name="Choose the best response",
      value=lb_types.Radio(answer=lb_types.ClassificationAnswer(
          name="Response B")))
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  radio_annotation_ndjson = {
      "name": "radio_convo",
      "answer": {
          "name": "first_radio_answer"
      }
  }
  ```
</CodeGroup>

### Checklist

<CodeGroup>
  ```python Python Annotation theme={"theme":"css-variables"}
  checklist_annotation= lb_types.ClassificationAnnotation(
    name="checklist_convo", # must match your ontology feature"s name
    value=lb_types.Checklist(
        answer = [
          lb_types.ClassificationAnswer(
              name = "first_checklist_answer"
          ),
          lb_types.ClassificationAnswer(
              name = "second_checklist_answer"
          )
        ]
      ),
    message_id="message-1" # Message specific annotation
   )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  checklist_annotation_ndjson = {
      "name": "checklist_convo",
      "answers": [
          {"name": "first_checklist_answer"},
          {"name": "second_checklist_answer"}
      ]
  }
  ```
</CodeGroup>

### Free-form text

<CodeGroup>
  ```python Python Annotation theme={"theme":"css-variables"}
  text_annotation = lb_types.ClassificationAnnotation(
      name="Provide a reason for your choice",
      value=lb_types.Text(answer="the answer to the text questions right here")
  )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  text_annotation_ndjson = {
      "name": "text_convo",
      "answer": "the answer to the text questions right here",
  }
  ```
</CodeGroup>

## Example: Import pre-labels or ground truths

The steps to import annotations as pre-labels (machine-assisted learning) are similar to the steps to import annotations as ground truth labels, and we will describe the slight differences for each scenario.

### Before you start

The below imports are needed to use the code examples in this section.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  import labelbox as lb
  import uuid
  import labelbox.types as lb_types

  from labelbox.types import (
  Label,
  MessageEvaluationTaskAnnotation,
  MessageInfo,
  MessageMultiSelectionTask,
  MessageRankingTask,
  MessageSingleSelectionTask,
  OrderedMessageInfo,
  )
  ```
</CodeGroup>

### Step 1: Import data rows

You need to import data rows to **Catalog** to attach annotations.

This example shows how to create a data row in **Catalog** by attaching it to a [dataset](/horizon/developer-guides/dataset) .

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  mmc_asset = "https://storage.googleapis.com/labelbox-datasets/conversational_model_evaluation_sample/offline-model-chat-evaluation.json"
  global_key = "offline-multimodal_chat_evaluation"

  # Upload data rows
  convo_data = {
    "row_data": mmc_asset ,
    "global_key": global_key
  }

  # Create a dataset
  dataset = client.create_dataset(name="offline-multimodal_chat_evaluation_demo")
  # Create a datarow
  task = dataset.create_data_rows([convo_data])
  task.wait_till_done()
  print("Errors:",task.errors)
  print("Failed data rows:", task.failed_data_rows)
  ```

  ```json Sample import theme={"theme":"css-variables"}
  {
    "type": "application/vnd.labelbox.conversational.model-chat-evaluation",
    "version": 2,
    "actors": {
      "clxfzhair00013b6vlatdodq2": {
        "role": "human",
        "metadata": {
          "name": "User"
        }
      },
      "clxfzhbwl00033b6vbq14ufri": {
        "role": "model",
        "metadata": {
          "modelConfigName": "GPT 4 with temperature 0.7"
        }
      },
      "clxfzhbwl00043b6vgajlwovx": {
        "role": "model",
        "metadata": {
          "modelConfigName": "GPT 5"
        }
      }
    },
    "messages": {
      "clxfzhair00023b6vb607bqo6": {
        "actorId": "clxfzhair00013b6vlatdodq2",
        "content": [
          {
            "type": "text",
            "content": "The best restaurant in Wroclaw"
          }
        ],
        "childMessageIds": [
          "clxfzi3r400053b6vm5udpdgo",
          "clxfzi3r400063b6vuaeajylo"
        ]
      },
      "clxfzi3r400053b6vm5udpdgo": {
        "actorId": "clxfzhbwl00033b6vbq14ufri",
        "content": [
          {
            "type": "text",
            "content": "It's impossible to definitively say which restaurant in Wroclaw is \"the best\" because it's subjective and depends on individual preferences. However, I can provide you with a list of some of the most popular and highly-rated restaurants in Wroclaw, representing a variety of cuisines and dining experiences:\n\n**Fine Dining:**\n\n* **JaDka:** Offers modern Polish cuisine with a focus on local and seasonal ingredients. \n* **Szara Ges:** Located in the heart of the city, this restaurant is known for its elegant atmosphere and refined Polish and European dishes. \n* **Młoda Polska Bistro & Pianino:** Serves traditional Polish cuisine with a modern twist in a cozy and stylish setting.\n\n**Casual Dining:**\n\n* **Bernard:** A popular spot for traditional Polish food, known for its generous portions and affordable prices. \n* **Konspira:** Offers a unique dining experience in a former safe house from the communist era, serving Polish and European cuisine. \n* **U Gruzina:** A Georgian restaurant with delicious and authentic dishes, including khachapuri (cheese-filled bread) and khinkali (dumplings).\n\n**International Cuisine:**\n\n* **Dinette:** Serves up delicious and creative breakfast and brunch dishes all day long.\n* **Pizza Pany:** A great option for Neapolitan-style pizza with fresh ingredients and a lively atmosphere. \n* **Curry House:** Offers a wide variety of authentic Indian curries and dishes. \n\n**Vegetarian/Vegan:**\n\n* **Vega:** A vegetarian restaurant with a diverse menu featuring Polish and international dishes. \n* **Zielony Szpinak:** Offers vegan versions of traditional Polish dishes as well as international options. \n\n**For a truly unique experience:**\n\n* **Wrocławskie Podwale:** Dine in a medieval cellar with vaulted ceilings and enjoy traditional Polish cuisine. \n* **Stary Młyn:** Located in a former mill, this restaurant offers a romantic atmosphere and serves Polish and European dishes. \n\nTo narrow down your search, consider what kind of cuisine you're in the mood for, your budget, and the occasion. I recommend checking online reviews and making a reservation, especially if you're dining during peak hours. \n"
          }
        ],
        "childMessageIds": ["clxfznjb800073b6v43ppx9ca"]
      },
      "clxfzi3r400063b6vuaeajylo": {
        "actorId": "clxfzhbwl00043b6vgajlwovx",
        "content": [
          {
            "type": "text",
            "content": "There are several highly-rated restaurants in Wroclaw, but one of the best is \"Szynkarnia\". It's known for its delicious Polish cuisine, cozy atmosphere, and excellent service. Other top contenders include \"The White Dog\" known for its modern European cuisine, and \"Restauracja JaDka\" for those who want to try traditional Polish dishes."
          }
        ],
        "childMessageIds": ["clxfznjb800073b6v43ppx9ca"]
      },
      "clxfznjb800073b6v43ppx9ca": {
        "actorId": "clxfzhair00013b6vlatdodq2",
        "content": [
          {
            "type": "text",
            "content": "Tell me more"
          }
        ],
        "childMessageIds": [
          "clxfzocbm00083b6v8vczsept",
          "clxfzocbm00093b6vx4ndisub"
        ]
      },
      "clxfzocbm00083b6v8vczsept": {
        "actorId": "clxfzhbwl00033b6vbq14ufri",
        "content": [
          {
            "type": "text",
            "content": "Alright, tell me more about what you're looking for in a restaurant! To give you a more personalized recommendation, I need some more information.  Could you tell me:\n\n* **What kind of food are you craving?** (e.g., Italian, Polish, Asian, seafood, etc.) \n* **What's your budget?** (e.g., fine dining, mid-range, budget-friendly)\n* **What's the occasion?** (e.g., romantic dinner, family meal, night out with friends)\n* **What kind of atmosphere are you looking for?** (e.g., lively, cozy, romantic, trendy)\n* **Are there any dietary restrictions or preferences I should know about?** (e.g., vegetarian, vegan, gluten-free)\n\nThe more information you can give me, the better I can tailor my recommendation to your specific needs and preferences. \n"
          }
        ],
        "childMessageIds": []
      },
      "clxfzocbm00093b6vx4ndisub": {
        "actorId": "clxfzhbwl00043b6vgajlwovx",
        "content": [
          {
            "type": "text",
            "content": "Sure, here's more information about these restaurants:\n\n1. Szynkarnia: This is a traditional Polish restaurant located in the heart of Wroclaw. It offers a variety of Polish dishes, including pierogi (dumplings), bigos (hunter's stew), and kielbasa (sausage). The restaurant is also known for its wide selection of local beers. The interior is rustic and cozy, with wooden furniture and warm lighting.\n\n2. The White Dog: This restaurant offers a modern take on European cuisine. The menu changes seasonally to take advantage of fresh, local ingredients. The White Dog also has an extensive wine list and a stylish, contemporary interior. It's a great place for a romantic dinner or a special occasion.\n\n3. Restauracja JaDka: This restaurant is located in a beautiful, historic building in Wroclaw. It specializes in Polish cuisine, with dishes like duck with apples, beetroot soup, and a variety of pierogi. The interior is elegant and atmospheric, with exposed brick walls and vintage furniture.\n\nRemember, it's always a good idea to make a reservation in advance, as these popular spots can fill up quickly."
          }
        ],
        "childMessageIds": []
      }
    },
    "rootMessageIds": ["clxfzhair00023b6vb607bqo6"]
  }
  ```
</CodeGroup>

### Step 2: Set up ontology

Your project ontology needs to support the classifications required by your annotations. To ensure accurate schema feature mapping, the value used as the name parameter needs to match the value of the name field in your annotation.

For example, if you provide a name `annotation_name` for your message ranking annotation, you need to name the message ranking tool as `anotations_name` when setting up your ontology. The same alignment must hold true for the other tools and classifications that you create in the ontology.

This example shows how to create an ontology containing all supported [annotation types](#supported-annotations) .

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  ontology_builder = lb.OntologyBuilder(
      tools=[
          lb.Tool(
              tool=lb.Tool.Type.MESSAGE_SINGLE_SELECTION,
              name="Single message selection",
          ),
          lb.Tool(
              tool=lb.Tool.Type.MESSAGE_MULTI_SELECTION,
              name="Multi message selection",
          ),
          lb.Tool(tool=lb.Tool.Type.MESSAGE_RANKING, name="Message ranking"),
      ],
    classifications=[
      lb.Classification(
        class_type=lb.Classification.Type.RADIO,
        scope=lb.Classification.Scope.GLOBAL,
        name="Choose the best response",
        options=[lb.Option(value="Response A"), lb.Option(value="Response B"), lb.Option(value="Tie")]
      ),
      lb.Classification(
        class_type=lb.Classification.Type.TEXT,
        name="Provide a reason for your choice"
      ),
      lb.Classification(
        class_type=lb.Classification.Type.CHECKLIST,
        scope=lb.Classification.Scope.INDEX,
        name="checklist_convo",
        options=[
          lb.Option(value="first_checklist_answer"),
          lb.Option(value="second_checklist_answer")
        ]
      )
    ]
  )
  # Create ontology
  ontology = client.create_ontology(
      "MMC ontology",
      ontology_builder.asdict(),
      media_type=lb.MediaType.Conversational,
      ontology_kind=lb.OntologyKind.ModelEvaluation,
  )
  ```
</CodeGroup>

### Step 3: Set up a labeling project

Use the following code to create an offline multimodal evaluation project:

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Create Labelbox project
  project = client.create_offline_model_evaluation_project(
      name="Offline MMC Import Demo",
      description="<project_description>",  # optional
  )

  # Setup your ontology

  project.connect_ontology(ontology) # Connect the ontology to your project
  ```
</CodeGroup>

### Step 4: Send data rows to the project

Use the following code to send data rows to the project you just created:

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Create a batch to send to your project
  batch = project.create_batch(
    "first-batch-convo-demo", # Each batch in a project must have a unique name
    global_keys=[global_key], # Paginated collection of data row objects, list of data row ids or global keys
    priority=5 # priority between 1(Highest) - 5(lowest)
  )

  print("Batch: ", batch)
  ```
</CodeGroup>

### Step 5: Create annotation payloads

To declare payloads, you can use Python annotation types (*preferred*) or NDJSON objects. To understand annotation payloads, see [overview](#overview).

These examples demonstrate each format and how to compose annotations into labels attached to data rows.

<Info>
  ### Replace placeholder fields with actual values

  Replace `message_id` and `model_config_name` with the actual message ID and model configuration name before appending annotations.
</Info>

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  label = []
  label.append(
    lb_types.Label(
      data={"global_key" : global_key },
      annotations=[
        message_ranking_annotation,
        single_message_selection_annotation,
        multiple_message_selection_annotation,
        text_annotation,
        checklist_annotation,
        radio_annotation,
      ]
    )
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  label_ndjson = []
  for annotations in [
      message_ranking_annotation_ndjson,
      single_message_selection_annotation_ndjson,
      multiple_message_selection_annotation_ndjson,
      text_annotation_ndjson,
      checklist_annotation_ndjson,
      radio_annotation_ndjson,
      nested_checklist_annotation_ndjson,
      nested_radio_annotation_ndjson
      ]:
    annotations.update({
        "dataRow": {
            "globalKey": global_key
        }
    })
    label_ndjson.append(annotations)
  ```
</CodeGroup>

### Step 6: Import annotation payload

For prelabeled (model-assisted labeling) scenarios, pass your payload as the value of the `predictions` parameter. For ground truths, pass the payload to the `labels` parameter.

#### Option A: import as [prelabels (model assisted labeling)](/horizon/guides/model-assisted-labeling)

This option is helpful for speeding up the initial labeling process and reducing the manual labeling workload for high-volume datasets.

<CodeGroup>
  ```python MAL import theme={"theme":"css-variables"}
  # Upload MAL label for this data row in project
  upload_job = lb.MALPredictionImport.create_from_objects(
      client = client,
      project_id = project.uid,
      name="mal_job"+str(uuid.uuid4()),
      predictions=label
  )

  print(f"Errors: {upload_job.errors}", )
  print(f"Status of uploads: {upload_job.statuses}")
  ```
</CodeGroup>

#### Option B: Import as [ground truth](/horizon/guides/import-ground-truth) labels

This option is helpful for loading high-confidence labels from another platform or previous projects that just need review rather than manual labeling effort.

<CodeGroup>
  ```python Label import theme={"theme":"css-variables"}
  # Upload label for this data row in project
  upload_job = lb.LabelImport.create_from_objects(
      client = client,
      project_id = project.uid,
      name="label_import_job"+str(uuid.uuid4()),
      labels=label
  )

  print(f"Errors: {upload_job.errors}", )
  print(f"Status of uploads: {upload_job.statuses}")
  ```
</CodeGroup>


# Import multimodal chat data
Source: https://docs.labelbox.com/horizon/developer-guides/import-multimodal-chat-data

How to import multimodal chat data and sample import formats.

You need to set up a multimodal chat evaluation project before importing data. The two types of multimodal chat evaluation projects have different project creation methods and data row setups:

* For **offline multimodal chat evaluation** projects, use `create_offline_model_evaluation_project` and import data rows of existing conversations.

* For **live multimodal chat evaluation projects**, use `client.create_model_evaluation_project` and either:

  * (Recommended) Create data rows and send them to projects, like other types of projects.
  * Generate empty data rows upon project creation, which can't create data rows with attachments and metadata.

For a full walk-through of setting up a multimodal chat evaluation project, see [Multimodal chat evaluation](/horizon/guides/multimodal-chat-evaluation-editor).

## Set up live multimodal chat evaluation projects

Use `client.create_model_evaluation_project` to create a live multimodal chat evaluation project. This method takes the same parameters as the traditional `client.create_project`, with a few additional parameters specific to multimodal chat evaluation projects.

The `client.create_model_evaluation_project` methods require the following parameters:

* `name`: The name of your new project.

* `description`: An optional description of your project.

* `dataset_name` (optional): The name of the dataset where the generated data rows will be located. Include this parameter only if you want to create a new dataset.

* `dataset_id` (optional): The dataset ID of an existing Labelbox dataset. Include this parameter if you want to append it to an existing dataset.

* `data_row_count` (optional): The number of data row assets that will be generated and used with your project. Defaults to 100 if a `dataset_name` or `dataset_id` is included.

### Option A: Create and send data rows to projects

<CodeGroup>
  ```python Python expandable theme={"theme":"css-variables"}
  # Create the project
  project = client.create_model_evaluation_project(
      name="Example live multimodal chat project",
      description="<project_description>",  # optional
  )

  def make_data_rows(dataset_id=None):
      # If a dataset ID is provided, fetch the dataset using that ID.
      # Otherwise, create a new dataset with the specified name.
      if dataset_id:
          dataset = client.get_dataset(dataset_id)
      else:
          dataset = client.create_dataset(name="example live mmc dataset")

      # Helper function to generate a single data row
      def generate_data(ind):
          return {
              "row_data": {  # The chat data format
                  'type': 'application/vnd.labelbox.conversational.model-chat-evaluation',
                  'draft': True,
                  'rootMessageIds': [],
                  'actors': {},
                  'version': 2,
                  'messages': {}
              },
              "global_key": f"global_key_{dataset.uid}_{ind}",
              "metadata_fields": [{"name": "tag", "value": "val_tag"}],
              "attachments": [
                  {
                      "type": "IMAGE_OVERLAY",
                      "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/rgb.jpg"
                  }
              ]
          }

      # Generate a list of 100 data rows
      data_list = [generate_data(ind) for ind in range(100)]

      # Upload the generated data rows to the dataset
      task = dataset.create_data_rows(data_list)
      print("Processing task ", task.uid)  # Print the unique ID of the task
      task.wait_till_done()

      # Ensure that the task status is 'COMPLETE' to confirm success
      assert task.status == "COMPLETE"

      # Return the dataset object
      return dataset

  # Create a new data set. Alternatively, pass an existing dataset ID
  dataset = make_data_rows()

  # Retrieve the data row IDs from the dataset
  data_row_ids = [data_row.uid for data_row in dataset.data_rows()]

  # Send data rows to the project
  batch = project.create_batch(
      name="mmc-batch",  # each batch in a project must have a unique name
      data_rows=data_row_ids, # data row IDs to include in the batch
      priority=1  # priority between 1(highest) - 5(lowest)
  )

  print(f"Batch: {batch}")
  ```
</CodeGroup>

### Option B: Generate empty data rows

<Info>
  ### No metadata support

  Only use this option if your project doesn't require metadata attachments or embeddings for data rows.
</Info>

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Create the project and generate data rows
  project = client.create_model_evaluation_project(
      name="Example live multimodal chat project",
      description="<project_description>",  # optional
      dataset_name="Example live multimodal chat dataset",
      data_row_count=100,
  )

  # Connect the project to the created ontology
  project.connect_ontology(ontology)
  ```
</CodeGroup>

## Set up offline multimodal chat evaluation projects

Use `client.create_offline_model_evaluation_project` to create offline multimodal chat evaluation projects. This method uses the same parameters as `client.create_project` and adds validation to ensure the project is set up correctly.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  project = client.create_offline_model_evaluation_project(
      name="<project_name>",
      description="<project_description>",  # optional
  )
  ```
</CodeGroup>

After creating the project, you can import conversational version 2 data rows to the project. To learn how to import annotations, see [Import multimodal chat annotations](/horizon/developer-guides/import-multimodal-chat-annotations).

## Specifications

File format: chat data JSON in [conversation v2 format](#sample-conversation-v2-json)

Import methods:

* Local upload (maximum character count: 2,621,440)
* IAM Delegated Access
* Signed URLs (`https` URLs only)

When importing conversation or thread data to Labelbox, include the following information for each data row in your JSON file.

| Parameter | Required | Description |
| - | - | - |
| `row_data` | Yes | `https` path to a cloud-hosted conversational text JSON file. See the section below for details on our conversation format. |
| `global_key` | No | Unique user-generated file name or ID for the file. [Global keys](/horizon/developer-guides/data-row-global-keys) are enforced to be unique in your org. Data rows will not be imported if their global keys are duplicated to existing data rows. |
| `media_type` | No | `"CONVERSATIONAL"` (optional media type to provide better validation and error messaging) |
| `metadata_fields` | No | See [metadata](/horizon/guides/datarow-metadata) |

## Import format

<CodeGroup>
  ```json Local upload expandable theme={"theme":"css-variables"}
  {
    "row_data": {
      "type": "application/vnd.labelbox.conversational.model-chat-evaluation",
      "version": 2,
      "actors": {
          "cm1qu8krf00063b72cutnbn5l": {
          "role": "human",
          "metadata": { "name": "User" }
          },
          "cm1vjleif00023b6y4fw4ew94": {
          "role": "model",
          "metadata": {
              "modelConfigName": "Gem Pro-Copy"
          }
          },
          "cm1vjleif00033b6yifzroser": {
          "role": "model",
          "metadata": {
              "modelConfigName": "gpt 4-Copy"
          }
          }
      },
      "messages": {
          "cm1qu8krf00073b72fyar00vh": {
          "actorId": "cm1qu8krf00063b72cutnbn5l",
          "content": [{ "type": "text", "content": "Hello " }],
          "childMessageIds": [
              "cm1vjlitg00043b6y1tgssq1r",
              "cm1vjlitg00053b6y19ve1qra"
          ]
          },
          "cm1vjlitg00043b6y1tgssq1r": {
          "actorId": "cm1vjleif00023b6y4fw4ew94",
          "content": [
              {
              "type": "text",
              "content": "Hello! 👋 How can I assist you today? 😊 \\n"
              }
          ],
          "childMessageIds": []
          },
          "cm1vjlitg00053b6y19ve1qra": {
          "actorId": "cm1vjleif00033b6yifzroser",
          "content": [
              { "type": "text", "content": "Hi! How can I assist you today?" }
          ],
          "childMessageIds": []
          }
      },
      "rootMessageIds": ["cm1qu8krf00073b72fyar00vh"]
      },
    "global_key": "global_key"
  }
  ```

  ```json Cloud storage theme={"theme":"css-variables"}
  [
      {
        "row_data": "https://storage.googleapis.com/labelbox-datasets/conversational-sample-data/pairwise_shopping_1.json",
        "global_key": "global_key_1"
      },
      {
          "row_data": "https://storage.googleapis.com/labelbox-datasets/conversational-sample-data/pairwise_shopping_2.json",
          "global_key": "global_key_2"
      },
      {
          "row_data": "https://storage.googleapis.com/labelbox-datasets/conversational-sample-data/pairwise_shopping_3.json",
          "global_key": "global_key_3"
      }
  ]
  ```
</CodeGroup>

## Python example

<CodeGroup>
  ```python Local upload expandable theme={"theme":"css-variables"}
  # Embed the chat conversation data
  row_data = {
      "type": "application/vnd.labelbox.conversational.model-chat-evaluation",
      "version": 2,
      "actors": {
          "cm1qu8krf00063b72cutnbn5l": {
              "role": "human",
              "metadata": { "name": "User" }
          },
          "cm1vjleif00023b6y4fw4ew94": {
              "role": "model",
              "metadata": {
                  "modelConfigName": "Gem Pro-Copy"
              }
          },
          "cm1vjleif00033b6yifzroser": {
              "role": "model",
              "metadata": {
                  "modelConfigName": "gpt 4-Copy"
              }
          }
      },
      "messages": {
          "cm1qu8krf00073b72fyar00vh": {
              "actorId": "cm1qu8krf00063b72cutnbn5l",
              "content": [{ "type": "text", "content": "Hello " }],
              "childMessageIds": [
                  "cm1vjlitg00043b6y1tgssq1r",
                  "cm1vjlitg00053b6y19ve1qra"
              ]
          },
          "cm1vjlitg00043b6y1tgssq1r": {
              "actorId": "cm1vjleif00023b6y4fw4ew94",
              "content": [
                  {
                      "type": "text",
                      "content": "Hello! 👋 How can I assist you today? 😊 \\n"
                  }
              ],
              "childMessageIds": []
          },
          "cm1vjlitg00053b6y19ve1qra": {
              "actorId": "cm1vjleif00033b6yifzroser",
              "content": [
                  { "type": "text", "content": "Hi! How can I assist you today?" }
              ],
              "childMessageIds": []
          }
      },
      "rootMessageIds": ["cm1qu8krf00073b72fyar00vh"]
  }

  # Create a dataset
  dataset = client.create_dataset(
      name="mmc_dataset",
  )

  # Upload the conversation data to the dataset as a data row.
  task = dataset.create_data_rows([{"row_data": row_data}])
  task.wait_till_done()

  # Output any errors that occurred during the import.
  print("Errors:", task.errors)
  print("Failed data rows:", task.failed_data_rows)
  ```

  ```python Cloud storage theme={"theme":"css-variables"}
  # Generate dummy global keys
  global_key_1 = str(uuid.uuid4())
  global_key_2 = str(uuid.uuid4())
  global_key_3 = str(uuid.uuid4())

  # Create a dataset
  dataset = client.create_dataset(
      name="pairwise_demo_"+str(uuid.uuid4()),
      iam_integration=None
  )
  # Upload data rows
  task = dataset.create_data_rows([
      {
        "row_data": "https://storage.googleapis.com/labelbox-datasets/conversational-sample-data/pairwise_shopping_1.json",
        "global_key": global_key_1
      },
      {
          "row_data": "https://storage.googleapis.com/labelbox-datasets/conversational-sample-data/pairwise_shopping_2.json",
          "global_key": global_key_2
      },
      {
          "row_data": "https://storage.googleapis.com/labelbox-datasets/conversational-sample-data/pairwise_shopping_3.json",
          "global_key": global_key_3
      }
    ])
  task.wait_till_done()
  print("Errors:",task.errors)
  print("Failed data rows:", task.failed_data_rows)
  ```
</CodeGroup>

## Conversation v2 JSON

| Parameter | Required | Description |
| - | - | - |
| `type` | Yes | Populate with `application/vnd.labelbox.conversational.model-chat-evaluation` |
| `version` | Yes | Populate with `2` |
| `actors` | Yes | An object of [actors](#actor-object) of the chat conversation. |
| `messages` | Yes | An object of messages from each actor. |
| `rootMessageIds` | Yes | An array of message ids. You would include the id of first message given from a human actor. |

### Actor object

Actor objects start with a key value of a unique user given id.

Each actor object has a `role` key and a `metadata` key. The metadata contains the specifics of the actor and will vary depending on the actor's role.

| Parameter | Required | Description |
| - | - | - |
| `role` | Yes | The role the actor receives. Either `human` or `model`. |
| `name` | No | The name of the actor. This is applicable and required for actors with the human role. Placed inside the `metadata` actor key. |
| `modelConfigName` | Yes | The model config name of the actor. This is required for actors with the model role. Placed inside the `metadata` actor key. |

### Message object

Message objects start with a key value of a unique user given id.

| Parameter | Required | Description |
| - | - | - |
| `actorId` | Yes | The id of the actor who produced the message. |
| `content` | Yes | An array of content for the message. See [message content](#message-content). |
| `childMessageIds` | No | An array of message ids that are children of the message object. Typically this would be the next series of messages. If you were comparing more then one model response, multiple message ids can be included. |

#### Message content

| Parameter | Required | Description |
| - | - | - |
| `type` | Yes | The type of message. This will be `fileData` for attachments, `text` for raw text, and `dataRowAttachment` for attachments on data rows. |
| `content` | No | The raw text content of your message. This field supports markdown. This field is used for `text` type messages. |
| `fileUri` | No | `https` path to a public cloud-hosted attachment file. This field is used for `fileData` type messages. If you want to use [IAM delegated access](/horizon/guides/iam-delegated-access) to store conversation files, you should first add them as data row attachments. See [attachments](/horizon/developer-guides/attachments) on how to add an attachment to a data row. After you add your attachments to your data row, you can use the `type` and `attachmentName` keys to include your attachment inside your conversational data. |
| `attachmentName` | No | The name of the attachment on the data row. |
| `mimeType` | No | The `mimeType` of your attachment `fileUri` data. The following types are supported: - `video/mp4` - `image/png` - `application/pdf` |

#### Embed images

You can either embed images directly in the message `content` or add them as attachments.\
For `dataRowAttachment`, the value of `attachmentName` must exist in the `attachments` section.

<CodeGroup>
  ```json Embedded image content theme={"theme":"css-variables"}
  // Message
  {
    "actorId": "",
    "childMessageIds": [],
    "content": [
       {
          "type": "text",
          "content": f"<img title= {model_name2} alt={model_name2} src={model_url2}>"
       }
    ]
  }
  ```

  ```json Image attachment theme={"theme":"css-variables"}
  // Message
  {
    "actorId": "",
    "childMessageIds": [],
    "content": [
       {
          "type": "text",
          "content": "What do you see in this image?"
       },
       {
          "type": "fileData",
          "fileUri": "https://link-to-my-image",
          "mimeType": "image/png"
       },
       {
          "type": "dataRowAttachment",
          "attachmentName": "attachment_name_1",
       }       
    ]
  }
  ```
</CodeGroup>

## Sample conversation v2 JSON

<Info>
  ### SDK import only

  You can't upload the following file from the web interface directly. You must use an import file as described in [Import format](#import-format).
</Info>

<CodeGroup>
  ```json Sample conversation v2 json expandable theme={"theme":"css-variables"}
      "type": "application/vnd.labelbox.conversational.model-chat-evaluation",
      "version": 2,
      "actors": {
          "actor1": {
              "role": "human",
              "metadata": {
                  "name": "User"
              }
          },
          "actor2": {
              "role": "model",
              "metadata": {
                  "modelConfigName": "Model 1"
              }
          },
          "actor3": {
              "role": "model",
              "metadata": {
                  "modelConfigName": "Model 2"
              }
          }
      },
      "messages": {
          "clxcboi1e00053p6n0ya733nn": {
              "actorId": "actor1",
              "content": [
                  {
                      "type": "text",
                      "content": "What's in the images?"
                  },
                  {
                      "type": "fileData",
                      "fileUri": "https://storage.googleapis.com/labelbox-developer-testing-assets/image/giraffe-combunetes.png",
                      "mimeType": "image/png"
                  },
                  {
                      "type": "fileData",
                      "fileUri": "https://storage.googleapis.com/labelbox-developer-testing-assets/image/label-blocks.png",
                      "mimeType": "image/png"
                  },
                  {
                      "type": "fileData",
                      "fileUri": "https://storage.googleapis.com/labelbox-developer-testing-assets/image/tim-test.png",
                      "mimeType": "image/png"
                  }
              ],
              "childMessageIds": [
                  "clxcboue900083p6no6emql83",
                  "clxcboue900093p6nrepe8jjd"
              ]
          },
          "clxcboue900083p6no6emql83": {
              "actorId": "actor2",
              "content": [
                  {
                      "type": "text",
                      "content": "The images show:\n 1. **A giraffe in an office:**  The giraffe is standing in front of a desk with a computer showing code on the screen. There's also a whiteboard with diagrams about Kubernetes."
                  }
              ],
              "childMessageIds": [
                  "clxmrt0hh00023p6qykkdaqtk"
              ]
          },
          "clxcboue900093p6nrepe8jjd": {
              "actorId": "actor3",
              "content": [
                  {
                      "type": "text",
                      "content": "1. The first image shows a giraffe standing in an office environment, interacting with a computer. The giraffe is looking at a large monitor displaying code, and there is a whiteboard in the background with various diagrams and notes.\n2."
                  }
              ],
              "childMessageIds": [
                  "clxmrt0hh00023p6qykkdaqtk"
              ]
          },
          "clxmrt0hh00023p6qykkdaqtk": {
              "actorId": "actor1",
              "content": [
                  {
                      "type": "text",
                      "content": "What's in this PDF file?"
                  },
                  {
                      "type": "fileData",
                      "fileUri": "https://storage.googleapis.com/labelbox-developer-testing-assets/pdf/data_files/south-platte-tree-map.pdf",
                      "mimeType": "application/pdf"
                  }
              ],
              "childMessageIds": [
                  "clxmrtgxg00033p6qqzl2596o",
                  "clxmrtgxg00043p6qiehsvww4"
              ]
          },
          "clxmrtgxg00033p6qqzl2596o": {
              "actorId": "actor2",
              "content": [
                  {
                      "type": "text",
                      "content": "This PDF file contains directions and maps for Christmas tree cutting areas managed by the South Platte Ranger District.  Here's a breakdown:\n**Page 1: Directions**\n* Provides written directions to three Christmas tree cutting areas:\n    **Buffalo Creek:** Located southeast of Pine Junction.\n    **Camp Fickes:** Located west of Buffalo Creek.\n    **Sugar Creek:** Located along County Road\n"
                  }
              ],
              "childMessageIds": [
                  "clxmru9j600053p6q0qh89zm4"
              ]
          },
          "clxmrtgxg00043p6qiehsvww4": {
              "actorId": "actor3",
              "content": [
                  {
                      "type": "text",
                      "content": "Via fugit referre [duasque longumque](http://luminavale.com/) fateri sumite\ncalidumque arma spatiis fuerit genialiter errore iacent; cuncta hausit memori.\nAestus a omnia nomenque inlimis captantur ipsumque fuga. Aeneadae dona tenero\nclipei tamen, sed de amor flagellari quas; corpore, grande.\n[Pectore inclinatcadunt](http://tardoset.com/uni-et.html), Hectoreis defensatque virga altera\nsecum caliturasque militia pennas."
                  }
              ],
              "childMessageIds": [
                  "clxmru9j600053p6q0qh89zm4"
              ]
          },
          "clxmru9j600053p6q0qh89zm4": {
              "actorId": "actor1",
              "content": [
                  {
                      "type": "text",
                      "content": "What have astronauts brought back from the moon?"
                  }
              ],
              "childMessageIds": [
                  "clxmrupyh00063p6q4wxj97sz",
                  "clxmrupyh00073p6qeszn06l7"
              ]
          },
          "clxmrupyh00063p6q4wxj97sz": {
              "actorId": "actor2",
              "content": [
                  {
                      "type": "text",
                      "content": "## Petebat semine\nDiurnis parsque, tanti nuper novi, extremae caede *Psophidaque spiro* dum visa.\nUsu dicebat obstet meritos."
                  }
              ],
              "childMessageIds": []
          },
          "clxmrupyh00073p6qeszn06l7": {
              "actorId": "actor3",
              "content": [
                  {
                      "type": "text",
                      "content": "## Ossa custos captabat insanis humus Cipe temptatum\nLorem markdownum adflatuque est Tydides medios. Notatas te Pandrose **solent**\npartes saucius animal certamen, plures opem corpora. Est magni duce, illiarcus: Iuno atque aderat amplexo genusque."
                  }
              ],
              "childMessageIds": []
          }
      },
      "rootMessageIds": [
          "clxcboi1e00053p6n0ya733nn"
      ]
  ```
</CodeGroup>

<Info>
  ### LaTeX support

  To add LaTeX formatting, wrap your math expressions using backticks and dollar signs. The editor supports both inline and block LaTeX formatting. For example, to add LaTeX formatting for `x=2`, put `$$x = 2$$`.
</Info>


# Import prompt and response annotations
Source: https://docs.labelbox.com/horizon/developer-guides/import-prompt-and-response-annotations

Developer guide for importing annotations on prompt and response projects.

<CardGroup>
  <Card title="Open In Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/annotation_import/prompt_response.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/tree/main/annotation_import/prompt_response.ipynb" />
</CardGroup>

## Overview

To import annotations in Labelbox, you need to create an annotations payload. In this section, we provide this payload for every supported annotation type.

### Annotation payload types

Labelbox supports two formats for the annotations payload:

* Python annotation types (recommended)

  * Provides a seamless transition between third-party platforms, machine learning pipelines, and Labelbox.
  * Allows you to build annotations locally with local file paths, numpy arrays, or URLs.
  * Supports easy conversion to NDJSON format to quickly import annotations to Labelbox.
  * Supports one-level nested classification (radio, checklist, or free-form text) under a tool or classification annotation.

* JSON

  * Skips formatting annotation payload in the Labelbox Python annotation type.
  * Supports any levels of nested classification (radio, checklist, or free-form text) under a tool or classification annotation.

### Label Import Types

Labelbox supports two types of label imports:

* [Model-assisted labeling (MAL)](/horizon/guides/model-assisted-labeling) allows you to import computer-generated predictions and simple annotations created outside of Labelbox as pre-labels on an asset.
* [Ground truth](/horizon/guides/import-ground-truth) allows you to bulk import ground truth annotations from an external or third-party labeling system into Labelbox *Annotate*. Using the label import API to import external data can consolidate and migrate all annotations into Labelbox as a single source of truth.

## Supported Annotations

Prompt and response generated projects support the following annotations data row:

* Prompt and response creation projects

  * Prompt text
  * Radio
  * Checklist
  * Response text

* Prompt creation projects

  * Prompt text

* Response creation projects

  * Radio
  * Checklist
  * Response text

## Prompt

### Free-form text

<Info>
  ### Information

  Only one prompt annotation is allowed per label.
</Info>

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  prompt_annotation = lb_types.PromptClassificationAnnotation(
      name = "Follow the prompt and select answers",
      value = lb_types.PromptText(answer = "This is an example of a prompt")
  )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  prompt_annotation_ndjson = {
    "name": "Follow the prompt and select answers",
    "answer": "This is an example of a prompt"
  }
  ```
</CodeGroup>

## Responses

### Radio (single choice)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  response_radio_annotation = lb_types.ClassificationAnnotation(
      name="response radio feature",
      value=lb_types.Radio(answer =
          lb_types.ClassificationAnswer(name = "first_radio_answer")
      )
  )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  response_radio_annotation_ndjson = {
    "name": "response radio feature",
    "answer": {
        "name": "first_radio_answer"
      }
  }
  ```
</CodeGroup>

### Checklist (multiple choice)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  response_checklist_annotation = lb_types.ClassificationAnnotation(
      name="response checklist feature",
      value=lb_types.Checklist(answer = [
          lb_types.ClassificationAnswer(name = "option_1"),
          lb_types.ClassificationAnswer(name = "option_2"),
      ])
    )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  response_checklist_annotation_ndjson = {
    "name": "response checklist feature",
    "answer": [
      {
        "name": "option_1"
      },
      {
        "name": "option_2"
      }
    ]
  }
  ```
</CodeGroup>

### Free-form text

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  response_text_annotation = lb_types.ClassificationAnnotation(
      name = "Provide a reason for your choice",
      value = lb_types.Text(answer = "This is an example of a response text")
  )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  response_text_annotation_ndjson = {
    "name": "Provide a reason for your choice",
    "answer": "This is an example of a response text"
  }
  ```
</CodeGroup>

### Nested classification

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  nested_response_radio_annotation = lb_types.ClassificationAnnotation(
  name="nested_response_radio_question",
  value=lb_types.Radio(
    answer=lb_types.ClassificationAnswer(
      name="first_radio_answer",
      classifications=[
        lb_types.ClassificationAnnotation(
          name="sub_radio_question",
          value=lb_types.Radio(
            answer=lb_types.ClassificationAnswer(
              name="first_sub_radio_answer"
            )
          )
        )
      ]
    )
  )
  )

  nested_response_checklist_annotation = lb_types.ClassificationAnnotation(
  name="nested_response_checklist_question",
  value=lb_types.Checklist(
    answer=[lb_types.ClassificationAnswer(
      name="first_checklist_answer",
      classifications=[
        lb_types.ClassificationAnnotation(
          name="sub_checklist_question",
          value=lb_types.Checklist(
            answer=[lb_types.ClassificationAnswer(
            name="first_sub_checklist_answer"
          )]
        ))
      ]
    )]
  )
  )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  nested_response_radio_annotation_ndjson = {
    "name": "nested_radio_question",
    "answer": [{
        "name": "first_radio_answer",
        "classifications" : [
          {
            "name": "sub_radio_question",
            "answer": {"name": "first_sub_radio_answer"}
          }
        ]
    }]
  }

  nested_response_checklist_annotation_ndjson = {
    "name": "nested_checklist_question",
    "answer": [{
        "name": "first_checklist_answer",
        "classifications" : [
          {
            "name": "sub_checklist_question",
            "answer": {"name": "first_sub_checklist_answer"}
          }
        ]
    }]
  }
  ```
</CodeGroup>

## Example: Import pre-labels or ground truths

The steps to import annotations as pre-labels (machine-assisted learning) are similar to the steps to import annotations as ground truth labels, and we will describe the slight differences for each scenario.

### Before you start

The below imports are needed to use the code examples in this section.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  import labelbox as lb
  import labelbox.types as lb_types
  import uuid
  import time
  ```
</CodeGroup>

Replace the value of `API_KEY` with a valid [API key](/horizon/developer-guides/create-api-key) to connect to the Labelbox client.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  API_KEY = None
  client = lb.Client(API_KEY)
  ```
</CodeGroup>

### Step 1: Import or generate data rows and create projects

Each type of the prompt and response generation project requires different setup. See [prompt and response project](/horizon/developer-guides/prompt-and-response-projects) for more details on the differences.

#### Prompt response and prompt creation

A prompts and responses creation project automatically generates empty data rows upon creation. You will then need to obtain either the `global_keys` or `data_row_ids` attached to the generated data rows by exporting them from the created project or obtaining them directly on the [data row tab](/horizon/guides/data-rows-activity) using the UI.

<CodeGroup>
  ```python Prompt and response creation theme={"theme":"css-variables"}
  prompt_response_project = client.create_prompt_response_generation_project(
      name="Demo prompt response project",
      media_type=lb.MediaType.LLMPromptResponseCreation,
      dataset_name="Demo prompt response dataset",
      data_row_count=1,
  )
  ```

  ```python Prompt creation theme={"theme":"css-variables"}
  prompt_project = client.create_prompt_response_generation_project(
      name="Demo prompt project",
      media_type=lb.MediaType.LLMPromptCreation,
      dataset_name="Demo prompt dataset",
      data_row_count=1,
  )
  ```
</CodeGroup>

#### Response creation

For response creation projects, text data rows are used and are not generated upon project creation. The following steps create a dataset with a text data row attached, create a response creation project, and batch the created data row towards the project.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Create dataset with text data row
  global_key = "lorem-ipsum.txt"
  text_asset = {
      "row_data": "https://storage.googleapis.com/labelbox-sample-datasets/nlp/lorem-ipsum.txt",
      "global_key": global_key,
      "media_type": "TEXT",
      }

  dataset = client.create_dataset(name="text_annotation_import_demo_dataset")
  task = dataset.create_data_rows([text_asset])
  task.wait_till_done()

  print("Errors:",task.errors)
  print("Failed data rows:", task.failed_data_rows)

  # Create response creation project

  project = client.create_response_creation_project(
    name="Demo response project",
  )

  # Create a batch of data rows for newly created project

  batch = project.create_batch(
    name="Demo response batch", # each batch in a project must have a unique name
    global_keys=[global_key], # paginated collection of data row objects, list of data row ids or global keys
    priority=1 # priority between 1(highest) - 5(lowest)
  )
  ```
</CodeGroup>

### Step 2: Set up ontology

Your project ontology needs to support the classifications required by your annotations. To ensure accurate schema feature mapping, the value used as the `name` parameter needs to match the value of the `name` field in your annotation.

For example, if you provide a name `annotation_name` for your created annotation, you need to name the bounding box tool as `anotations_name` when setting up your ontology. The same alignment must hold true for the other tools and classifications that you create in the ontology.

This example shows how to create an ontology containing all supported by prompt and response project types [annotation types](#supported-annotations).

<CodeGroup>
  ```python Prompt and response creation expandable theme={"theme":"css-variables"}
  ontology_builder = lb.OntologyBuilder(
      tools=[],
      classifications=[
          lb.PromptResponseClassification(
              class_type=lb.PromptResponseClassification.Type.PROMPT,
              name="prompt text",
              character_min = 1, # Minimum character count of prompt field (optional)
              character_max = 20, # Maximum character count of prompt field (optional)
          ),
          lb.PromptResponseClassification(
              class_type=lb.PromptResponseClassification.Type.RESPONSE_CHECKLIST,
              name="response checklist feature",
              options=[
                  lb.ResponseOption(value="option_1", label="option_1"),
                  lb.ResponseOption(value="option_2", label="option_2"),
              ],
          ),
          lb.PromptResponseClassification(
              class_type=lb.PromptResponseClassification.Type.RESPONSE_RADIO,
              name="response radio feature",
              options=[
                  lb.ResponseOption(value="first_radio_answer"),
                  lb.ResponseOption(value="second_radio_answer"),
              ],
          ),
          lb.PromptResponseClassification(
              class_type=lb.PromptResponseClassification.Type.RESPONSE_TEXT,
              name="response text",
              character_min = 1, # Minimum character count of response text field (optional)
              character_max = 20, # Maximum character count of response text field (optional)
          ),
          lb.PromptResponseClassification(
              class_type=lb.PromptResponseClassification.Type.RESPONSE_RADIO,
              name="nested_response_radio_question",
              options=[
                  lb.ResponseOption("first_radio_answer",
                                    options=[
                                      lb.PromptResponseClassification(
                                        class_type=lb.PromptResponseClassification.RESPONSE_RADIO,
                                        name="sub_radio_question",
                                        options=[lb.ResponseOption("first_sub_radio_answer")]
                                      )
                                    ])
              ],
          ),
          lb.PromptResponseClassification(
              class_type=lb.PromptResponseClassification.Type.RESPONSE_CHECKLIST,
              name="nested_response_checklist_question",
              options=[
                  lb.ResponseOption("first_checklist_answer",
                                    options=[
                                      lb.PromptResponseClassification(
                                        class_type=lb.PromptResponseClassification.RESPONSE_CHECKLIST,
                                        name="sub_checklist_question",
                                        options=[lb.ResponseOption("first_sub_checklist_answer")]
                                      )
                                    ])
              ],
          ),
      ],
  )

  # Create ontology
  ontology = client.create_ontology(
      "Prompt and response ontology",
      ontology_builder.asdict(),
      media_type=lb.MediaType.LLMPromptResponseCreation,
  )

  # Connect ontology
  prompt_response_project.connect_ontology(ontology)
  ```

  ```python Prompt creation theme={"theme":"css-variables"}
  ontology_builder = lb.OntologyBuilder(
      tools=[],
      classifications=[
          lb.PromptResponseClassification(
              class_type=lb.PromptResponseClassification.Type.PROMPT,
              name=f"prompt text",
              character_min = 1, # Minimum character count of prompt field (optional)
              character_max = 20, # Maximum character count of prompt field (optional)
          )
      ],
  )

  # Create ontology
  ontology = client.create_ontology(
      "Prompt ontology",
      ontology_builder.asdict(),
      media_type=lb.MediaType.LLMPromptCreation,
  )

  # Connect ontology
  prompt_project.connect_ontology(ontology)
  ```

  ```python Response creation theme={"theme":"css-variables"}
  ontology_builder = lb.OntologyBuilder(
      tools=[],
      classifications=[
          lb.PromptResponseClassification(
              class_type=lb.PromptResponseClassification.Type.RESPONSE_CHECKLIST,
              name="response checklist feature",
              options=[
                  lb.ResponseOption(value="option 1", label="option 1"),
                  lb.ResponseOption(value="option 2", label="option 2"),
              ],
          ),
          lb.PromptResponseClassification(
              class_type=lb.PromptResponseClassification.Type.RESPONSE_RADIO,
              name="response radio feature",
              options=[
                  lb.ResponseOption(value="first_radio_answer"),
                  lb.ResponseOption(value="second_radio_answer"),
              ],
          ),
          lb.PromptResponseClassification(
              class_type=lb.PromptResponseClassification.Type.RESPONSE_TEXT,
              name="response text",
              character_min = 1, # Minimum character count of response text field (optional)
              character_max = 20, # Maximum character count of response text field (optional)
          ),
          lb.PromptResponseClassification(
              class_type=lb.PromptResponseClassification.Type.RESPONSE_RADIO,
              name="nested_response_radio_question",
              options=[
                  lb.ResponseOption("first_radio_answer",
                                    options=[
                                      lb.PromptResponseClassification(
                                        class_type=lb.PromptResponseClassification.RESPONSE_RADIO
                                        name="sub_radio_question",
                                        options=[lb.ResponseOption("first_sub_radio_answer")]
                                      )
                                    ])
              ],
          ),
          lb.PromptResponseClassification(
              class_type=lb.PromptResponseClassification.Type.RESPONSE_CHECKLIST,
              name="nested_response_checklist_question",
              options=[
                  lb.ResponseOption("first_checklist_answer",
                                    options=[
                                      lb.PromptResponseClassification(
                                        class_type=lb.PromptResponseClassification.RESPONSE_CHECKLIST
                                        name="sub_checklist_question",
                                        options=[lb.ResponseOption("first_sub_checklist_answer")]
                                      )
                                    ])
              ],
          ),
      ],
  )

  # Create ontology
  ontology = client.create_ontology(
      "Response ontology",
      ontology_builder.asdict(),
      media_type=lb.MediaType.Text,
      ontology_kind=lb.OntologyKind.ResponseCreation
  )

  # Connect ontology
  response_project.connect_ontology(ontology)
  ```
</CodeGroup>

### Step 3: Export for global\_keys

For prompt response creation and prompt creation projects you will need to obtain either the `global_keys` or `data_row_ids` attached to the generated data rows by exporting them from the created project. Since the generation of data rows is an async process you will need to wait for the project data rows to be completed before exporting.

<CodeGroup>
  ```python Prompt and response export theme={"theme":"css-variables"}
  time.sleep(20)

  export_task = prompt_response_project.export()
  export_task.wait_till_done()

  # Stream the export using a callback function

  def json_stream_handler(output: labelbox.BufferedJsonConverterOutput):
    print(output.json)

  export_task.get_buffered_stream(stream_type=labelbox.StreamType.RESULT).start(stream_handler=json_stream_handler)

  # Collect all exported data into a list

  export_json = [data_row.json for data_row in export_task.get_buffered_stream()]

  # Obtain global keys to be used later on

  global_keys = [dr.json["data_row"]["global_key"] for dr in stream]

  ```

  ```python Prompt export theme={"theme":"css-variables"}
  time.sleep(20)

  export_task = prompt_project.export()
  export_task.wait_till_done()

  # Stream the export using a callback function
  def json_stream_handler(output: labelbox.BufferedJsonConverterOutput):
    print(output.json)

  export_task.get_buffered_stream(stream_type=labelbox.StreamType.RESULT).start(stream_handler=json_stream_handler)

  # Collect all exported data into a list
  export_json = [data_row.json for data_row in export_task.get_buffered_stream()]

  # Obtain global keys to be used later on
  global_keys = [dr.json["data_row"]["global_key"] for dr in stream]
  ```
</CodeGroup>

### Step 4: Import Annotations

For prelabeled (model-assisted labeling) scenarios, pass your payload as the value of the `predictions` parameter. For ground truths, pass the payload to the `labels` parameter.

#### Payload

Depending on the type of prompt and response project you are using your payload might look different. For the response payload, you can also use `is_benchmark_reference` to specify [benchmarks](/horizon/guides/benchmark).

<CodeGroup>
  ```python Prompt and response payload theme={"theme":"css-variables"}
  # Python annotation objects
  label = []
  annotations = [
    prompt_annotation,
    response_radio_annotation,
    response_checklist_annotation,
    response_text_annotation,
    nested_response_radio_annotation,
    nested_response_checklist_annotation
  ]
  label.append(
      lb_types.Label(data={"global_key" : global_keys[0] },
                     annotations=annotations)
              )

  # NDJSON
  label_ndjson = []
  annotations = [
    prompt_annotation_ndjson,
    response_radio_annotation_ndjson,
    response_checklist_annotation_ndjson,
    response_text_annotation_ndjson,
    nested_response_radio_annotation_ndjson,
    nested_response_checklist_annotation_ndjson
  ]
  for annotation in annotations:
      annotation.update({
          "dataRow": {
              "globalKey": global_keys[0]
          },
      })
      label_ndjson.append(annotation)
  ```

  ```python Prompt payload theme={"theme":"css-variables"}
  # Python annotation objects
  label = []
  annotations = [
    prompt_annotation
  ]
  label.append(
      lb_types.Label(data={"global_key" : global_keys[0] },
                     annotations=annotations)
              )

  # NDJSON
  label_ndjson = []
  annotations = [
    prompt_annotation_ndjson
  ]
  for annotation in annotations:
      annotation.update({
          "dataRow": {
              "globalKey": global_keys[0]
          },
      })
      label_ndjson.append(annotation)
  ```

  ```python Response payload theme={"theme":"css-variables"}
  # Python annotation objects
  label = []
  annotations = [
    response_radio_annotation,
    response_checklist_annotation,
    response_text_annotation,
    nested_response_radio_annotation,
    nested_response_checklist_annotation
  ]
  label.append(
      lb_types.Label(data={"global_key" : global_key },
                     annotations=annotations
                     # Optional: set the label as a benchmark
                     # Only supported for groud truth imports
                     is_benchmark_reference = True)
              )

  # NDJSON
  label_ndjson = []
  annotations = [
    response_radio_annotation_ndjson,
    response_checklist_annotation_ndjson,
    response_text_annotation_ndjson,
    nested_response_radio_annotation_ndjson,
    nested_response_checklist_annotation_ndjson
  ]
  for annotation in annotations:
      annotation.update({
          "dataRow": {
              "globalKey": global_key
          },
      })
      label_ndjson.append(annotation)
  ```
</CodeGroup>

#### Option A: Upload as [prelabels (model assisted labeling)](/horizon/guides/model-assisted-labeling)

This option is helpful for speeding up the initial labeling process and reducing the manual labeling workload for high-volume datasets.

<CodeGroup>
  ```python MAL import theme={"theme":"css-variables"}
  # Upload MAL label for this data row in project
  upload_job = lb.MALPredictionImport.create_from_objects(
      client = client,
      project_id = prompt_response_project.uid, # Replace with project of different prompt and response project types
      name = "mal_job"+str(uuid.uuid4()),
      predictions = label
  )

  print(f"Errors: {upload_job.errors}", )
  print(f"Status of uploads: {upload_job.statuses}")

  ```
</CodeGroup>

#### Option B: Upload to a labeling project as [ground truth](/horizon/guides/import-ground-truth)

This option is helpful for loading high-confidence labels from another platform or previous projects that just need review rather than manual labeling effort.

<CodeGroup>
  ```python Label import theme={"theme":"css-variables"}
  # Upload label for this data row in project
  upload_job = lb.LabelImport.create_from_objects(
      client = client,
      project_id = prompt_response_project.uid, # Replace with project of different prompt and response project types
      name = "label_import_job" + str(uuid.uuid4()),
      labels = label
  )

  print(f"Errors: {upload_job.errors}", )
  print(f"Status of uploads: {upload_job.statuses}")
  ```
</CodeGroup>


# Import text annotations
Source: https://docs.labelbox.com/horizon/developer-guides/import-text-annotations

Developer guide for importing annotations on text data and sample import formats.

<CardGroup>
  <Card title="Open In Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/annotation_import/text.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/annotation_import/text.ipynb" />
</CardGroup>

## Overview

To import annotations in Labelbox, you need to create an annotations payload. In this section, we provide this payload for every supported annotation type.

### Annotation payload types

Labelbox supports two formats for the annotations payload:

* Python annotation types (recommended)
  * Provides a seamless transition between third-party platforms, machine learning pipelines, and Labelbox.
  * Allows you to build annotations locally with local file paths, numpy arrays, or URLs
  * Easily convert Python Annotation Type format to NDJSON format to quickly import annotations to Labelbox
  * Supports one-level nested classification (radio, checklist, or free-form text) under a tool or classification annotation.
* JSON
  * Skips formatting annotation payload in the Labelbox Python annotation type
  * Supports any levels of nested classification (radio, checklist, or free-form text) under a tool or classification annotation.

### Label import types

Labelbox additionally supports two types of label imports:

* [Model-assisted labeling (MAL)](/horizon/guides/model-assisted-labeling)
  * This workflow allows you to import computer-generated predictions (or simply annotations created outside of Labelbox) as pre-labels on an asset.
* [Ground truth](https://www.google.com/url?q=https%3A%2F%2Fdocs.labelbox.com%2Fdocs%2Fimport-ground-truth)
  * This workflow functionality allows you to bulk import your ground truth annotations from an external or third-party labeling system into Labelbox *Annotate*. Using the label import API to import external data is a useful way to consolidate and migrate all annotations into Labelbox as a single source of truth.

## Supported annotations

The following annotations are supported for an video data row:

* Radio
* Checklist
* Free-form text
* Entity
* Relationships

## Classifications

### Radio (single choice)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  radio_annotation = lb_types.ClassificationAnnotation(
      name="radio_question",
      value=lb_types.Radio(answer =
          lb_types.ClassificationAnswer(name = "first_radio_answer")
      )
  )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  radio_annotation_ndjson = {
    "name": "radio_question",
    "answer": {"name": "first_radio_answer"}
  }
  ```
</CodeGroup>

### Checklist (multiple choice)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  checklist_annotation = lb_types.ClassificationAnnotation(
      name="checklist_question",
      value=lb_types.Checklist(answer = [
          lb_types.ClassificationAnswer(name = "first_checklist_answer"),
          lb_types.ClassificationAnswer(name = "second_checklist_answer"),
          lb_types.ClassificationAnswer(name = "third_checklist_answer")
      ])
    )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  checklist_annotation_ndjson = {
    "name": "checklist_question",
    "answer": [
      {"name": "first_checklist_answer"},
      {"name": "second_checklist_answer"},
      {"name": "third_checklist_answer"},
    ]
  }
  ```
</CodeGroup>

### Free-form text

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  text_annotation = lb_types.ClassificationAnnotation(
      name = "free_text",
      value = lb_types.Text(answer="sample text")
  )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  text_annotation_ndjson = {
    "name": "free_text",
    "answer": "sample text",
  }
  ```
</CodeGroup>

### Nested classifications

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  nested_radio_annotation = lb_types.ClassificationAnnotation(
    name="nested_radio_question",
    value=lb_types.Radio(
      answer=lb_types.ClassificationAnswer(
        name="first_radio_answer",
        classifications=[
          lb_types.ClassificationAnnotation(
            name="sub_radio_question",
            value=lb_types.Radio(
              answer=lb_types.ClassificationAnswer(
                name="first_sub_radio_answer"
              )
            )
          )
        ]
      )
    )
  )

  nested_checklist_annotation = lb_types.ClassificationAnnotation(
    name="nested_checklist_question",
    value=lb_types.Checklist(
      answer=[lb_types.ClassificationAnswer(
        name="first_checklist_answer",
        classifications=[
          lb_types.ClassificationAnnotation(
            name="sub_checklist_question",
            value=lb_types.Checklist(
              answer=[lb_types.ClassificationAnswer(
              name="first_sub_checklist_answer"
            )]
          ))
        ]
      )]
    )
  )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  nested_radio_annotation_ndjson= {
    "name": "nested_radio_question",
    "answer": {
        "name": "first_radio_answer",
        "classifications": [
          {
            "name": "sub_radio_question",
            "answer": {"name": "first_sub_radio_answer"}
          }
        ]
      }
  }

  nested_checklist_annotation_ndjson = {
    "name": "nested_checklist_question",
    "answer": [{
        "name": "first_checklist_answer",
        "classifications" : [
          {
            "name": "sub_checklist_question",
            "answer": {"name": "first_sub_checklist_answer"}
          }
        ]
    }]
  }
  ```
</CodeGroup>

## Tools

### Entity

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  named_entity = lb_types.TextEntity(start=10, end=20)
  named_entitity_annotation = lb_types.ObjectAnnotation(value=named_entity, name = "named_entity")
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  entities_ndjson = {
      "name": "named_entity",
      "location": {
          "start": 67,
          "end": 128
      }
  }
  ```
</CodeGroup>

### Tool with nested classification

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  tool_with_radio_subclass_annotation = lb_types.ObjectAnnotation(
      name=# Feature name,
      value=# Add tool annotation (lb_types."tool"),
      classifications=[
          lb_types.ClassificationAnnotation(
              name="sub_radio_question",
              value=lb_types.Radio(answer=lb_types.ClassificationAnswer(
                  name="first_sub_radio_answer")))
      ])
  ```

  ```python Entity Example theme={"theme":"css-variables"}
  bbox_with_radio_subclass_annotation = lb_types.ObjectAnnotation(
      name="entity_with_radio_subclass",
      value=lb_types.TextEntity(start=10, end=20),
      classifications=[
          lb_types.ClassificationAnnotation(
              name="sub_radio_question",
              value=lb_types.Radio(answer=lb_types.ClassificationAnswer(
                  name="first_sub_radio_answer")))
      ])
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  bbox_with_radio_subclass_ndjson = {
      "name": "bbox_with_radio_subclass",
      "classifications": [{
          "name": "sub_radio_question",
          "answer": {
              "name": "first_sub_radio_answer"
          }
      }],
      "bbox": {
          "top": 933,
          "left": 541,
          "height": 191,
          "width": 330
      }
  }
  ```
</CodeGroup>

## Relationship

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  relationship = lb_types.RelationshipAnnotation(
      name=# Relationship name,
      value=lb_types.Relationship(
          source=# Source tool,
          target=# Target tool,
          type=lb_types.Relationship.Type.UNIDIRECTIONAL,
      ))
  ```

  ```python Entity Example theme={"theme":"css-variables"}
  ner_source = lb_types.ObjectAnnotation(
      name="named_entity",
      value=lb_types.TextEntity(
        start=133,
        end=140
      )
  )

  ner_target = lb_types.ObjectAnnotation(
      name="named_entity",
      value=lb_types.TextEntity(
        start=143,
        end=159
      )
  )

  ner_relationship = lb_types.RelationshipAnnotation(
      name="relationship",
      value=lb_types.Relationship(
          source=ner_source, # UUID is not required for annotation types
          target=ner_target,
          type=lb_types.Relationship.Type.UNIDIRECTIONAL,
      ))
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  uuid_source = str(uuid.uuid4())
  uuid_target = str(uuid.uuid4())

  entity_source_ndjson = {
    "name": "named_entity",
    "uuid": uuid_source,
    "location": {
            "start" : 133,
            "end": 140
        }
  }

  entity_target_ndjson = {
    "name": "named_entity",
    "uuid": uuid_target,
    "location": {
      "start": 143,
      "end": 159
    }
  }

  ner_relationship_annotation_ndjson = {
      "name": "relationship",
      "relationship": {
        "source": uuid_source, # UUID reference to entity source annotation
        "target": uuid_target, # UUID reference to target source annotation
        "type": "unidirectional"
      }
  }
  ```
</CodeGroup>

## Example: Import pre-labels or ground truths

The steps to import annotations as pre-labels (machine-assisted learning) are similar to those to import annotations as ground truth labels. However, they vary slightly, and we will describe the differences for each scenario.

### Before you start

The below imports are needed to use the code examples in this section.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  import labelbox as lb
  import labelbox.types as lb_types
  import uuid
  import json
  ```
</CodeGroup>

Replace the value of `API_KEY` with a valid [API key](/horizon/developer-guides/create-api-key) to connect to the Labelbox client.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  API_KEY = None
  client = lb.Client(API_KEY)
  ```
</CodeGroup>

### Step 1: Import data rows

Data rows must first be uploaded to **Catalog** to attach annotations.

This example shows how to create a data row in **Catalog** by attaching it to a [dataset](/horizon/developer-guides/dataset) .

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # You can now include ohter fields like attachments, media type and metadata in the data row creation step: /horizon/developer-guides/import-text-data
  global_key = "lorem-ipsum.txt"
  text_asset = {
      "row_data": "https://storage.googleapis.com/labelbox-sample-datasets/nlp/lorem-ipsum.txt",
      "global_key": global_key,
      "media_type": "TEXT",
      "attachments": [{"type": "TEXT_URL", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt"}]
      }

  dataset = client.create_dataset(name="text_annotation_import_demo_dataset")
  task = dataset.create_data_rows([text_asset])
  task.wait_till_done()
  print("Errors:",task.errors)
  print("Failed data rows:", task.failed_data_rows)
  ```
</CodeGroup>

### Step 2: Set up ontology

Your project ontology should support the tools and classifications required by your annotations. To ensure accurate schema feature mapping, the value used as the `name` parameter should match the value of the `name` field in your annotation.

For example, when we created an annotation above, we provided a name`annotation_name`. Now, when we set up our ontology, we must ensure that the name of our bounding box tool is also `anotations_name`. The same alignment must hold true for the other tools and classifications we create in our ontology.

This example shows how to create an ontology containing all supported [annotation types](#supported-annotations) .

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}

  ontology_builder = lb.OntologyBuilder(
  classifications=[ # List of Classification objects
    lb.Classification(
      class_type=lb.Classification.Type.RADIO,
      name="radio_question",
      options=[lb.Option(value="first_radio_answer")]
    ),
    lb.Classification(
      class_type=lb.Classification.Type.RADIO,
      name="nested_radio_question",
      options=[
        lb.Option(value="first_radio_answer",
          options=[
              lb.Classification(
                class_type=lb.Classification.Type.RADIO,
                name="sub_radio_question",
                options=[
                  lb.Option(value="first_sub_radio_answer")
                ]
            ),
          ]
        ),
      ],
    ),
     lb.Classification(
      class_type=lb.Classification.Type.CHECKLIST,
      name="nested_checklist_question",
      options=[
          lb.Option("first_checklist_answer",
            options=[
              lb.Classification(
                  class_type=lb.Classification.Type.CHECKLIST,
                  name="sub_checklist_question",
                  options=[lb.Option("first_sub_checklist_answer")]
              )
          ]
        )
      ]
    ),
    lb.Classification(
      class_type=lb.Classification.Type.CHECKLIST,
      name="checklist_question",
      options=[
        lb.Option(value="first_checklist_answer"),
        lb.Option(value="second_checklist_answer"),
        lb.Option(value="third_checklist_answer")
      ]
    ),
     lb.Classification(
      class_type=lb.Classification.Type.TEXT,
      name="free_text"
    )
  ],
  tools=[ # List of Tool objects
         lb.Tool(
            tool=lb.Tool.Type.NER,
            name="named_entity"
          ),
         lb.Tool(
            tool=lb.Tool.Type.RELATIONSHIP,
            name="relationship"
          )
    ]
  )
  ontology = client.create_ontology("Ontology Text Annotations", ontology_builder.asdict())
  ```
</CodeGroup>

### Step 3: Set Up a Labeling Project

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Project defaults to batch mode with benchmark quality settings if this argument is not provided
  # Queue mode will be deprecated once dataset mode is deprecated

  project = client.create_project(name="text_project_demo",
  queue_mode=lb.QueueMode.Batch,
  media_type=lb.MediaType.Text)

  project.connect_ontology(ontology)
  ```
</CodeGroup>

### Step 4: Send Data Rows to Project

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Set up batches and ontology

  # Create a batch to send to your MAL project
  batch = project.create_batch(
    "first-batch-text-demo", # Each batch in a project must have a unique name
    global_keys=[global_key] , # a list of global keys, data rows, or data row ids
    priority=5 # priority between 1(highest) - 5(lowest)
  )

  print("Batch: ", batch)
  ```
</CodeGroup>

### Step 5: Create annotation payloads

For help understanding annotation payloads, see [overview](#overview). To declare payloads, you can use Python annotation types (*preferred*) or NDJSON objects. For annotations that you want to import as ground truth labels, you can also specify [benchmarks](/horizon/guides/benchmark) using the `is_benchmark_reference` flag.

These examples demonstrate each format and how to compose annotations into labels attached to data rows.

<CodeGroup>
  ```python Python annotation payload theme={"theme":"css-variables"}
  labels = []
  labels.append(
      lb_types.Label(
          data={"global_key" : global_key }
          annotations = [
              named_entitity_annotation,
              radio_annotation,
              checklist_annotation,
              text_annotation,
              ner_source,
              ner_target,
              ner_relationship,
              nested_checklist_annotation,
              nested_radio_annotation
          ],
          # optional: set the label as a benchmark
           is_benchmark_reference = True
      )
  )
  labels = []
  labels.append(
      lb_types.Label(
          data={"global_key" : global_key },
          annotations = [
              named_entitity_annotation,
              radio_annotation,
              checklist_annotation,
              text_annotation,
              nested_checklist_annotation,
              nested_radio_annotation
          ],
          # Optional: set the label as a benchmark
          # Only supported for groud truth imports
          is_benchmark_reference = True
      )
  )
  ```

  ```python NDJSON Payload theme={"theme":"css-variables"}
  label_ndjson = []
  for annotations in [entities_ndjson,
                     radio_annotation_ndjson,
                     checklist_annotation_ndjson,
                     text_annotation_ndjson,
                     nested_radio_annotation_ndjson,
                     nested_checklist_annotation_ndjson,
                     entity_source_ndjson,
                     entity_target_ndjson,
                     ner_relationship_annotation_ndjson,
                      ] :
    annotations.update({
        "dataRow": { "globalKey": global_key }
    })
    label_ndjson.append(annotations)
  ```
</CodeGroup>

### Step 6: Import annotation payload

For prelabeled (model-assisted labeling) scenarios, pass your payload as the value of the `predictions` parameter. For ground truths, pass the payload to the `labels` parameter.

<Warning>
  ### Warning

  Relationship annotations are not supported for ground truth import jobs.
</Warning>

#### Option A: Upload as [prelabels (model assisted labeling)](/horizon/guides/model-assisted-labeling)

This option is helpful for speeding up the initial labeling process and reducing the manual labeling workload for high-volume datasets.

<CodeGroup>
  ```python MAL import theme={"theme":"css-variables"}
  # Upload MAL label for this data row in project
  upload_job = lb.MALPredictionImport.create_from_objects(
      client = client,
      project_id = project.uid,
      name="mal_job"+str(uuid.uuid4()),
      predictions=label
  )

  print(f"Errors: {upload_job.errors}", )
  print(f"Status of uploads: {upload_job.statuses}")
  ```
</CodeGroup>

#### Option B: Upload to a labeling project as [ground truth](/horizon/guides/import-ground-truth)

This option is helpful for loading high-confidence labels from another platform or previous projects that just need review rather than manual labeling effort.

<CodeGroup>
  ```python Label import theme={"theme":"css-variables"}
  # Upload label for this data row in project
  upload_job = lb.LabelImport.create_from_objects(
      client = client,
      project_id = project.uid,
      name="label_import_job"+str(uuid.uuid4()),
      labels=label
  )

  print(f"Errors: {upload_job.errors}", )
  print(f"Status of uploads: {upload_job.statuses}")
  ```
</CodeGroup>


# Import text data
Source: https://docs.labelbox.com/horizon/developer-guides/import-text-data

How to import annotations on text data and sample import formats.

## Specifications

File format: TXT Text encoding: UTF-8 (Note: The Editor does not process any special character sequences like HTML Entities, Unicode Escape Sequence, or colon emoji aliases.)

Import methods:

* IAM Delegated Access
* Signed URLs (`https` URLs only)
* Direct upload of local files(256 MB max file size) Note: Direct upload currently does not support adding additional metadata and attachments, see below Python example.

## Parameters

| Parameter | Required | Description |
| - | - | - |
| `row_data` | Yes | You can upload text in two ways: 1) A `https` url to cloud hosted text file. The txt file must be encoded as UTF-8. 2) A text string (up to 16MB) |
| `global_key` | No | Unique user-generated file name or ID for the file. [Global keys](/horizon/developer-guides/data-row-global-keys) are enforced to be unique in your org. Data rows will not be imported if its global keys are duplicated to existing data rows. |
| `media_type` | No | `"TEXT"` (optional media type to provide better validation and error messaging) |
| `metadata_fields` | No | See [Metadata](/horizon/guides/datarow-metadata). |
| `attachments` | No | See [Attachments](/horizon/guides/label-data) and [Asset overlays](/horizon/guides/label-data). |

## Import format

<CodeGroup>
  ```json Delegated Access URL theme={"theme":"css-variables"}
  [
    {
      "row_data": "https://lb-test-data.s3.us-west-1.amazonaws.com/text-samples/sample-text-1.txt",
      "global_key": "https://lb-test-data.s3.us-west-1.amazonaws.com/text-samples/sample-text-1.txt",
      "media_type": "TEXT",
      "metadata_fields": [{"name": "<metadata_field_name>", "value": "tag_string"}],
      "attachments": [{"type": "TEXT_URL", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt"}]
    },
    {
      "row_data": "https://lb-test-data.s3.us-west-1.amazonaws.com/text-samples/sample-text-2.txt",
      "global_key": "https://lb-test-data.s3.us-west-1.amazonaws.com/text-samples/sample-text-2.txt",
      "media_type": "TEXT",
      "metadata_fields": [{"name": "<metadata_field_name>", "value": "tag_string"}],
      "attachments": [{"type": "TEXT_URL", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt"}]
    }
  ]
  ```

  ```json Standard URL theme={"theme":"css-variables"}
  [
    {
      "row_data": "https://lb-test-data.s3.us-west-1.amazonaws.com/text-samples/sample-text-1.txt",
      "global_key": "https://lb-test-data.s3.us-west-1.amazonaws.com/text-samples/sample-text-1.txt",
      "media_type": "TEXT",
      "metadata_fields": [{"schema_id": "cko8s9r5v0001h2dk9elqdidh", "value": "tag_string"}],
      "attachments": [{"type": "HTML", "value": "https://www.wikipedia.org/" }]
    }
  ]
  ```

  ```json Raw Text Upload theme={"theme":"css-variables"}
  [
    {
      "row_data": "👋 I am a raw text string with emojis  🙏 😃",
      "global_key": "483e64b0-c2fb-41bc-9f69-82c036f1ca5c",
      "media_type": "TEXT",
      "metadata_fields": [{"schema_id": "cko8s9r5v0001h2dk9elqdidh", "value": "tag_string"}],
      "attachments": [{"type": "TEXT_URL", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt"}]
    },
    {
      "row_data": "Active learning is a method of learning in which students are actively or experientially involved in the learning process and where there are different levels of active learning, depending on student involvement.[1] Bonwell & Eison (1991) states that students participate [in active learning] when they are doing something besides passively listening. According to Hanson and Moser (2003) using active teaching techniques in the classroom create better academic outcomes for students. Scheyvens, Griffin, Jocoy, Liu, & Bradford (2008) further noted that “by utilizing learning strategies that can include small-group work, role-play and simulations, data collection and analysis, active learning is purported to increase student interest and motivation and to build students ‘critical thinking, problem-solving and social skills”. In a report from the Association for the Study of Higher Education (ASHE), authors discuss a variety of methodologies for promoting active learning. They cite literature that indicates students must do more than just listen in order to learn. They must read, write, discuss, and be engaged in solving problems. This process relates to the three learning domains referred to as knowledge, skills and attitudes (KSA). This taxonomy of learning behaviors can be thought of as the goals of the learning process.[2] In particular, students must engage in such higher-order thinking tasks as analysis, synthesis, and evaluation.[3]",
      "global_key": "6a12ff36-8a3e-4e7d-aff1-261840500c96",
      "media_type": "TEXT",
      "metadata_fields": [{"schema_id": "cko8s9r5v0001h2dk9elqdidh", "value": "tag_string"}],
      "attachments": [{"type": "TEXT_URL", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt"}]
    }
  ]
  ```
</CodeGroup>

## Python example

<CodeGroup>
  ```python bulk import example theme={"theme":"css-variables"}
  from labelbox import Client
  from uuid import uuid4 ## to generate unique IDs
  import datetime

  client = Client(api_key="<YOUR_API_KEY>")

  dataset = client.create_dataset(name="Bulk import example - Text")

  assets = [
    {
      "row_data": "https://lb-test-data.s3.us-west-1.amazonaws.com/text-samples/sample-text-1.txt",
      "global_key": "https://lb-test-data.s3.us-west-1.amazonaws.com/text-samples/sample-text-1.txt",
      "media_type": "TEXT",
      "metadata_fields": [{"name": "<metadata_field_name>", "value": "tag_string"}],
      "attachments": [{"type": "TEXT_URL", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt"}]
    }
  ]
  task = dataset.create_data_rows(assets)
  task.wait_till_done()
  print(task.errors)
  ```

  ```python local files theme={"theme":"css-variables"}
  from labelbox import Client

  client = Client(api_key="<YOUR_API_KEY>")

  dataset = client.create_dataset(name="Bulk import example - Text")

  assets = [
    {
      "row_data": "path/to/local/file1",
      "global_key": "path/to/local/file1",
      "media_type": "TEXT"
    },
    {
      "row_data": "path/to/local/file2",
      "global_key": "path/to/local/file2",
      "media_type": "TEXT"
    },
  ]
  task = dataset.create_data_rows(assets)
  task.wait_till_done()
  print(task.errors)
  ```
</CodeGroup>


# Import video annotations
Source: https://docs.labelbox.com/horizon/developer-guides/import-video-annotations

Developer guide for importing annotations on video data and sample import formats.

<CardGroup>
  <Card title="Open In Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/annotation_import/video.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/annotation_import/video.ipynb" />
</CardGroup>

## Overview

To import annotations in Labelbox, you need to create an annotations payload. In this section, we provide this payload for every supported annotation type.

### Annotation payload types

Labelbox supports two formats for the annotations payload:

* Python annotation types (recommended)
  * Provides a seamless transition between third-party platforms, machine learning pipelines, and Labelbox.
  * Allows you to build annotations locally with local file paths, numpy arrays, or URLs
  * Easily convert Python Annotation Type format to NDJSON format to quickly import annotations to Labelbox
  * Supports one-level nested classification (radio, checklist, or free-form text) under a tool or classification annotation.
* JSON
  * Skips formatting annotation payload in the Labelbox Python annotation type
  * Supports any levels of nested classification (radio, checklist, or free-form text) under a tool or classification annotation.

### Label import types

Labelbox additionally supports two types of label imports:

* [Model-assisted labeling (MAL)](/horizon/guides/model-assisted-labeling)
  * This workflow allows you to import computer-generated predictions (or simply annotations created outside of Labelbox) as pre-labels on an asset.
* [Ground truth](https://www.google.com/url?q=https%3A%2F%2Fdocs.labelbox.com%2Fdocs%2Fimport-ground-truth)
  * This workflow functionality allows you to bulk import your ground truth annotations from an external or third-party labeling system into Labelbox *Annotate*. Using the label import API to import external data is a useful way to consolidate and migrate all annotations into Labelbox as a single source of truth.

## Supported annotations

The following annotations are supported for an video data row:

* Radio
* Checklist
* Free-form text
* Bounding box
* Point
* Polyline
* Segmentation masks

<Info>
  ### Tool features are frame-based

  Global and frame-based classifications are supported on video assets. However, tool annotations are required to be frame-based.
</Info>

## Classifications

You can use per-framed annotations or temporal annotations interchangeably.

### Global classifications

#### Checklist (global)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  # Use ClassificationAnnotation instead of VideoClassification for global annotations
  global_checklist_annotation=[lb_types.ClassificationAnnotation(
    name="checklist_class_global",
    value=lb_types.Checklist(
        answer = [
          lb_types.ClassificationAnswer(
              name = "first_checklist_answer"
          ),
          lb_types.ClassificationAnswer(
              name = "second_checklist_answer"
          )
        ]
      )
   )]
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  global_checklist_classification_ndjson = {
      "name": "checklist_class_global",
      "answer": [
          { "name": "first_checklist_answer" },
          { "name": "second_checklist_answer"}
    ]
  }
  ```
</CodeGroup>

#### Radio (global)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  # Use ClassificationAnnotation instead of VideoClassification for global annotations
  global_radio_annotation = [lb_types.ClassificationAnnotation(
      name="radio_class_global",
      value=lb_types.Radio(answer = lb_types.ClassificationAnswer(name = "first_radio_answer"))
  )]
  ```

  ```r NDJSON theme={"theme":"css-variables"}
  global_radio_classification_ndjson = {
      "name": "radio_class_global",
      "answer": { "name": "first_radio_answer"}
  }
  ```
</CodeGroup>

#### Free-form text (global)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  text_annotation = lb_types.ClassificationAnnotation(
    name="free_text_global",
    value=lb_types.Text(answer="sample text")
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  text_annotation_ndjson = {
    "name": "free_text_global",
    "answer": "sample text",
  }
  ```
</CodeGroup>

#### Nested classifications (global)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  nested_checklist_annotation = [lb_types.ClassificationAnnotation(
    name="nested_checklist_question",
    value=lb_types.Checklist(
      answer=[lb_types.ClassificationAnswer(
        name="first_checklist_answer",
        classifications=[
          lb_types.ClassificationAnnotation(
            name="sub_checklist_question",
            value=lb_types.Checklist(
              answer=[lb_types.ClassificationAnswer(
              name="first_sub_checklist_answer"
            )]
          ))
        ]
      )]
    )
  )]

  nested_radio_annotation =[lb_types.ClassificationAnnotation(
    name="nested_radio_question",
    value=lb_types.Radio(
      answer=lb_types.ClassificationAnswer(
        name="first_radio_answer",
        classifications=[
          lb_types.ClassificationAnnotation(
            name="sub_radio_question",
            value=lb_types.Radio(
              answer=lb_types.ClassificationAnswer(
                name="first_sub_radio_answer"
              )
            )
          )
        ]
      )
    )
  )]
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  nested_checklist_annotation_ndjson = {
    "name": "nested_checklist_question",
    "answer": [{
        "name": "first_checklist_answer",
        "classifications" : [
          {
            "name": "sub_checklist_question",
            "answer": {"name": "first_sub_checklist_answer"}
          }
        ]
    }]
  }

  nested_radio_annotation_ndjson = {
    "name": "nested_radio_question",
    "answer": {
      "name": "first_radio_answer",
      "classifications": [
        {"name": "sub_radio_question", "answer": {"name": "first_sub_radio_answer"}}
      ],
    },
  }
  ```
</CodeGroup>

### Frame-based classifications

#### Checklist (frame-based)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  checklist_annotation= [
      lb_types.VideoClassificationAnnotation(
          name="checklist_class",
          frame=29,
          segment_index=0,
          value=lb_types.Checklist(
              answer = [
                  lb_types.ClassificationAnswer(
                      name = "first_checklist_answer"
                  ),
                  lb_types.ClassificationAnswer(
                      name = "second_checklist_answer"
                  )
              ]
              )
          ),
      lb_types.VideoClassificationAnnotation(
          name="checklist_class",
          frame=35,
          segment_index=0,
          value=lb_types.Checklist(
              answer = [
                  lb_types.ClassificationAnswer(
                      name = "first_checklist_answer"
                  ),
                   lb_types.ClassificationAnswer(
                      name = "second_checklist_answer"
                  )
              ]
              )
          )
  ]
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  frame_checklist_classification_ndjson = {
      "name": "checklist_class",
      "answer": [
          { "name": "first_checklist_answer" , "frames": [{"start": 29, "end": 35 }]},
          { "name": "second_checklist_answer" , "frames": [{"start": 29, "end": 35 }]}
    ]
  }
  ```
</CodeGroup>

#### Radio (frame-based)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  radio_annotation = [
      lb_types.VideoClassificationAnnotation(
          name="radio_class",
          frame=9,
          segment_index=0,
          value=lb_types.Radio(answer = lb_types.ClassificationAnswer(name = "first_radio_answer"))
      ),
      lb_types.VideoClassificationAnnotation(
          name="radio_class",
          frame=15,
          segment_index=0,
          value=lb_types.Radio(answer = lb_types.ClassificationAnswer(name = "first_radio_answer"))
      )
  ]
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  frame_radio_classification_ndjson = {
      "name": "radio_class",
      "answer": { "name": "first_radio_answer", "frames": [{"start": 9, "end": 15}]}
  }
  ```
</CodeGroup>

#### Free-form text (frame-based)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  text_annotation = [
  	lb_types.TemporalClassificationText(
  	    name="your_classification_name",
  	    value=[
  	        (start_frame, end_frame, "your text value"),
  	        (start_frame, end_frame, "another text value"),
  	    ],
  	    classifications=[
  	        # optional nested classifications
  	    ],
  	)
  ]
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  frame_text_classification_ndjson = {
      "name": "free_text",
      "answer": [
        { "value": "sample text", "frames": [{"start": 13, "end": 19}]}
      ]
  }
  ```
</CodeGroup>

### Temporal classifications (frames)

Use temporal classifications to attach **time-based classification labels** to a video asset. Temporal ranges are represented in **frame indices** from the start of the asset.

When using NDJSON, temporal classifications use a unified `answer: [...]` list structure:

* For **temporal radio**, `answer` must be a single-item list (unlike global radio, which uses an object).
* For **temporal checklist**, `answer` is a multi-item list.
* For **temporal text**, `answer` is a list of: `{ "value": <text>, "frames": [{"start": <frame>, "end": <frame>}] }`.

Temporal classifications support nested hierarchies (for example: Radio → Radio → Radio, or Text → Text → Text).

Child classifications must use frame ranges that fall within the parent classification's frame range.

#### Checklist (temporal)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  temporal_checklist_annotation = lb_types.TemporalClassificationQuestion(
    name="road_conditions",
    value=[
      lb_types.TemporalClassificationAnswer(
        name="rain",
        frames=[(0, 450), (900, 1200)],
      ),
      lb_types.TemporalClassificationAnswer(
        name="construction_zone",
        frames=[(600, 1000)],
      ),
      lb_types.TemporalClassificationAnswer(
        name="low_visibility",
        frames=[(950, 1200)],
      ),
    ],
  )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  temporal_checklist_annotation_ndjson = {
    "name": "road_conditions",
    "answer": [
      {
        "name": "rain",
        "frames": [{"start": 0, "end": 450}, {"start": 900, "end": 1200}]
      },
      {
        "name": "construction_zone",
        "frames": [{"start": 600, "end": 1000}]
      },
      {
        "name": "low_visibility",
        "frames": [{"start": 950, "end": 1200}]
      }
    ]
  }
  ```
</CodeGroup>

#### Radio (temporal)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  temporal_radio_annotation = lb_types.TemporalClassificationQuestion(
    name="traffic_light_state",
    value=[
      lb_types.TemporalClassificationAnswer(
        name="red",
        frames=[(0, 300)],
      ),
      lb_types.TemporalClassificationAnswer(
        name="green",
        frames=[(301, 800)],
      ),
    ],
  )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  temporal_radio_annotation_ndjson = {
    "name": "traffic_light_state",
    "answer": [
      { "name": "red", "frames": [{"start": 0, "end": 300}] },
      { "name": "green", "frames": [{"start": 301, "end": 800}] }
    ]
  }
  ```
</CodeGroup>

#### Free-form text (temporal)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  temporal_text_annotation = lb_types.TemporalClassificationText(
    name="play_description",
    value=[
      (120, 240, "Quarterback throws a deep pass"),
      (241, 310, "Wide receiver catches the ball"),
      (311, 380, "Touchdown scored"),
    ],
  )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  temporal_text_annotation_ndjson = {
    "name": "play_description",
    "answer": [
      { "value": "Quarterback throws a deep pass", "frames": [{"start": 120, "end": 240}] },
      { "value": "Wide receiver catches the ball", "frames": [{"start": 241, "end": 310}] },
      { "value": "Touchdown scored", "frames": [{"start": 311, "end": 380}] }
    ]
  }
  ```
</CodeGroup>

#### Nested classifications (temporal)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  nested_temporal_radio_annotation = lb_types.TemporalClassificationQuestion(
    name="driver_behavior",
    value=[
      lb_types.TemporalClassificationAnswer(
        name="distracted",
        frames=[(500, 900)],
        classifications=[
          lb_types.TemporalClassificationQuestion(
            name="distraction_type",
            value=[
              lb_types.TemporalClassificationAnswer(
                name="mobile_phone_use",
                frames=[(550, 850)],
                classifications=[
                  lb_types.TemporalClassificationQuestion(
                    name="risk_level",
                    value=[
                      lb_types.TemporalClassificationAnswer(
                        name="high",
                        frames=[(700, 850)],
                      )
                    ],
                  )
                ],
              )
            ],
          )
        ],
      )
    ],
  )
  ```

  ```json NDJSON theme={"theme":"css-variables"}
  nested_temporal_radio_annotation_ndjson = {
    "name": "driver_behavior",
    "answer": [
      {
        "name": "distracted",
        "frames": [{"start": 500, "end": 900}],
        "classifications": [
          {
            "name": "distraction_type",
            "answer": [
              {
                "name": "mobile_phone_use",
                "frames": [{"start": 550, "end": 850}],
                "classifications": [
                  {
                    "name": "risk_level",
                    "answer": [
                      { "name": "high", "frames": [{"start": 700, "end": 850}] }
                    ]
                  }
                ]
              }
            ]
          }
        ]
      }
    ]
  }
  ```
</CodeGroup>

## Tools

### Bounding box (frame-based)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  # Confidence scores are not supported for frame specific bounding box annotations and VideoObjectAnnotation class
  bbox_dm = {
    "top":617,
    "left":1371,
    "height":419,
    "width":505
  }

  # Python Annotation
  bbox_annotation = [
    lb_types.VideoObjectAnnotation(
      name = "bbox_video",
      keyframe=True,
      frame=13,
      segment_index=0,
      value = lb_types.Rectangle(
            start=lb_types.Point(x=bbox_dm["left"], y=bbox_dm["top"]), # x = left, y = top
            end=lb_types.Point(x=bbox_dm["left"] + bbox_dm["width"], y=bbox_dm["top"] + bbox_dm["height"]), # x= left + width , y = top + height
        )
    ),
    lb_types.VideoObjectAnnotation(
      name = "bbox_video",
      keyframe=True,
      frame=19,
      segment_index=0,
      value = lb_types.Rectangle(
            start=lb_types.Point(x=bbox_dm["left"], y=bbox_dm["top"]),
            end=lb_types.Point(x=bbox_dm["left"] + bbox_dm["width"], y=bbox_dm["top"] + bbox_dm["height"]),
        )
    )
  ]
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  bbox_dm = {
    "top":617,
    "left":1371,
    "height":419,
    "width":505
  }

  bbox_annotation_ndjson = {
      "name" : "bbox_video",
      "segments" : [{
          "keyframes" : [
              {
                "frame": 13,
                "bbox" : bbox_dm
             },
             {
                "frame": 19,
                "bbox" : bbox_dm
             }
          ]
        }
      ]
  }
  ```
</CodeGroup>

### Point (frame-based)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  point_annotation = [
      lb_types.VideoObjectAnnotation(
          name = "point_video",
          keyframe=True,
          frame=17,
          value = lb_types.Point(x=660.134, y=407.926),
          )
  ]
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  point_annotation_ndjson = {
      "name": "point_video",
      "segments": [{
          "keyframes": [{
              "frame": 17,
              "point" : {
                  "x": 660.134 ,
                  "y": 407.926
              }
          }]
      }]
  }
  ```
</CodeGroup>

### Polyline (frame-based)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  polyline_annotation = [
    lb_types.VideoObjectAnnotation(
      name = "line_video_frame",
      keyframe=True,
      frame=5,
      segment_index=0,
      value=lb_types.Line(
            points=[lb_types.Point(x=680, y=100), lb_types.Point(x=100, y=190)]
        )
    ),
    lb_types.VideoObjectAnnotation(
      name = "line_video_frame",
      keyframe=True,
      frame=20,
      segment_index=0,
      value=lb_types.Line(
            points=[lb_types.Point(x=680, y=100), lb_types.Point(x=100, y=190)]
        )
    ),
    lb_types.VideoObjectAnnotation(
      name = "line_video_frame",
      keyframe=True,
      frame=24,
      segment_index=1,
      value=lb_types.Line(
            points=[lb_types.Point(x=680, y=100), lb_types.Point(x=100, y=190)]
        )
    ),
    lb_types.VideoObjectAnnotation(
      name = "line_video_frame",
      keyframe=True,
      frame=45,
      segment_index=1,
      value=lb_types.Line(
            points=[lb_types.Point(x=680, y=100), lb_types.Point(x=100, y=190)]
        )
    )

  ]
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  polyline_frame_annotation_ndjson = {
    "name": "line_video_frame",
    "segments": [
        {
          "keyframes": [
            {
              "frame": 5,
              "line": [{
                "x": 680,
                "y": 100
              },{
                "x": 100,
                "y": 190
              },{
                "x": 190,
                "y": 220
              }]
            },
            {
              "frame": 20,
              "line": [{
                "x": 680,
                "y": 180
              },{
                "x": 100,
                "y": 200
              },{
                "x": 200,
                "y": 260
              }]
            }
          ]
        },
        {
          "keyframes": [
            {
              "frame": 24,
              "line": [{
                "x": 300,
                "y": 310
              },{
                "x": 330,
                "y": 430
              }]
            },
            {
              "frame": 45,
              "line": [{
                "x": 600,
                "y": 810
              },{
                "x": 900,
                "y": 930
              }]
            }
          ]
        }
      ]
  }
  ```
</CodeGroup>

### Masks (frame-based)

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Uploading single mask instances 
  ##  For this example we are going to to pass all the annotations payload in a single VideoMaskAnnotation (line 56) 

  # Single mask
  url = "https://storage.googleapis.com/labelbox-datasets/image_sample_data/frame_24_composite_mask.png"
  response = requests.get(url)
  img_bytes = base64.b64encode(response.content).decode('utf-8')

  # We are generating our frames and instances in this step, and will later add them to the VideoMaskAnnotation that will contain
  # all frames and instances
  frames_mask_single=[
      lb_types.MaskFrame(
          index=20,
          im_bytes=response.content # Instead of bytes you could also pass an instance URI : instance_uri=url
      )
  ]
  instances_mask_single=[
      lb_types.MaskInstance(color_rgb=(76, 104, 177), name= "video_mask")
  ]

  ####################################################################################################################################
  # Uploading annotations with composite masks 

  ## Add multiple masks using multiple tools in different frames - Note that only once composite mask can exist per frame
  frames_cp_mask_url = [
      {"1": "https://storage.googleapis.com/labelbox-datasets/image_sample_data/frame_1_composite_mask.png"},
      {"24": "https://storage.googleapis.com/labelbox-datasets/image_sample_data/frame_24_composite_mask.png"},
      {"26": "https://storage.googleapis.com/labelbox-datasets/image_sample_data/frame_26_composite_mask.png" }
  ]

  rgb_mask_tool = [(227, 135, 126) ,(169, 248, 152),(83, 152, 103)]
  cp_masks = []
  unique_colors = set()


  lb_frames = []
  lb_instances = []
  counter = 0

  for d in frames_cp_mask_url:
      for frame_no, v in d.items():
          response = requests.get(v)
          colors = extract_rgb_colors_from_url(v)
          for color in colors:
              if not color in unique_colors:
                  unique_colors.add(color)
                  name = "video_mask" if color in rgb_mask_tool else "mask_with_text_subclass"
                  lb_instances.append(lb_types.MaskInstance(color_rgb=color, name=name))
                  counter += 1
          lb_frames.append(
              lb_types.MaskFrame(
                  index=frame_no,
                  im_bytes=response.content
              )
          )
  cp_masks.append(lb_types.VideoMaskAnnotation(
      frames=lb_frames + frames_mask_single,
      instances=lb_instances + instances_mask_single
  ))

  pp.pprint(lb_frames)
  pp.pprint(cp_masks)
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  url = "https://storage.googleapis.com/labelbox-datasets/image_sample_data/color_mask.png"
  response = requests.get(url)
  img_bytes = base64.b64encode(response.content).decode('utf-8')

  # NDJSON - single tool
  video_mask_ndjson_bytes_2 = {
      'masks': {
        'frames': [
            {
                "index" : 31,
                "imBytes": img_bytes,
            },
            {
                "index" : 34,
                "imBytes": img_bytes,
            }
        ],
        'instances': [
            {
                "colorRGB" : [76, 104, 177],
                "name" : "video_mask"
            }
        ]
      }
   }
  ```
</CodeGroup>

### Bounding box with sub-classifications (frame-based)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  bbox_dm2 = {
    "top": 146.0,
    "left": 98.0,
    "height": 382.0,
    "width": 341.0
  }

  # Python Annotation
  frame_bbox_with_checklist_subclass_annotation = [
    lb_types.VideoObjectAnnotation(
      name = "bbox_class",
      keyframe=True,
      frame=10,
      segment_index=0,
      value = lb_types.Rectangle(
            start=lb_types.Point(x=bbox_dm2["left"], y=bbox_dm2["top"]), # x = left, y = top
            end=lb_types.Point(x=bbox_dm2["left"] + bbox_dm2["width"], y=bbox_dm2["top"] + bbox_dm2["height"]), # x= left + width , y = top + height
        )
    ),
    lb_types.VideoObjectAnnotation(
      name = "bbox_class",
      keyframe=True,
      frame=11,
      segment_index=0,
      value = lb_types.Rectangle(
            start=lb_types.Point(x=bbox_dm2["left"], y=bbox_dm2["top"]),
            end=lb_types.Point(x=bbox_dm2["left"] + bbox_dm2["width"], y=bbox_dm2["top"] + bbox_dm2["height"]),
        ),
      classifications=[
                  lb_types.ClassificationAnnotation(
                      name="checklist_class",
                      value=lb_types.Checklist(answer=[lb_types.ClassificationAnswer(
                          name="first_checklist_answer")])
                  )
              ]
    ),
    lb_types.VideoObjectAnnotation(
      name = "bbox_class",
      keyframe=True,
      frame=13,
      segment_index=0,
      value = lb_types.Rectangle(
            start=lb_types.Point(x=bbox_dm2["left"], y=bbox_dm2["top"]),
            end=lb_types.Point(x=bbox_dm2["left"] + bbox_dm2["width"], y=bbox_dm2["top"] + bbox_dm2["height"]),
        ),
      classifications=[
                lb_types.ClassificationAnnotation(
                    name="checklist_class",
                    value=lb_types.Checklist(answer=[lb_types.ClassificationAnswer(
                        name="second_checklist_answer")])
                )
            ]
    )
  ]
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  bbox_dm2 = {
    "top": 146.0,
    "left": 98.0,
    "height": 382.0,
    "width": 341.0
  }

  frame_bbox_with_checklist_subclass_annotation_ndjson = {
      "name": "bbox_class",
      "segments": [{
          "keyframes": [
              {
              "frame": 10,
              "bbox": bbox_dm2
            },
            {
            "frame": 11,
              "bbox": bbox_dm2,
              "classifications": [
                {
                  "name": "checklist_class",
                  "answer": [{"name": "first_checklist_answer"}]
                }
              ]
            },
            {
            "frame": 13,
              "bbox": bbox_dm2,
              "classifications": [
                {
                  "name": "checklist_class",
                  "answer": [{"name": "second_checklist_answer"}]
                }
              ]
            }
          ]
        }
      ]
  }
  ```
</CodeGroup>

### Multiple instances of bounding box annotations in the same frame

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}

  bbox_dm = {
    "top":617,
    "left":1371,
    "height":419,
    "width":505
  }

  # Fist instance of bounding box ranging from frame 22 to 27
  bbox_annotation_1 = [
    lb_types.VideoObjectAnnotation(
      name = "bbox_video",
      keyframe=True,
      frame=22,
      segment_index=0,
      value = lb_types.Rectangle(
            start=lb_types.Point(x=bbox_dm["left"], y=bbox_dm["top"]), # x = left, y = top
            end=lb_types.Point(x=bbox_dm["left"] + bbox_dm["width"], y=bbox_dm["top"] + bbox_dm["height"]), # x= left + width , y = top + height
        )
    ),
    lb_types.VideoObjectAnnotation(
      name = "bbox_video",
      keyframe=True,
      frame=27,
      segment_index=0,
      value = lb_types.Rectangle(
            start=lb_types.Point(x=bbox_dm["left"], y=bbox_dm["top"]),
            end=lb_types.Point(x=bbox_dm["left"] + bbox_dm["width"], y=bbox_dm["top"] + bbox_dm["height"]),
        )
    )
  ]

  # Second instance of bounding box ranging from frame 22 to 27
  bbox_annotation_2 = [
    lb_types.VideoObjectAnnotation(
      name = "bbox_video",
      keyframe=True,
      frame=22,
      segment_index=0,
      value = lb_types.Rectangle(
            start=lb_types.Point(x=bbox_dm2["left"], y=bbox_dm2["top"]),
            end=lb_types.Point(x=bbox_dm2["left"] + bbox_dm2["width"], y=bbox_dm2["top"] + bbox_dm2["height"]),
        )
    ),
    lb_types.VideoObjectAnnotation(
      name = "bbox_video",
      keyframe=True,
      frame=27,
      segment_index=0,
      value = lb_types.Rectangle(
            start=lb_types.Point(x=bbox_dm2["left"], y=bbox_dm2["top"]),
            end=lb_types.Point(x=bbox_dm2["left"] + bbox_dm2["width"], y=bbox_dm2["top"] + bbox_dm2["height"]),
        )
    )
  ]
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  bbox_dm = {
    "top":617,
    "left":1371,
    "height":419,
    "width":505
  }
  # Fist instance of bounding box ranging from frame 22 to 27
  bbox_frame_annotation_ndjson = {
      "name": "bbox_video",
      "segments": [{
          "keyframes": [
            {
              "frame": 22,
              "bbox":  bbox_dm
            },
            {
              "frame": 27,
              "bbox": bbox_dm2
            }

          ]
        }]
  }
  # Second instance of bounding box ranging from frame 22 to 27
  bbox_frame_annotation_ndjson2 = {
        "name": "bbox_video",
        "segments": [{
            "keyframes": [
              {
                "frame": 22,
                "bbox": bbox_dm
              },
              {
                "frame": 27,
                "bbox": bbox_dm2
              }
            ]
          }]
      }
  ```
</CodeGroup>

## Example: Import pre-labels or ground truths

The steps to import annotations as pre-labels (machine-assisted learning) are similar to those to import annotations as ground truth labels. However, they vary slightly, and we will describe the differences for each scenario.

### Before you start

The below imports are needed to use the code examples in this section.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  import labelbox as lb
  import labelbox.types as lb_types
  import uuid
  from PIL import Image
  import requests
  import base64
  from io import BytesIO
  ```
</CodeGroup>

Replace the value of `API_KEY` with a valid [API key](/horizon/developer-guides/create-api-key) to connect to the Labelbox client.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  API_KEY = None
  client = lb.Client(API_KEY)
  ```
</CodeGroup>

### Step 1: Import data rows

Data rows must first be uploaded to **Catalog** to attach annotations.

This example shows how to create a data row in **Catalog** by attaching it to a [dataset](/horizon/developer-guides/dataset) .

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  global_key = "sample-video-jellyfish.mp4"
  asset = {
      "row_data": "https://storage.googleapis.com/labelbox-datasets/video-sample-data/sample-video-2.mp4",
      "global_key": global_key,
      "media_type": "VIDEO"
   }

  dataset = client.create_dataset(
      name="video_demo_dataset",
      iam_integration=None # If this argument is removed, labelbox will use the default integration for your organization.
  )
  task = dataset.create_data_rows([asset])
  task.wait_till_done()
  print(f"Failed data rows: {task.failed_data_rows}")
  print(f"Errors: {task.errors}")

  if task.errors:
      for error in task.errors:
          if 'Duplicate global key' in error['message'] and dataset.row_count == 0:
              # If the global key already  exists in the workspace the dataset will be created empty, so we can delete it.
              print(f"Deleting empty dataset: {dataset}")
              dataset.delete()
  ```
</CodeGroup>

### Step 2: Set up ontology

Your project ontology should support the tools and classifications required by your annotations. To ensure accurate schema feature mapping, the value used as the `name` parameter should match the value of the `name` field in your annotation.

For example, when we created an annotation above, we provided a name`annotation_name`. Now, when we set up our ontology, we must ensure that the name of our bounding box tool is also `anotations_name`. The same alignment must hold true for the other tools and classifications we create in our ontology.

This example shows how to create an ontology containing all supported [annotation types](#supported-annotations) .

<CodeGroup>
  ```python Python expandable theme={"theme":"css-variables"}
  ontology_builder = lb.OntologyBuilder(
      tools=[
          lb.Tool(tool=lb.Tool.Type.BBOX, name="bbox_video"),
          lb.Tool(tool=lb.Tool.Type.POINT, name="point_video"),
          lb.Tool(tool=lb.Tool.Type.LINE, name="line_video_frame"),
          lb.Tool(tool=lb.Tool.Type.RASTER_SEGMENTATION, name="video_mask"),
          lb.Tool(
            tool=lb.Tool.Type.BBOX, name="bbox_class",
            classifications=[
              lb.Classification(
                class_type=lb.Classification.Type.RADIO,
                name="checklist_class",
                scope = lb.Classification.Scope.INDEX,
                options=[
                  lb.Option(value="first_checklist_answer"),
                  lb.Option(value="second_checklist_answer")
                ]
              )
            ]
          ),
        	lb.Tool(tool=lb.Tool.Type.RASTER_SEGMENTATION,
                  name="mask_with_text_subclass",
                  classifications=[
                      lb.Classification(
                          class_type=lb.Classification.Type.TEXT,
                          name="sub_free_text")
                      ]
                  )
      ],
      classifications=[
          lb.Classification(
              class_type=lb.Classification.Type.CHECKLIST,
              name="checklist_class",
              scope = lb.Classification.Scope.INDEX, ## Need to defined scope for frame classifications
              options=[
                  lb.Option(value="first_checklist_answer"),
                  lb.Option(value="second_checklist_answer")
              ]
          ),
          lb.Classification(
              class_type=lb.Classification.Type.RADIO,
              name="radio_class",
              scope = lb.Classification.Scope.INDEX,
              options=[
                  lb.Option(value="first_radio_answer"),
                  lb.Option(value="second_radio_answer")
              ]
          ),
           lb.Classification(
                class_type=lb.Classification.Type.RADIO,
                name="nested_radio_question",
                options=[
                    lb.Option("first_radio_answer",
                          options=[
                              lb.Classification(
                                  class_type=lb.Classification.Type.RADIO,
                                  name="sub_radio_question",
                                  options=[lb.Option("first_sub_radio_answer")]
                              )
                          ]
                    )
                ]
          ),
          lb.Classification(
            class_type=lb.Classification.Type.CHECKLIST,
            name="nested_checklist_question",
            options=[
                lb.Option("first_checklist_answer",
                  options=[
                    lb.Classification(
                        class_type=lb.Classification.Type.CHECKLIST,
                        name="sub_checklist_question",
                        options=[lb.Option("first_sub_checklist_answer")]
                    )
                ]
              )
            ]
          ),
          lb.Classification(
            class_type=lb.Classification.Type.RADIO,
            name="radio_class_global",
            options=[
                  lb.Option(value="first_radio_answer"),
                  lb.Option(value="second_radio_answer")
              ]
          ),
          lb.Classification(
            class_type=lb.Classification.Type.CHECKLIST,
            name="checklist_class_global",
            options=[
                  lb.Option(value="first_checklist_answer"),
                  lb.Option(value="second_checklist_answer")
            ]
          ),
          lb.Classification(
            class_type=lb.Classification.Type.TEXT,
            name="free_text"
          )
      ]
  )

  ontology = client.create_ontology("Video Annotation Import Demo Ontology",
                                    ontology_builder.asdict(),
                                    media_type=lb.MediaType.Video)
  ```
</CodeGroup>

### Step 3: Set Up a Labeling Project

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  project = client.create_project(name="Video Annotation Import Demo",
                                      media_type=lb.MediaType.Video)

  ## connect ontology to your project
  project.connect_ontology(ontology)
  ```
</CodeGroup>

### Step 4: Send Data Rows to Project

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  batch = project.create_batch(
    "first-batch-video-demo2", # Each batch in a project must have a unique name
    global_keys=[global_key], # A paginated collection of data row objects, a list of data rows or global keys
    priority=5 # priority between 1(Highest) - 5(lowest)
  )

  print("Batch: ", batch)
  ```
</CodeGroup>

### Step 5: Create annotation payloads

For help understanding annotation payloads, see [overview](#overview). To declare payloads, you can use Python annotation types (*preferred*) or NDJSON objects. For annotations that you want to import as ground truth labels, you can also specify [benchmarks](/horizon/guides/benchmark) using the `is_benchmark_reference` flag.

These examples demonstrate each format and how to compose annotations into labels attached to data rows.

<CodeGroup>
  ```python Python annotation payload theme={"theme":"css-variables"}
  label = []
  annotations_list = [
            checklist_annotation,
            radio_annotation,
            bbox_annotation,
            frame_bbox_with_checklist_subclass_annotation,
            bbox_annotation_1,
            bbox_annotation_2,
            point_annotation,
            polyline_annotation,
            global_checklist_annotation,
            global_radio_annotation,
            nested_checklist_annotation,
            nested_radio_annotation,
            text_annotation,
            cp_masks
        ]

  for annotation in annotations_list:
      label.append(
          lb_types.Label(
              data= {"global_key": global_key},
              annotations = annotation,
              # Optional: set the label as a benchmark
              # Only supported for groud truth imports
              is_benchmark_reference = True
          )
      )
  ```

  ```python NDJSON payload theme={"theme":"css-variables"}
  label_ndjson = []

  annotations_list_ndjson = [
      point_annotation_ndjson,
      bbox_annotation_ndjson,
      polyline_frame_annotation_ndjson,
      frame_checklist_classification_ndjson,
      frame_radio_classification_ndjson,
      frame_text_classification_ndjson,
      nested_radio_annotation_ndjson,
      nested_checklist_annotation_ndjson,
      frame_bbox_with_checklist_subclass_annotation_ndjson,
      global_radio_classification_ndjson,
      global_checklist_classification_ndjson,
      text_annotation_ndjson,
      bbox_frame_annotation_ndjson,
      bbox_frame_annotation_ndjson2,
      video_mask_ndjson_bytes_2
  ]

  for annotation in annotations_list_ndjson:
    annotation.update({
        "dataRow": {
            "globalKey": global_key
        }
    })
    label_ndjson.append(annotation)
  ```
</CodeGroup>

### Step 6: Import annotation payload

For prelabeled (model-assisted labeling) scenarios, pass your payload as the value of the `predictions` parameter. For ground truths, pass the payload to the `labels` parameter.

#### Option A: Upload as [prelabels (model assisted labeling)](/horizon/guides/model-assisted-labeling)

This option is helpful for speeding up the initial labeling process and reducing the manual labeling workload for high-volume datasets.

<CodeGroup>
  ```python MAL import theme={"theme":"css-variables"}
  # Upload MAL label for this data row in project
  upload_job = lb.MALPredictionImport.create_from_objects(
      client = client,
      project_id = project.uid,
      name="mal_job"+str(uuid.uuid4()),
      predictions=label
  )

  print(f"Errors: {upload_job.errors}", )
  print(f"Status of uploads: {upload_job.statuses}")
  ```
</CodeGroup>

#### Option B: Upload to a labeling project as [ground truth](/horizon/guides/import-ground-truth)

This option is helpful for loading high-confidence labels from another platform or previous projects that just need review rather than manual labeling effort.

<CodeGroup>
  ```python Label import theme={"theme":"css-variables"}
  # Upload label for this data row in project
  upload_job = lb.LabelImport.create_from_objects(
      client = client,
      project_id = project.uid,
      name="label_import_job"+str(uuid.uuid4()),
      labels=label
  )

  print(f"Errors: {upload_job.errors}", )
  print(f"Status of uploads: {upload_job.statuses}")
  ```
</CodeGroup>


# Import video data
Source: https://docs.labelbox.com/horizon/developer-guides/import-video-data

How to import video data and sample import formats.

## Specifications

Supported file format: MP4

Video encoding: H.264 (also referred to as AVC), H.265

Limits:

* Maximum file size: 1024 MB
* Maximum frame count: 20,000

Import methods:

* IAM Delegated Access
* Signed URLs (`https` URLs only)
* Direct upload of local files **Note**: Direct upload currently does not support adding additional metadata and attachments.

Videos have to pass the **frame-count consistency** check as well. Read below for details.

<Warning>
  ### Frame-Count Consistency

  Please note that the Labelbox Editor only supports videos which have **frame-count consistency**. This is achieved when

  `Frames Per Second (FPS) x Video Duration (Number of secs) == Total Number of Frames`

  While real-world videos may not always satisfy this requirement due to different encoding schemas, we warn you about this on upload with the message *"Unsupported video detected"* so you can proceed with caution when labeling.

  **Why do we check for this?** Because while labeling, every frame unaccounted for is a potential error.

  Some common examples where frame-count consistency is not achieved:

  * Variable frame rate
  * Negative frame presentation times (PTS)

  You can utilize the [FFmpeg](https://ffmpeg.org/) command line tool to process videos into a compatible format with the following command: `ffmpeg -i <input.mp4> -vcodec libx264 -acodec aac <output.mp4>`
</Warning>

<Danger>
  **Unsupported attributes**

  Please note that the Labelbox Editor does not support videos with:

  * Non-square sample aspect ratios (may appear stretched or squashed)

  Videos that contain these attributes will be flagged or blocked in the Editor, and will soon be flagged at the time of import as well.
</Danger>

When importing video data to Labelbox, your JSON file must include the following information for each video.

| Parameter | Required | Description |
| - | - | - |
| `row_data` | Yes | `https` path to a cloud-hosted video. For IAM Delegated Access, this URL must be in [virtual-hosted-style format](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#virtual-hosted-style-access). For older regions, your S3 bucket may be in `https://<bucket-name>.s3.<region>.amazonaws.com/<key>` format. If your object URLs are formatted this way, make sure they are in the virtual-hosted-style format before importing. |
| `global_key` | No | Unique user-generated file name or ID for the file. [Global keys](/horizon/developer-guides/data-row-global-keys) are enforced to be unique in your org. Data rows will not be imported if its global keys are duplicated to existing data rows. |
| `media_type` | No | `"VIDEO"` (optional media type to provide better validation and error messaging) |
| `metadata_fields` | No | See [Metadata](/horizon/guides/datarow-metadata). |
| `attachments` | No | See [Attachments](/horizon/guides/label-data) and [Asset overlays](/horizon/guides/label-data). |

## Import format

<CodeGroup>
  ```json Delegated Access URL theme={"theme":"css-variables"}
  [
    {
      "row_data": "https://lb-test-data.s3.us-west-1.amazonaws.com/video-samples/sample-video-1.mp4",
      "global_key": "https://lb-test-data.s3.us-west-1.amazonaws.com/video-samples/sample-video-1.mp4",
      "media_type": "VIDEO",
      "metadata_fields": [{"name": "<metadata_field_name>", "value": "tag_string"}],
      "attachments": [{"type": "VIDEO", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/drone_video.mp4" }]
    },
    {
      "row_data": "https://lb-test-data.s3.us-west-1.amazonaws.com/video-samples/sample-video-2.mp4",
      "global_key": "https://lb-test-data.s3.us-west-1.amazonaws.com/video-samples/sample-video-2.mp4",
      "media_type": "VIDEO",
      "metadata_fields": [{"name": "<metadata_field_name>", "value": "tag_string"}],
      "attachments": [{"type": "TEXT_URL", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt"}]
    }
  ]
  ```

  ```json Standard URL theme={"theme":"css-variables"}
  [
    {
      "row_data": "https://storage.googleapis.com/labelbox-datasets/video-sample-data/sample-video-1.mp4",
      "global_key": "https://storage.googleapis.com/labelbox-datasets/video-sample-data/sample-video-1.mp4",
      "media_type": "VIDEO",
      "metadata_fields": [{"schema_id": "cko8s9r5v0001h2dk9elqdidh", "value": "tag_string"}],
      "attachments": [{"type": "VIDEO", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/drone_video.mp4" }]
    },
    {
      "row_data": "https://storage.googleapis.com/labelbox-datasets/video-sample-data/sample-video-2.mp4",
      "global_key": "https://storage.googleapis.com/labelbox-datasets/video-sample-data/sample-video-2.mp4",
      "media_type": "VIDEO",
      "metadata_fields": [{"schema_id": "cko8s9r5v0001h2dk9elqdidh", "value": "tag_string"}],
      "attachments": [{"type": "TEXT_URL", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt"}]
    }
  ]
  ```
</CodeGroup>

## Python example

<CodeGroup>
  ```python bulk import example theme={"theme":"css-variables"}
  from labelbox import Client
  from uuid import uuid4 ## to generate unique IDs
  import datetime

  client = Client(api_key="<YOUR_API_KEY>")

  dataset = client.create_dataset(name="Bulk import example - Video")

  assets = [
    {
      "row_data": "https://storage.googleapis.com/labelbox-datasets/video-sample-data/sample-video-1.mp4",
      "global_key": "https://storage.googleapis.com/labelbox-datasets/video-sample-data/sample-video-1.mp44",
      "media_type": "VIDEO",
      "metadata_fields": [{"name": "<metadata_field_name>", "value": "tag_string"}],
      "attachments": [{"type": "VIDEO", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/drone_video.mp4" }]
    },
    {
      "row_data": "https://storage.googleapis.com/labelbox-datasets/video-sample-data/sample-video-2.mp4",
      "global_key": "https://storage.googleapis.com/labelbox-datasets/video-sample-data/sample-video-2.mp4",
      "media_type": "VIDEO",
      "metadata_fields": [{"name": "<metadata_field_name>", "value": "tag_string"}],
      "attachments": [{"type": "TEXT_URL", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt"}]
    }
  ]

  task = dataset.create_data_rows(assets)
  task.wait_till_done()
  print(task.errors)
  ```

  ```python local files theme={"theme":"css-variables"}
  local_file_paths = ['path/to/local/file1', 'path/to/local/file1'] # limit: 15k files

  new_dataset = client.create_dataset(name = "Local files upload")

  try:
      task = new_dataset.create_data_rows(local_file_paths)
      task.wait_till_done()
  except Exception as err:
      print(f'Error while creating labelbox dataset -  Error: {err}')
  ```
</CodeGroup>


# Installation
Source: https://docs.labelbox.com/horizon/developer-guides/install-python-sdk

Describes how to install, update, and authenticate to the Labelbox Python SDK.

Many actions and workflows available in the Labelbox app can be automated using the [Python SDK](https://github.com/Labelbox/labelbox-python/), which provides access to the [Labelbox API](https://labelbox-python.readthedocs.io/en/latest/).

## Requirements

* Make sure `pip` is installed and up-to-date.

* Use the **Workspace settings** menu of the Labelbox app to [create an API key](/horizon/developer-guides/create-api-key). Store your key in a safe place, as you cannot view it after it has been created.

* Python: 3.9 - 3.13

## Install or upgrade SDK

After you [create an API key](/horizon/developer-guides/create-api-key), you can start using our Python SDK.

To install, run either `pip install labelbox` or`pip install "labelbox[data]"` in your command line.

<CodeGroup>
  ```shell shell theme={"theme":"css-variables"}
  pip install "labelbox[data]"
  # or
  pip install labelbox
  ```
</CodeGroup>

| Module | Description |
| - | - |
| `"labelbox[data]"` | Installs all [required dependencies](https://github.com/Labelbox/labelbox-python/blob/develop/requirements.lock) (libraries, tools to manipulate annotations, and more.) |
| `labelbox` | Installs all required libraries for client-only functionality |

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # install latest labelbox version
  !pip install labelbox

  import labelbox as lb

  # Enter your Labelbox API key here
  LB_API_KEY = None

  # Create Labelbox client
  client = lb.Client(api_key=LB_API_KEY)
  ```
</CodeGroup>

To upgrade to the latest version, use your package manager: `pip install --upgrade labelbox`

To run the Python SDK in a Docker container, use the [included Dockerfile](https://github.com/Labelbox/labelbox-python/blob/develop/libs/labelbox/Dockerfile) as a starting place.

Related info:

* The Python SDK [GitHub repository](https://github.com/Labelbox/labelbox-python/)
* The most recent update to the Python SDK can be found at: [`https://github.com/Labelbox/labelbox-python/releases/latest/`](https://github.com/Labelbox/labelbox-python/releases/latest)
* [Changelog](https://github.com/Labelbox/labelbox-python/blob/develop/libs/labelbox/CHANGELOG.md) of recent changes.

## Authentication

You need an [API key](/horizon/developer-guides/create-api-key) to authenticate your client. You can:

* Save your key to an environment variable:

  <CodeGroup>
    ```shell shell theme={"theme":"css-variables"}
    user@machine:~$ export LABELBOX_API_KEY="<your_api_key>"
    user@machine:~$ python3
    ```
  </CodeGroup>

  Then, import and initialize the API Client.

  <CodeGroup>
    ```python Python theme={"theme":"css-variables"}
    import labelbox as lb

    client = lb.Client()
    ```
  </CodeGroup>

* Paste your API key directly into your script (*not recommended*)

  <CodeGroup>
    ```python Python theme={"theme":"css-variables"}
    import labelbox as lb

    if __name__ == '__main__':
        API_KEY = "<your_api_key_here>"
        client = lb.Client(API_KEY)
    ```
  </CodeGroup>

Remember that your API key authorizes access to Labelbox, your account, and your data assets; treat it as carefully as any other set of credentials, and make sure the [scope of the key](/horizon/developer-guides/create-api-key) is set according to the usage is meant for.


# Issues & comments
Source: https://docs.labelbox.com/horizon/developer-guides/issues

Developer guide for creating and modifying issues and comments via the Python SDK.

## Issue categories

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  project = client.get_project("<PROJECT_ID>")

  # Create
  category = project.create_issue_category(
      name="Quality",
      description="Quality-related issues",
  )

  # List all
  categories = project.get_issue_categories()

  # Update
  category = category.update(name="Renamed", description="New description")

  # Delete
  category.delete()
  ```
</CodeGroup>

## Issues

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  from labelbox import IssueStatus, ImageIssuePosition, VideoIssuePosition, VideoFrameRange

  # Create -- minimal
  issue = project.create_issue(
      content="General quality issue",
      data_row_id="<DATA_ROW_ID>",
  )

  # Create -- with label, category, and position
  issue = project.create_issue(
      content="Bounding box is misaligned",
      data_row_id="<DATA_ROW_ID>",
      label_id="<LABEL_ID>",
      category_id=category.id,
      position=ImageIssuePosition(x=100, y=200),
  )

  # Also accepts DataRow/Label/IssueCategory objects directly
  issue = project.create_issue(
      content="Using objects instead of IDs",
      data_row_id=data_row,       # DataRow object
      label_id=label,             # Label object
      category_id=category,       # IssueCategory object
  )

  # Fetch with filters (returns PaginatedCollection)
  for issue in project.get_issues(status=IssueStatus.OPEN):
      print(issue.friendly_id, issue.content)

  # Other available filters
  project.get_issues(
      data_row_id="<DATA_ROW_ID>",
      category_id="<CATEGORY_ID>",
      created_by_ids=["<USER_ID>"],
      content="search text",
  )

  # Fetch single issue
  issue = project.get_issue("<ISSUE_ID>")

  # Update (only provided fields are changed)
  issue = issue.update(content="Updated text")
  issue = issue.update(category_id="<NEW_CATEGORY_ID>")

  # Resolve / Reopen
  issue = issue.resolve()
  issue = issue.reopen()

  # Lazy-loaded related objects (each makes an API call)
  data_row = issue.data_row()     # Optional[DataRow]
  label = issue.label()           # Optional[Label]
  category = issue.category()     # Optional[IssueCategory]

  # Delete single
  issue.delete()

  # Bulk delete
  project.delete_issues(["<ISSUE_ID_1>", "<ISSUE_ID_2>"])
  ```
</CodeGroup>

## Comments

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Create
  comment = issue.create_comment(content="I agree, this needs fixing")

  # List all comments on an issue
  comments = issue.comments()

  # Update
  comment = comment.update(content="Revised comment")

  # Delete
  comment.delete()
  ```
</CodeGroup>

## Video position variants

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Single frame
  VideoIssuePosition(frames=[
      VideoFrameRange(start=5, end=5, x=100, y=200)
  ])

  # Contiguous range (stationary pin)
  VideoIssuePosition(frames=[
      VideoFrameRange(start=5, end=11, x=450, y=300)
  ])

  # Contiguous range (moving pin)
  VideoIssuePosition(frames=[
      VideoFrameRange(start=5, end=11, x=450, y=300, end_x=500, end_y=350)
  ])

  # Multiple separated ranges
  VideoIssuePosition(frames=[
      VideoFrameRange(start=5, end=11, x=450, y=300),
      VideoFrameRange(start=20, end=25, x=100, y=100),
  ])
  ```
</CodeGroup>

## Addendum: Position formats for different data types

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # PDF (%)
  PdfIssuePosition(x=0.5, y=0.75, page=2)

  # Text (Char range)
  TextIssuePosition(text_block_id="b1", start_char_index=10, end_char_index=25)

  # Video (Multiple frames/paths)
  VideoIssuePosition(frames=[
      VideoFrameRange(start=5, end=11, x=450, y=300, end_x=500, end_y=350),
      VideoFrameRange(start=20, end=25, x=100, y=100)
  ])
  ```
</CodeGroup>


# Labeling Workforce
Source: https://docs.labelbox.com/horizon/developer-guides/labeling-workforce

Python SDK methods for requesting and managing your labeling workforce.

You can request labeling service for your Annotate projects and manage the requested labeling workforce using the Labelbox Python SDK.

<Info>
  ### Experimental feature

  This is an [experimental](/horizon/guides/product-release-phases#experimental) feature that requires you to enable experimental features before using its methods.
</Info>

# Methods

## Get and request labeling service

Use `project.labeling_service = project.get_labeling_service()` to get the labeling service associated with the project. Then use `labeling_service.request()` to request a workforce of trained labelers for your data.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Enable experimental features
  client.enable_experimental = True

  # Get labeling service
  labeling_service = project.get_labeling_service()

  # Validate all project requirements and request labeling service
  labeling_service.request()
  ```
</CodeGroup>

## Get labeling service status

Use `project.get_labeling_service_status()` to check the status of your requested labeling service.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Enable experimental features
  client.enable_experimental = True

  # Get status of requested labeling service
  project.get_labeling_service_status()
  ```
</CodeGroup>

This method returns the following value defined in the `LabelingServiceStatus` enum, corresponding to the statuses of your requested labeling service:

* `SET_UP`: Your request has been submitted, but you need to set up your request by adding data rows, ontologies, and labeling instructions to proceed the request.
* `REQUESTED`: Your request has been validated and is pending approval for connecting to labeling workforce.
* `ACCEPTED`: Your request has satisfied the prerequisite and been accepted.
* `CALIBRATION`: Labelers are labeling a sample of your data rows that you can review and confirm alignment on the expectation.
* `PRODUCTION`: Labelers are labeling all data rows.
* `COMPLETE`: Labelers have finished adding labels to your projects.

## Get labeling service details

Use `project.get_labeling_service_dashboard()` to retrieve the following attributes of the project for which you requested labeling service:

* `id`: The project ID
* `name`: The project name `status`: The [status](#get-labeling-service-status) of the labeling service
* `data_rows_count`: The total number of data rows batched in the project
* `tasks_completed_count`: The number of completed tasks (with the `COMPLETE` status)
* `tasks_remaining_count`: The number of incomplete tasks (queued data rows with statuses other than `COMPLETE`)
* `tags`: [Project tags](/horizon/guides/project-tagging) associated with the project
* `media_type`: The [media type](/horizon/developer-guides/project#create-a-project) of the project
* client: The Labelbox client

If you are an admin, you can also use `client.get_labeling_service_dashboards()` to retrieve details of all projects with labeling service requested in your workspace.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Enable experimental features
  client.enable_experimental = True

  # Get details of a project with labeling service requested
  project.get_labeling_service_dashboard()

  # Get details of all projects with labeling service requested
  client.get_labeling_service_dashboards()
  ```
</CodeGroup>


# Media attributes
Source: https://docs.labelbox.com/horizon/developer-guides/media-attributes

Overview of Labelbox computed media attributes.

Labelbox automatically computes media attributes when you create new data rows in the platform. If the data is not accessible when you create the data rows, the media attributes will not be computed for those data rows.

Below is a list of the available media attributes.

| Parameter | Asset type | Description |
| - | - | - |
| `mimeType` | Video Image Text Tiled imagery | Media type. Will be one of the following: - `video/mp4` - `image/jpeg` - `text/plain` - `application/x-tms-geo` (tiled imagery: EPSG3857) - `application/x-tms-simple` (tiled imagery: Simple) - `application/json` (other) |
| `width` | Video Image | Image or frame width in pixels. |
| `height` | Video Image | Image or frame height in pixels. |
| `codec` | Video | Indicates the video compression standard. |
| `duration` | Video | Duration of the video in seconds. |
| `frameRate` | Video | Frame rate per second. |
| `frameCount` | Video | Number of frames in the video. |

Use this sample script to retrieve the media attributes for a data row.

<CodeGroup>
  ```python Get media attributes theme={"theme":"css-variables"}
  datarow = client.get_data_row(data_row.uid)
  print(datarow.media_attributes)
  ```
</CodeGroup>

Below are some sample responses by asset type.

<CodeGroup>
  ```json Image theme={"theme":"css-variables"}
  {
    "data": {
      "dataRow": {
        "mediaAttributes": {
          "width": 3872,
          "height": 2592,
          "mimeType": "image/jpeg"
        }
     }
  }
  ```

  ```json Video theme={"theme":"css-variables"}
  {
    "data": {
      "dataRow": {
        "mediaAttributes": {
          "codec": "avc1",
          "width": 1920,
          "height": 1080,
          "duration": 10,
          "mimeType": "video/mp4",
          "frameRate": 10,
          "frameCount": 100
        }
     }
  }
  ```

  ```json Text theme={"theme":"css-variables"}
  {
    "data": {
      "dataRow": {
        "mediaAttributes": {
          "mimeType": "text/plain"
        }
     }
  }
  ```

  ```json Tiled imagery theme={"theme":"css-variables"}
  {
    "data": {
      "dataRow": {
        "mediaAttributes": {
          "mimeType": "application/x-tms-geo"
        }
     }
  }
  ```
</CodeGroup>


# Metadata
Source: https://docs.labelbox.com/horizon/developer-guides/metadata

Developer guide for creating, importing, exporting, and modifying metadata fields via the Python SDK.

<CardGroup>
  <Card title="Open in Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/basics/data_row_metadata.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/basics/data_row_metadata.ipynb" />
</CardGroup>

## Metadata types

All metadata needs to be one of the following types:

| Type | Descriptions | Enum value |
| - | - | - |
| `DateTime` | An ISO 8601 datetime field. All times must be in UTC timezone | `DataRowMetadataKind.datetime` |
| `Number` | Floating-point value (max: 64-bit float) | `DataRowMetadataKind.number` |
| `String` | Free text field. Max 4,096 characters. | `DataRowMetadataKind.string` |
| `Enum` | Enum field with options. Multiple options can be imported. | `DataRowMetadataKind.enum` |
| `Option (Enum)` | Option of an enum. Max 64 options can be created per Enum type. 128 for enterprise customers and can be further increased upon request. | `DataRowMetadataKind.enumoption` |
| `Embedding` | 128 float 32 vector used for similarity | `DataRowMetadataKind.embedding` |

## Reserved fields

The following field names are reserved and cannot be used as custom metadata field names:

| Name | Type | Description |
| - | - | - |
| `tag` | `String` | The tags of the data row |
| `split` | `Enum` | The split of the dataset that the data row belongs, including`train`, `valid`, and `test` |
| `captureDateTime` | `DateTime` | The timestamp when the data is captured |
| `skipNFrames` | `Number` | (Video data only) The number of frames to skip |
| `turnInstructions` | `String` | JSON string that contains instructions for each turn in a Multi-modal chat conversation. |

## Construct metadata fields

To construct a metadata field, you must provide the Schema ID for the field and the value that will be uploaded. You can do this in two ways:

* Option 1: Specify the metadata using the *DataRowMetadataField* object (comes with validation for metadata fields)
* Option 2: Specify the metadata fields in *dictionary* format without declaring the DataRowMetadataField objects

The Metadata `value` attribute should not be `null`.

<CodeGroup>
  ```python DataRowMetadataField object theme={"theme":"css-variables"}
  metadata_fields = []

  ## Construct a metadata field of string kind

  tag_schema = metadata_ontology.get_by_name("tag")
  tag_metadata_field = DataRowMetadataField(
  schema_id=tag_schema.uid, # specify the schema id
  value="tag_string", # typed inputs
  )
  metadata_fields.append(tag_metadata_field)

  # Construct an metadata field of datetime

  datetime_schema = metadata_ontology.get_by_name("captureDateTime")
  capture_datetime_field = DataRowMetadataField(
  name=datetime_schema.name, # specify the schema id
  value=datetime.datetime.utcnow(), # typed inputs
  )
  metadata_fields.append(capture_datetime_field)

  # # Construct a metadata field of Enums options. You can import multiple options.

  test_schema = metadata_ontology.get_by_name("split")["test"]
  test_schema_field = DataRowMetadataField(
  schema_id=test_schema.parent, # specify the schema id
  value=test_schema.uid, # typed inputs
  )
  metadata_fields.append(test_schema_field)

  valid_schema = metadata_ontology.get_by_name("split")["valid"]
  valid_schema_field = DataRowMetadataField(
  schema_id=valid_schema.parent, # specify the schema id
  value=valid_schema.uid, # typed inputs
  )
  metadata_fields.append(valid_schema_field)
  ```

  ```python Dictionary theme={"theme":"css-variables"}
  metadata_fields = []

  ## Construct a metadata field of string kind
  tag_schema = metadata_ontology.get_by_name("tag")
  metadata_fields.append({"name": tag_schema.name, "value": "tag_value"})

  # Construct an metadata field of datetime
  datetime_schema = metadata_ontology.get_by_name("captureDateTime")
  metadata_fields.append({"name": datetime_schema.name, "value": datetime.datetime.utcnow()})

  ## Construct a metadata field of Enums options. You can import multiple options.
  train_schema = metadata_ontology.get_by_name("split")["valid"]
  metadata_fields.append({"schema_id": train_schema.parent, "value": train_schema.uid})

  test_schema = metadata_ontology.get_by_name("split")["test"]
  metadata_fields.append({"schema_id": test_schema.parent, "value": test_schema.uid})
  ```
</CodeGroup>

## Create custom metadata schema

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  import labelbox
  from labelbox.schema.data_row_metadata import DataRowMetadataKind

  client = labelbox.Client(api_key="LABELBOX_API_KEY")
  metadata_ontology = client.get_data_row_metadata_ontology()

  # create a custom metadata schema (set the kind value to a supported data type)

  metadata_schema = metadata_ontology.create_schema(name="metadata_name", kind=DataRowMetadataKind.string)

  # get the schema id

  schema_id = metadata_schema.uid
  ```
</CodeGroup>

## Upload data rows with metadata

Custom metadata field limits vary according to your subscription; for details, see [Limits](/horizon/guides/limits).

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  data_row = {
    "row_data": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/basic.jpg",
    "global_key": "metadata_tutorial",
    "metadata_fields": metadata_fields
  }

  dataset = client.create_dataset(name="Create data row with metadata")
  task = dataset.create_data_rows([data_row])
  task.wait_till_done()
  ```
</CodeGroup>

## Get metadata schema

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # you can look up a schema by name.
  metadata_schema = metadata_ontology.get_by_name("tag")
  metadata_schema = metadata_ontology.get_by_name("enum_metadata_name")

  # check the schema

  print(metadata_schema)
  schema_id = metadata_schema.uid
  ```
</CodeGroup>

## Get metadata fields (ontology)

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  ## Fetch metadata schema ontology. A Labelbox workspace has a single metadata ontology.
  metadata_ontology = client.get_data_row_metadata_ontology()

  # List all available fields
  metadata_ontology.fields
  ```
</CodeGroup>

## Get metadata fields

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  datarow = next(dataset.data_rows())
  for metadata_field in datarow.metadata_fields:
    print(metadata_field['name'], ":", metadata_field['value'])
  ```
</CodeGroup>

Result:

<CodeGroup>
  ```json Output theme={"theme":"css-variables"}
  tag : custom_tag
  split : train
  captureDateTime : 2023-04-04T15:24:37.229417Z
  ```
</CodeGroup>

## Bulk export data rows with metadata

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  export_params= {
    "performance_details": True,
    "label_details": True,
    "metadata_fields": True
  }

  export_task = dataset.export(params=export_params)
  export_task.wait_till_done()

  # Stream the export using a callback function

  def json_stream_handler(output: labelbox.BufferedJsonConverterOutput):
    print(output.json)

  export_task.get_buffered_stream(stream_type=labelbox.StreamType.RESULT).start(stream_handler=json_stream_handler)

  # Collect all exported data into a list

  export_json = [data_row.json for data_row in export_task.get_buffered_stream()]
  ```
</CodeGroup>

Result:

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "data_row": {
      "id": "clflxqzty07fj077qa3dd4v27",
      "global_key": "metadata_tutorial",
      "row_data": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/basic.jpg"
    },
    "media_attributes": {
      "height": 1285,
      "width": 2258,
      "mime_type": "image/jpeg"
    },
    "metadata_fields": [{
      "schema_id": "cko8s9r5v0001h2dk9elqdidh",
      "schema_name": "tag",
      "value": "tag_string"
    }, {
      "schema_id": "cko8sbczn0002h2dkdaxb5kal",
      "schema_name": "split",
      "value": [{
        "schema_id": "cko8sc2yr0004h2dk69aj5x63",
        "schema_name": "valid"
      }, {
        "schema_id": "cko8scbz70005h2dkastwhgqt",
        "schema_name": "test"
      }]
    }, {
      "schema_id": "cko8sdzv70006h2dk8jg64zvb",
      "schema_name": "captureDateTime",
      "value": "2023-03-24T02:40:40.832576+00:00"
    }]
  }
  ```
</CodeGroup>

## Export metadata by data row ID

You can bulk export metadata by data row with the SDK.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  data_row_ids = ['<data_row_id>']
  global_keys = ['<global_key>']

  #The data row identifiers methods (lb.DataRowIds and lb.GlobalKeys) validate whether the provided ID is a global key or a data row ID.
  #Additionally, they ensure that all IDs from the list provided are unique

  datarow_identifiers = lb.DataRowIds(data_row_ids)
  global_key_identifiers = lb.GlobalKeys(global_keys)

  # Use one of the identifiers

  mdo.bulk_export(data_row_ids=global_key_identifiers)

  # mdo.bulk_export(data_row_ids=datarow_identifiers)
  ```
</CodeGroup>

## Delete metadata fields from a data row

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  global_key = '<global_key>'
  schema_ids_to_delete =['<metadata_schema_id>']
  data_row_id = '<data_row_id>'

  deletions = [
      lb.DeleteDataRowMetadata(data_row_id=lb.GlobalKey(global_key), fields=schema_ids_to_delete)
      ]

  # Delete the specified metadata on the data row
  mdo.bulk_delete(deletes=deletions)
  ```
</CodeGroup>

## Upsert metadata to existing data rows

Labelbox supports individual or bulk metadata upsert of data rows. Metadata overwrites occur on a per-field basis.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  tag_schema = metadata_ontology.get_by_name("tag")

  # Construct a string field

  field = DataRowMetadataField(
  schema_id=tag_schema.uid, # specify the schema id
  value="updated", # typed inputs
  )

  # Completed object ready for import

  metadata_payload = DataRowMetadata(
  global_key="<global key>", # optionally, set the argument to data_row_id to use a data row ID
  fields=[field]
  )

  # Provide a list of DataRowMetadata objects to upload

  metadata_ontology.bulk_upsert([metadata_payload])
  ```
</CodeGroup>

## Update metadata schema

You can update any custom metadata schema's name. However, the type cannot be modified. You also cannot modify the names of reserved fields.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # update a metadata schema's name
  metadata_schema = metadata_ontology.update_schema(name="metadata_name", new_name="metadata_name_updated")

  # Enum metadata schema is a bit different since it contains options.
  # create an Enum metadata with options
  enum_schema = metadata_ontology.create_schema(name="enum_metadata_name", kind=DataRowMetadataKind.enum,
                                              options=["option 1", "option 2"])

  # update an Enum metadata schema's name, similar to other metadata schema types
  enum_schema = metadata_ontology.update_schema(name="enum_metadata_name", new_name="enum_metadata_name_updated")

  # update an Enum metadata schema option's name, this only applies to Enum metadata schema.
  enum_schema = metadata_ontology.update_enum_option(name="enum_metadata_name_option_updated", option="option 1",
                                                   new_option="option 3")
  ```
</CodeGroup>

## Delete metadata schema

You can delete a metadata schema by name.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  status = metadata_ontology.delete_schema(name=metadata_schema.name)
  # returns True if successfully deleted
  ```
</CodeGroup>


# Model
Source: https://docs.labelbox.com/horizon/developer-guides/model

A developer guide for creating and managing model experiments.

## Model

An *experiment* is a container in Labelbox that houses all of the information related to the iterative development of a specific model. It contains the data rows for training, model error analysis metrics, model versioning, and versioned snapshots (model runs) of data rows, predictions, etc, associated with a model’s development.

Experiments are designed to help you track and compare all of the iterations associated with your model development.

### Create a model

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  model = client.create_model(
    name="<model_name>",
    ontology_id="<ontology_id>"
  )
  ```
</CodeGroup>

### Get a model

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  model = client.get_model("<model_id>")

  # from labelbox import Model
  models = client.get_models(where=(Model.name == "<model_name>"))
  ```
</CodeGroup>

## Methods

### Create a model run

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  model.create_model_run(name="<model_run_name>"

  # optionally, you can supply details of the model run config
  model.create_model_run(
    name="<model_run_with_config>",
    config={
      "learning_rate": 0.001,
      "batch_size": 32
    }
  )
  ```
</CodeGroup>

### Delete a model, and its model runs

Deleting a model also deletes its model runs.

This action is permanent; it cannot be undone or rolled back.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  model.delete()
  ```
</CodeGroup>

## Attributes

### Get the basics

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # name (str)
  model.name
  ```
</CodeGroup>

### Get the model runs

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # get the model runs (relationship to ModelRun objects)
  model_runs = model.model_runs()

  # inspect one model run
  next(model_runs)

  # inspect all model runs
  for model_run in model_runs:
    print(model_run)
  ```
</CodeGroup>

# Model run

A *model run* represents a single iteration within a model training experiment. Each model run contains a versioned snapshot of the data rows, annotations (predictions and/or ground truth), and [data splits](/horizon/guides/splits) for each iteration within a model training experiment.

Model runs make it easy for you to reproduce a model training experiment using different parameters. You can also use model runs to track and compare model runs trained on different data versions.

## Get all model runs inside a Model

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  model_runs = model.model_runs()
  ```
</CodeGroup>

## Create a model run

Creates a model run belonging to this model.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  model_run_name = "<your_model_run_name>"
  example_config = {
      "learning_rate": 0.001,
      "batch_size": 32,
  }
  model_run = model.create_model_run(name=model_run_name, config=example_config)
  ```
</CodeGroup>

## Get model run

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  model_run_id = "<your_model_run_id>"
  model_run = client.get_model_run(model_run_id=model_run_id)

  model_run_data = model_run.model_run_data_rows()
  model_run_config = model_run.get_config())
  ```
</CodeGroup>

## Add data rows to a model run

Add data rows to a model run **without** any associated labels. You can use either `data_row_id` or `global_key` to specify the data rows.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Using data row ids
  data_row_ids = ["<data_row_id_1>", "<data_row_id_2>"]
  model_run.upsert_data_rows(data_row_ids=data_row_ids)

  # Using global keys
  global_keys = ["<global_key1>", "<global_key_2>"]
  model_run.upsert_data_rows(global_keys=global_keys)
  ```
</CodeGroup>

## Assign data row training, validation, and test split

Note that `assign_data_rows_to_split` only works on data rows or labels that are already in a model run. You can assign them to one of "TRAINING", "VALIDATION", "TEST" split.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  client.enable_experimental=True

  dataset = client.get_dataset("<Dataset_id>") # Your training dataset

  # using data row ids
  model_run.assign_data_rows_to_split(
    data_row_ids=data_row_ids[:100],
    split="TRAINING",
  )
  model_run.assign_data_rows_to_split(
    data_row_ids=data_row_ids[100:150],
    split="VALIDATION",
  )
  model_run.assign_data_rows_to_split(
    data_row_ids=data_row_ids[150:200],
    split="TEST",
  )

  # using global keys
  model_run.assign_data_rows_to_split(
    global_keys=global_keys[:100],
    split="TRAINING",
  )
  model_run.assign_data_rows_to_split(
    global_keys=global_keys[100:150],
    split="VALIDATION",
  )
  model_run.assign_data_rows_to_split(
    global_keys=global_keys[150:200],
    split="TEST",
  )
  ```
</CodeGroup>

## Add labels to a model run

Adds data rows and labels to a model run. By adding labels, the associated data rows will also be upserted to the model run.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # upsert using label ids
  label_ids = ["<label_id_1>","<label_id_2>", ...]
  model_run.upsert_labels(label_ids)
  ```
</CodeGroup>

Alternatively, you can add all labels from a project to a Model run directly. This will also add all data rows from that project to the model run.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # upsert using project id
  model_run.upsert_labels(project_id=<project_id>)
  ```
</CodeGroup>

## Export labels from a model run

See [Export for Model Runs](/horizon/guides/export-data-for-model-training) for more details and export JSON format.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Set the export params to include/exclude certain fields. Make sure each of these fields are correctly grabbed
  export_params= {
      "attachments": True,
      "metadata_fields": True,
      "data_row_details": True,
  }

  export_task = model_run.export(params=export_params)
  export_task.wait_till_done()

  # Stream the export using a callback function
  def json_stream_handler(output: labelbox.BufferedJsonConverterOutput):
    print(output.json)

  export_task.get_buffered_stream(stream_type=labelbox.StreamType.RESULT).start(stream_handler=json_stream_handler)

  # Collect all exported data into a list
  export_json = [data_row.json for data_row in export_task.get_buffered_stream()]
  ```
</CodeGroup>

## Create, modify, and delete model run config to track hyperparameters.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  example_config = {
      "learning_rate": 0.001,
      "checkpoint_path": "/path/to/checkpoint/file",
      "early_stopping": False,
      "batch_size": 32,
      "optimizer": {
        "adam": {
          "beta1": 0.899999976158,
          "beta2": 0.999000012875,
          "epsilon": 9.99999993923e-9
        }
      },
      "ngpu": 1,
      }

  model_run_1 = model.create_model_run(name="run 1", config=example_config)
  # You can also create a model with config specified, see above.
  # Here is how to create a model run first and update the model config field.
  model_run_2 = model.create_model_run(name="run 2")
  #The update will repace the previous model run config with the new json input.
  model_run_2.update_config(example_config)
  ```
</CodeGroup>

### Get model run config

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  model_run_parameters = model_run.get_config()
  ```
</CodeGroup>

### Delete the model run config

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  model_run.reset_config()
  ```
</CodeGroup>

### Delete data rows from a model run

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  data_row_ids = ["<data_row_id_1>","<data_row_id_2>", ...]
  model_run.delete_model_run_data_rows(data_row_ids=data_row_ids)
  ```
</CodeGroup>

### Delete model run

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  model_run.delete()
  ```
</CodeGroup>


# Model run custom metrics
Source: https://docs.labelbox.com/horizon/developer-guides/model-run-custom-metrics

Developer guide on how to import custom model run metrics

<CardGroup>
  <Card title="Open in Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/model_experiments/custom_metrics_demo.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/model_experiments/custom_metrics_demo.ipynb" />
</CardGroup>

## Upload custom metrics

If the auto-generated metrics are not sufficient for your use case, you can upload custom metrics to your model run. This will help you even more precisely evaluate your model performance in Labelbox.

### Upload custom metrics to individual prediction annotations.

To upload custom metrics to individual predictions, you can append the following list of dictionaries to the respective prediction. The custom metric fields are supported for all annotation types except raster segmentation.

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  # Include the following list of dictionaries in your prediction annotation built with python classes
  custom_metrics = [
    { 'name': 'iou', 'value': 0.5 },
    { 'name': 'f1', 'value': 0.33 },
    { 'name': 'precision', 'value': 0.55 },
    { 'name': 'recall', 'value': 0.33 },
    { 'name': 'tagsCount', 'value': 43 },
    { 'name': 'metric_with_a_very_long_name', 'value': 0.334332 }
  ]
  ```

  ```python NDJSON annotation theme={"theme":"css-variables"}
  # Include the following list of dictionaries in your prediction annotation built with ndjson
  'customMetrics': [
    { 'name': 'iou', 'value': 0.5 },
    { 'name': 'f1', 'value': 0.33 },
    { 'name': 'precision', 'value': 0.55 },
    { 'name': 'recall', 'value': 0.33 },
    { 'name': 'tagsCount', 'value': 43 },
    { 'name': 'metric_with_a_very_long_name', 'value': 0.334332 }
  ]
  ```
</CodeGroup>

### Scalar custom metrics

A `ScalarMetric` is a custom metric with a single scalar value. It can be uploaded at the following levels of granularity:

<CardGroup>
  <Card title="Data rows" icon="square-1" />

  <Card title="Features" icon="square-2" />

  <Card title="Nested features" icon="square-3" />
</CardGroup>

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  from labelbox.data.annotation_types import (ScalarMetric,
                                              ScalarMetricAggregation,
                                              ConfusionMatrixMetric)
  ```
</CodeGroup>

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # custom metric on a data row
  data_row_metric = ScalarMetric(metric_name="iou_custom", value=0.5)

  # custom metric on a feature

  feature_metric = ScalarMetric(metric_name="iou_custom", feature_name="cat", value=0.5)

  # custom metric on a nested feature

  subclass_metric = ScalarMetric(metric_name="iou_custom",
  feature_name="cat",
  subclass_name="orange",
  value=0.5)

  ```
</CodeGroup>

### Aggregation of custom metrics

This is an optional field on the `ScalarMetric` object, to control how custom metrics are aggergated. By default, the aggregation uses `ARITHMETIC_MEAN`.

Aggregations occur in the following cases:

* When you provide a feature or nested-feature metric, Labelbox automatically aggregates the metric across features and nested-features on the data row. *For example, say you provide a custom metric Bounding Box Width (BBW) on the features "cat" and "dog" . The data row-level metric for BBW is the average of these two values.*
* When you create slices, the custom metric is aggregated across data rows of the Slice.
* When you filter data inside a Model Run, the custom metric is aggregated across the filtered data rows.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  """
  If the following metrics are uploaded then
  in the Labelbox App, users will see:
  true positives dog = 4
  true positives cat = 3
  true positives = 7
  """

  feature_metric = ScalarMetric(metric_name="true_positives",
                              feature_name="cat",
                              value=3,
                              aggregation=ScalarMetricAggregation.SUM)

  feature_metric = ScalarMetric(metric_name="true_positives",
                              feature_name="dog",
                              value=4,
                              aggregation=ScalarMetricAggregation.SUM)
  ```
</CodeGroup>

## Add labels to a model run

Adds data rows and labels to a model run. By adding labels, the associated data rows will also be upserted to the model run.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # upsert using label ids
  label_ids = ["<label_id_1>","<label_id_2>", ...]
  model_run.upsert_labels(label_ids)
  ```
</CodeGroup>

Alternatively, you can add all labels from a project to a Model run directly. This will also add all data rows from that project to the model run.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # upsert using project id
  model_run.upsert_labels(project_id=<project_id>)
  ```
</CodeGroup>


# Model run slice
Source: https://docs.labelbox.com/horizon/developer-guides/model-slice

Developer guide for interacting with model slices using the Python SDK.

<CardGroup>
  <Card title="Open in Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/model_experiments/model_slices.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/model_experiments/model_slices.ipynb" />
</CardGroup>

[Model slices](/horizon/guides/slices) operate much like [Catalog slices](/horizon/guides/slices), both essentially functioning as saved searches that help filter large datasets into more manageable units. While Catalog slices function support **Catalog**, Model slices work within model runs. Model slices let you filter model run data rows into manageable units.

Here, we show how to use the SDK to:

* Create an experiment and a model run.
* Create a slice from model run data.

## Example: create and retrieve model slice

### Before you start

You must import these libraries to use the code examples in this section.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  import labelbox as lb
  import uuid
  ```
</CodeGroup>

Replace the value of `API_KEY` with a valid [API key](/horizon/developer-guides/create-api-key) to connect to the Labelbox client.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  API_KEY = None
  client = lb.Client(API_KEY)
  ```
</CodeGroup>

## Create a model experiment and a model run

To interact with model slices, you must create a model experiment with a model run and then create a model slice through the platform. The steps below go over this process. See Slices [Slices](/horizon/guides/slices)for more information

### Create a model experiment

To create a model experiment, you will need to create an ontology. See [Ontology](/horizon/developer-guides/ontology) for more information

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  #Create an ontology
  classification_features = [
      lb.Classification(
          class_type=lb.Classification.Type.CHECKLIST,
          name="Quality Issues",
          options=[
              lb.Option(value="blurry", label="Blurry"),
              lb.Option(value="distorted", label="Distorted")
          ]
      )

  ]

  ontology_builder = lb.OntologyBuilder(
  tools=[],
  classifications=classification_features
  )

  ontology = client.create_ontology(
  "Ontology from new features",
  ontology_builder.asdict(),
  media_type=lb.MediaType.Image
  )

  ```
</CodeGroup>

Attach the ontology during model creation.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  model = client.create_model(
    name="Model Slice Demo",
    ontology_id=ontology.uid
  )
  ```
</CodeGroup>

### Create a model run from a model experiment

In this step, you will need to create a dataset to attach data rows to our model run.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # create a sample dataset to send to your model
  global_key = str(uuid.uuid4())

  test_img_url = {
  "row_data":
  "https://storage.googleapis.com/labelbox-datasets/image_sample_data/2560px-Kitano_Street_Kobe01s5s4110.jpeg",
  "global_key":
  global_key
  }

  dataset = client.create_dataset(name="foundry-demo-dataset")
  task = dataset.create_data_rows([test_img_url])
  task.wait_till_done()

  print(f"Errors: {task.errors}")
  print(f"Failed data rows: {task.failed_data_rows}")

  # Create a model configuration

  model_run_name = "Model Slice Demo"
  example_config = {
  "learning_rate": 0.001,
  "batch_size": 32,
  }

  # Create a model run

  model_run = model.create_model_run(name=model_run_name, config=example_config)

  # Send the data rows to your model

  model_run.upsert_data_rows(global_keys=[global_key])

  ```
</CodeGroup>

## Create a model slice

Use the Labelbox app to create a model slice.

<Steps>
  <Step>
    From **Model**, select **Experiment** and select your model experiment.
  </Step>

  <Step>
    A filter must be active to create a slice. From the **Search** menu, select **Data row** and then set the **Condition** menu to **is not one of**.
  </Step>

  <Step>
    Set **Search for an id** to *test* and then press *Enter* to activate the filter. You can add additional filter conditions as needed.
  </Step>

  <Step>
    Select **Save slice**, name your slice, and then select **Save**. When you do this, the **Slice** menu displays the name of your slice and the number of matching data rows.
  </Step>

  <Step>
    From the **Slice** menu, select **Copy slice ID**.
  </Step>

  <Step>
    Paste the **Slice ID** into your your code:
  </Step>
</Steps>

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  SLICE_ID = ""
  ```
</CodeGroup>

### Get model slice

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  model_slice = client.get_model_slice(SLICE_ID)
  ```
</CodeGroup>

### Obtain data row IDs from model slice

Data row identifiers are objects that contain both data row IDs and global keys.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  data_row_identifiers = model_slice.get_data_row_identifiers(model_run.uid)

  drids = [dr for dr in data_row_identifiers]

  # get both global keys and data row ids

  # and utilize the hash method to combine both global keys and data row ids into a dictionary

  for dr in drids:
  print(f"Data row: {dr.id}, Global Key: {dr.global_key}, dr_gk: {dr.to_hash()}")

  ```
</CodeGroup>

### Obtain data row identifiers

```

data_rows = model_slice.get_data_row_identifiers(model_run.uid)

for data_row in data_rows:
print(data_row)

```

### Model slice attributes

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # name (str)
  model_slice.name

  # description (str)
  model_slice.description

  # updated at (datetime)
  model_slice.updated_at

  # created at (datetime)
  model_slice.created_at

  # filter (list[dict])
  model_slice.filter
  ```
</CodeGroup>


# Multimodal chat evaluation
Source: https://docs.labelbox.com/horizon/developer-guides/multimodal-chat-evaluation-project

Learn how to set up a multimodal chat evaluation project using the SDK.

<CardGroup>
  <Card title="Open in Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/project_configuration/multimodal_chat_project.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/project_configuration/multimodal_chat_project.ipynb" />
</CardGroup>

The [multimodal chat evaluation editor](/horizon/guides/multimodal-chat-evaluation-editor) allows you to evaluate generative model responses across multiple data types. This guide walks through setting up a multimodal chat evaluation project using the SDK.

## Initialize the SDK

Import the Labelbox Python SDK:

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  import labelbox as lb
  ```
</CodeGroup>

## API key and client

Provide a valid API key to authenticate the Labelbox client. See [Labelbox API keys](/horizon/developer-guides/create-api-key) to learn how to generate your key.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  API_KEY = "" # Your API key
  client = lb.Client(api_key=API_KEY)
  ```
</CodeGroup>

## Create a multimodal chat ontology

You can create ontologies for multimodal chat projects using either `client.create_ontology()` or `client.create_ontology_from_feature_schemas()`, with the `ontology_kind` parameter set to `lb.OntologyKind.ModelEvaluation`. See [Supported annotation types](/horizon/guides/multimodal-chat-evaluation-editor#supported-annotation-types) for the annotation types you can include in a multimodal chat evaluation ontology.

### Option A: `create_ontology`

Use `create_ontology` to create an ontology and define its schema:

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  ontology_builder = lb.OntologyBuilder(ontology_builder = lb.OntologyBuilder(
      tools=[
          lb.Tool(
              tool=lb.Tool.Type.MESSAGE_SINGLE_SELECTION,
              name="single select feature",
          ),
          lb.Tool(
              tool=lb.Tool.Type.MESSAGE_MULTI_SELECTION,
              name="multi select feature",
          ),
          lb.Tool(tool=lb.Tool.Type.MESSAGE_RANKING, name="ranking feature"),
          StepReasoningTool(name="step reasoning"),
          FactCheckingTool(name="fact checking"),
          PromptIssueTool(name="prompt rating"),
      ],
      classifications=[
          lb.Classification(
              class_type=lb.Classification.Type.CHECKLIST,
              name="checklist feature",
              options=[
                  lb.Option(value="option 1", label="option 1"),
                  lb.Option(value="option 2", label="option 2"),
              ],
          ),
          lb.Classification(
              class_type=lb.Classification.Type.RADIO,
              name="radio_question",
              options=[
                  lb.Option(value="first_radio_answer"),
                  lb.Option(value="second_radio_answer"),
              ],
          ),
      ],
  )

  # Create ontology
  ontology = client.create_ontology(
      "Example ontology",
      ontology_builder.asdict(),
      media_type=lb.MediaType.Conversational,
      ontology_kind=lb.OntologyKind.ModelEvaluation,
  )
  ```
</CodeGroup>

### Option B: `create_ontology_from_feature_schemas`

Use `create_ontology_from_feature_schemas` with feature schema IDs to create ontologies that reuse existing feature schemas instead of defining new ones. To obtain these IDs, go to the **Schema** tab.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  ontology = client.create_ontology_from_feature_schemas(
      "Example ontology",
      feature_schema_ids=["<list of feature schema ids"],
      media_type=lb.MediaType.Conversational,
      ontology_kind=lb.OntologyKind.ModelEvaluation,
  )
  ```
</CodeGroup>

## Create multimodal chat evaluation projects

The two types of multimodal chat evaluation projects have different project creation methods and data row setups:

* For **offline** multimodal chat evaluation projects, use `create_offline_model_evaluation_project` and import data rows of existing conversations.
* For **live** multimodal chat evaluation projects, use `client.create_model_evaluation_project` and either:
  * (Recommended) Create data rows and send them to projects, like other types of projects.
  * Generate empty data rows upon project creation, which can't create data rows with attachments and metadata.

## Set up offline multimodal chat evaluation projects

Use `client.create_offline_model_evaluation_project` to create offline multimodal chat evaluation projects. This method takes the same parameters as `client.create_project`: a `name` and an optional `description`.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  project = client.create_offline_model_evaluation_project(
      name="<project_name>",
      description="<project_description>",  # optional
  )
  ```
</CodeGroup>

After creating the project, import conversational version 2 data rows for further processing. For instructions, see [Import multimodal chat evaluation data](/horizon/developer-guides/import-multimodal-chat-data). To learn how to import annotations, see [Import multimodal chat annotations](/horizon/developer-guides/import-multimodal-chat-annotations).

## Set up live multimodal chat evaluation projects

Use `client.create_model_evaluation_project` to create a live multimodal chat evaluation project. This method takes the same `name` and optional `description` parameters as `client.create_project`, with a few additional parameters specific to multimodal chat evaluation projects:

* `data_row_count` (optional): The number of data rows to generate for your project. Defaults to 100 if a `dataset_name` or `dataset_id` is included.
* `dataset_name` (optional): The name of a new dataset. Include this parameter only if you want to create a new dataset for the generated data rows.
* `dataset_id` (optional): The dataset ID of an existing Labelbox dataset. Include this parameter only if you want to append generated data rows to an existing dataset.

### Option A: Create and send data rows to projects

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Create the project
  project = client.create_model_evaluation_project(
      name="Example live multimodal chat project",
      description="<project_description>",  # optional
  )

  def make_data_rows(dataset_id=None):
      # If a dataset ID is provided, fetch the dataset using that ID.
      # Otherwise, create a new dataset with the specified name.
      if dataset_id:
          dataset = client.get_dataset(dataset_id)
      else:
          dataset = client.create_dataset(name="example live mmc dataset")

      # Helper function to generate a single data row
      def generate_data(ind):
          return {
              "row_data": {  # The chat data format
                  'type': 'application/vnd.labelbox.conversational.model-chat-evaluation',
                  'draft': True,
                  'rootMessageIds': [],
                  'actors': {},
                  'version': 2,
                  'messages': {}
              },
              "global_key": f"global_key_{dataset.uid}_{ind}",
              "metadata_fields": [{"name": "tag", "value": "val_tag"}],
              "attachments": [
                  {
                      "type": "IMAGE_OVERLAY",
                      "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/rgb.jpg"
                  }
              ]
          }

      # Generate a list of 100 data rows
      data_list = [generate_data(ind) for ind in range(100)]

      # Upload the generated data rows to the dataset
      task = dataset.create_data_rows(data_list)
      print("Processing task ", task.uid)  # Print the unique ID of the task
      task.wait_till_done()
      
      # Ensure that the task status is 'COMPLETE' to confirm success
      assert task.status == "COMPLETE"

      # Return the dataset object
      return dataset

  # Create a new data set. Alternatively, pass an existing dataset ID
  dataset = make_data_rows()

  # Retrieve the data row IDs from the dataset
  data_row_ids = [data_row.uid for data_row in dataset.data_rows()]

  # Send data rows to the project
  batch = project.create_batch(
      name="mmc-batch",  # each batch in a project must have a unique name
      data_rows=data_row_ids, # data row IDs to include in the batch
      priority=1  # priority between 1(highest) - 5(lowest)
  )

  print(f"Batch: {batch}")
  ```
</CodeGroup>

### Option B: Generate empty data rows

<Info>
  ### No metadata support

  Only use this option if your project doesn't require metadata attachments or embeddings.
</Info>

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Create the project and generate data rows
  project = client.create_model_evaluation_project(
      name="Example live multimodal chat project",
      description="<project_description>",  # optional
      dataset_name="Example live multimodal chat dataset",
      data_row_count=100,
  )

  # Connect the project to the created ontology
  project.connect_ontology(ontology)
  ```
</CodeGroup>

### Set up model configurations

You can create, delete, attach, and remove model configurations that evaluate your live multimodal chat responses.

#### Create model configs

Use `client.create_model_config` to create a model configuration. This method takes the following parameters:

* `name`: The name of the model configuration.
* `model_id`: The ID of the model to configure. To get this value, go to the **Model** tab, select your model, and copy the ID from the URL.
* `inference_params`: Model configuration parameters in the JSON format. Each model has unique parameters.

<Info>
  ### First-time setup

  If you're setting up model configurations for the first time, it's recommended to set one up using the web platform instead of the SDK to understand all the associated parameters.
</Info>

The following example creates a Google Gemini 1.5 Pro model configuration:

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  MODEL_ID = "270a24ba-b983-40d6-9a1f-98a1bbc2fb65"

  inference_params = {"max_new_tokens": 1024, "use_attachments": True}

  model_config = client.create_model_config(
      name="Example Model Config",
      model_id=MODEL_ID,
      inference_params=inference_params,
  )
  ```
</CodeGroup>

#### Attach model configurations to projects

Use `project.add_model_config to attach` or `project.remove_model_config` to remove model configurations. Both methods take just a `model_config` ID.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  project.add_model_config(model_config.uid)
  ```
</CodeGroup>

#### Delete model configurations

Use `project.delete_project_model_config()` or `client.delete_model_config` to delete model configurations. Both methods require the `model_config` ID as a parameter. You can obtain this ID in one of the following ways:

* From the response when you create a model configuration
* By accessing `project.project_model_configs` and iterating through the list of model configurations attached to your project

<CodeGroup>
  ```python project.delete_project_model_config() theme={"theme":"css-variables"}
  model_configs = project.project_model_configs()

  for model_config in model_configs:
      project.delete_project_model_config(model_config.uid)
  ```

  ```python client.delete_model_config theme={"theme":"css-variables"}
  model_configs = project.project_model_configs()

  for model_config in model_configs:
      client.delete_model_config(model_config.uid)
  ```
</CodeGroup>

### Mark project setup as complete

Once you've completed your project setup and model configuration, use `project.set_project_model_setup_complete()` to mark the setup as complete. After that, you can add more data rows to the project, but you can't further add, modify, or delete project model configurations.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  project.set_project_model_setup_complete()
  ```
</CodeGroup>

## Export multimodal chat evaluation projects

The `export()` method provides a unified way to export data from all project types, including multimodal chat evaluation projects. It retrieves model responses, annotations, and related metadata in a structured format. For detailed instructions on using `export()` to export multimodal chat evaluation projects and sample export formats, see [Export multimodal chat annotations](/horizon/developer-guides/export-multimodal-chat-annotations).


# Ontology
Source: https://docs.labelbox.com/horizon/developer-guides/ontology

Developer guide for creating, modifying, and connecting ontologies via the Python SDK.

<CardGroup>
  <Card title="Open in Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/basics/ontologies.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/basics/ontologies.ipynb" />
</CardGroup>

## Client

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  import labelbox as lb
  import labelbox.data.annotation_types as lb_types
  client = lb.Client(api_key="<YOUR_API_KEY>")
  ```
</CodeGroup>

## Create an ontology

Each `Tool` and `Classification` type requires a specific value to be passed when creating the feature.

| Feature | Class | Value |
| - | - | - |
| Bounding box | `Tool` | `BBOX` |
| Polygon | `Tool` | `POLYGON` |
| Polyline | `Tool` | `LINE` |
| Point | `Tool` | `POINT` |
| Segmentation mask | `Tool` | `RASTER_SEGMENTATION` |
| Relationship | `Tool` | `RELATIONSHIP` |
| Entity | `Tool` | `NER` |
| Message ranking | `Tool` | `MESSAGE_RANKING` |
| Single message selection | `Tool` | `MESSAGE_SINGLE_SELECTION` |
| Multiple message selection | `Tool` | `MESSAGE_MULTI_SELECTION` |
| Message step reasoning | `Tool` | `STEP_REASONING` |
| Fact-checking | `Tool` | `FACT_CHECKING` |
| Prompt rating | `Tool` | `PROMPT_ISSUE` |
| Radio | `Classification` | `RADIO` |
| Checklist | `Classification` | `CHECKLIST` |
| Text | `Classification` | `TEXT` |

When you create an ontology, specify the `media_type` parameter, which represents the modality of the data rows that the ontology will be used to label. Certain features are only compatible with certain media types.

<Warning>
  ### Specify an ontology media type

  The `media_type` parameter of `client.create_ontology` defaults to `None`, but it's highly recommended to specify one for validation and error handling.
</Warning>

The `media_type` parameter takes the following values:

* `lb.MediaType.Audio`
* `lb.MediaType.Conversational`
* `lb.MediaType.Document`
* `lb.MediaType.Geospatial_Tile`
* `lb.MediaType.Html`
* `lb.MediaType.Image`
* `lb.MediaType.Simple_Tile`
* `lb.MediaType.Text`
* `lb.MediaType.Video`

### Create ontologies from new features

You can define new features to build new ontologies. The following code sample shows how to create simple features for upserting into ontologies. To learn more about creating features, see [Features](/horizon/developer-guides/feature#create-a-feature).

<CodeGroup>
  ```python Create features theme={"theme":"css-variables"}
  object_features = [
      lb.Tool(
          tool=lb.Tool.Type.BBOX,
          name="regulatory-sign",
          color="#ff0000",
      )
  ]

  classification_features = [
  lb.Classification(
  class_type=lb.Classification.Type.CHECKLIST,
  name="Quality Issues",
  options=[
  lb.Option(value="blurry", label="Blurry"),
  lb.Option(value="distorted", label="Distorted")
  ]
  )

  ]

  ```
</CodeGroup>

You can then create an ontology with your defined features either using `ontology_builder` or `client.create_ontology_from_feature_schemas()`.

<CodeGroup>
  ```python ontology_builder theme={"theme":"css-variables"}
  ontology_builder = lb.OntologyBuilder(
      tools=object_features,
      classifications=classification_features
  )

  ontology = client.create_ontology(
    "Ontology from new features",
    ontology_builder.asdict(),
    media_type=lb.MediaType.Image
  )
  ```

  ```python create_ontology_from_feature_schemas theme={"theme":"css-variables"}
  # First create the feature schema
  feature_schema_cat = client.create_feature_schema(feature_schema_cat_normalized)
  # When we create the ontology it will not re-create the feature schema
  print(feature_schema_cat.uid)
  ontology = client.create_ontology_from_feature_schemas(ontology_name,
                                                         [feature_schema_cat.uid])
  ```
</CodeGroup>

For more complex ontology creation examples, see [Ontology examples guide](/horizon/developer-guides/ontology-examples) .

### Create from normalized JSON format

* Users can create ontologies from a JSON definition of the ontology.
* Each tool type requires a specific value to be passed:

| Tool | Value |
| - | - |
| Bounding box | `rectangle` |
| Polygon | `polygon` |
| Polyline | `line` |
| Point | `point` |
| Segmentation mask | `raster-segmentation` |
| Entity | `named-entity` |

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # This automatically creates new feature schema
  ontology_name = "sdk-ontology"
  feature_schema_cat_normalized = {
      'tool': 'polygon',
      'name': 'cat',
      'color': 'black'
  }

  ontology_normalized_json = {
  "tools": [feature_schema_cat_normalized],
  "classifications": []
  }
  ontology = client.create_ontology(name=ontology_name,
  normalized=ontology_normalized_json,
  media_type=lb.MediaType.Image)

  ```
</CodeGroup>

### Create from existing features

If you already have features created, you can query them by name or by schema id. Re-using an existing feature is highly recommended.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  ## Search feature by name in your org
  regulatory_sign_feature_schema = next(client.get_feature_schemas("regulatory-sign"))
  classification_feature = next(client.get_feature_schemas("Quality Issues"))

  ## Get feature by feature schema ID. Alternatively you can get this from the UI
  regulatory_sign_feature_schema = client.get_feature_schema("FEATURE_SCHEMA_ID")

  ontology = client.create_ontology_from_feature_schemas("Ontology from existing features", [regulatory_sign_feature_schema.uid, classification_feature.uid], media_type=MediaType.Image)
  ```
</CodeGroup>

### Create a nested ontology

You can create a child feature nested under a parent feature. Only classification features can be children of a nested ontology. For instance, if you want to create an ontology where a bounding box has a radio sub-classification, you will add a `classifications`field that contains a list of classification features as children.

<CodeGroup>
  ```python Tool with subclassification theme={"theme":"css-variables"}
  bbox_with_sub_classification = lb.Tool(tool=lb.Tool.Type.BBOX, name="cat",
           classifications=[
              lb.Classification(class_type=lb.Classification.Type.RADIO, name="radio", options=[
                  lb.Option(value="long-fur"),
                  lb.Option(value="short-fur")
              ])
           ])

  ontology_builder = lb.OntologyBuilder(tools=[
  bbox_with_sub_classification
  ])

  ontology = client.create_ontology("Simple nested ontology", ontology_builder.asdict(), media_type=MediaType.Image)

  ```

  ```python Classification with subclassification theme={"theme":"css-variables"}
  radio_with_sub_classification = Classification(
        class_type=Classification.Type.RADIO,
        instructions="radio_question_sub",
        options=[
          Option(value="first_radio_answer",
                 options=[
                     Classification(
                      class_type=Classification.Type.RADIO,
                      instructions="sub_radio_question",
                      options=[
                        Option(value="first_sub_radio_answer"),
                        Option(value="second_sub_radio_answer")
                      ]
                    ),
              ])],
      )

  ontology_builder = OntologyBuilder(classifications=[
      sub_radio_classification
  ])

  ontology = client.create_ontology("Simple nested ontology", ontology_builder.asdict())
  ```
</CodeGroup>

## Get ontologies

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # get by ID
  ontology = client.get_ontology("<ontology_id>")

  # get by name

  ontology = next(client.get_ontologies("<ontology_name>"))

  # get multiple ontologies by keyword

  ontologies = client.get_ontologies(name_contains="<keyword>")

  ```
</CodeGroup>

### Check if a feature is archived in an ontology

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  client.is_feature_schema_archived("<ontology_id>", "<feature_schema_id>")
  ```
</CodeGroup>

### Get all unused ontologies

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  client.get_unused_ontologies()
  ```
</CodeGroup>

### Delete an unused ontology

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  client.delete_unused_ontology("<ontology_id>")
  ```
</CodeGroup>

## Connect an ontology to a project

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # get (or create) a project
  project = client.get_project("<project_id>")

  # get (or create) an ontology

  ontology = client.get_ontology("<ontology_id>")

  # the argument must be an object of the Ontology class

  project.connect_ontology(ontology)

  ```
</CodeGroup>

## Methods

### Get the tools

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  tools = ontology.tools()

  for tool in tools:
  print(tool)

  ```
</CodeGroup>

### Get the classifications

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  classifications = ontology.classifications()

  for classification in classifications:
  print(classification)

  ```
</CodeGroup>

## Attributes

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # name (str)
  ontology.name

  # description (str)

  ontology.description

  # updated at (datetime)

  ontology.updated_at

  # created at (datetime)

  ontology.created_at

  # normalized output of the ontology (JSON)

  ontology.normalized

  # count of object tools in the ontology (int)

  ontology.object_schema_count

  # count of classifications in the ontology (int)

  ontology.classification_schema_count

  # created by (relationship to User object)

  user = ontology.created_by()

  ```
</CodeGroup>


# Ontology examples
Source: https://docs.labelbox.com/horizon/developer-guides/ontology-examples

Complex examples of creating a variety of ontologies.

## Image

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  ontology_builder = lb.OntologyBuilder(
      classifications=[  # list of classification objects
          lb.Classification(class_type=lb.Classification.Type.RADIO,
                            name="radio_question",
                            options=[
                                lb.Option(value="first_radio_answer"),
                                lb.Option(value="second_radio_answer")
                            ]),
          lb.Classification(class_type=lb.Classification.Type.CHECKLIST,
                            name="checklist_question",
                            options=[
                                lb.Option(value="first_checklist_answer"),
                                lb.Option(value="second_checklist_answer")
                            ]),
          lb.Classification(class_type=lb.Classification.Type.TEXT,
                            name="free_text"),
          lb.Classification(
              class_type=lb.Classification.Type.RADIO,
              name="nested_radio_question",
              options=[
                  lb.Option("first_radio_answer",
                            options=[
                                lb.Classification(
                                    class_type=lb.Classification.Type.RADIO,
                                    name="sub_radio_question",
                                    options=[lb.Option("first_sub_radio_answer")])
                            ])
              ]),
          lb.Classification(
              class_type=lb.Classification.Type.CHECKLIST,
              name="nested_checklist_question",
              options=[
                  lb.Option(
                      "first_checklist_answer",
                      options=[
                          lb.Classification(
                              class_type=lb.Classification.Type.CHECKLIST,
                              name="sub_checklist_question",
                              options=[lb.Option("first_sub_checklist_answer")])
                      ])
              ]),
      ],
      tools=[  # List of Tool objects
          lb.Tool(tool=lb.Tool.Type.BBOX, name="bounding_box"),
          lb.Tool(tool=lb.Tool.Type.BBOX,
                  name="bbox_with_radio_subclass",
                  classifications=[
                      lb.Classification(
                          class_type=lb.Classification.Type.RADIO,
                          name="sub_radio_question",
                          options=[lb.Option(value="first_sub_radio_answer")]),
                  ]),
          lb.Tool(tool=lb.Tool.Type.POLYGON, name="polygon"),
          lb.Tool(tool=lb.Tool.Type.RASTER_SEGMENTATION, name="mask"),
        	lb.Tool(tool=lb.Tool.Type.RASTER_SEGMENTATION,
                  name="mask_with_text_subclass",
                  classifications=[
                      lb.Classification(
                          class_type=lb.Classification.Type.TEXT,
                          name="sub_free_text")
                  ]),
          lb.Tool(tool=lb.Tool.Type.POINT, name="point"),
          lb.Tool(tool=lb.Tool.Type.LINE, name="polyline"),
          lb.Tool(tool=lb.Tool.Type.RELATIONSHIP, name="relationship")
      ])

  ontology = client.create_ontology("Image Annotation Import Demo Ontology",
                                    ontology_builder.asdict(),
                                    media_type=lb.MediaType.Image)
  ```
</CodeGroup>

## Video

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  ontology_builder = lb.OntologyBuilder(
      tools=[
          lb.Tool(tool=lb.Tool.Type.BBOX, name="bbox_video"),
          lb.Tool(tool=lb.Tool.Type.POINT, name="point_video"),
          lb.Tool(tool=lb.Tool.Type.LINE, name="line_video_frame"),
          lb.Tool(tool=lb.Tool.Type.RASTER_SEGMENTATION, name="video_mask"),
          lb.Tool(
            tool=lb.Tool.Type.BBOX, name="bbox_class",
            classifications=[
              lb.Classification(
                class_type=lb.Classification.Type.RADIO,
                name="checklist_class",
                scope = lb.Classification.Scope.INDEX,
                options=[
                  lb.Option(value="first_checklist_answer"),
                  lb.Option(value="second_checklist_answer")
                ]
              )
            ]
          ),
        	lb.Tool(tool=lb.Tool.Type.RASTER_SEGMENTATION,
                  name="mask_with_text_subclass",
                  classifications=[
                      lb.Classification(
                          class_type=lb.Classification.Type.TEXT,
                          name="sub_free_text")
                      ]
                  )
      ],
      classifications=[
          lb.Classification(
              class_type=lb.Classification.Type.CHECKLIST,
              name="checklist_class",
              scope = lb.Classification.Scope.INDEX, ## Need to defined scope for frame classifications
              options=[
                  lb.Option(value="first_checklist_answer"),
                  lb.Option(value="second_checklist_answer")
              ]
          ),
          lb.Classification(
              class_type=lb.Classification.Type.RADIO,
              name="radio_class",
              scope = lb.Classification.Scope.INDEX,
              options=[
                  lb.Option(value="first_radio_answer"),
                  lb.Option(value="second_radio_answer")
              ]
          ),
           lb.Classification(
                class_type=lb.Classification.Type.RADIO,
                name="nested_radio_question",
                options=[
                    lb.Option("first_radio_answer",
                          options=[
                              lb.Classification(
                                  class_type=lb.Classification.Type.RADIO,
                                  name="sub_radio_question",
                                  options=[lb.Option("first_sub_radio_answer")]
                              )
                          ]
                    )
                ]
          ),
          lb.Classification(
            class_type=lb.Classification.Type.CHECKLIST,
            name="nested_checklist_question",
            options=[
                lb.Option("first_checklist_answer",
                  options=[
                    lb.Classification(
                        class_type=lb.Classification.Type.CHECKLIST,
                        name="sub_checklist_question",
                        options=[lb.Option("first_sub_checklist_answer")]
                    )
                ]
              )
            ]
          ),
          lb.Classification(
            class_type=lb.Classification.Type.RADIO,
            name="radio_class_global",
            options=[
                  lb.Option(value="first_radio_answer"),
                  lb.Option(value="second_radio_answer")
              ]
          ),
          lb.Classification(
            class_type=lb.Classification.Type.CHECKLIST,
            name="checklist_class_global",
            options=[
                  lb.Option(value="first_checklist_answer"),
                  lb.Option(value="second_checklist_answer")
            ]
          ),
          lb.Classification(
            class_type=lb.Classification.Type.TEXT,
            name="free_text"
          )
      ]
  )

  ontology = client.create_ontology("Video Annotation Import Demo Ontology",
                                    ontology_builder.asdict(),
                                    media_type=lb.MediaType.Video)
  ```
</CodeGroup>

## Text

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}

  ontology_builder = lb.OntologyBuilder(
    classifications=[ # List of Classification objects
      lb.Classification(
        class_type=lb.Classification.Type.RADIO,
        name="radio_question",
        options=[lb.Option(value="first_radio_answer")]
      ),
      lb.Classification(
        class_type=lb.Classification.Type.RADIO,
        name="nested_radio_question",
        options=[
          lb.Option(value="first_radio_answer",
            options=[
                lb.Classification(
                  class_type=lb.Classification.Type.RADIO,
                  name="sub_radio_question",
                  options=[
                    lb.Option(value="first_sub_radio_answer")
                  ]
              ),
            ]
          ),
        ],
      ),
       lb.Classification(
        class_type=lb.Classification.Type.CHECKLIST,
        name="nested_checklist_question",
        options=[
            lb.Option("first_checklist_answer",
              options=[
                lb.Classification(
                    class_type=lb.Classification.Type.CHECKLIST,
                    name="sub_checklist_question",
                    options=[lb.Option("first_sub_checklist_answer")]
                )
            ]
          )
        ]
      ),
      lb.Classification(
        class_type=lb.Classification.Type.CHECKLIST,
        name="checklist_question",
        options=[
          lb.Option(value="first_checklist_answer"),
          lb.Option(value="second_checklist_answer"),
          lb.Option(value="third_checklist_answer")
        ]
      ),
       lb.Classification(
        class_type=lb.Classification.Type.TEXT,
        name="free_text"
      )
    ],
    tools=[ # List of Tool objects
           lb.Tool(
              tool=lb.Tool.Type.NER,
              name="named_entity"
            ),
           lb.Tool(
              tool=lb.Tool.Type.RELATIONSHIP,
              name="relationship"
            )
      ]
  )
  ontology = client.create_ontology("Ontology Text Annotations", ontology_builder.asdict())
  ```
</CodeGroup>

## Geospatial

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  ontology_builder = lb.OntologyBuilder(
      tools=[
          lb.Tool(tool=lb.Tool.Type.POINT, name="point_geo"),
          lb.Tool(tool=lb.Tool.Type.LINE, name="polyline_geo"),
          lb.Tool(tool=lb.Tool.Type.POLYGON, name="polygon_geo"),
          lb.Tool(tool=lb.Tool.Type.POLYGON, name="polygon_geo_2"),
          lb.Tool(tool=lb.Tool.Type.BBOX, name="bbox_geo"),
          lb.Tool(
            tool=lb.Tool.Type.BBOX,
            name="bbox_checklist_geo",
            classifications=[
                  lb.Classification(
                      class_type=lb.Classification.Type.CHECKLIST,
                      name="checklist_class_name",
                      options=[
                        lb.Option(value="first_checklist_answer")
                      ]
                  ),
              ]
            ),
          lb.Tool(
            tool=lb.Tool.Type.BBOX,
            name="bbox_text_geo",
            classifications=[
                  lb.Classification(
                      class_type=lb.Classification.Type.TEXT,
                      name="free_text_geo"
                  ),
              ]
            )
        ],
        classifications = [
            lb.Classification(
                class_type=lb.Classification.Type.CHECKLIST,
                name="checklist_question_geo",
                options=[
                    lb.Option(value="first_checklist_answer"),
                    lb.Option(value="second_checklist_answer"),
                    lb.Option(value="third_checklist_answer")
                ]
            ),
            lb.Classification(
                class_type=lb.Classification.Type.RADIO,
                name="radio_question_geo",
                options=[
                    lb.Option(value="first_radio_answer")
                ]
            ),

          lb.Classification(
            class_type=lb.Classification.Type.RADIO,
            name="nested_radio_question",
            options=[
              lb.Option(value="first_radio_answer",
                options=[
                    lb.Classification(
                      class_type=lb.Classification.Type.RADIO,
                      name="sub_radio_question",
                      options=[
                        lb.Option(value="first_sub_radio_answer")
                      ]
                  ),
                ]
              ),
            ],
          ),
          lb.Classification(class_type=lb.Classification.Type.TEXT,
                            name="free_text"),
          lb.Classification(
            class_type=lb.Classification.Type.CHECKLIST,
            name="nested_checklist_question",
            options=[
                lb.Option("first_checklist_answer",
                  options=[
                    lb.Classification(
                        class_type=lb.Classification.Type.CHECKLIST,
                        name="sub_checklist_question",
                        options=[lb.Option("first_sub_checklist_answer")]
                    )
                ]
              )
            ]
        )
      ]
  )

  ontology = client.create_ontology("Ontology Geospatial Annotations", ontology_builder.asdict(), media_type=lb.MediaType.Geospatial_Tile)
  ```
</CodeGroup>

## Document

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  ontology_builder = lb.OntologyBuilder(
    classifications=[ # List of Classification objects
      lb.Classification(
        class_type=lb.Classification.Type.RADIO,
        name="radio_question",
        scope = lb.Classification.Scope.GLOBAL,
        options=[
          lb.Option(value="first_radio_answer"),
          lb.Option(value="second_radio_answer")
        ]
      ),
      lb.Classification(
        class_type=lb.Classification.Type.CHECKLIST,
        name="checklist_question",
        scope = lb.Classification.Scope.GLOBAL,
        options=[
          lb.Option(value="first_checklist_answer"),
          lb.Option(value="second_checklist_answer")
        ]
      ),
      lb.Classification(
        class_type=lb.Classification.Type.TEXT,
        name="free_text",
        scope = lb.Classification.Scope.GLOBAL
      ),
      lb.Classification(
          class_type=lb.Classification.Type.RADIO,
          name="nested_radio_question",
          scope = lb.Classification.Scope.GLOBAL,
          options=[
              lb.Option("first_radio_answer",
                  options=[
                      lb.Classification(
                          class_type=lb.Classification.Type.RADIO,
                          name="sub_radio_question",
                          options=[lb.Option("first_sub_radio_answer")]
                      )
                  ])
            ]
      ),
      lb.Classification(
        class_type=lb.Classification.Type.CHECKLIST,
        name="nested_checklist_question",
        scope = lb.Classification.Scope.GLOBAL,
        options=[
            lb.Option("first_checklist_answer",
              options=[
                lb.Classification(
                    class_type=lb.Classification.Type.CHECKLIST,
                    name="sub_checklist_question",
                    options=[lb.Option("first_sub_checklist_answer")]
                )
            ])
        ]
      ),
    ],
    tools=[ # List of Tool objects
      lb.Tool( tool=lb.Tool.Type.BBOX,name="bounding_box"),
      lb.Tool(tool=lb.Tool.Type.NER, name="named_entity"),
      lb.Tool(tool=lb.Tool.Type.RELATIONSHIP,name="relationship"),
      lb.Tool(tool=lb.Tool.Type.NER,
              name="ner_with_checklist_subclass",
              classifications=[
                lb.Classification(
                  class_type=lb.Classification.Type.CHECKLIST,
                  name="sub_checklist_question",
                  options=[
                    lb.Option(value="first_sub_checklist_answer")
                  ]
                )
            ]),
      lb.Tool( tool=lb.Tool.Type.BBOX,
              name="bbox_with_radio_subclass",
              classifications=[
                lb.Classification(
                    class_type=lb.Classification.Type.RADIO,
                    name="sub_radio_question",
                    options=[
                      lb.Option(
                        value="first_sub_radio_answer" ,
                        options=[
                          lb.Classification(
                            class_type=lb.Classification.Type.RADIO,
                            name="second_sub_radio_question",
                            options=[lb.Option("second_sub_radio_answer")]
                          )]
                      )]
                  )]
        )]
  )

  ontology = client.create_ontology("Document Annotation Import Demo",
                                    ontology_builder.asdict(),
                                    media_type=lb.MediaType.Document)
  ```
</CodeGroup>

## Audio

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  ontology_builder = lb.OntologyBuilder(
    classifications=[
      lb.Classification(
        class_type=lb.Classification.Type.TEXT,
        name="text_audio"),
      lb.Classification(
        class_type=lb.Classification.Type.CHECKLIST,
        name="checklist_audio",
        options=[
          lb.Option(value="first_checklist_answer"),
          lb.Option(value="second_checklist_answer")
        ]
      ),
      lb.Classification(
        class_type=lb.Classification.Type.RADIO,
        name="radio_audio",
        options=[
          lb.Option(value="first_radio_answer"),
          lb.Option(value="second_radio_answer")
        ]
      )
    ]
  )

  ontology = client.create_ontology("Ontology Audio Annotations",
                                    ontology_builder.asdict(),
                                    media_type=lb.MediaType.Audio)
  ```
</CodeGroup>

## HTML

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  ontology_builder = lb.OntologyBuilder(
    classifications=[
      lb.Classification(
        class_type=lb.Classification.Type.TEXT,
        name="text_html"),
      lb.Classification(
        class_type=lb.Classification.Type.CHECKLIST,
        name="checklist_html",
        options=[
          lb.Option(value="first_checklist_answer"),
          lb.Option(value="second_checklist_answer")
        ]
      ),
      lb.Classification(
        class_type=lb.Classification.Type.RADIO,
        name="radio_html",
        options=[
          lb.Option(value="first_radio_answer"),
          lb.Option(value="second_radio_answer")
        ]
      )
    ]
  )

  ontology = client.create_ontology("Ontology HTML Annotations", ontology_builder.asdict(), media_type=lb.MediaType.Html)
  ```
</CodeGroup>

## Conversational

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  ontology_builder = lb.OntologyBuilder(
    tools=[
      lb.Tool(tool=lb.Tool.Type.NER,name="ner"),
      lb.Tool(tool=lb.Tool.Type.RELATIONSHIP,name="relationship")
      ],
    classifications=[
      lb.Classification(
        class_type=lb.Classification.Type.TEXT,
        scope=lb.Classification.Scope.INDEX,  # Remove this line or set scope to "GLOBAL" if importing global text annotations
        instructions="text_convo"),
      lb.Classification(
        class_type=lb.Classification.Type.CHECKLIST,
        scope=lb.Classification.Scope.INDEX,  # Remove this line or set scope to "GLOBAL" if importing global checklist annotations
        instructions="checklist_convo",
        options=[
          lb.Option(value="first_checklist_answer"),
          lb.Option(value="second_checklist_answer")
        ]
      ),
      lb.Classification(
        class_type=lb.Classification.Type.RADIO,
        instructions="radio_convo",
        scope=lb.Classification.Scope.INDEX, # Remove this line or set scope to "GLOBAL" if importing global radio  annotations
        options=[
          lb.Option(value="first_radio_answer"),
          lb.Option(value="second_radio_answer")
        ]
      )
    ]
  )

  ontology = client.create_ontology("Ontology Conversation Annotations", ontology_builder.asdict())
  ```
</CodeGroup>

## Model Chat Evaluation

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  ontology_builder = lb.OntologyBuilder(
      tools=[
          lb.Tool(
              tool=lb.Tool.Type.MESSAGE_SINGLE_SELECTION,
              name="single select feature",
          ),
          lb.Tool(
              tool=lb.Tool.Type.MESSAGE_MULTI_SELECTION,
              name="multi select feature",
          ),
          lb.Tool(tool=lb.Tool.Type.MESSAGE_RANKING, name="ranking feature"),
      ],
      classifications=[
          lb.Classification(
              class_type=lb.Classification.Type.CHECKLIST,
              name="checklist feature",
              options=[
                  lb.Option(value="option 1", label="option 1"),
                  lb.Option(value="option 2", label="option 2"),
              ],
          ),
          lb.Classification(
              class_type=lb.Classification.Type.RADIO,
              name="radio_question",
              options=[
                  lb.Option(value="first_radio_answer"),
                  lb.Option(value="second_radio_answer"),
              ],
          ),
      ],
  )

  # Create ontology
  ontology = client.create_ontology(
      "MCE ontology",
      ontology_builder.asdict(),
      media_type=lb.MediaType.Conversational,
      ontology_kind=lb.OntologyKind.ModelEvaluation,
  )
  ```
</CodeGroup>


# Organization
Source: https://docs.labelbox.com/horizon/developer-guides/organization

A developer guide for querying organization information via the Python SDK.

## Client

### Get the current organization

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  organization = client.get_organization()
  ```
</CodeGroup>

## Methods

### Get IAM integrations

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # get the default IAM integration
  default_iam_integration = organization.get_default_iam_integration()

  # get all of the IAM integrations (returns a list of IAMIntegration objects)
  iam_integrations = organization.get_iam_integrations()

  for integration in iam_integrations:
    print(integration)
  ```
</CodeGroup>

### Create a project tag

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  organization.create_resource_tag(
    {
      "text": "new-tag-name",
      "color": "ffffff"
    }
  )
  ```
</CodeGroup>

### Get the project tags

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # get the project tags (returns a list of ResourceTag objects)
  tags = organization.get_resource_tags()

  for tag in tags:
    print(tag)
  ```
</CodeGroup>

### Invite users

Depending on the level of access required, you can:

* Invite users to a workspace without assigning them to any projects or user groups.
* Invite users to a workspace and assign them to specific projects or user groups.

<CodeGroup>
  ```python Organization-only theme={"theme":"css-variables"}
  # get and view the available roles
  roles = client.get_roles()
  print("Roles: ", roles)

  # create an invitation for an organization-wide role (in this case, a labeler)
  organization.invite_user(
      email="<email_address>",
      role=roles["LABELER"]
  )
  ```

  ```python Project-based theme={"theme":"css-variables"}
  # get and view the available roles
  roles = client.get_roles()
  print("Roles: ", roles)

  # get a project
  project = client.get_project("<project_id>")

  # create a ProjectRole object for a project-based role, in this case a labeler
  project_role = lb.ProjectRole(project=project, role=roles["LABELER"])

  # create an invitation for the project-based role
  organization.invite_user(
    email="<email_address>",
    # for a project-based role, the org-wide role must be NONE
    role=roles["NONE"],
    project_roles=[project_role]
  )

  # Mass invite users to project
  emails = []
  for email in emails:
    invite = organization.invite_user(emails,
                                    roles["NONE"],
                                    project_roles=[project_role])
  ```

  ```python Group-based theme={"theme":"css-variables"}
  from labelbox import Client, UserGroupRole
  from labelbox.schema.user_group import UserGroup

  roles = client.get_roles()

  # Use an existing UserGroup id (e.g. from the UI / API)
  user_group = UserGroup(client=client, id="<user_group_id>")
  # Create a UserGroupRole object for a group-based role, in this case a reviewer
  user_group_role = UserGroupRole(user_group=user_group, role=roles["REVIEWER"])

  invite = organization.invite_user(
      email="<email_address>",
      # for a group-based role, the org-wide role must be NONE
      role=roles["NONE"],
      user_group_roles=[user_group_role]    
  )
  ```
</CodeGroup>

### Get pending invites

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  project_id = "<project_id>"

  # For the entire organization
  org_invites = organization.get_invites()

  # For a specific project
  project_invites = organization.get_project_invites(project_id)
  ```
</CodeGroup>

### Cancel invite

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  email = "<email_address>""
  project_id = "<project_id>"

  # Option 1: organization invite
  invite = next(
      invite for invite in organization.get_invites() if invite.email == email
  )

  # Option 2: project invite
  invite = next(
      invite for invite in organization.get_project_invites(project_id) if invite.email == email
  )

  invite.cancel()
  ```
</CodeGroup>

### View the invite limit

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # already accounts for users currently in the organization, thus:
  # 	used = users + invites_sent
  # 	remaining = limit - used
  organization.invite_limit()
  ```
</CodeGroup>

### Remove a user

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # the argument must be a User object
  organization.remove_user(user)
  ```
</CodeGroup>

## Attributes

### Get the basics

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # name (str)
  organization.name

  # created_at (datetime)
  organization.created_at

  # updated_at (datetime)
  organization.updated_at
  ```
</CodeGroup>

### Get the users

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # get the users (relationship to User objects)
  users = organization.users()

  # inspect one user
  next(users)

  # inspect all users
  for user in users:
    print(user)

  # for ease of use, you can convert the paginated collection to a list
  list(users)
  ```
</CodeGroup>

### Get the projects

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # get the projects (relationship to Project objects)
  projects = organization.projects()

  # inspect one project
  next(projects)

  # inspect all projects
  for project in projects:
    print(project)

  # for ease of use, you can convert the paginated collection to a list
  list(projects)
  ```
</CodeGroup>

### Get the webhooks

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # get the webhooks (relationship to Webhook objects)
  webhooks = organization.webhooks()

  # inspect one webhook
  next(webhooks)

  # inspect all webhooks
  for webhook in webhooks:
    print(webhook)

  # for ease of use, you can convert the paginated collection to a list
  list(webhooks)
  ```
</CodeGroup>


# Performance API
Source: https://docs.labelbox.com/horizon/developer-guides/performance-api

Read the numbers behind the Performance dashboard and the Monitor through the REST API or the Python SDK.

The Performance API returns the numbers shown on a project's [Performance dashboard](/horizon/guides/performance-dashboard) and on the workspace [Monitor](/horizon/guides/monitor): throughput, time, and quality metrics over a period, and totals for each labeler and reviewer. Use it to bring those numbers into your own reporting.

The API runs the same reports as the pages. For the same dates and filters it returns the same values, and the same rules decide what you can see.

## Before you start

You need:

* A Labelbox [API key](/horizon/developer-guides/create-api-key).
* Access to performance data. If you can open a project's Performance dashboard, you can read that project through the API. Workspace reports need permission to view performance across your organization, as the Monitor does.
* For the Python examples, Python SDK version 7.13.0 or later.

All endpoints are `GET` requests under this base URL, authenticated with your API key as a bearer token:

```text theme={"theme":"css-variables"}
https://api.labelbox.com/api/v1/performance
```

## Quick start

Get the number of labels created per day in a project:

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  import labelbox as lb

  client = lb.Client(api_key="<YOUR_API_KEY>")
  project = client.get_project("<PROJECT_ID>")

  series = project.get_performance_metric(
      "labels_created", "2026-01-01", "2026-01-31"
  )
  for bucket in series.buckets:
      print(bucket.start.date(), bucket.value)
  ```

  ```bash cURL theme={"theme":"css-variables"}
  curl -H "Authorization: Bearer $LABELBOX_API_KEY" \
    "https://api.labelbox.com/api/v1/performance/projects/$PROJECT_ID/metrics/labels_created?startDate=2026-01-01&endDate=2026-01-31"
  ```
</CodeGroup>

The response has one bucket per day:

```json theme={"theme":"css-variables"}
{
  "projectId": "<PROJECT_ID>",
  "metric": "labels_created",
  "unit": "count",
  "bucketType": "time",
  "buckets": [
    {
      "start": "2026-01-01T00:00:00.000Z",
      "end": "2026-01-02T00:00:00.000Z",
      "value": 412,
      "numerator": null,
      "denominator": null,
      "modes": null
    }
  ],
  "interval": "day",
  "startDate": "2026-01-01",
  "endDate": "2026-01-31",
  "generatedAt": "2026-02-01T09:30:00.000Z"
}
```

## Endpoints

| Endpoint | Python SDK | Returns |
| - | - | - |
| `GET /metrics` | `client.get_performance_metrics()` | The metrics you can query |
| `GET /projects/{projectId}/metrics/{metric}` | `project.get_performance_metric()` | One metric for a project, a value per interval |
| `GET /projects/{projectId}/labelers` | `project.get_labeler_performance()` | Totals for each labeler |
| `GET /projects/{projectId}/reviewers` | `project.get_reviewer_performance()` | Totals for each reviewer |
| `GET /projects/{projectId}/download` | `project.get_performance_report_download()` | A link to the report as a file |
| `GET /workspace/metrics/{metric}` | `client.get_workspace_performance_metric()` | One metric across your workspace |
| `GET /workspace/labelers` | `client.get_workspace_labeler_performance()` | Totals for each labeler on each project |

Every table is paged. [Filters](#filters) lists which parameters each report takes. See [Paging](#paging) and [Reliability](#reliability) before you build on the API.

## Units and dates

* **Time** is in seconds.
* **Percentages and agreement scores** are fractions from 0 to 1. A value of `0.9` means 90%.
* **Dates** are calendar days in UTC, written as `2026-01-31`. Reports are bucketed by UTC day. `startDate` and `endDate` are the first and last day of the period, both included, and a request can cover at most 366 days.
* **Wider buckets** keep their calendar bounds: a week runs from Monday, a month from the 1st. The first and last bucket of a series can reach beyond your period, but they count only the days inside it. For counts and time totals, a wider bucket is the sum of its days. Averages and percentages are worked out over the whole bucket, so they are not.

In the Python SDK you can pass a `date`, a `datetime`, or a string. A `datetime` with a time zone is converted to its UTC day.

## Metrics

List the metrics available to you, with their units and definitions:

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  for metric in client.get_performance_metrics():
      print(metric.name, metric.unit, metric.description)
  ```

  ```bash cURL theme={"theme":"css-variables"}
  curl -H "Authorization: Bearer $LABELBOX_API_KEY" \
    "https://api.labelbox.com/api/v1/performance/metrics"
  ```
</CodeGroup>

### Throughput

| Metric | Unit | What it measures |
| - | - | - |
| `done_datarows` | count | Data rows in the Done step of the workflow |
| `accepted_datarows` | count | Data rows in the Accepted step of the workflow |
| `labels_created` | count | Labels created in the period |
| `annotations_created` | count | Annotations created in the period |
| `approved_reviews` | count | Approve actions, through review tasks in the workflow, on labels created in the period |
| `rejected_reviews` | count | Reject actions, through review tasks in the workflow, on labels created in the period |
| `media_duration` | seconds | Total media duration of data rows in the Done step |
| `media_duration_in_review` | seconds | Total media duration of data rows currently in a Review step |
| `media_duration_accepted` | seconds | Total media duration of accepted data rows |
| `total_time` | seconds | Total labeling, review, and rework time spent on labels created in the period |
| `labeling_time` | seconds | Total time spent in labeling mode while the timer is on |
| `review_time` | seconds | Total time spent in review mode while the timer is on |
| `rework_time` | seconds | Total time spent in rework mode while the timer is on |

### Efficiency

| Metric | Unit | What it measures |
| - | - | - |
| `avg_time_per_label` | seconds | Total labeling time divided by the number of labels worked on, including skipped and unsaved labels |
| `avg_review_time` | seconds | Total review time divided by the number of data rows reviewed |
| `avg_rework_time` | seconds | Total rework time divided by the number of data rows reworked |
| `aht_labeled_datarows` | seconds | Average handling time per data row with at least one label |
| `aht_done_datarows` | seconds | Average handling time per data row marked Done |
| `aht_accepted_datarows` | seconds | Average handling time per data row marked Accepted |
| `aht_all_labels` | seconds | Average handling time per label created, counting skipped, abandoned, and submitted labels |
| `aht_submitted_labels` | seconds | Average handling time per submitted label |
| `aht_done_labels` | seconds | Average handling time per label on done data rows |
| `aht_accepted_labels` | seconds | Average handling time per label on accepted data rows |

Average handling time includes creating, reviewing, and reworking time. The `aht_` metrics are listed and can be queried only when average handling time charts are enabled for your organization. Their buckets also carry `modes`, the handling time split into `creating`, `reviewing`, and `reworking`.

### Quality

| Metric | Unit | What it measures |
| - | - | - |
| `benchmark_percentage` | percentage | Average benchmark agreement score of labels |
| `benchmark_distribution` | count | Histogram of benchmark agreement scores |
| `consensus_percentage` | percentage | Average consensus agreement score of labels |
| `consensus_distribution` | count | Histogram of consensus agreement scores |

The two distribution metrics return score ranges instead of time intervals. Their response has `"bucketType": "score"` and ten buckets, each with `rangeStart`, `rangeEnd`, and `value`, the number of labels whose score falls in that range. The ranges are fractions like the scores themselves: `0` to `0.1`, up to `0.9` to `1`.

## Filters

These parameters narrow a report. Lists are comma-separated in a request, or a Python list in the SDK.

| Parameter | Python SDK | Applies to | Description |
| - | - | - | - |
| `startDate`, `endDate` | `start_date`, `end_date` | All reports | Required. The period, as UTC days. |
| `interval` | `interval` | Metrics | Bucket width: `day` (default), `week`, `month`, `quarter`, or `year`. |
| `userIds` | `user_ids` | All reports | Count only work by these users. Up to 100. |
| `userGroupIds` | `user_group_ids` | Project reports | Count only work by the current members of these user groups. Up to 25. With `userIds` too, only users in both are counted. |
| `batchIds` | `batch_ids` | All reports | Count only these batches. Up to 100. |
| `projectIds` | `project_ids` | Workspace reports | Narrow the report to these projects. Up to 50. |
| `memberType` | `member_type` | Workspace reports | `internal` for members of your organization, `external` for members of other organizations. See [Shared projects and workforce providers](#shared-projects-and-workforce-providers). |
| `ownerOrganizationIds` | `owner_organization_ids` | Workspace reports | Count only projects owned by these organizations. Up to 100. |
| `deletedLabels` | `deleted_labels` | All reports | `include` (default), `exclude`, or `only`. |

A user group filter follows the group's current membership. A group that is empty, is no longer attached to the project, or that you cannot view matches no work.

A parameter that an endpoint does not have is refused by name, so a misspelled filter never runs an unfiltered report.

A list that is sent must name at least one ID. An empty list is refused with `400` (`userIds=`), and the Python SDK raises `ValueError` for one (`user_ids=[]`). It is not read as "no filter", which would report on everyone. To leave a filter off, leave the parameter out, or pass `None` in the SDK.

## Labeler and reviewer totals

These are the two tables on the Performance dashboard: one row per member with their totals for the period.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  labelers = project.get_labeler_performance(
      "2026-01-01",
      "2026-01-31",
      sort_by="labels_created",
      descending=True,
  )
  print(labelers.total)
  for labeler in labelers:
      print(labeler.email, labeler.labels_created, labeler.avg_time_per_label)

  for reviewer in project.get_reviewer_performance("2026-01-01", "2026-01-31"):
      print(reviewer.email, reviewer.labels_reviewed)
  ```

  ```bash cURL theme={"theme":"css-variables"}
  curl -H "Authorization: Bearer $LABELBOX_API_KEY" \
    "https://api.labelbox.com/api/v1/performance/projects/$PROJECT_ID/labelers?startDate=2026-01-01&endDate=2026-01-31&sort=labelsCreated&order=desc"

  curl -H "Authorization: Bearer $LABELBOX_API_KEY" \
    "https://api.labelbox.com/api/v1/performance/projects/$PROJECT_ID/reviewers?startDate=2026-01-01&endDate=2026-01-31"
  ```
</CodeGroup>

Both tables are paged and sortable. See [Paging](#paging).

The response wraps the rows in `data` and adds `page`, `perPage`, and `total`:

```json theme={"theme":"css-variables"}
{
  "projectId": "<PROJECT_ID>",
  "data": [
    {
      "userId": "<USER_ID>",
      "projectId": "<PROJECT_ID>",
      "email": "labeler@example.com",
      "memberType": "internal",
      "labelsCreated": 128,
      "labelsSkipped": 3,
      "memberTotalTime": 21640,
      "labelingTimeSubmitted": 19200,
      "labelingTimeSkipped": 240,
      "reviewsReceived": 40,
      "reviewTimeAll": 1800,
      "reworkTimeAll": 600,
      "avgTimePerLabel": 148.4,
      "avgReviewTimeAll": 45,
      "avgReworkTimeAll": 150,
      "avgBenchmarkAgreement": 0.92,
      "avgConsensusAgreement": 0.87,
      "reworkPercentage": 0.03,
      "approvalPercentage": 0.95,
      "avgRating": 4.5,
      "isRemoved": false
    }
  ],
  "page": 1,
  "perPage": 50,
  "total": 37,
  "startDate": "2026-01-01",
  "endDate": "2026-01-31",
  "cachedAt": null,
  "generatedAt": "2026-02-01T09:30:00.000Z"
}
```

### Labeler fields

The Python SDK uses the same names in snake case, such as `labels_created`.

| Field | Description |
| - | - |
| `userId`, `email` | The member. See [Access and privacy](#access-and-privacy) for when the email is masked. |
| `memberType` | `internal` if the member's account belongs to your organization, `external` if it belongs to another one. `null` if the account no longer exists. |
| `labelsCreated` | Labels created by the member in the period. |
| `labelsSkipped` | Labels skipped by the member in the period. |
| `memberTotalTime` | Total time the member spent in the editor during the period. |
| `labelingTimeSubmitted` | Labeling time spent by the member on submitted or unsaved labels. |
| `labelingTimeSkipped` | Labeling time spent by the member on skipped labels. |
| `reviewsReceived` | Approve and Reject actions, through the workflow review queue, on labels the member created. |
| `reviewTimeAll` | Total review time spent on labels the member created. |
| `reworkTimeAll` | Total rework time spent on labels the member created. |
| `avgTimePerLabel` | Total labeling time divided by the number of labels worked on, including skipped and unsaved labels. |
| `avgReviewTimeAll` | Total review time divided by the number of labels reviewed, for labels the member created. |
| `avgReworkTimeAll` | Total rework time divided by the number of labels reworked, for labels the member created. |
| `reworkPercentage` | Labels the member created that were reworked, divided by the labels they created. |
| `approvalPercentage` | Labels the member created that received an Approve action, divided by those with any review action. |
| `avgConsensusAgreement` | Average agreement with other consensus labels, for labels the member created. |
| `avgBenchmarkAgreement` | Average agreement with benchmark labels, for labels the member created. |
| `avgRating` | The member's average rating on the project, or `null` if they have none. |
| `isRemoved` | `true` when the member is no longer on the project. |

### Reviewer fields

| Field | Description |
| - | - |
| `userId`, `email` | The member. |
| `memberType` | `internal` or `external`, as for labelers. |
| `labelsReviewed` | Labels that had review time from the member in the period. |
| `labelsReworked` | Labels that had rework time from the member in the period. |
| `memberTotalTime` | Total time the member spent in the editor during the period. |
| `reviewTime` | Total review time spent by the member. |
| `reworkTime` | Total rework time spent by the member. |
| `avgReviewTime` | Total review time divided by the number of labels reviewed. |
| `avgReworkTime` | Total rework time divided by the number of labels reworked. |
| `reworkPercentage` | Labels reviewed by the member that were reworked, divided by the labels they reviewed. |
| `approvalPercentage` | Labels with an Approve action by the member, divided by labels with any review action by the member. |
| `isRemoved` | `true` when the member is no longer on the project. |

## Download the report

Get a link to the project's performance report as a file. This is the file the Download button on the Performance dashboard gives you.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  report = project.get_performance_report_download("2026-01-01", "2026-01-31")
  print(report.url)
  ```

  ```bash cURL theme={"theme":"css-variables"}
  curl -H "Authorization: Bearer $LABELBOX_API_KEY" \
    "https://api.labelbox.com/api/v1/performance/projects/$PROJECT_ID/download?startDate=2026-01-01&endDate=2026-01-31"
  ```
</CodeGroup>

The response has the `url` and `expiresInSeconds`. The link works for 24 hours.

<Warning>
  Anyone who has the link can open the file until it expires. Treat it like the data it points to.
</Warning>

## Workspace reports

Workspace reports are the Monitor's data. They cover every project your organization owns and every project another organization shares with yours, or only the projects you name in `projectIds`.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  series = client.get_workspace_performance_metric(
      "labels_created", "2026-01-01", "2026-01-31", interval="week"
  )

  rows = client.get_workspace_labeler_performance(
      "2026-01-01", "2026-01-31", sort_by="labels_created", descending=True
  )
  print(rows.total)
  for row in rows:
      print(row.project_name, row.email, row.member_type, row.labels_created)
  ```

  ```bash cURL theme={"theme":"css-variables"}
  curl -H "Authorization: Bearer $LABELBOX_API_KEY" \
    "https://api.labelbox.com/api/v1/performance/workspace/metrics/labels_created?startDate=2026-01-01&endDate=2026-01-31&interval=week"

  curl -H "Authorization: Bearer $LABELBOX_API_KEY" \
    "https://api.labelbox.com/api/v1/performance/workspace/labelers?startDate=2026-01-01&endDate=2026-01-31&sort=labelsCreated&order=desc"
  ```
</CodeGroup>

`workspace/labelers` returns one row for each labeler on each project they worked on in the period. A row has the [labeler fields](#labeler-fields), without `avgRating`, plus:

| Field | Description |
| - | - |
| `projectId`, `projectName` | The project. `projectName` is `null` for a project that can no longer be found. |
| `projectOrganizationId`, `projectOrganizationName` | The organization that owns the project. It is not yours for a project shared with you. Both are `null` for a project that can no longer be found. |
| `isProjectDeleted` | `true` when the project has been deleted. Its rows and figures stay in the report. |

The table is [paged](#paging) like the project tables. Without `sort`, rows are ordered by project, then member.

Workspace reports do not take a user group filter.

## Shared projects and workforce providers

A project belongs to one organization and can be shared with another, for example when you add a workforce provider to it. The provider's members then work in your project, and the project appears in the provider's workspace as a shared project. The API reflects both sides of that connection.

**What a workspace report covers.** The projects your organization owns, and the projects shared with it. Two fields tell them apart on every row of `workspace/labelers`:

* `projectOrganizationId` is the organization that owns the project.
* `memberType` is whose member did the work: `internal` for a member of your organization, `external` for a member of another organization or an Alignerr member. The Monitor calls external members **Workforce members**.

Both are told from your side. To a workforce provider its own labelers are `internal`, and a customer's project has the customer's `projectOrganizationId`. Alignerr members are the exception: they are `external` to everyone, including to Alignerr's own organization.

**Split a report.** Use `memberType` and `ownerOrganizationIds` on either workspace report. These are the Monitor's **Member Type** and **Owned by** filters.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Labels created each week by your workforce provider's members
  external = client.get_workspace_performance_metric(
      "labels_created",
      "2026-01-01",
      "2026-01-31",
      interval="week",
      member_type="external",
  )

  # The same for your own members
  internal = client.get_workspace_performance_metric(
      "labels_created",
      "2026-01-01",
      "2026-01-31",
      interval="week",
      member_type="internal",
  )

  # As a workforce provider: your members' totals on one customer's projects
  rows = client.get_workspace_labeler_performance(
      "2026-01-01",
      "2026-01-31",
      owner_organization_ids=["<CUSTOMER_ORGANIZATION_ID>"],
      member_type="internal",
  )
  ```

  ```bash cURL theme={"theme":"css-variables"}
  curl -H "Authorization: Bearer $LABELBOX_API_KEY" \
    "https://api.labelbox.com/api/v1/performance/workspace/metrics/labels_created?startDate=2026-01-01&endDate=2026-01-31&interval=week&memberType=external"

  curl -H "Authorization: Bearer $LABELBOX_API_KEY" \
    "https://api.labelbox.com/api/v1/performance/workspace/labelers?startDate=2026-01-01&endDate=2026-01-31&ownerOrganizationIds=$CUSTOMER_ORGANIZATION_ID&memberType=internal"
  ```
</CodeGroup>

A few rules apply:

* **A split covers members and projects with activity in the period.** As on the Monitor, `memberType` and `ownerOrganizationIds` select from the members and projects that have rows in the report for your dates and other filters. If none match, every bucket is zero. It is never the unfiltered report.
* **A split by owner leaves out projects that are gone.** A project that can no longer be found has no owner, so `ownerOrganizationIds` never selects it, and the splits by owner can add up to slightly less than the whole workspace.
* **A split by owner leaves out projects you cannot report on.** Your workspace can include a project you have no role in. A metric split by `ownerOrganizationIds` does not count it, and lists it in `skippedProjectIds` so that you know.
* **A split of a metric needs permission to view everyone's performance.** With `projectIds`, `memberType` and `ownerOrganizationIds` on `workspace/metrics` need permission to view all performance in every project you name. Otherwise the request returns `403`. On `workspace/labelers` they filter the rows you can already see.
* **Project reports carry `memberType` too.** The labeler and reviewer tables of a project give each member's `memberType`.
* **Names and emails are masked between a customer and a workforce provider.** See [Access and privacy](#access-and-privacy). IDs are never masked, so you can always group and filter by `userId`, `projectId`, and `projectOrganizationId`.
* **User groups belong to the project's organization.** A `userGroupIds` filter on a project shared with you matches no work.

## Paging

The three tables (`projects/{projectId}/labelers`, `projects/{projectId}/reviewers`, and `workspace/labelers`) are paged. In the Python SDK, iterating fetches one page at a time, and `total` gives the number of rows without reading them all:

```python Python theme={"theme":"css-variables"}
rows = client.get_workspace_labeler_performance(
    "2026-01-01", "2026-01-31", page_size=100
)
print(rows.total)
for row in rows:  # fetches page after page as you go
    ...
```

In a request, use these parameters:

| Parameter | Description |
| - | - |
| `page` | Page number, starting at 1. Written in digits only. |
| `perPage` | Rows per page, up to 100. The default is 50. In the SDK, `page_size`. |
| `sort` | A field of the row, such as `labelsCreated`. In the SDK, `sort_by="labels_created"`. |
| `order` | `asc` (default) or `desc`. In the SDK, `descending=True`. |
| `useCache` | `false` reads the report again instead of using rows read in the last few minutes. In the SDK, `use_cache=False`. Not available on reviewers. |

Every response has `page`, `perPage`, and `total`, the number of rows across all pages. Request pages until you have `total` rows. A page past the end returns an empty `data`.

```bash cURL theme={"theme":"css-variables"}
curl -H "Authorization: Bearer $LABELBOX_API_KEY" \
  "https://api.labelbox.com/api/v1/performance/workspace/labelers?startDate=2026-01-01&endDate=2026-01-31&perPage=100&page=2"
```

Keep these in mind when you read a whole table:

* **Leave the cache on.** The pages of `workspace/labelers` are cut from one reading of the report, kept for five minutes, so they agree with each other and the rows are in a fixed order. `cachedAt` says when that reading was taken. With `useCache=false` every page reads the report again.
* **Sorting.** Every table sorts by the figures in its rows, such as `labelsCreated` or `avgTimePerLabel`, except `avgRating`. Without `sort`, the project tables come ordered by email, as on the Performance dashboard, and rows that tie under a sort are in that order too. A masked email keeps the place of the real one. `workspace/labelers` comes ordered by project, then member.
* **Metrics are not paged.** A metric returns at most one bucket per day of the period, and the period is limited to 366 days.

## Access and privacy

The API shows you what the pages show you.

* **Your own performance only.** If you can only view your own performance on a project, every project report returns only your data, whatever `userIds` you pass.
* **Workspace reports.** Without `projectIds`, you need permission to view performance across your organization. With `projectIds`, you need performance access to each project you name.
* **Projects you cannot see.** A project you have no performance access to returns `404`, the same as a project that does not exist.
* **Splitting by member type or owner.** On `workspace/metrics`, `memberType` and `ownerOrganizationIds` need permission to view all performance in every project you name in `projectIds`. A project found through its owner that you cannot report on is left out and listed in `skippedProjectIds`.
* **Masked emails.** The email of a member you may not identify is replaced with a masked value, as on the pages. For example, a workforce provider's members appear masked to the project's owner, and the owner's members appear masked to the provider.
* **Masked project and organization names.** A workforce provider does not see the names of the projects shared with it or of the organizations that own them. In `workspace/labelers`, `projectName` and `projectOrganizationName` are replaced with masked values for those projects. The IDs are unchanged.
* **Restricted API keys.** A key created with a limited set of permissions needs a performance permission, and is limited to the ones it has. A key that only has permission to view your own performance returns only your own.

## Data freshness

The API reads the same data as the Performance dashboard and the Monitor, so it is never more current than those pages. That data is rebuilt periodically, not continuously: the current day's figures can trail real activity by anything from a quarter of an hour to over an hour. A response served from the cache can be up to five minutes older again. `generatedAt` is when the response was produced. `cachedAt`, on the labeler tables, is set when the rows came from a result computed in the last few minutes. Neither tells you how recent the underlying activity data is, and no response carries an as-of time for it.

Two things follow for the current day:

* **Today's bucket keeps growing.** Read settled days for totals you will not revisit.
* **Charts and tables do not refresh together.** For settled days a metric and the labeler table normally add up to the same total. For the current day they can differ until both have caught up.

## Limits

| Limit | Value |
| - | - |
| Period | 366 days per request |
| `userIds`, `batchIds` | 100 IDs each |
| `userGroupIds` | 25 groups, with at most 5,000 members between them |
| `projectIds` | 50 projects |
| `ownerOrganizationIds` | 100 organizations |
| `perPage` | 100 rows |

Requests count toward your [API rate limit](/horizon/guides/limits), like the rest of the Labelbox API.

## Reliability

* **Requests are safe to repeat.** Every endpoint is a `GET` and changes nothing, so you can retry any request.
* **Retry `5xx` responses and timeouts** with a growing delay between attempts. The Python SDK does this for you, for up to four minutes, and tries once more after a request that timed out. When it stops retrying, it raises the error that kept happening.
* **Do not retry `4xx` responses** without changing the request. For `429`, wait before sending more.
* **Allow for long requests.** Reports are computed when you ask, and a long period over a large workspace can take a while. The Python SDK waits up to 120 seconds for a response.
* **Prefer fewer, wider requests.** One request for a month at `interval=day` costs less than a request for each day. To read every member's totals for a project as a file, use [Download the report](#download-the-report).
* **Expect new fields.** Responses can gain fields over time. Ignore the ones you do not know. The Python SDK does.

## Errors

| Status | Meaning | Python SDK exception |
| - | - | - |
| `400` | A parameter is invalid. The response lists each one in `details`. | `InvalidQueryError` |
| `401` | The API key is missing or invalid. | `AuthenticationError` |
| `403` | Your role or API key does not include the permission, or you named a project in a split by member type or owner where you cannot view everyone's performance. | `AuthorizationError` |
| `404` | The metric does not exist, or the project does not exist or is not visible to you. | `ResourceNotFoundError` |
| `429` | You have made too many requests. | `ApiLimitError` |
| `500` | The report could not be produced. Try again. | `InternalServerError` |
| `503` | Performance reports are under maintenance. Try again later. | `InternalServerError` |

The Python SDK raises `TimeoutError` when a request gets no response in time, and `NetworkError` when it cannot connect. All of these exceptions are in `lbox.exceptions`. It raises Python's `ValueError`, before sending anything, for a filter given as an empty list.

A `400` response names each parameter that was refused:

```json theme={"theme":"css-variables"}
{
  "statusCode": 400,
  "error": "Bad Request",
  "message": "Invalid query parameters",
  "details": [
    { "parameter": "endDate", "message": "must not be before startDate" }
  ]
}
```


# Project
Source: https://docs.labelbox.com/horizon/developer-guides/project

Developer guide for creating and modifying projects using the Python SDK.

<CardGroup>
  <Card title="Open in Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/basics/projects.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/basics/projects.ipynb" />
</CardGroup>

## Client

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  import labelbox as lb
  client = lb.Client(api_key="<YOUR_API_KEY>")
  ```
</CodeGroup>

## Create a project

When creating a project, specify a `media_type` using one of the following values:

* `lb.MediaType.Audio`
* `lb.MediaType.Conversational`
* `lb.MediaType.Document`
* `lb.MediaType.Geospatial_Tile`
* `lb.MediaType.Html`
* `lb.MediaType.Image`
* `lb.MediaType.Simple_Tile`
* `lb.MediaType.Text`
* `lb.MediaType.Video`

<CodeGroup>
  ```python Create a project theme={"theme":"css-variables"}
  # Create a new project
  project = client.create_project(
      name="<project_name>",
      description="<project_description>",    # optional
      media_type=lb.MediaType.Image           # specify the media type
  )
  ```
</CodeGroup>

## Get a project

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  project = client.get_project("<project_id>")

  # alternatively, you can get a dataset by name
  project = client.get_projects(where=lb.Project.name == "<project_name>").get_one()
  ```
</CodeGroup>

## Methods

### Create a batch

When creating a batch to send to a project, one of either `global_keys` or `data_rows` must be supplied as an argument. If using the `data_rows` argument, you can supply either a list of data row IDs or a list of `DataRow` class objects.

Optionally, you can supply a `priority`, ranging from 1 (highest) to 5 (lowest), for which the batch should be labeled. This will determine the order in which the included data rows appear in the labeling queue compared to other batches. If no value is provided, the batch will assume the lowest priority.

For more details, see [Batch](/horizon/developer-guides/batch).

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  project.create_batch(
    name="<unique_batch_name>",
    global_keys=["key1", "key2", "key3"],
    priority=5,
  )

  # if the project uses consensus, you can optionally supply a dictionary with consensus settings
  # if provided, the batch will use consensus with the specificed coverage and votes
  project.create_batch(
    name="<unique_batch_name>",
    data_rows=["<data_row_id>", "<data_row_id>"],
    priority=1,
    consensus_settings={"number_of_labels": 3, "coverage_percentage": 0.1}
  )
  ```
</CodeGroup>

#### Create multiple batches

The `project.create_batches()` method accepts up to 1 million data rows. Batches are chunked into groups of 100k data rows (if necessary), which is the maximum batch size.

This method takes in a list of either data row IDs or `DataRow`objects into a `data_rows` argument or global keys into a `global_keys` argument, but both approaches cannot be used in the same method. Batches will be created with the specified `name_prefix` argument and a unique suffix to ensure unique batch names. The suffix will be a 4-digit number starting at `0000`.

For example, if the name prefix is `demo-create-batches-` and three batches are created, the names will be `demo-create-batches-0000`, `demo-create-batches-0001`, and `demo-create-batches-0002`. This method will throw an error if a batch with the same name already exists.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  task = project.create_batches(
    name_prefix="demo-create-batches-",
    global_keys=global_keys,
    priority=5
  )

  print("Errors: ", task.errors())
  print("Result: ", task.result())
  ```
</CodeGroup>

#### Create batches from a dataset

If you wish to create batches in a project using all the data rows of a dataset, instead of gathering global keys or IDs and iterating over subsets of data rows, you can use the `project.create_batches_from_dataset()` method.

This method takes in a dataset ID and creates a batch (or batches if there are more than 100k data rows) comprised of all data rows not already in the project. The same logic applies to the `name_prefix` argument and the naming of batches as described in the section immediately above.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  dataset = client.get_dataset("<dataset_id>")

  task = project.create_batches_from_dataset(
      name_prefix="demo-dataset-",
      dataset_id=dataset.uid,
      priority=5
  )

  print("Errors: ", task.errors())
  print("Result: ", task.result())
  ```
</CodeGroup>

### Get the batches

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # get the batches (objects of the Batch class)
  batches = project.batches()

  # inspect one batch
  next(batches)

  # inspect all batches
  for batch in batches:
    print(batch)

  # for ease of use, you can convert the paginated collection to a list
  list(batches)
  ```
</CodeGroup>

### Get a batch

You can retrieve the batch of a particular project with`client.get_batch()`.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  project_id = "<project_id>"
  batch_id = "<batch_id>"

  # returns a Batch object
  batch = client.get_batch(project_id, batch_id)
  ```
</CodeGroup>

### Connect an ontology

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # the argument must be an object of the Ontology class
  project.connect_ontology(ontology)
  ```
</CodeGroup>

### Get the members and their roles

The scope of a member is provided by the attribute `access_from` from the class `ProjectMember`.

It can have one of the following values:

* ORGANIZATION: project membership is derived from the organization role
* PROJECT\_MEMBERSHIP: access is given specifically to the project
* USER\_GROUP: access is given via a group

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # get the members (objects of the ProjectMember class with relationships to a User and Role)
  members = project.members()

  # inspect one member
  member = next(members)
  print(member.user(), member.role(), member.access_from)

  # Display member info:
  print(member.user().uid, member.user().email, member.role().name, access_from, sep="\t")

  # inspect all members
  for member in members:
    print(member.user(), member.role(), access_from)

  # for ease of use, you can convert the paginated collection to a list
  list(members)
  ```
</CodeGroup>

### Upload labeling instructions

Note that if the ontology connected to your project is connected to other projects, calling this method will attach the instructions to those projects as well.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # must be a PDF or HTML file
  project.upsert_instructions("<local_file_path>")
  ```
</CodeGroup>

### Get the workflow tasks

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # get the task queues (relationship to TaskQueue objects)
  task_queues = project.task_queues()

  # inspect all task queues
  for task_queue in task_queues:
    print(task_queue)
  ```
</CodeGroup>

### Move data rows to a workflow task

Note that data rows need labels attached before being moved to a different workflow task. They can not be moved from "Initial Labeling."

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  project.move_data_rows_to_task_queue(
    data_row_ids=lb.GlobalKeys(["<global_key>", "<global_key>"]), # Use "lb.UniqueIds" for "<data_row_ids>"
    # Optional: If not included, defaults to "None", which moves data rows to the "Done" bucket
    task_queue_id="<task_queue_id>"
  )
  ```
</CodeGroup>

### Modify data row priority

Once a batch has been added to a project, you can set [the priority](/horizon/guides/labeling-queue) of its data rows. To do so, define a list of label parameter overrides (LPOs), which are [tuples](https://docs.python.org/3/library/ast.html?highlight=tuple#ast.Tuple) that set the priority for individual data rows.

Each override has three values: an object of the `DataRow` class or a `DataRowIdentifier` object, the new priority. All values must be integers that match the range of the list.

The priority is an integer between `-2,147,483,648` to `2,147,483,647`. The lowest value has the highest priority.

Override lists are limited to 1,000 items; larger lists trigger an error.

Once the override list is defined, pass it to `project.set_labeling_parameter_overrides` to change the priority of the corresponding data rows. Use `project.labeling_parameter_overrides` to get a list of [data row priorities](#get-data-row-priority) and `project.update_data_row_labeling_priority` to update existing data row priority.

#### Set data row priority

<CodeGroup>
  ```python Python (add LPOS) theme={"theme":"css-variables"}
  # Extract the global keys
  export_params = {
      "data_row_details": True
  }

  export_task = project.export(params=export_params)

  # Wait until the export task is complete
  export_task.wait_till_done()

  # Stream the export using a callback function
  def json_stream_handler(output: labelbox.BufferedJsonConverterOutput):
    print(output.json)

  export_task.get_buffered_stream(stream_type=labelbox.StreamType.RESULT).start(stream_handler=json_stream_handler)

  # Collect all exported data into a list
  export_json = [data_row.json for data_row in export_task.get_buffered_stream()]

  global_keys = [item["data_row"]["global_key"] for item in export_json]

  # Add LPOs
  lpos = []
  priority=1

  # With global keys
  for global_key in global_keys:
    lpos.append((lb.GlobalKey(global_key), priority))
    priority += 1

  # With data row ids
  # data_row_ids = ["clw7jlmav35yn0768xrpawwrc", "clw7jlmav35yo0768a5amfztu"]
  # for dr_id in data_row_ids:
  #   lpos.append((lb.UniqueId(dr_id), priority))
  #   priority+=1

  # With data row objects
  # data_rows = [data_row_1, data_row_2]

  # for data_row in data_rows:
  #   lpos.append((data_row, priority))
  #   priority+=1

  # Set data row priorities
  project.set_labeling_parameter_overrides(lpos)

  # Check results
  project_lpos = list(project.labeling_parameter_overrides())
  for lpo in project_lpos:
    print(lpo)
  ```
</CodeGroup>

#### Update data row priority

<CodeGroup>
  ```python Python (update LPOs) theme={"theme":"css-variables"}
  # Update LPOs

  # With global keys
  global_keys = ["global_key1", "global_key2"]
  project.update_data_row_labeling_priority(data_rows=lb.GlobalKeys(global_keys), priority=1)

  # With data row ids
  # data_row_ids = ["clw7jlmav35yn0768xrpawwrc", "clw7jlmav35yo0768a5amfztu"]
  # project.update_data_row_labeling_priority(data_rows=lb.UniqueIds(data_row_ids), priority=1)

  # With data row objects
  # data_rows = [data_row_1, data_row_2]
  # project.update_data_row_labeling_priority(data_rows=data_rows, priority=1)

  # Check results
  project_lpos = list(project.labeling_parameter_overrides())

  for lpo in project_lpos:
    print(lpo)
  ```
</CodeGroup>

### Add project tags

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  tags = project.update_project_resource_tags(["<project_tag_id>", "<project_tag_id>"])
  ```
</CodeGroup>

### Get project tags

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  tags = project.get_resource_tags()
  ```
</CodeGroup>

The `tags` variable is a list where each element is an object of type `ResourceTag` with the attributes, `uid`, `color`(ex: "008856") and`text`.

### Get the project overview

With `project.get_overview(details)`you can obtain some of the data from the Project Overview tab.

#### Output

The boolean parameter `details` will change the output to display the distribution of data rows between the queues.

When `details` is to false:

| Attribute | Description | Name in the Overview tab |
| - | - | - |
| `to_label` | Number of data rows that are yet to be labeled | To Label |
| `in_review` | Number of data rows to be reviewed | In Review |
| `in_rework` | Number of data rows to be reworked | In Rework |
| `skipped` | Number of skipped data rows | Skipped |
| `done` | Number of data rows marked as Done | Done |
| `issues` | Number of data rows with associated issues | Issues |
| `labeled` | Number of data rows with one or more labels | - |
| `total_data_rows` | Total number of data rows in the project | - |

When `details` is set to true, the output will be the same as before, except for the following:

| Attribute | Description |
| - | - |
| `in_review` | `data`: List of task queues in review with the associated number of data rows |
| | `total`: Number of data rows to be reviewed |
| `in_rework` | `data`: List of task queues in rework with the associated number of data rows |
| | `total`: Number of data rows to be reworked |

#### Equivalences

The following are equal:

| Attribute | Sum of attributes |
| - | - |
| `overview.labeled` | `overview.in_review + overview.in_rework + overview.done` |
| `overview.total_data_rows` | `overview.to_label + overview.in_review + overview.in_rework + overview.done` |

#### Example project overview

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Example without details
  overview = project.get_overview()

  # Selection of some attributes
  print(f"""
  To label:\t{overview.to_label / overview.total_data_rows:.2%}
  Labeled:\t{overview.labeled / overview.total_data_rows:.2%}
  """)

  # To label: 18.37%
  # Labeled: 81.63%

  # Example with details
  detailed_overview = project.get_overview(details=True)

  # Task queues in review
  print(f"""
  Number of data rows {detailed_overview.total_data_rows},
  In review:,
  \tQueues {detailed_overview.in_review["data"]},
  \tNumber of data rows {detailed_overview.in_review["total"]},
  In rework:
  \tQueues {detailed_overview.in_rework["data"]},
  \tNumber of data rows {detailed_overview.in_rework["total"]}
  """,
  sep="\n")

  # Number of data rows 23220,
  # In review:,
  # Queues [{'Initial review task': 7}],
  # Number of data rows 7,
  # In rework:
  # Queues [{'Rework (all rejected)': 1830}],
  # Number of data rows 1830
  ```
</CodeGroup>

### Manage bulk imports

Use the following methods to manage bulk imports:

* `project.get_mal_prediction_imports()` to retrieve the list of MAL import jobs.
* `project.get_label_imports()` to to retrieve the list of ground-truth import jobs.
* `MALPredictionImport.delete()` to delete a MAL import.

<Warning>
  ### Delete label imports

  The `MALPredictionImport.delete()` method can only delete MAL imports. To delete a ground-truth label, use the [Data Rows tab](/horizon/guides/data-rows-activity) on the web platform. Deleting an import is permanent and can't be undone.
</Warning>

### Export a project

For complete details, see [Export overview](/horizon/developer-guides/export-overview#export-data-rows-from-a-project).

<CodeGroup>
  ```python Export theme={"theme":"css-variables"}
    # The return type of this method is an `ExportTask`, which is a wrapper of a`Task`
    # Most of `Task` features are also present in `ExportTask`.
    export_params= {
      "attachments": True,
      "metadata_fields": True,
      "data_row_details": True,
      "project_details": True,
      "label_details": True,
      "performance_details": True,
      "interpolated_frames": True
    }

    # Note: Filters follow AND logic, so typically using one filter is sufficient.
    filters= {
      "last_activity_at": ["2000-01-01 00:00:00", "2050-01-01 00:00:00"],
      "label_created_at": ["2000-01-01 00:00:00", "2050-01-01 00:00:00"],
      "workflow_status": "InReview",
      "batch_ids": ["batch_id_1", "batch_id_2"],
      "data_row_ids": ["data_row_id_1", "data_row_id_2"],
      "global_keys": ["global_key_1", "global_key_2"]
    }

    export_task = project.export(params=export_params, filters=filters)
    export_task.wait_till_done()

    # Stream the export using a callback function
    def json_stream_handler(output: labelbox.BufferedJsonConverterOutput):
      print(output.json)

    export_task.get_buffered_stream(stream_type=labelbox.StreamType.RESULT).start(stream_handler=json_stream_handler)

    # Collect all exported data into a list
    export_json = [data_row.json for data_row in export_task.get_buffered_stream()]

    print("file size: ", export_task.get_total_file_size(stream_type=lb.StreamType.RESULT))
    print("line count: ", export_task.get_total_lines(stream_type=lb.StreamType.RESULT))
  ```
</CodeGroup>

### Export issues and comments

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  import requests

  url = project.export_issues()
  issues = requests.get(url).json()

  # optionally, you can export only the open or resolved issues
  open_issues_url = project.export_issues(status="Open")
  resolved_issues_url = project.export_issues(status="Resolved")
  ```
</CodeGroup>

### Duplicate a project

See the section [Duplicate a project](/horizon/guides/create-a-project#duplicate-a-project) for the scope of this method.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  cloned_project = project.clone()
  ```
</CodeGroup>

### Update a project

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  project.update(name="<new_project_name>")
  ```
</CodeGroup>

### Delete a project

<Danger>
  **Deleting a project cannot be undone**

  This method deletes the project along with all labels made in the project. This action cannot be reverted.
</Danger>

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  project.delete()
  ```
</CodeGroup>

## Attributes

### Get the basics

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # name (str)
  project.name

  # description (str)
  project.description

  # updated at (datetime)
  project.updated_at

  # created at (datetime)
  project.created_at

  # last activity time (datetime)
  project.last_activity_time

  # number of required labels per consensus data row (int)
  project.auto_audit_number_of_labels

  # default percentage of consensus data rows per batch (float)
  project.auto_audit_percentage

  # created by (relationship to User object)
  user = project.created_by()

  # organization (relationship to Organization object)
  organization = project.organization()
  ```
</CodeGroup>

### Get the ontology

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # get the ontology connected to the project (relationship to Ontology object)
  ontology = project.ontology()
  ```
</CodeGroup>

### Get the benchmarks

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # get the benchmarks (relationship to Benchmark objects)
  benchmarks = project.benchmarks()

  # inspect one benchmark
  next(benchmarks)

  # inspect all benchmarks
  for benchmark in benchmarks:
    print(benchmark)

  # for ease of use, you can convert the paginated collection to a list
  list(benchmarks)
  ```
</CodeGroup>

### Get the webhooks

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # get the webhooks connected to the project (relationship to Webhook objects)
  webhooks = project.webhooks()

  # inspect one webhook
  next(webhooks)

  # inspect all webhooks
  for webhook in webhooks:
    print(webhook)

  # for ease of use, you can convert the paginated collection to a list
  list(webhooks)
  ```
</CodeGroup>

### Get data row priority

Use `project.labeling_parameter_overrides` to get a list of labeling parameter overrides (LPOs), which define the priority for each label in the override list. Use `set_labeling_parameter_overrides` and `update_data_row_labeling_priority` to [modify data row priority](#set-data-row-priority).

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # gets the LPOs created in the project (relationship to LabelingParameterOverride objects)
  lpos = project.labeling_parameter_overrides()

  # inspect one LPO
  next(lpos)

  # inspect all LPOs
  for lpo in lpos:
    print(lpo)

  # for ease of use, you can convert the paginated collection to a list
  list(lpos)

  # Get the data row id
  for lpo in lpos:
    print(lpo)
    print("Data row:", lpo.data_row().uid)
  ```
</CodeGroup>

### Get the number of labels

Use `project.get_label_count()` to return the sum of labels in the different task queues of a project.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Return the number of
  project.get_label_count()
  ```
</CodeGroup>

## Copy data rows and labels

To copy our data rows and labels to a different project from a source project, use the `client.send_to_annotate_from_catalog` method with our Labelbox client.

Send to Annotate does not currently support consensus projects.

### Parameters

When you send data rows with labels to our destination project, you may choose to include or exclude certain parameters inside a Python dictionary, at a minimum, a `source_project_id` will need to be provided:

* `source_project_id`
  * The id of the project where our data rows with labels will originate.
* `annotation_ontology_mapping`
  * A dictionary containing the mapping of the source project's ontology feature schema IDs to the destination project's ontology feature schema IDs. If left empty, only the data rows with no labels will be sent to our destination project.
  * `{"<source_feature_schema_id>" : "<destination_feature_schema_id>"}`
* `exclude_data_rows_in_project`
  * Excludes data rows that are already in the project.
* `override_existing_annotations_rule`
  * The strategy defines how to handle conflicts in classifications between the data rows that already exist in the project and incoming labels from the source project.
    * Defaults to `ConflictResolutionStrategy.KeepExisting`
    * Options include:
      * `ConflictResolutionStrategy.KeepExisting`
      * `ConflictResolutionStrategy.OverrideWithPredictions`
      * `ConflictResolutionStrategy.OverrideWithAnnotations`
* `param batch_priority`
  * The priority of the batch.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  from labelbox.schema.conflict_resolution_strategy import ConflictResolutionStrategy

  send_to_annotate_params = {
    "source_project_id": project.uid,
    "annotations_ontology_mapping": annotation_ontology_mapping, # to be defined
    "exclude_data_rows_in_project": False,
    "override_existing_annotations_rule": ConflictResolutionStrategy.OverrideWithPredictions,
    "batch_priority": 5,
  }

  # Get task id to workflow you want to send data rows. If sent to initial labeling queue, labels will be pre-labels.
  queue_id = [queue.uid for queue in destination_project.task_queues()
    if queue.queue_type == "MANUAL_REVIEW_QUEUE" ][0]

  task = client.send_to_annotate_from_catalog(
    destination_project_id=destination_project.uid,
    task_queue_id=queue_id, # ID of workflow task, set ID to None if you want to send data rows with labels to the Done queue.
    batch_name="Prediction Import Demo Batch",
    data_rows=lb.GlobalKeys(
      global_keys # Provide a list of global keys from source project
    ),
    params=send_to_annotate_params
  )

  task.wait_till_done()

  print(f"Errors: {task.errors}")
  ```
</CodeGroup>


# Prompt and response projects
Source: https://docs.labelbox.com/horizon/developer-guides/prompt-and-response-projects

How to set up a prompt and response type project

<CardGroup>
  <Card title="Open in Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/project_configuration/prompt_response_projects.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/tree/main/project_configuration/prompt_response_projects.ipynb" />
</CardGroup>

Prompt and response creation projects are set up differently than other Labelbox projects. They have unique methods and modifications to existing methods. This guide will showcase the differences and provide an example workflow.

## Before you start

The below imports are needed to use the code examples in this section.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  import labelbox as lb
  ```
</CodeGroup>

## API key and client

Please provide a valid API key below to connect to the Labelbox client properly. For more information, please review the [Create API key guide](/horizon/developer-guides/create-api-key).

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  API_KEY = None
  client = lb.Client(api_key=API_KEY)
  ```
</CodeGroup>

## Create a prompt and response ontology

You can create ontologies for prompt and response projects in the same way as other project ontologies using two methods: `client.create_ontology` and `client.create_ontology_from_feature_schemas`. For response creation projects the only difference between other projects is the media\_type for the project needs to be set to `lb.MediaType.Text`. For prompt and prompt response creation projects you need to include their respective media type: `lb.MediaType.LLMPromptCreation` and `lb.MediaType.LLMPromptResponseCreation`. Additional you also need to provide an additional parameter of `ontology_kind`, which needs to be set to `lb.OntologyKind.ResponseCreation` this is only applicable for prompt and prompt response creation projects.

### Option A: `create_ontology`

Typically, you create ontologies and generate the associated features simultaneously. Below is an example of creating an ontology for your prompt and response projects using supported classifications; for information on supported annotation types, visit our [prompt and response generation](/horizon/guides/prompt-and-response-generation-editor) guide.

Depending on whether you are creating a prompt, a response, or both, certain classifications in your ontologies might not be necessary. See [supported annotation types](/horizon/guides/prompt-and-response-generation-editor#supported-prompt-formats) for more information.

<CodeGroup>
  ```python Prompt and response creation theme={"theme":"css-variables"}
  ontology_builder = lb.OntologyBuilder(
      tools=[],
      classifications=[
          lb.PromptResponseClassification(
              class_type=lb.PromptResponseClassification.Type.PROMPT,
              name="prompt text",
              character_min = 1, # Minimum character count of prompt field (optional)
              character_max = 20, # Maximum character count of prompt field (optional)
          ),
          lb.PromptResponseClassification(
              class_type=lb.PromptResponseClassification.Type.RESPONSE_CHECKLIST,
              name="response checklist feature",
              options=[
                  lb.Option(value="option 1", label="option 1"),
                  lb.Option(value="option 2", label="option 2"),
              ],
          ),
          lb.PromptResponseClassification(
              class_type=lb.PromptResponseClassification.Type.RESPONSE_RADIO,
              name="response radio feature",
              options=[
                  lb.Option(value="first_radio_answer"),
                  lb.Option(value="second_radio_answer"),
              ],
          ),
          lb.PromptResponseClassification(
              class_type=lb.PromptResponseClassification.Type.RESPONSE_TEXT,
              name="response text",
              character_min = 1, # Minimum character count of response text field (optional)
              character_max = 20, # Maximum character count of response text field (optional)
          )
      ],
  )

  # Create ontology

  ontology = client.create_ontology(
  "Prompt and response ontology",
  ontology_builder.asdict(),
  media_type=lb.MediaType.LLMPromptResponseCreation,
  )

  ```

  ```python Prompt creation theme={"theme":"css-variables"}
  ontology_builder = lb.OntologyBuilder(
      tools=[],
      classifications=[
          lb.PromptResponseClassification(
              class_type=lb.PromptResponseClassification.Type.PROMPT,
              name=f"prompt text",
              character_min = 1, # Minimum character count of prompt field (optional)
              character_max = 20, # Maximum character count of prompt field (optional)
          )
      ],
  )

  # Create ontology
  ontology = client.create_ontology(
      "Prompt ontology",
      ontology_builder.asdict(),
      media_type=lb.MediaType.LLMPromptCreation,
  )
  ```

  ```python Response creation theme={"theme":"css-variables"}
  ontology_builder = lb.OntologyBuilder(
      tools=[],
      classifications=[
          lb.PromptResponseClassification(
              class_type=lb.PromptResponseClassification.Type.RESPONSE_CHECKLIST,
              name="response checklist feature",
              options=[
                  lb.Option(value="option 1", label="option 1"),
                  lb.Option(value="option 2", label="option 2"),
              ],
          ),
          lb.PromptResponseClassification(
              class_type=lb.PromptResponseClassification.Type.RESPONSE_RADIO,
              name="response radio feature",
              options=[
                  lb.Option(value="first_radio_answer"),
                  lb.Option(value="second_radio_answer"),
              ],
          ),
          lb.PromptResponseClassification(
              class_type=lb.PromptResponseClassification.Type.RESPONSE_TEXT,
              name=f"response text",
              character_min = 1, # Minimum character count of response text field (optional)
              character_max = 20, # Maximum character count of response text field (optional)
          )
      ],
  )

  # Create ontology
  ontology = client.create_ontology(
      "Response ontology",
      ontology_builder.asdict(),
      media_type=lb.MediaType.Text,
      ontology_kind=lb.OntologyKind.ResponseCreation
  )
  ```
</CodeGroup>

### Option B: `create_ontology_from_feature_schemas`

You can also create ontologies using feature schema IDs. This makes your ontologies come with existing features instead of generating new features. You can get these features by going to the **Schema** tab inside Labelbox.

<CodeGroup>
  ```python Prompt and response creation theme={"theme":"css-variables"}
  ontology = client.create_ontology_from_feature_schemas(
      "Prompt and response ontology",
      feature_schema_ids=["<list of feature schema ids"],
      media_type=lb.MediaType.LLMPromptResponseCreation
  )
  ```

  ```python Prompt creation theme={"theme":"css-variables"}
  ontology = client.create_ontology_from_feature_schemas(
      "Prompt ontology",
      feature_schema_ids=["<list of feature schema ids"],
      media_type=lb.MediaType.LLMPromptCreation
  )
  ```

  ```python Response creation theme={"theme":"css-variables"}
  ontology = client.create_ontology_from_feature_schemas(
      "Response ontology",
      feature_schema_ids=["<list of feature schema ids"],
      media_type=lb.MediaType.Text,
      ontology_kind=lb.OntologyKind.ResponseCreation
  )
  ```
</CodeGroup>

## Create response creation projects

You can create response creation projects using `client.create_response_creation_project`, which uses the same parameters as `client.create_project` but provides better validation to ensure the project is set up correctly. Additionally, you need to import text data rows to be used as prompts.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  project = client.create_response_creation_project(
      name="<project_name>",
      description="<project_description>",  # optional
  )
  ```
</CodeGroup>

## Create prompt response and prompt creation projects

When creating a prompt response or prompt creation project using `client.create_prompt_response_generation_project`, you do not need to create data rows because they are generated automatically. This method takes the same parameters as the traditional `client.create_project` but with a few specific additional parameters.

### Parameters

The `client.create_prompt_response_generation_project` method requires the following parameters:

* `create_prompt_response_generation_project` parameters:

  * `name` (required): The name of your new project.

  * `description`: An optional description of your project.

  * `media_type` (required): The type of assets this project accepts. Can be either `lb.MediaType.LLMPromptCreation` or `MediaType.LLMPromptResponseCreation`, depending on the project type you are setting up.

  * `dataset_name`: The name of the dataset where the generated data rows will be located. Include this parameter only if you want to create a new dataset.

  * `dataset_id`: An optional dataset ID of an existing Labelbox dataset. Include this parameter if you want to append it to an existing dataset.

  * `data_row_count`: The number of data row assets that will be generated and used with your project.

<CodeGroup>
  ```python Prompt and response creation theme={"theme":"css-variables"}
  project = client.create_prompt_response_generation_project(
      name="Demo prompt response project",
      media_type=lb.MediaType.LLMPromptResponseCreation,
      dataset_name="Demo prompt response dataset",
      data_row_count=100,
  )

  # Setup project with ontology created above

  project.connect_ontology(ontology)

  ```

  ```python Prompt creation theme={"theme":"css-variables"}
  project = client.create_prompt_response_generation_project(
      name="Demo prompt project",
      media_type=lb.MediaType.LLMPromptCreation,
      dataset_name="Demo prompt dataset",
      data_row_count=100,
  )

  # Setup project with ontology created above
  project.connect_ontology(ontology)
  ```
</CodeGroup>

## Exporting prompt response, prompt or response create project

Exporting from a prompt and response type project works the same as exporting from other projects. In this example, your export will be empty unless you create labels within the Labelbox platform. See [prompt and response export](/horizon/developer-guides/export-prompt-response-data) for a sample export.

<CodeGroup>
  ```python Export theme={"theme":"css-variables"}
  # The return type of this method is an `ExportTask`, which is a wrapper of a`Task`
  # Most of `Task` features are also present in `ExportTask`.

  export_params = {
  "attachments": True,
  "metadata_fields": True,
  "data_row_details": True,
  "project_details": True,
  "label_details": True,
  "performance_details": True,
  "interpolated_frames": True,
  }

  # Note: Filters follow AND logic, so typically using one filter is sufficient.

  filters = {
  "last_activity_at": ["2000-01-01 00:00:00", "2050-01-01 00:00:00"],
  "label_created_at": ["2000-01-01 00:00:00", "2050-01-01 00:00:00"],
  "workflow_status": "InReview",
  "batch_ids": ["batch_id_1", "batch_id_2"],
  "data_row_ids": ["data_row_id_1", "data_row_id_2"],
  "global_keys": ["global_key_1", "global_key_2"],
  }

  export_task = project.export(params=export_params, filters=filters)
  export_task.wait_till_done()

  # Stream the export using a callback function

  def json_stream_handler(output: labelbox.BufferedJsonConverterOutput):
  print(output.json)

  export_task.get_buffered_stream(stream_type=labelbox.StreamType.RESULT).start(stream_handler=json_stream_handler)

  # Collect all exported data into a list

  export_json = [data_row.json for data_row in export_task.get_buffered_stream()]

  print("file size: ", export_task.get_total_file_size(stream_type=lb.StreamType.RESULT))
  print("line count: ", export_task.get_total_lines(stream_type=lb.StreamType.RESULT))

  ```
</CodeGroup>


# Quick start
Source: https://docs.labelbox.com/horizon/developer-guides/quick-start

Short introduction towards Labelbox Python SDK, geared mainly for new users.

<CardGroup>
  <Card title="Open in Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/basics/quick_start.ipynb" />

  <Card title="Open in GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/basics/quick_start.ipynb" />
</CardGroup>

This short introduction to Labelbox Python SDK is designed mainly for new users. The example will introduce you to Labelbox Python SDK concepts that you will use commonly. We will demonstrate a simple but common workflow of importing an image data row, setting up a project, and exporting.

## Example: Import data rows to exporting from project

### Before you start

We must first install the `Labelbox` library and then import the SDK module. It is recommended to install `"labelbox[data]"` over `labelbox` to obtain all the correct dependencies. We will also be importing the Python UUID library to generate universal unique IDs for the variety of objects that will be created with this notebook.

<CodeGroup>
  ```shell Shell theme={"theme":"css-variables"}
  pip install -q "labelbox[data]"
  ```
</CodeGroup>

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  import labelbox as lb
  import uuid
  ```
</CodeGroup>

Replace the value of `API_KEY` with a valid [API key](/horizon/developer-guides/create-api-key) to connect to the Labelbox client.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  API_KEY = None
  client = lb.Client(API_KEY)
  ```
</CodeGroup>

## Step 1: Create dataset and import data row

Below, we will create a <Tooltip>dataset</Tooltip> and then attach a publicly hosted image <Tooltip>data row</Tooltip>. Typically, you would import data rows hosted on a cloud provider (*recommended*) or import them locally. For more information, visit our [import image data section](/horizon/developer-guides/image) in our developer guides. You can find your dataset inside the [*Catalog*](/horizon/guides/catalog-overview) section of Labelbox.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Create dataset from client
  dataset = client.create_dataset(name="Quick Start Example Dataset")

  global_key = str(uuid.uuid4()) # Unique user specified ID

  # Data row structure

  image_data_rows = [{
    "row_data": "https://storage.googleapis.com/labelbox-datasets/image_sample_data/2560px-Kitano_Street_Kobe01s5s4110.jpeg",
    "global_key": global_key,
    "media_type": "IMAGE",
  }]

  # Bulk import data row

  task = dataset.create_data_rows(image_data_rows) # List of data rows
  task.wait_till_done()
  print(task.errors) # Print any errors
  ```
</CodeGroup>

## Step 2: Creating an ontology

Before sending our data row to a labeling project, we must create an <Tooltip>ontology</Tooltip>. The example below will create a simple ontology with a bounding box tool and a checklist classification feature. For more information, visit the [ontology section](/horizon/developer-guides/ontology) inside our developer guides.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Bounding box feature
  object_features = [
      lb.Tool(
          tool=lb.Tool.Type.BBOX,
          name="regulatory-sign",
          color="#ff0000",
      )
  ]

  # Checklist feature
  classification_features = [
      lb.Classification(
          class_type=lb.Classification.Type.CHECKLIST,
          name="Quality Issues",
          options=[
              lb.Option(value="blurry", label="Blurry"),
              lb.Option(value="distorted", label="Distorted"),
          ],
      )
  ]

  # Builder function
  ontology_builder = lb.OntologyBuilder(tools=object_features,
                                        classifications=classification_features)

  # Create ontology
  ontology = client.create_ontology(
      "Ontology from new features",
      ontology_builder.asdict(),
      media_type=lb.MediaType.Image,
  )
  ```
</CodeGroup>

## Step 3: Creating a project and attaching our ontology

Now that we have made our ontology, we are ready to create a <Tooltip>project</Tooltip> where we can label our data row.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Create a new project
  project = client.create_project(
      name="Quick Start Example Project",
      media_type=lb.MediaType.Image,
  )

  # Attach created ontology

  project.connect_ontology(ontology)
  ```
</CodeGroup>

## Step 4: Sending our data row to our project by creating a batch

With our project created, we can send our data rows by creating a <Tooltip>batch</Tooltip>. Our data rows will start in the initial labeling queue, where labelers are able to annotate our data row. For more information on batches, review the [batches section](/horizon/developer-guides/batch#create-a-batch) of our developer guides.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  project.create_batch(
      name="Quick Start Example Batch" + str(uuid.uuid4()),
      global_keys=[
          global_key
      ],  # Global key we used earlier in this guide to create our dataset
      priority=5,
  )
  ```
</CodeGroup>

## Step 5: Exporting from our project

We have now successfully set up a project for labeling using only the SDK!

From here, you can either label our data row directly inside the [labeling queue](/horizon/guides/labeling-queue) or [import annotations](/horizon/developer-guides/import-image-annotations) directly through our SDK. Below we will demonstrate the final step of this guide by exporting from our project. Since we did not label any data rows or import annotations within this guide, no labels will be presented on our data row. For a full overview of exporting, visit our [export overview](/horizon/developer-guides/export-overview) developer guide.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Start export from project
  export_task = project.export()
  export_task.wait_till_done()

  # Stream the export using a callback function

  def json_stream_handler(output: labelbox.BufferedJsonConverterOutput):
  print(output.json)

  export_task.get_buffered_stream(stream_type=labelbox.StreamType.RESULT).start(stream_handler=json_stream_handler)

  # Collect all exported data into a list

  export_json = [data_row.json for data_row in export_task.get_buffered_stream()]
  ```
</CodeGroup>

## Clean up

This section serves as an optional clean-up step to delete the Labelbox assets created within this guide. You will need to uncomment the delete methods shown.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # project.delete()
  # client.delete_unused_ontology(ontology.uid)
  # dataset.delete()
  ```
</CodeGroup>


# Fundamental concepts
Source: https://docs.labelbox.com/horizon/developer-guides/sdk-fundamental-concepts

Overview of concepts used throughout Labelbox-Python SDK.

## DbObject

The Labelbox SDK is primarily used to interact with Labelbox's database. The core abstraction used to represent a database entity is a DbObject. It has special attributes called Fields and Relationships that facilitate database queries.

Most objects within the Labelbox SDK are DbOjects. You will see `Project` and `Dataset`, for example, among others throughout the documentation.

## Field caching

When you fetch an object from the server, the client obtains all field values for that object. When you access that obtained field value, the cached value is returned. There is no round-trip fetch to the server to get the field value you have already fetched. Server-side updates made after the client-side fetch are not auto-propagated, meaning the values returned will still be the cached values.

## Fields

Each DbObject has a set of fields. The below sections explain how to access or update one or more fields on a DbObject.

### Fetch

DbObject fields must be accessed as attributes (e.g.,`project.name`). Unlike relationships, fields are cached.

In this example, `client.get_projects()` returns a `PaginatedCollection` object. You can iterate through the `PaginatedCollection` object and specify the fields you want to be returned in the fetch. To fetch multiple fields from an object, you must separate them by commas.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  from labelbox import Client

  client = Client()
  projects = client.get_projects()
  for project in projects:
      print(project.name, project.uid)
  ```
</CodeGroup>

### Update

To update a field, use the `DbObject.update()` method to pass the field and the new value. Each data update that uses`DbObject.update()` on the client side immediately performs the same update on the server side. If the client-side update does not raise an exception, you can assume the update was successfully passed on the server side.

This example uses the`project.update()` method to change a project's name and description.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  project = client.get_project("<project_id>")
  project.update(name="Project Name", description="New description")
  ```
</CodeGroup>

## Use comparison operators

Paginated collections support comparisons for filtering a query.

For example, you can use the `get_datasets` method with a `where` parameter with a comparison operator to filter. Any of the standard comparison operators (`==`, `!=`, `>`, `=>`, `<`, `<=`) will work. The `get_datasets` method will give you a `PaginatedCollection`. You can iterate over the `PaginatedCollection` to get your dataset.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  from labelbox import Dataset
  datasets_x = client.get_datasets(where=Dataset.name == "X")
  for x in datasets_x:
      print(x)
  ```
</CodeGroup>

### Use combined comparisons

Combine comparisons using logical expressions to filter projects. Currently, the `where` clause supports the logical AND operator.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  from labelbox import Project, Client

  client = Client()
  projects = client.get_projects(where=(Project.name == "X") & (Project.description == "Y"))
  for x in projects:
      print(x)
  ```
</CodeGroup>

## Pagination

Some API calls return a large number of results. To prevent too many results from being returned at once, the Labelbox API limits the number of returned objects, and the Python SDK automatically creates a `PaginatedCollection` instead.

1. For both the top-level object fetch, `client.get_projects()`, and the relationship call, `project.batches()`, a `PaginatedCollection` object is returned. This `PaginatedCollection` object takes care of the paginated fetching.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  projects = client.get_projects()
  type(projects)
  ```
</CodeGroup>

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  project = projects[0]
  datasets = project.batches()
  type(batches)
  for batch in batches:
      batch.name
  ```
</CodeGroup>

2. Note that nothing is fetched immediately when the `PaginatedCollection` object is created.

3. Round-trips to the server are made only as you iterate through a `PaginatedCollection`. In this code block, that happens when a `list` is initialized with a `PaginatedCollection`, and when a `PaginatedCollection` is iterated over in a for-loop.

4. You cannot get a count of objects in the relationship from a PaginatedCollection nor access objects within it like a Python list (using squared-bracket indexing). You can only iterate over it.

### Iterate over PaginatedCollection

Be careful about converting a `PaginatedCollection` into a list. This will cause all objects in that collection to be fetched from the server.

In cases when you need only some objects, you can use two convenient methods: `get_one()` and `get_many(n)`.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  data_row = dataset.data_rows().get_one() # returns first result
  # or
  data_rows = dateset.date_rows.get_many(10) # returns list of 10 first results
  ```
</CodeGroup>

You can still iterate over a PaginatedCollection object as with a normal iterator object:

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  data_rows = dataset.data_rows()
  for data_row in data_rows:
    	# your business logic
      pass
  ```
</CodeGroup>


# Catalog slice
Source: https://docs.labelbox.com/horizon/developer-guides/slice

How to programmatically retrieve data from a Catalog slice through the SDK

## Get Catalog slices via SDK

You can programmatically retrieve a slice's data rows and all associated information via our Python SDK. From there, you can use Catalog to inspect the data rows you retrieved via the SDK visually.

Retrieving a slice programmatically is a convenient way to curate a new batch or a model run dataset directly from a saved slice.

# End-to-end example: create and retrieve Catalog slices

### Before you start

You must import these libraries to use the code examples in this section

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  import labelbox as lb
  import uuid
  ```
</CodeGroup>

### Replace with your API key

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  API_KEY = ""
  # To get your API key go to: Workspace settings -> API -> Create API Key
  client = lb.Client(api_key=API_KEY)
  ```
</CodeGroup>

## Create a Catalog slice

Currently, we do not support creating slices through the SDK; for the purpose of this demo, we will create a catalog slice through the UI.

<Steps>
  <Step>
    Navigate to the **Catalog** section of the Labelbox Platform, and select **All datasets** or a particular dataset you would like to create a slice from.
  </Step>

  <Step>
    Navigate to **Search your data** dropdown menu or use a **similarity search** to create a filter.
  </Step>

  <Step>
    Hit **Enter** and select **Save slice**
  </Step>

  <Step>
    Give the slice name and select **Save**
  </Step>

  <Step>
    Copy the **Slice ID**
  </Step>

  <Step>
    Paste **Slice ID**

    <CodeGroup>
      ```python Python theme={"theme":"css-variables"}
      catalog_slice_id = "<CATALOG_SLICE_ID_FROM_UI>"
      ```
    </CodeGroup>
  </Step>
</Steps>

### Get Catalog Slice

<CodeGroup>
  ```python Get Catalog slices by ID theme={"theme":"css-variables"}
  catalog_slice = client.get_catalog_slice(catalog_slice_id)
  ```

  ```python Get Catalog slices by name theme={"theme":"css-variables"}
  # May return a list of slices if multiple slices have the same name
  catalog_slice = client.get_catalog_slice(name="slice to search")
  ```

  ```python Get all Catalog slices theme={"theme":"css-variables"}
  catalog_slices = client.get_catalog_slices()
  ```
</CodeGroup>

### Obtain Data Row IDs and Data Row objects from the Catalog slice

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Get a data row id
  slice_data_rows_ids = catalog_slice.get_data_row_ids()

  # Get a data row objects
  for data_row_id in slice_data_rows_ids:
    print(client.get_data_row(data_row_id))
  ```
</CodeGroup>

### Obtain Data Row identifiers

Data row identifiers are objects that contain both the data row ID(s) and global keys.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  data_row_identifiers = catalog_slice.get_data_row_identifiers()

  drids = [dr for dr in data_row_identifiers]

  # get both global keys and data row ids
  # and utilize the hash method to combine both global keys and data row ids into a dictionary
  for dr in drids:
    print(f"Data row: {dr.id}, Global Key: {dr.global_key}, dr_gk: {dr.to_hash()}")
  ```
</CodeGroup>

## Curate a batch from a Catalog slice via SDK

You can create a new batch from your slice or create a random sample from a slice using our Python SDK. See the Python example below to learn how to do this.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Optional: sample Data rows from your Slice
  sampled_data_row_ids = random.sample(slice_data_rows_ids, 5)

  batch = project.create_batch(
    "test batch", # name of the batch
    sampled_data_row_ids, # list of Data Rows
    1 # priority between 1-5
  )
  ```
</CodeGroup>

You can append data rows to your model runs for inference from your slice. See the Python example below to learn how to do this.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  model_run.upsert_data_rows(list(slice_data_rows_ids))
  ```
</CodeGroup>


# Task objects
Source: https://docs.labelbox.com/horizon/developer-guides/tasks

Summary of task objects

[`Task`](https://labelbox-python.readthedocs.io/en/latest/task.html) objects represent operations you performed using the SDK or the UI, such as importing and exporting annotations. The `Task` object can make your workflow synchronous, especially for longer operations.

`DataUpsertTask` is a subclass of `Task` that represents data row upsert operations.

## Methods

The main method associated with a task object is `wait_till_done`, which waits until the task is completed by periodically querying the server to update the task attributes. This method takes two parameters:

1. `timeout_seconds`: Represents the maximum time `wait_till_done` can block your code. The default value is five minutes, which should be increased for longer tasks.
2. `check_frequency`: Defines how often your task queries the server to update the task attributes, which defaults to two seconds. This is useful to limit the amount of API calls against our server in a given time frame.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  dataset = client.get_dataset("<dataset id>")

  task = dataset.create_data_rows([{
    "row_data": item_url,
    "global_key": "<unique_global_key>"
  }])
  task.wait_till_done(timeout_seconds=5_000, check_frequency=10)
  ```
</CodeGroup>

### Task Results

The task result is a collection of details, typically information on the data rows associated with the task.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  dataset = client.get_dataset("<dataset id>")

  task = dataset.create_data_rows([{
    "row_data": item_url,
    "global_key": "<unique_global_key>"
  }])
  task.wait_till_done()

  print(task.result) # prints the data rows associated with the create_data_rows operation
  ```
</CodeGroup>

<Warning>
  ### Warning

  Large results (over 150,000 data rows) can take up to 10 mins to process.
</Warning>

### Task Errors

The task errors work similarly to the task results but contain information on any associated errors.

<CodeGroup>
  ```python errors theme={"theme":"css-variables"}
  dataset = client.get_dataset("<dataset id>")

  task = dataset.create_data_rows([{
    "row_data": item_url,
    "global_key": "<unique_global_key>"
  }])
  task.wait_till_done()

  print(task.errors) # prints the data rows associated with the create_data_rows operation
  ```
</CodeGroup>

## Get tasks

You can view tasks created in the past 30 days in the UI on the [Notifications](https://app.labelbox.com/notifications) page. You can also fetch a specific task by its ID using `client.get_task_by_id()`.

To retrieve tasks programmatically, you can query all tasks in an organization or filter them by status using the `where` parameter on `organization.tasks`. Task statuses are exposed via `Task.status_as_enum`.

Valid statuses are:

* `TaskStatus.In_Progress`
* `TaskStatus.Complete`
* `TaskStatus.Canceling`
* `TaskStatus.Canceled`
* `TaskStatus.Failed`

<CodeGroup>
  ```python Get task by ID theme={"theme":"css-variables"}
  id = "" # Your task ID string
  client.get_task_by_id(id)
  ```

  ```python Get tasks by status theme={"theme":"css-variables"}
  # All tasks
  tasks = organization.tasks()

  # Tasks in progress
  org_tasks_in_progress = organization.tasks(
      where=Task.status_as_enum == TaskStatus.In_Progress
  )

  # Failed tasks
  org_task_failed = organization.tasks(
      where=Task.status_as_enum == TaskStatus.Failed
  )
  ```
</CodeGroup>

<Warning>
  ### Export tasks are not supported

  Currently, you can't retrieve export tasks using this method.
</Warning>

## Cancel tasks

You can cancel the following types of tasks before they complete using their task id strings:

* [Streamable export tasks](/horizon/developer-guides/export-overview#cancel-export-tasks)
* [Sending to annotate](/horizon/developer-guides/foundry-app-developer-guide#step-7-send-annotations-from-catalog-to-annotate)
* [Sending to experiment](/horizon/developer-guides/model#add-data-rows-to-a-model-run)
* Bulk-updating queue priority
* Classification creation
* Agreement backfill for benchmark and consensus

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Cancel the task before it completes
  success = client.cancel_task(id)
  assert success is True

  # Verify the task was cancelled
  cancelled_task = client.get_task_by_id(id)
  assert cancelled_task.status in ["CANCELING", "CANCELED"]
  ```
</CodeGroup>


# Upload conversational text predictions
Source: https://docs.labelbox.com/horizon/developer-guides/upload-conversational-text-predictions

How to upload predictions on conversational text data in a model run and sample upload formats.

<CardGroup>
  <Card title="Open In Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/prediction_upload/conversational_predictions.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/prediction_upload/conversational_predictions.ipynb" />
</CardGroup>

## Supported predictions

To upload predictions in Labelbox, you need to create a predictions payload. In this section, we provide this payload for every supported prediction type.

Labelbox supports two formats for the predictions payload:

* Python annotation types (recommended)
* NDJSON

Both are described below.

### Entity

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  ner_prediction = lb_types.ObjectAnnotation(
      name="ner",
      confidence=0.5,
      value=lb_types.ConversationEntity(
          start=0,
          end=8,
          message_id="4"
      )
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  ner_prediction_ndjson = {
    "name": "ner",
    "confidence":0.5,
    "location": {
      "start": 0,
      "end": 8
    },
    "messageId": "4"
  }
  ```
</CodeGroup>

### Global classification: Radio

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  radio_prediction = lb_types.ClassificationAnnotation(
      name="radio_question",
      value=lb_types.Radio(answer = lb_types.ClassificationAnswer(name = "second_radio_answer", confidence=0.5))
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  radio_prediction_ndjson = {
    "name": "radio_question",
    "answer": {"name": "second_radio_answer", "confidence": 0.5}
  }
  ```
</CodeGroup>

### Message-based classification: Radio

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  radio_prediction = lb_types.ClassificationAnnotation(
      name="radio_convo",
      value=lb_types.Radio(answer = lb_types.ClassificationAnswer(name = "first_radio_answer", confidence=0.5)),
      message_id="0"
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  radio_prediction_ndjson = {
      "name": "radio_convo",
      "answer": {
          "name": "first_radio_answer", "confidence":0.5
      },
      "messageId": "0",
  }
  ```
</CodeGroup>

### Message-based classification: Free-form text

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  # Confidence scores are not supported for text predictions
  text_prediction = lb_types.ClassificationAnnotation(
      name="text_convo",
      value=lb_types.Text(answer="the answer to the text questions are right here"),
      message_id="0"
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  text_prediction_ndjson = {
      "name": "text_convo",
      "answer": "the answer to the text questions are right here",
      "messageId": "0"
  }
  ```
</CodeGroup>

### Message-based classification: Checklist

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  checklist_prediction= lb_types.ClassificationAnnotation(
    name="checklist_convo", # must match your ontology feature"s name
    value=lb_types.Checklist(
        answer = [
          lb_types.ClassificationAnswer(
              name = "first_checklist_answer", confidence=0.5
          ),
          lb_types.ClassificationAnswer(
              name = "second_checklist_answer", confidence=0.5
          )
        ]
      ),
    message_id="2"
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  checklist_prediction_ndjson = {
      "name": "checklist_convo",
      "answers": [
          {"name": "first_checklist_answer", "confidence":0.5},
          {"name": "second_checklist_answer", "confidence":0.5}
      ],
      "messageId": "2"
  }
  ```
</CodeGroup>

### Message-based nested classifications: Checklist and radio

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  nested_checklist_prediction = lb_types.ClassificationAnnotation(
    name="nested_checklist_question",
    message_id="10",
    value=lb_types.Checklist(
      answer=[lb_types.ClassificationAnswer(
        name="first_checklist_answer",
        confidence=0.5, # Confidence scores should be added to the answer 
        classifications=[
          lb_types.ClassificationAnnotation(
            name="sub_checklist_question",
            value=lb_types.Checklist(
              answer=[lb_types.ClassificationAnswer(
              name="first_sub_checklist_answer",
              confidence=0.5 # Confidence scores should be added to the answer 
            )]
          ))
        ]
      )]
    )
  )

  nested_radio_prediction = lb_types.ClassificationAnnotation(
    name="nested_radio_question",
    value=lb_types.Radio(
      answer=lb_types.ClassificationAnswer(
        name="first_radio_answer",
        confidence=0.5, # Confidence scores should be added to the answer 
        classifications=[
          lb_types.ClassificationAnnotation(
            name="sub_radio_question",
            value=lb_types.Radio(
              answer=lb_types.ClassificationAnswer(
                name="first_sub_radio_answer",
                confidence=0.5 # Confidence scores should be added to the answer
              )
            )
          )
        ]
      )
    )
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  nested_checklist_prediction_ndjson = {
    "name": "nested_checklist_question",
    "messageId": "10",
    "answer": [{
        "name": "first_checklist_answer",
        "confidence": 0.5, # Confidence scores should be added to the answer
        "classifications" : [
          {
            "name": "sub_checklist_question",
            "answer": {
              "name": "first_sub_checklist_answer",
              "confidence": 0.5, # Confidence scores should be added to the answer
            }
          }
        ]
    }]
  }

  nested_radio_prediction_ndjson = {
    "name": "nested_radio_question",
    "answer": {
        "name": "first_radio_answer",
        "confidence": 0.5,
        "classifications": [{
            "name":"sub_radio_question",
            "answer": { "name" : "first_sub_radio_answer",
                       "confidence": 0.5}
          }]
      }
  }
  ```
</CodeGroup>

## End-to-end example: Upload predictions to a model run

Here are the steps to upload predictions to a model run:

### Before you start

You will need to import these libraries to use the code examples in this section:

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  import labelbox as lb
  import labelbox.types as lb_types
  import uuid
  ```
</CodeGroup>

Replace the value of `API_KEY` with a valid [API key](/horizon/developer-guides/create-api-key) to connect to the Labelbox client.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  API_KEY = None
  client = lb.Client(API_KEY)
  ```
</CodeGroup>

### Step 1: Import data rows into Catalog

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  global_key = "conversation-1.json"

  asset = {
    "row_data": "https://storage.googleapis.com/labelbox-developer-testing-assets/conversational_text/1000-conversations/conversation-1.json",
    "global_key": global_key,
  }

  dataset = client.create_dataset(name="conversational_annotation_import_demo_dataset")

  task = dataset.create_data_rows([asset])
  task.wait_till_done()

  print("Errors:", task.errors)
  print("Failed data rows: ", task.failed_data_rows)
  ```
</CodeGroup>

### Step 2: Create/select an ontology for your model predictions

For best results, verify your ontology before submitting a model run. Verify that:

* Your ontology supports all features and classifications that might be needed during the model run.
* Tool names and classification instructions match the `name` and `instructions` fields in your annotation payloads.

This helps ensure that feature schemas match your intentions.

<CodeGroup>
  ```python Python expandable theme={"theme":"css-variables"}
  ontology_builder = lb.OntologyBuilder(
    tools=[
      lb.Tool(tool=lb.Tool.Type.NER,name="ner")
      ],
    classifications=[
      lb.Classification(
        class_type=lb.Classification.Type.TEXT,
        scope=lb.Classification.Scope.INDEX,
        instructions="text_convo"),
      lb.Classification(
        class_type=lb.Classification.Type.CHECKLIST,
        scope=lb.Classification.Scope.INDEX,
        instructions="checklist_convo",
        options=[
          lb.Option(value="first_checklist_answer"),
          lb.Option(value="second_checklist_answer")
        ]
      ),
      lb.Classification(
        class_type=lb.Classification.Type.RADIO,
        instructions="radio_convo",
        scope=lb.Classification.Scope.INDEX,
        options=[
          lb.Option(value="first_radio_answer"),
          lb.Option(value="second_radio_answer")
        ]
      ),
      lb.Classification(
        class_type=lb.Classification.Type.CHECKLIST,
        name="nested_checklist_question",
        scope = lb.Classification.Scope.INDEX,
        options=[
            lb.Option("first_checklist_answer",
              options=[
                lb.Classification(
                    class_type=lb.Classification.Type.CHECKLIST,
                    name="sub_checklist_question",
                    options=[lb.Option("first_sub_checklist_answer")]
                )
            ])
        ]
      ),
      lb.Classification(
          class_type=lb.Classification.Type.RADIO,
          name="nested_radio_question",
          scope = lb.Classification.Scope.GLOBAL,
          options=[
              lb.Option("first_radio_answer",
                  options=[
                      lb.Classification(
                          class_type=lb.Classification.Type.RADIO,
                          name="sub_radio_question",
                          options=[lb.Option("first_sub_radio_answer")]
                      )
                  ])
            ]
      )
    ]
  )

  ontology = client.create_ontology("Ontology Conversation Annotations", ontology_builder.asdict())
  ```
</CodeGroup>

### Step 3: Create a model and a model run

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # create model
  model = client.create_model(name="Conversational_model_run_"+ str(uuid.uuid4()),
                              ontology_id=ontology.uid)
  # create model run
  model_run = model.create_model_run("iteration 1")

  ```
</CodeGroup>

### Step 4: Send data rows to the model run

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  model_run.upsert_data_rows(global_keys=[global_key])
  ```
</CodeGroup>

### Step 5: Create the predictions payload

Create the prediction payload using the code snippets in the section above.

Labelbox supports two formats for the annotations payload: NDJSON and Python annotation types. Both are described below to compose your annotations into labels attached to the data rows.

The resulting payload should have exactly the same content for the annotations supported by both formats.

#### Python annotations

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  label_prediction = []
  label_prediction.append(
    lb_types.Label(
  		data= {"global_key": global_key}),
      annotations=[
        ner_prediction,
        checklist_prediction,
        text_prediction,
        radio_prediction,
        nested_checklist_prediction,
        nested_radio_prediction
      ]
    )
  )
  ```
</CodeGroup>

#### NDJSON

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  label_prediction_ndjson = []
  for annotations in [
      ner_prediction_ndjson,
      text_prediction_ndjson,
      checklist_prediction_ndjson,
      radio_prediction_ndjson,
      nested_checklist_prediction_ndjson,
      nested_radio_prediction_ndjson
      ]:
    annotations.update({
        "dataRow": {
            "globalKey": global_key
        }
    })
    label_prediction_ndjson.append(annotations)
  ```
</CodeGroup>

### Step 6: Upload the predictions payload to the model run

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Upload the prediction label to the Model Run
  upload_job_prediction = model_run.add_predictions(
      name="prediction_upload_job"+str(uuid.uuid4()),
      predictions=label_prediction)

  # Errors will appear for annotation uploads that failed.

  print("Errors:", upload_job_prediction.errors)
  print("Status of uploads: ", upload_job_prediction.statuses)

  ```
</CodeGroup>

### Step 7: Send annotations to the model run

To send annotations to a model run, we must first import them into a project, create a label payload and then send them to the model run.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # 7.1 Create a labelbox project
  project = client.create_project(name="Conversational Text Prediction Import Demo",
                                      media_type=lb.MediaType.Conversational)
  project.connect_ontology(ontology)

  # 7.2 Create a batch to send to the project
  project.create_batch(
    "batch_convo_prediction_demo", # Each batch in a project must have a unique name
    global_keys=[global_key], # Paginated collection of data row objects, list of data row ids or global keys
    priority=5 # priority between 1(Highest) - 5(lowest)
  )

  # 7.3 Create the annotations payload
  ner_annotation = lb_types.ObjectAnnotation(
      name="ner",
      value=lb_types.ConversationEntity(
          start=0,
          end=8,
          message_id="4"
      )
  )

  text_annotation = lb_types.ClassificationAnnotation(
      name="text_convo",
      value=lb_types.Text(answer="the answer to the text questions are right here"),
      message_id="0"
  )

  checklist_annotation= lb_types.ClassificationAnnotation(
    name="checklist_convo", # must match your ontology feature"s name
    value=lb_types.Checklist(
        answer = [
          lb_types.ClassificationAnswer(
              name = "first_checklist_answer"
          ),
          lb_types.ClassificationAnswer(
              name = "second_checklist_answer"
          )
        ]
      ),
    message_id="2"
   )

  radio_annotation = lb_types.ClassificationAnnotation(
      name="radio_convo",
      value=lb_types.Radio(answer = lb_types.ClassificationAnswer(name = "first_radio_answer")),
      message_id="0"
  )

  nested_checklist_annotation = lb_types.ClassificationAnnotation(
    name="nested_checklist_question",
    message_id="10",
    value=lb_types.Checklist(
      answer=[lb_types.ClassificationAnswer(
        name="first_checklist_answer",
        classifications=[
          lb_types.ClassificationAnnotation(
            name="sub_checklist_question",
            value=lb_types.Checklist(
              answer=[lb_types.ClassificationAnswer(
              name="first_sub_checklist_answer"
            )]
          ))
        ]
      )]
    )
  )

  nested_radio_annotation = lb_types.ClassificationAnnotation(
    name="nested_radio_question",
    value=lb_types.Radio(
      answer=lb_types.ClassificationAnswer(
        name="first_radio_answer",
        classifications=[
          lb_types.ClassificationAnnotation(
            name="sub_radio_question",
            value=lb_types.Radio(
              answer=lb_types.ClassificationAnswer(
                name="first_sub_radio_answer"
              )
            )
          )
        ]
      )
    )
  )

  # 7.4 Create the label object
  label = []
  label.append(
    lb_types.Label(
  		data= {"global_key": global_key},
      annotations=[
        ner_annotation,
        text_annotation,
        checklist_annotation,
        radio_annotation,
        nested_radio_annotation,
        nested_checklist_annotation
      ]
    )
  )

  # 7.5 Upload annotations to the project using label import
  upload_job_annotation = lb.LabelImport.create_from_objects(
      client = client,
      project_id = project.uid,
      name="text_label_import_job"+ str(uuid.uuid4()),
      labels=label)

  upload_job_annotation.wait_until_done()

  # Errors will appear for annotation uploads that failed.
  print("Errors:", upload_job_annotation.errors)
  print("Status of uploads: ", upload_job_annotation.statuses)

  # 7.6 Send the annotations to the model run
  # get the labels from the project
  model_run.upsert_labels(project_id=project.uid)
  ```
</CodeGroup>


# Upload document predictions
Source: https://docs.labelbox.com/horizon/developer-guides/upload-document-prediction

How to upload predictions on geospatial data in a model run and sample upload formats.

<CardGroup>
  <Card title="Open In Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/prediction_upload/pdf_predictions.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/prediction_upload/pdf_predictions.ipynb" />
</CardGroup>

## Supported predictions

To upload predictions in Labelbox, you need to create a predictions payload. In this section, we provide this payload for every supported prediction type.

Labelbox supports two formats for the predictions payload:

* Python annotation types (recommended)
* NDJSON

Both are described below.

### Entity

The `textSelections` field is required in the payload for each entity annotation. Each `textSelections` item in the list requires the following fields:

* A list of `token_ids` for each word in the group of words.
* The `group_id` associated with a group of words.
* The `page` of the document (1-indexed).

Both the `token_ids`and the `group_id` are extracted from the text layer URL attached to the data row. Please follow the end-to-end demo to learn how to construct an entity annotation for documents.

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  entities_prediction = lb_types.ObjectAnnotation(
      name="named_entity",
      confidence=0.5,
      value= lb_types.DocumentEntity(
          name="named_entity",
          textSelections=[
              lb_types.DocumentTextSelection(
                  token_ids=[],
                  group_id="",
                  page=1
              )
          ]
      )
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  entities_prediction_ndjson = {
      "name": "named_entity",
      "confidence": 0.5,
      "textSelections": [
          {
              "tokenIds": [
                  "<UUID>",
              ],
              "groupId": "<UUID>",
              "page": 1,
          }
      ]
  }
  ```
</CodeGroup>

### Classification: Radio

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  radio_prediction = lb_types.ClassificationAnnotation(
      name="radio_question",
      value=lb_types.Radio(answer =
          lb_types.ClassificationAnswer(name = "first_radio_answer", confidence=0.5)
      )
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  radio_prediction_ndjson = {
    "name": "radio_question",
    "answer": {"name": "first_radio_answer", "confidence": 0.5}
  }
  ```
</CodeGroup>

### Classification: Checklist

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  checklist_prediction = lb_types.ClassificationAnnotation(
      name="checklist_question",
      value=lb_types.Checklist(answer = [
          lb_types.ClassificationAnswer(name = "first_checklist_answer", confidence=0.5),
          lb_types.ClassificationAnswer(name = "second_checklist_answer", confidence=0.5)
      ])
    )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  checklist_prediction_ndjson = {
    "name": "checklist_question",
    "answer": [
      {"name": "first_checklist_answer", "confidence": 0.5},
      {"name": "second_checklist_answer", "confidence": 0.5}
    ]
  }
  ```
</CodeGroup>

### Bounding box

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  bbox_dim_1 = {
    "top": 135.3,
    "left": 102.771,
    "height": 109.843,
    "width": 415.8
  }

  bbox_prediction = lb_types.ObjectAnnotation(
    name="bounding_box",  # must match your ontology feature"s name
    value=lb_types.DocumentRectangle(
      start=lb_types.Point(x=bbox_dim_1["left"], y=bbox_dim_1["top"]),  # x = left, y = top 
      end=lb_types.Point(x=bbox_dim_1["left"] + bbox_dim_1["width"], y=bbox_dim_1["top"]+ bbox_dim_1["height"]),  # x= left + width , y = top + height
      page=0,
      unit=lb_types.RectangleUnit.POINTS
    )
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  bbox_dim_1 = {
    "top": 135.3,
    "left": 102.771,
    "height": 109.843,
    "width": 415.8
  }

  bbox_prediction_ndjson = {
    "name": "bounding_box",
    "bbox": bbox_dim_1,
    "page": 0,
    "unit": "POINTS"
  }
  ```
</CodeGroup>

### Nested classifications: Checklist and radio

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  nested_checklist_prediction = lb_types.ClassificationAnnotation(
    name="nested_checklist_question",
    value=lb_types.Checklist(
      answer=[lb_types.ClassificationAnswer(
        name="first_checklist_answer",
        confidence=0.5, # Confidence scores should be added to the answer 
        classifications=[
          lb_types.ClassificationAnnotation(
            name="sub_checklist_question",
            value=lb_types.Checklist(
              answer=[lb_types.ClassificationAnswer(
              name="first_sub_checklist_answer",
              confidence=0.5 # Confidence scores should be added to the answer 
            )]
          ))
        ]
      )]
    )
  )

  nested_radio_prediction = lb_types.ClassificationAnnotation(
    name="nested_radio_question",
    value=lb_types.Radio(
      answer=lb_types.ClassificationAnswer(
        name="first_radio_answer",
        confidence=0.5, # Confidence scores should be added to the answer 
        classifications=[
          lb_types.ClassificationAnnotation(
            name="sub_radio_question",
            value=lb_types.Radio(
              answer=lb_types.ClassificationAnswer(
                name="first_sub_radio_answer",
                confidence=0.5 # Confidence scores should be added to the answer 
              )
            )
          )
        ]
      )
    )
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  nested_checklist_prediction_ndjson = {
    "name": "nested_checklist_question",
    "answer": [{
        "name": "first_checklist_answer", 
        "confidence": 0.5, # Confidence scores should be added to the answer 
        "classifications" : [
          {
            "name": "sub_checklist_question", 
            "answer": {
              "name": "first_sub_checklist_answer", 
              "confidence": 0.5, # Confidence scores should be added to the answer 
            }
          }          
        ]         
    }]
  }

  nested_radio_prediction_ndjson = {
    "name": "nested_radio_question",
    "answer": {
        "name": "first_radio_answer",
        "confidence": 0.5,
        "classifications": [{
            "name":"sub_radio_question",
            "answer": { "name" : "first_sub_radio_answer", 
                       "confidence": 0.5}
          }]
      }
  }
  ```
</CodeGroup>

### Classification: Free-form text

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  text_prediction = lb_types.ClassificationAnnotation(
    name="free_text",  # must match your ontology feature"s name
    value=lb_types.Text(answer="sample text")
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  text_prediction_ndjson = {
    "name": "free_text",
    "answer": "sample text"
  }
  ```
</CodeGroup>

### Bounding box with nested classification

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  bbox_dim = {
    "top": 226.757,
    "left": 317.271,
    "height": 194.229,
    "width": 249.386
  }

  bbox_with_radio_subclass_prediction = lb_types.ObjectAnnotation(
    name="bbox_with_radio_subclass",
    confidence=0.5, 
    value=lb_types.DocumentRectangle(
      start=lb_types.Point(x=bbox_dim["left"], y=bbox_dim["top"]), # x = left, y = top 
      end=lb_types.Point(x=bbox_dim["left"] + bbox_dim["width"], y=bbox_dim["top"] + bbox_dim["height"]), # x= left + width , y = top + height
      unit=lb_types.RectangleUnit.POINTS,
      page=1
    ),
    classifications=[
      lb_types.ClassificationAnnotation(
        name="sub_radio_question",
       	value=lb_types.Radio(
          answer=lb_types.ClassificationAnswer(
            name="first_sub_radio_answer",
            confidence=0.5, 
            classifications=[
              lb_types.ClassificationAnnotation(
                name="second_sub_radio_question",
                value=lb_types.Radio(
                  answer=lb_types.ClassificationAnswer(
                    name="second_sub_radio_answer",
                    confidence=0.5
                  )
                )
              )
            ]
          )
        )
      )
    ]
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  bbox_with_radio_subclass_prediction_ndjson = {
    "name": "bbox_with_radio_subclass",
    "classifications": [
      {
        "name": "sub_radio_question", 
        "answer": {
          "name": "first_sub_radio_answer", 
          "confidence": 0.5,
          "classifications": [
            {
              "name": "second_sub_radio_question", 
              "answer": {
                "name": "second_sub_radio_answer", "confidence": 0.5}
            }
          ]
        }
      }
    ],
    "bbox":bbox_dim,
    "page": 1,
    "unit": "POINTS"
  }
  ```
</CodeGroup>

### Entity with nested classification

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  ner_with_checklist_subclass_prediction = lb_types.ObjectAnnotation(
    name="ner_with_checklist_subclass",
   confidence=0.5,
    value=lb_types.DocumentEntity(
      name="ner_with_checklist_subclass",
      text_selections=[
        lb_types.DocumentTextSelection(
          token_ids=[],
          group_id="",
          page=1
        )
      ]
    ),
    classifications=[
      lb_types.ClassificationAnnotation(
        name="sub_checklist_question",
        value=lb_types.Checklist(
        answer=[lb_types.ClassificationAnswer(name="first_sub_checklist_answer", confidence=0.5)]
        )
      )
    ]
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  ner_with_checklist_subclass_prediction_ndjson = {
    "name": "ner_with_checklist_subclass",
    "classifications":[
      {
        "name": "sub_checklist_question",
        "answer": [{"name": "first_sub_checklist_answer", "confidence":0.5 }]
      }
    ],
    "textSelections": [
        {
            "tokenIds": [
                ""
            ],
            "groupId": "",
            "page": 1
        }
    ]
  }
  ```
</CodeGroup>

## End-to-end example: Upload predictions to a model run

Here are the steps to upload predictions to a model run:

### Before you start

You will need to import these libraries to use the code examples in this section:

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  import uuid
  import json
  import requests
  import labelbox as lb
  import labelbox.types as lb_types
  ```
</CodeGroup>

Replace the value of `API_KEY` with a valid [API key](/horizon/developer-guides/create-api-key) to connect to the Labelbox client.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  API_KEY = None
  client = lb.Client(API_KEY)
  ```
</CodeGroup>

### Step 1: Import data rows into Catalog

<CodeGroup>
  ```python theme={"theme":"css-variables"}
  ## Text layer url is required for uploading entity annotations
  global_key = "0801.3483.pdf"
  img_url = {
      "row_data": {
        "pdf_url": "https://storage.googleapis.com/labelbox-datasets/arxiv-pdf/data/99-word-token-pdfs/0801.3483.pdf",
        "text_layer_url": "https://storage.googleapis.com/labelbox-datasets/arxiv-pdf/data/99-word-token-pdfs/0801.3483-lb-textlayer.json"
      },
      "global_key": global_key
  }


  dataset = client.create_dataset(name="pdf_demo_dataset")
  task = dataset.create_data_rows([img_url])
  task.wait_till_done()
  print(f"Failed data rows: {task.failed_data_rows}")
  print(f"Errors: {task.errors}")

  if task.errors:
      for error in task.errors:
          if 'Duplicate global key' in error['message'] and dataset.row_count == 0:
              # If the global key already  exists in the workspace the dataset will be created empty, so we can delete it.
              print(f"Deleting empty dataset: {dataset}")
              dataset.delete()
  ```
</CodeGroup>

### Step 2: Create/select an ontology for your model predictions

Your model run should have the correct ontology setup with all the tools and classifications supported for your predictions, and the tool names and classification instructions should match the `name`/`instructions` fields in your annotation payloads to ensure the correct feature schemas are matched.

<CodeGroup>
  ```python Python expandable theme={"theme":"css-variables"}
  ## Setup the ontology and link the tools created above.

  ontology_builder = lb.OntologyBuilder(
    classifications=[ # List of Classification objects
      lb.Classification(
        class_type=lb.Classification.Type.RADIO,
        name="radio_question",
        scope = lb.Classification.Scope.GLOBAL,
        options=[
          lb.Option(value="first_radio_answer"),
          lb.Option(value="second_radio_answer")
        ]
      ),
      lb.Classification(
        class_type=lb.Classification.Type.CHECKLIST,
        name="checklist_question",
        scope = lb.Classification.Scope.GLOBAL,
        options=[
          lb.Option(value="first_checklist_answer"),
          lb.Option(value="second_checklist_answer")
        ]
      ),
      lb.Classification(
        class_type=lb.Classification.Type.TEXT,
        name="free_text",
        scope = lb.Classification.Scope.GLOBAL
      ),
      lb.Classification(
          class_type=lb.Classification.Type.RADIO,
          name="nested_radio_question",
          scope = lb.Classification.Scope.GLOBAL,
          options=[
              lb.Option("first_radio_answer",
                  options=[
                      lb.Classification(
                          class_type=lb.Classification.Type.RADIO,
                          name="sub_radio_question",
                          options=[lb.Option("first_sub_radio_answer")]
                      )
                  ])
            ]
      ),
      lb.Classification(
        class_type=lb.Classification.Type.CHECKLIST,
        name="nested_checklist_question",
        scope = lb.Classification.Scope.GLOBAL,
        options=[
            lb.Option("first_checklist_answer",
              options=[
                lb.Classification(
                    class_type=lb.Classification.Type.CHECKLIST,
                    name="sub_checklist_question",
                    options=[lb.Option("first_sub_checklist_answer")]
                )
            ])
        ]
      ),
    ],
    tools=[ # List of Tool objects
      lb.Tool( tool=lb.Tool.Type.BBOX,name="bounding_box"),
      lb.Tool(tool=lb.Tool.Type.NER, name="named_entity"),
      lb.Tool(tool=lb.Tool.Type.NER,
              name="ner_with_checklist_subclass",
              classifications=[
                lb.Classification(
                  class_type=lb.Classification.Type.CHECKLIST,
                  name="sub_checklist_question",
                  options=[
                    lb.Option(value="first_sub_checklist_answer")
                  ]
                )
            ]),
      lb.Tool( tool=lb.Tool.Type.BBOX,
              name="bbox_with_radio_subclass",
              classifications=[
                lb.Classification(
                    class_type=lb.Classification.Type.RADIO,
                    name="sub_radio_question",
                    options=[
                      lb.Option(
                        value="first_sub_radio_answer" ,
                        options=[
                          lb.Classification(
                            class_type=lb.Classification.Type.RADIO,
                            name="second_sub_radio_question",
                            options=[lb.Option("second_sub_radio_answer")]
                          )]
                      )]
                  )]
        )]
  )

  ontology = client.create_ontology("Document Annotation Import Demo",
                                    ontology_builder.asdict(),
                                    media_type=lb.MediaType.Document)
  ```
</CodeGroup>

### Step 3: Create a model and a model run

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # create model
  model = client.create_model(name="PDF_model_run_"+ str(uuid.uuid4()),
                              ontology_id=ontology.uid)
  # create model run
  model_run = model.create_model_run("iteration 1")

  ```
</CodeGroup>

### Step 4: Send data rows to the model run

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  model_run.upsert_data_rows(global_keys=[global_key])
  ```
</CodeGroup>

### Step 5: Create the predictions payload

To import ner annotations, you can either pass a`text_layer_url` or use Labelbox-generated `text_layer_url`.

To extract the generated text layer url we first need to export the data row

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  task = lb.DataRow.export(client=client,global_keys=[global_key])
  task.wait_till_done()

  if task.has_result():
  stream = task.get_buffered_stream()

      text_layer = ""
      for output in stream:
          output_json = output.json
          text_layer = output_json['media_attributes']['text_layer_url']
      print(text_layer)

  ```
</CodeGroup>

<CodeGroup>
  ```python theme={"theme":"css-variables"}
  import requests
  import json

  # Helper method
  def update_text_selections(annotation, group_id, list_tokens, page):
    return annotation.update({
      "textSelections": [
        {
          "groupId": group_id,
          "tokenIds": list_tokens,
          "page": page
        }
      ]
    })
    

  # Fetch the content of the text layer
  res = requests.get(text_layer) 

  # Phrases that we want to annotate obtained from the text layer url
  content_phrases = ["Metal-insulator (MI) transitions have been one of the", "T. Sasaki,* N. Yoneyama, and N. Kobayashi"]

  # Parse the text layer
  text_selections = []
  text_selections_ner = []


  for obj in json.loads(res.text):
    for group in obj["groups"]:
      if group["content"] == content_phrases[0]:
        list_tokens = [x["id"] for x in group["tokens"]]
        # build text selections for Python annotations
        document_text_selection = lb_types.DocumentTextSelection(groupId=group["id"], tokenIds=list_tokens, page=1)
        text_selections.append(document_text_selection)
        # build text selection for the NDJson annotations
        update_text_selections(annotation=entities_prediction_ndjson,
                               group_id=group["id"], # id representing group of words 
                               list_tokens=list_tokens, # ids representing individual words from the group
                               page=1)
      if group["content"] == content_phrases[1]:
        list_tokens_2 = [x["id"] for x in group["tokens"]]
        # build text selections for Python annotations
        ner_text_selection = lb_types.DocumentTextSelection(groupId=group["id"], tokenIds=list_tokens_2, page=1)
        text_selections_ner.append(ner_text_selection)
        # build text selection for the NDJson annotations
        update_text_selections(annotation=ner_with_checklist_subclass_prediction_ndjson,
                               group_id=group["id"], # id representing group of words 
                               list_tokens=list_tokens_2, # ids representing individual words from the group
                               page=1)


  #re-write the entity annotation with text selections 
  entities_prediction_document_entity = lb_types.DocumentEntity(name="named_entity",confidence=0.5, textSelections = text_selections)
  entities_prediction = lb_types.ObjectAnnotation(name="named_entity",value=entities_prediction_document_entity)


  # re-write the entity annotation + subclassification with text selections 
  classifications = [
      lb_types.ClassificationAnnotation(
        name="sub_checklist_question",
        value=lb_types.Checklist(
        answer=[lb_types.ClassificationAnswer(name="first_sub_checklist_answer", confidence=0.5)]
        )
      )
    ]
  ner_annotation_with_subclass = lb_types.DocumentEntity(name="ner_with_checklist_subclass",confidence=0.5, textSelections= text_selections_ner)
  ner_with_checklist_subclass_annotation = lb_types.ObjectAnnotation(name="ner_with_checklist_subclass", 
                                                                     confidence=0.5,
                                                                     value=ner_annotation_with_subclass, 
                                                                     classifications=classifications)

  # Final NDJSON and python annotations        
  print(f"entities_annotations_ndjson={entities_prediction_ndjson}")
  print(f"entities_annotation={entities_prediction}")
  print(f"nested_entities_annotation_ndjson={ner_with_checklist_subclass_prediction_ndjson}")
  print(f"nested_entities_annotation={ner_with_checklist_subclass_annotation}")
  ```
</CodeGroup>

### Step 5: Create the predictions payload

Create the predictions payload using the snippets of code shown [above](#supported-predictions).

Labelbox supports two formats for the annotations payload: NDJSON and Python annotation types. Both approaches are described below with instructions to compose annotations into Labels attached to the data rows.

The resulting `label_predictions_ndjson` and `label_predictions` payloads should have exactly the same prediction content (with the exception of the uuid strings that are generated).

<CodeGroup>
  ```python Python annotation payload theme={"theme":"css-variables"}
  label_predictions = []

  label_predictions.append(
      lb_types.Label(
          data={"global_key" : global_key },
          annotations = [
              entities_prediction,
              checklist_prediction,
              nested_checklist_prediction, 
              text_prediction,
              radio_prediction,
              nested_radio_prediction,
              bbox_prediction,
              bbox_with_radio_subclass_prediction,
              ner_with_checklist_subclass_prediction
          ]
    )
  )
  ```

  ```json NDJSON payload theme={"theme":"css-variables"}
  label_predictions_ndjson = []
  for annot in [
      entities_prediction_ndjson,
      checklist_prediction_ndjson,
      nested_checklist_prediction_ndjson,
      text_prediction_ndjson,
      radio_prediction_ndjson,
      nested_radio_prediction_ndjson,
      bbox_prediction_ndjson,
      bbox_with_radio_subclass_prediction_ndjson,
      ner_with_checklist_subclass_prediction_ndjson
    ]:
    annot.update({
        "dataRow": {"globalKey": global_key},
    })
    label_predictions_ndjson.append(annot)
  ```
</CodeGroup>

### Step 6: Upload the predictions payload to the model run

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Upload the prediction label to the Model Run
  upload_job_prediction = model_run.add_predictions(
      name="prediction_upload_job"+str(uuid.uuid4()),
      predictions=label_predictions)

  # Errors will appear for annotation uploads that failed.

  print("Errors:", upload_job_prediction.errors)
  print("Status of uploads: ", upload_job_prediction.statuses)

  ```
</CodeGroup>

### Step 7: Send annotations to the model run

To send annotations to a model run, we must first import them into a project, create a label payload and then send them to the model run.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # 7.1 Create a labelbox project
  project = client.create_project(name="Document Prediction Import Demo",
                                      media_type=lb.MediaType.Document)
  project.connect_ontology(ontology)

  # 7.2 Create a batch to send to the project
  project.create_batch(
    "batch_text_prediction_demo", # Each batch in a project must have a unique name
    global_keys=[global_key], # Paginated collection of data row objects, list of data row ids or global keys
    priority=5 # priority between 1(Highest) - 5(lowest)
  )

  # 7.3 Create the annotations payload
  entities_annotation = lb_types.ObjectAnnotation(
      name="named_entity",
      value= lb_types.DocumentEntity(
          name="named_entity",
          textSelections=text_selections
      )
  )

  radio_annotation = lb_types.ClassificationAnnotation(
      name="radio_question",
      value=lb_types.Radio(answer =
          lb_types.ClassificationAnswer(name = "first_radio_answer")
      )
  )

  checklist_annotation = lb_types.ClassificationAnnotation(
      name="checklist_question",
      value=lb_types.Checklist(answer = [
          lb_types.ClassificationAnswer(name = "first_checklist_answer"),
          lb_types.ClassificationAnswer(name = "second_checklist_answer"),
      ])
    )

  bbox_dim_1 = {
              "top": 135.3,
              "left": 102.771,
              "height": 109.843,
              "width": 415.8
        }
  bbox_annotation = lb_types.ObjectAnnotation(
      name="bounding_box",  # must match your ontology feature"s name
      value=lb_types.DocumentRectangle(
          start=lb_types.Point(x=bbox_dim_1["left"], y=bbox_dim_1["top"]),  # x = left, y = top
          end=lb_types.Point(x=bbox_dim_1["left"] + bbox_dim_1["width"], y=bbox_dim_1["top"]+ bbox_dim_1["height"]),  # x= left + width , y = top + height
          page=0,
          unit=lb_types.RectangleUnit.POINTS
          )
      )

  nested_checklist_annotation = lb_types.ClassificationAnnotation(
    name="nested_checklist_question",
    value=lb_types.Checklist(
      answer=[lb_types.ClassificationAnswer(
        name="first_checklist_answer",
        classifications=[
          lb_types.ClassificationAnnotation(
            name="sub_checklist_question",
            value=lb_types.Checklist(
              answer=[lb_types.ClassificationAnswer(
              name="first_sub_checklist_answer",
            )]
          ))
        ]
      )]
    )
  )

  nested_radio_annotation = lb_types.ClassificationAnnotation(
    name="nested_radio_question",
    value=lb_types.Radio(
      answer=lb_types.ClassificationAnswer(
        name="first_radio_answer",
        classifications=[
          lb_types.ClassificationAnnotation(
            name="sub_radio_question",
            value=lb_types.Radio(
              answer=lb_types.ClassificationAnswer(
                name="first_sub_radio_answer",
              )
            )
          )
        ]
      )
    )
  )

  text_annotation = lb_types.ClassificationAnnotation(
    name="free_text",
    value=lb_types.Text(answer="sample text")
  )

  bbox_dim = {
          "top": 226.757,
          "left": 317.271,
          "height": 194.229,
          "width": 249.386
      }

  bbox_with_radio_subclass_annotation = lb_types.ObjectAnnotation(
      name="bbox_with_radio_subclass",

      value=lb_types.DocumentRectangle(
          start=lb_types.Point(x=bbox_dim["left"], y=bbox_dim["top"]), # x = left, y = top
          end=lb_types.Point(x=bbox_dim["left"] + bbox_dim["width"], y=bbox_dim["top"] + bbox_dim["height"]), # x= left + width , y = top + height
          unit=lb_types.RectangleUnit.POINTS,
          page=1
      ),
      classifications=[
      	lb_types.ClassificationAnnotation(
          	name="sub_radio_question",
        		value=lb_types.Radio(
            answer=lb_types.ClassificationAnswer(
              name="first_sub_radio_answer",

              classifications=[
                lb_types.ClassificationAnnotation(
                  name="second_sub_radio_question",

                  value=lb_types.Radio(
                    answer=lb_types.ClassificationAnswer(
                      name="second_sub_radio_answer"
                    )
                  )
                )
              ]
            )
            )
          )
      ]
  )

  ner_with_checklist_subclass_annotation = lb_types.ObjectAnnotation(
    name="ner_with_checklist_subclass",
    value=lb_types.DocumentEntity(
      name="ner_with_checklist_subclass",
      text_selections=text_selections_ner
    ),
    classifications=[
      lb_types.ClassificationAnnotation(
        name="sub_checklist_question",
        value=lb_types.Checklist(
        answer=[lb_types.ClassificationAnswer(name="first_sub_checklist_answer")]
        )
      )
    ]
  )

  # 7.4 Create the label object
  labels = []

  labels.append(
      lb_types.Label(
          data={"global_key" : global_key },
          annotations = [
              entities_annotation,
              checklist_annotation,
              nested_checklist_annotation,
              text_annotation,
              radio_annotation,
              nested_radio_annotation,
              bbox_annotation,
              bbox_with_radio_subclass_annotation,
              ner_with_checklist_subclass_annotation
          ]
    )
  )

  # 7.5 Upload annotations to the project using label import
  upload_job_annotation = lb.LabelImport.create_from_objects(
      client = client,
      project_id = project.uid,
      name="text_label_import_job"+ str(uuid.uuid4()),
      labels=labels)

  upload_job_annotation.wait_until_done()
  # Errors will appear for annotation uploads that failed.
  print("Errors:", upload_job_annotation.errors)
  print("Status of uploads: ", upload_job_annotation.statuses)

  # 7.6 Send the annotations to the model run

  # get the labels id from the project
  model_run.upsert_labels(project_id=project.uid)
  ```
</CodeGroup>


# Upload geospatial predictions
Source: https://docs.labelbox.com/horizon/developer-guides/upload-geospatial-predictions

How to upload predictions on geospatial data in a model run and sample upload formats.

<CardGroup>
  <Card title="Open In Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/prediction_upload/geospatial_predictions.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/prediction_upload/geospatial_predictions.ipynb" />
</CardGroup>

## Supported predictions

To upload predictions in Labelbox, you need to create the prediction payload. This section provides this payload for every prediction type.

Labelbox supports two formats for the predictions payload:

* Python annotation types (recommended)
* NDJSON

Both are described below.

<Info>
  ### Uploading confidence scores is optional

  If you do not specify a confidence score, the prediction will be treated as if it had a confidence score of 1.
</Info>

### Point

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  point_prediction = lb_types.ObjectAnnotation(
    name = "point_geo",
    confidence=0.4,
    value = lb_types.Point(x=-99.20647859573366, y=19.40018029091072),
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  point_prediction_ndjson = {
      "name": "point_geo",
      "confidence": 0.4,
      "point": {
           "x": -99.20647859573366,
           "y": 19.40018029091072
       }
  }
  ```
</CodeGroup>

### Polyline

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  # Coordinates
  coords = [
              [
                  -99.20842051506044,
                  19.40032196622975
              ],
              [
                  -99.20809864997865,
                  19.39758963475322
              ],
              [
                  -99.20758366584778,
                  19.39776167179227
              ],
              [
                  -99.20728325843811,
                  19.3973265189299
              ]
          ]

  line_points = []

  for sub in coords: 
    line_points.append(lb_types.Point(x=sub[0], y=sub[1]))

  # Python Annotation 
  polyline_prediction = lb_types.ObjectAnnotation(
    name = "polyline_geo",
    confidence=0.5,
    value = lb_types.Line(points=line_points),
  )

  ```

  ```python NDJSON theme={"theme":"css-variables"}
  coords = [
              [
                  -99.20842051506044,
                  19.40032196622975
              ],
              [
                  -99.20809864997865,
                  19.39758963475322
              ],
              [
                  -99.20758366584778,
                  19.39776167179227
              ],
              [
                  -99.20728325843811,
                  19.3973265189299
              ]
          ]

  line_points_ndjson = []

  for sub in coords: 
    line_points_ndjson.append({"x":sub[0], "y":sub[1]})
    
  polyline_prediction_ndjson = {
      "name": "polyline_geo",
      "confidence": 0.5,
      "line": line_points_ndjson
  }
  ```
</CodeGroup>

### Polygon

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  polygon_points = []

  for sub in coords_polygon: 
    polygon_points.append(lb_types.Point(x=sub[0], y=sub[1]))

  polygon_prediction = lb_types.ObjectAnnotation(
    name = "polygon_geo",
    confidence=0.5,
    value = lb_types.Polygon(points=polygon_points),
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  polygon_points_ndjson = []

  for sub in coords_polygon:
    polygon_points_ndjson.append({"x":sub[0], "y":sub[1]})

  polygon_prediction_ndjson = {
      "name": "polygon_geo",
      "confidence": 0.5,
      "polygon": polygon_points_ndjson
  }
  ```

  ```python coords_polygon expandable theme={"theme":"css-variables"}
  coords_polygon = [
      [
          -99.21042680740356,
          19.40036244486966
      ],
      [
          -99.2104160785675,
          19.40017017124035
      ],
      [
          -99.2103409767151,
          19.400008256428897
      ],
      [
          -99.21014785766603,
          19.400008256428897
      ],
      [
          -99.21019077301027,
          19.39983622176518
      ],
      [
          -99.21022295951845,
          19.399674306621385
      ],
      [
          -99.21029806137086,
          19.39951239131646
      ],
      [
          -99.2102873325348,
          19.399340356128437
      ],
      [
          -99.21025514602663,
          19.399117722085677
      ],
      [
          -99.21024441719057,
          19.39892544698541
      ],
      [
          -99.2102336883545,
          19.39874329141769
      ],
      [
          -99.21021223068239,
          19.398561135646027
      ],
      [
          -99.21018004417421,
          19.398399219233365
      ],
      [
          -99.21011567115785,
          19.39822718286836
      ],
      [
          -99.20992255210878,
          19.398136104719125
      ],
      [
          -99.20974016189577,
          19.398085505725305
      ],
      [
          -99.20957922935487,
          19.398004547302467
      ],
      [
          -99.20939683914186,
          19.39792358883935
      ],
      [
          -99.20918226242067,
          19.39786286996558
      ],
      [
          -99.20899987220764,
          19.397822390703805
      ],
      [
          -99.20891404151918,
          19.397994427496787
      ],
      [
          -99.20890331268312,
          19.398176583902874
      ],
      [
          -99.20889258384706,
          19.398368859888045
      ],
      [
          -99.20889258384706,
          19.398540896103246
      ],
      [
          -99.20890331268312,
          19.39872305189756
      ],
      [
          -99.20889258384706,
          19.39890520748796
      ],
      [
          -99.20889258384706,
          19.39907724313608
      ],
      [
          -99.20889258384706,
          19.399259398329956
      ],
      [
          -99.20890331268312,
          19.399431433603585
      ],
      [
          -99.20890331268312,
          19.39961358840092
      ],
      [
          -99.20890331268312,
          19.399785623300048
      ],
      [
          -99.20897841453552,
          19.399937418648214
      ],
      [
          -99.20919299125673,
          19.399937418648214
      ],
      [
          -99.2093861103058,
          19.39991717927664
      ],
      [
          -99.20956850051881,
          19.39996777770086
      ],
      [
          -99.20961141586305,
          19.40013981222548
      ],
      [
          -99.20963287353517,
          19.40032196622975
      ],
      [
          -99.20978307724,
          19.4004130431554
      ],
      [
          -99.20996546745302,
          19.40039280384301
      ],
      [
          -99.21019077301027,
          19.400372564528084
      ],
      [
          -99.21042680740356,
          19.40036244486966
      ]

  ]
  ```
</CodeGroup>

### Bounding box

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  bbox_top_left = lb_types.Point(x= -99.20746564865112, y=19.39799442829336)
  bbox_bottom_right = lb_types.Point(x=-99.20568466186523, y=19.39925939999194)

  bbox_prediction = lb_types.ObjectAnnotation(
    name = "bbox_geo",
    confidence=0.5,
    value = lb_types.Rectangle(start=bbox_top_left, end=bbox_bottom_right)
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  coord_object =  {
      "coordinates" : [[
              [
                  -99.20746564865112,
                  19.39799442829336
              ],
              [
                  -99.20746564865112,
                  19.39925939999194
              ],
              [
                  -99.20568466186523,
                  19.39925939999194
              ],
              [
                  -99.20568466186523,
                  19.39799442829336
              ],
              [
                  -99.20746564865112,
                  19.39799442829336
              ]
          ]]
  }

  bbox_prediction_ndjson = {
      "name" : "bbox_geo",
      "confidence": 0.5,
      "bbox" : {
          'top': coord_object["coordinates"][0][1][1],
          'left': coord_object["coordinates"][0][1][0],
          'height': coord_object["coordinates"][0][3][1] - coord_object["coordinates"][0][1][1],
          'width': coord_object["coordinates"][0][3][0] - coord_object["coordinates"][0][1][0]
      }
  }
  ```
</CodeGroup>

### Classification: Radio (single choice)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  radio_prediction = lb_types.ClassificationAnnotation(
      name="radio_question_geo",
      value=lb_types.Radio(answer=lb_types.ClassificationAnswer(name="first_radio_answer", confidence=0.5))
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  radio_prediction_ndjson = {
      "name": "radio_question_geo",
      "answer": { "name": "first_radio_answer", "confidence": 0.5}
  }
  ```
</CodeGroup>

### Bounding box with nested checklist classification

<CodeGroup>
  ```python Annotation Types theme={"theme":"css-variables"}
  bbox_with_checklist_subclass = lb_types.ObjectAnnotation(
      name="bbox_checklist_geo",
      confidence=0.5,
      value=lb_types.Rectangle(
          start=lb_types.Point(x=-99.210266, y=19.39540372195134), # Top left
          end=lb_types.Point(x=-99.20621067903966, y=19.396901), # Bottom right
      ),
      classifications=[
          lb_types.ClassificationAnnotation(
              name="checklist_class_name",
              value=lb_types.Checklist(
                  answer=[lb_types.ClassificationAnswer(name="first_checklist_answer", confidence=0.5)]
              )
          )
      ]
  )
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  coord_object_checklist = {
      "coordinates": [
         [
              [
                  -99.210266,
                  19.39540372195134
              ],
              [
                  -99.210266,
                  19.396901
              ],
              [
                  -99.20621067903966,
                  19.396901
              ],
              [
                  -99.20621067903966,
                  19.39540372195134
              ],
              [
                  -99.210266,
                  19.39540372195134
              ]
        ]
      ]
  }

  bbox_with_checklist_subclass_ndjson = {
      "name": "bbox_checklist_geo",
      "confidence": 0.5,
      "classifications": [{
          "name": "checklist_class_name",
          "answer": [
              { "name":"first_checklist_answer", "confidence": 0.5}
          ]
      }],
      "bbox": {
          'top': coord_object_checklist["coordinates"][0][1][1],
          'left': coord_object_checklist["coordinates"][0][1][0],
          'height': coord_object_checklist["coordinates"][0][3][1] - coord_object_checklist["coordinates"][0][1][1],
          'width': coord_object_checklist["coordinates"][0][3][0] - coord_object_checklist["coordinates"][0][1][0]
      }
  }
  ```
</CodeGroup>

### Bounding box with nested free-text classification

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  bbox_with_free_text_subclass = lb_types.ObjectAnnotation(
      name="bbox_text_geo",
      value=lb_types.Rectangle(
          start=lb_types.Point(x=-99.21019613742828, y=19.397447957052933), # Top left
          end=lb_types.Point(x=-99.20986354351044, y=19.39772119262215), # Bottom right
      ),
      classifications=[
          lb_types.ClassificationAnnotation(
              name="free_text_geo",
              value=lb_types.Text(answer="sample text")
          )
      ]
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}

  coord_object_text ={
      "coordinates": [
        [
          [
              -99.21019613742828,
              19.397447957052933
          ],
          [
              -99.21019613742828,
              19.39772119262215
          ],
          [
              -99.20986354351044,
              19.39772119262215
          ],
          [
              -99.20986354351044,
              19.397447957052933
          ],
          [
              -99.21019613742828,
              19.397447957052933
          ]
        ]
      ]
  }

  bbox_with_free_text_subclass_ndjson = {
      "name":"bbox_text_geo",
      "confidence": 0.5,
      "classifications": [{
          "name": "free_text_geo",
          "confidence": 0.5,
          "answer": "sample text"
      }],
      "bbox": {
          'top': coord_object_text["coordinates"][0][1][1],
          'left': coord_object_text["coordinates"][0][1][0],
          'height': coord_object_text["coordinates"][0][3][1] - coord_object_text["coordinates"][0][1][1],
          'width': coord_object_text["coordinates"][0][3][0] - coord_object_text["coordinates"][0][1][0]
      }
  }
  ```
</CodeGroup>

### Classification: Checklist (multi-choice)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  checklist_prediction = lb_types.ClassificationAnnotation(
      name="checklist_question_geo",
      confidence = 0.5,
      value=lb_types.Checklist(answer = [
          lb_types.ClassificationAnswer(name = "first_checklist_answer", confidence = 0.5),
          lb_types.ClassificationAnswer(name = "second_checklist_answer", confidence = 0.5),
          lb_types.ClassificationAnswer(name = "third_checklist_answer", confidence = 0.5)
      ])
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  checklist_prediction_ndjson = {
    'name': 'checklist_question_geo',
    "confidence": 0.5,
    'answer': [
      {'name': 'first_checklist_answer', "confidence": 0.5},
      {'name': 'second_checklist_answer', "confidence": 0.5},
      {'name': 'third_checklist_answer', "confidence": 0.5},
    ]
  }
  ```
</CodeGroup>

### Classification: Nested radio

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  nested_radio_prediction = lb_types.ClassificationAnnotation(
  name="nested_radio_question",
  value=lb_types.Radio(
      answer=lb_types.ClassificationAnswer(
      name="first_radio_answer",
      confidence=0.5,
      classifications=[
          lb_types.ClassificationAnnotation(
          name="sub_radio_question",
          value=lb_types.Radio(
              answer=lb_types.ClassificationAnswer(
              name="first_sub_radio_answer",
              confidence=0.2
              )
          )
          )
      ]
      )
  )
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  nested_radio_prediction_ndjson= {
    "name": "nested_radio_question",
    "answer": {
        "name": "first_radio_answer",
        "confidence": 0.2,
        "classifications": [{
            "name":"sub_radio_question",
            "answer": { "name" : "first_sub_radio_answer", "confidence": 0.3}
          }]
      }
  }
  ```
</CodeGroup>

### Classification: Nested checklist

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  nested_checklist_prediction = lb_types.ClassificationAnnotation(
    name="nested_checklist_question",
    value=lb_types.Checklist(
      answer=[lb_types.ClassificationAnswer(
        name="first_checklist_answer",
        confidence=0.5,
        classifications=[
          lb_types.ClassificationAnnotation(
            name="sub_checklist_question",
            value=lb_types.Checklist(
              answer=[lb_types.ClassificationAnswer(
              name="first_sub_checklist_answer",
              confidence=0.5
            )]
          ))
        ]
      )]
    )
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  nested_checklist_prediction_ndjson = {
    "name": "nested_checklist_question",
    "answer": [{
        "name": "first_checklist_answer",
        "confidence":0.5,
        "classifications" : [
          {
            "name": "sub_checklist_question",
            "answer": {"name": "first_sub_checklist_answer", "confidence": 0.5}
          }
        ]
    }]
  }
  ```
</CodeGroup>

## End-to-end example: Upload predictions to a model run

Follow the steps below to upload predictions to a model run.

### Before you start

You will need to import these libraries to use the code examples in this section.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  import os
  import uuid
  import numpy as np
  from PIL import Image
  import cv2

  import labelbox as lb
  import labelbox.types as lb_types
  ```
</CodeGroup>

Replace the value of `API_KEY` with a valid [API key](/horizon/developer-guides/create-api-key) to connect to the Labelbox client.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  API_KEY = None
  client = lb.Client(API_KEY)
  ```
</CodeGroup>

### Step 1: Import data rows into Catalog

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  top_left_bound = lb_types.Point(x=-99.21052827588443, y=19.400498983095076)
  bottom_right_bound = lb_types.Point(x=-99.20534818927473, y=19.39533555271248)

  epsg = lb_types.EPSG.EPSG4326
  bounds = lb_types.TiledBounds(epsg=epsg, bounds=[top_left_bound, bottom_right_bound])
  global_key = "mexico_city"

  tile_layer = lb_types.TileLayer(
      url="https://s3-us-west-1.amazonaws.com/lb-tiler-layers/mexico_city/{z}/{x}/{y}.png"
  )

  tiled_image_data = lb_types.TiledImageData(tile_layer=tile_layer,
                                    tile_bounds=bounds,
                                    zoom_levels=[17, 23])

  asset = {
      "row_data": tiled_image_data.asdict(),
      "global_key": global_key,
      "media_type": "TMS_GEO"
  }

  dataset = client.create_dataset(name="geo_demo_dataset")
  task= dataset.create_data_rows([asset])
  print("Errors:",task.errors)
  print("Failed data rows:", task.failed_data_rows)
  ```
</CodeGroup>

### Step 2: Create/select an ontology for your model predictions

Your model run should have the correct ontology set up with all the tools and classifications supported for your predictions.

Here is an example of creating an ontology programmatically for all the example predictions above:

<CodeGroup>
  ```python Python expandable theme={"theme":"css-variables"}
  ontology_builder = lb.OntologyBuilder(
      tools=[
          lb.Tool(tool=lb.Tool.Type.POINT, name="point_geo"),
          lb.Tool(tool=lb.Tool.Type.LINE, name="polyline_geo"),
          lb.Tool(tool=lb.Tool.Type.POLYGON, name="polygon_geo"),
          lb.Tool(tool=lb.Tool.Type.POLYGON, name="polygon_geo_2"),
          lb.Tool(tool=lb.Tool.Type.BBOX, name="bbox_geo"),
          lb.Tool(
            tool=lb.Tool.Type.BBOX,
            name="bbox_checklist_geo",
            classifications=[
                  lb.Classification(
                      class_type=lb.Classification.Type.CHECKLIST,
                      name="checklist_class_name",
                      options=[
                        lb.Option(value="first_checklist_answer")
                      ]
                  ),
              ]
            ),
          lb.Tool(
            tool=lb.Tool.Type.BBOX,
            name="bbox_text_geo",
            classifications=[
                  lb.Classification(
                      class_type=lb.Classification.Type.TEXT,
                      name="free_text_geo"
                  ),
              ]
            )
        ],
        classifications = [
            lb.Classification(
                class_type=lb.Classification.Type.CHECKLIST,
                name="checklist_question_geo",
                options=[
                    lb.Option(value="first_checklist_answer"),
                    lb.Option(value="second_checklist_answer"),
                    lb.Option(value="third_checklist_answer")
                ]
            ),
            lb.Classification(
                class_type=lb.Classification.Type.RADIO,
                name="radio_question_geo",
                options=[
                    lb.Option(value="first_radio_answer")
                ]
            ),

          lb.Classification(
            class_type=lb.Classification.Type.RADIO,
            name="nested_radio_question",
            options=[
              lb.Option(value="first_radio_answer",
                options=[
                    lb.Classification(
                      class_type=lb.Classification.Type.RADIO,
                      name="sub_radio_question",
                      options=[
                        lb.Option(value="first_sub_radio_answer")
                      ]
                  ),
                ]
              ),
            ],
          ),
          lb.Classification(
            class_type=lb.Classification.Type.CHECKLIST,
            name="nested_checklist_question",
            options=[
                lb.Option("first_checklist_answer",
                  options=[
                    lb.Classification(
                        class_type=lb.Classification.Type.CHECKLIST,
                        name="sub_checklist_question",
                        options=[lb.Option("first_sub_checklist_answer")]
                    )
                ]
              )
            ]
        )
      ]

  )

  ontology = client.create_ontology("Ontology Geospatial Annotations",
  ontology_builder.asdict(),
  media_type=lb.MediaType.Geospatial_Tile)

  ```
</CodeGroup>

### Step 3: Create a Model and model run

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # create Model
  model = client.create_model(name="geospatial_model_run_" + str(uuid.uuid4()),
                              ontology_id=ontology.uid)
  # create model run
  model_run = model.create_model_run("iteration 1")
  ```
</CodeGroup>

### Step 4: Send data rows to the model run

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  model_run.upsert_data_rows(global_keys=[global_key])
  ```
</CodeGroup>

### Step 5: Create the predictions payload

Create the annotations payload using the snippets of code shown [above](#supported-predictions).

Labelbox supports two formats for the annotations payload: NDJSON and Python annotation types. Both approaches are described below with instructions to compose annotations into Labels attached to the data rows.

The resulting `labels` and `ndjson_labels` from each approach will include every annotation (created above) supported by the respective method.

<CodeGroup>
  ```python Python annotation payload theme={"theme":"css-variables"}
  labels =[]
  labels.append(
      lb_types.Label(
           data={
              "global_key": global_key,
              "tile_layer": tile_layer,
              "tile_bounds":bounds,
              "zoom_levels": [12, 20]
          },
          annotations = [
              point_prediction,
              polyline_prediction,
              polygon_prediction,
              bbox_prediction,
              radio_prediction,
              bbox_with_checklist_subclass,
              bbox_with_free_text_subclass,
              checklist_prediction,
              nested_checklist_prediction,
              nested_radio_prediction
          ]
      )
  )
  ```

  ```python NDJSON payload theme={"theme":"css-variables"}
  label_ndjson = []
  for prediction in [
      radio_prediction_ndjson,
      checklist_prediction_ndjson,
      bbox_with_free_text_subclass_ndjson,
      bbox_with_checklist_subclass_ndjson,
      bbox_prediction_ndjson,
      point_prediction_ndjson,
      polyline_prediction_ndjson,
      polygon_prediction_ndjson,
      nested_checklist_prediction_ndjson,
      nested_radio_prediction_ndjson
  ]:
    prediction.update({
        'dataRow': {'globalKey': global_key},
    })
    label_ndjson.append(prediction)
  ```
</CodeGroup>

### Step 6: Upload the predictions payload to the Model Run

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Upload the prediction label to the Model Run
  upload_job_prediction = model_run.add_predictions(
      name="prediction_upload_job"+str(uuid.uuid4()),
      predictions=labels)

  # Errors will appear for annotation uploads that failed.

  print("Errors:", upload_job_prediction.errors)
  print("Status of uploads: ", upload_job_prediction.statuses)
  ```
</CodeGroup>

### Step 7: Send annotations to the Model Run (Optional)

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # 7.1. Create a labelbox project

  # Create a Labelbox project
  project = client.create_project(name="geospatial_prediction_demo",                                    
                                      queue_mode=lb.QueueMode.Batch,
                                      media_type=lb.MediaType.Geospatial_Tile)
  project.connect_ontology(ontology)

  # 7.2. Create a batch to send to the project

  project.create_batch(
    "batch_geospatial_prediction_demo", # Each batch in a project must have a unique name
    global_keys=[global_key], # A list of data rows or data row ids
    priority=5 # priority between 1(Highest) - 5(lowest)
  )


  # 7.3 Create the annotations payload as explained in:
  # https://docs.labelbox.com/horizon/developer-guides/import-geospatial-annotations#supported-annotations
  point_annotation...
  polyline_annotation...
  polygon_annotation...
  bbox_annotation...
  radio_annotation...
  bbox_with_checklist_subclass...
  bbox_with_free_text_subclass...
  checklist_annotation...
  nested_radio_annotation...
  nested_checklist_annotation...

  # 7.4 Create the label object 
  labels =[]
  labels.append(
      lb_types.Label(
          data={
              "global_key": global_key,
              "tile_layer": tile_layer,
              "tile_bounds":bounds,
              "zoom_levels": [12, 20]
          },
          annotations = [
              point_annotation,
              polyline_annotation,
              polygon_annotation,
              bbox_annotation,
              radio_annotation,
              bbox_with_checklist_subclass,  
              bbox_with_free_text_subclass,
              checklist_annotation,
              nested_checklist_annotation, 
              nested_radio_annotation
          ]
      )
  ) 

  # 7.5 Upload annotations to the project using Label Import
  upload_job_annotation = lb.LabelImport.create_from_objects(
      client = client,
      project_id = project.uid,
      name="geospatial_annotations_import_" + str(uuid.uuid4()),
      labels=labels)

  upload_job_annotation.wait_until_done()
  # Errors will appear for annotation uploads that failed.
  print("Errors:", upload_job_annotation.errors)
  print("Status of uploads: ", upload_job_annotation.statuses)

  # 7.6. Send the annotations to the Model Run

  # get the labels id from the project
  model_run.upsert_labels(project_id=project.uid)
  ```
</CodeGroup>


# Upload HTML predictions
Source: https://docs.labelbox.com/horizon/developer-guides/upload-html-predictions

How to upload predictions on HTML data in a model run and sample upload formats.

<CardGroup>
  <Card title="Open In Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/prediction_upload/html_predictions.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/prediction_upload/html_predictions.ipynb" />
</CardGroup>

## Supported annotations

To import annotations in Labelbox, you need to create the annotations payload. This section shows how to declare the payloads for each supported annotation type.

### Classification: Radio (single-choice)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  radio_prediction = lb_types.ClassificationAnnotation(
    name="radio_question",
    value=lb_types.Radio(answer = lb_types.ClassificationAnswer(name = "second_radio_answer", confidence=0.5))
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  radio_prediction_ndjson = {
    "name": "radio_question",
    "answer": {"name": "first_radio_answer"}
  }
  ```
</CodeGroup>

### Classification: Nested radio

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  nested_radio_prediction = lb_types.ClassificationAnnotation(
    name="nested_radio_question",
    value=lb_types.Radio(
      answer=lb_types.ClassificationAnswer(
        name="first_radio_answer",
        confidence=0.5,# Confidence scores should be added to the answer
        classifications=[
          lb_types.ClassificationAnnotation(
            name="sub_radio_question",
            value=lb_types.Radio(
              answer=lb_types.ClassificationAnswer(
                name="first_sub_radio_answer", confidence=0.5)
            )
          )
        ]
      )
    )
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  nested_radio_prediction_ndjson = {
    "name": "nested_radio_question",
    "answer": {
        "name": "first_radio_answer",
        "confidence": 0.5, # Confidence scores should be added to the answer
        "classifications": [{
            "name":"sub_radio_question",
            "answer": { "name" : "first_sub_radio_answer", "confidence": 0.5 }
          }]
      }
  }
  ```
</CodeGroup>

### Classification: Nested checklist

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  nested_checklist_prediction = lb_types.ClassificationAnnotation(
    name="nested_checklist_question",
    value=lb_types.Checklist(
      answer=[lb_types.ClassificationAnswer(
        name="first_checklist_answer",
        confidence=0.5, # Confidence scores should be added to the answer
        classifications=[
          lb_types.ClassificationAnnotation(
            name="sub_checklist_question",
            value=lb_types.Checklist(
              answer=[lb_types.ClassificationAnswer(
              name="first_sub_checklist_answer",
              confidence=0.5
            )]
          ))
        ]
      )]
    )
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  nested_checklist_prediction_ndjson = {
    "name": "nested_checklist_question",
    "answer": [{
        "name": "first_checklist_answer",
        "confidence": 0.5, # Confidence scores should be added to the answer
        "classifications" : [
          {
            "name": "sub_checklist_question",
            "answer": {"name": "first_sub_checklist_answer", "confidence": 0.5 }
          }
        ]
    }]
  }
  ```
</CodeGroup>

### Classification: Checklist (multi-choice)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  checklist_prediction = lb_types.ClassificationAnnotation(
      name="checklist_question",
      value=lb_types.Checklist(
          answer = [
              lb_types.ClassificationAnswer(
                  name = "first_checklist_answer",
                  confidence=0.5
              ),
              lb_types.ClassificationAnswer(
                  name = "second_checklist_answer",
                  confidence=0.5
              ),
              lb_types.ClassificationAnswer(
                  name = "third_checklist_answer",
                  confidence=0.5
              )
      ])
    )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  checklist_prediction_ndjson = {
    "name": "checklist_question",
    "answer": [
      {"name": "first_checklist_answer", "confidence": 0.5}
    ]
  }
  ```
</CodeGroup>

### Classification: Free-form text

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  text_prediction = lb_types.ClassificationAnnotation(
      name = "free_text",
      value = lb_types.Text(answer="sample text", confidence=0.5)
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  text_prediction_ndjson = {
    "name": "free_text",
    "answer": "sample text",
    "confidence":0.5
  }
  ```
</CodeGroup>

## Example: Upload predictions to model run

To upload predictions to a model run:

### Before you start

You will need to import these libraries to use the code examples in this section:

<CodeGroup>
  ```python Library Import theme={"theme":"css-variables"}
  import labelbox as lb
  import labelbox.types as lb_types
  import uuid
  import numpy as np
  ```
</CodeGroup>

Replace the value of `API_KEY` with a valid [API key](/horizon/developer-guides/create-api-key) to connect to the Labelbox client.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  API_KEY = None
  client = lb.Client(API_KEY)
  ```
</CodeGroup>

### Step 1: Import data rows into Catalog

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  global_key ="sample_html_2.html"

  test_img_url = {
    "row_data": "https://storage.googleapis.com/labelbox-datasets/html_sample_data/sample_html_2.html",
    "global_key": global_key,
  }
  dataset = client.create_dataset(
    name="html prediction demo dataset",
    iam_integration=None # Removing this argument will default to the organization"s default IAM integration
  )

  task = dataset.create_data_rows([test_img_url])
  task.wait_till_done()

  print("Errors:",task.errors)
  print("Failed data rows:", task.failed_data_rows)

  ```
</CodeGroup>

### Step 2: Set up ontology

Your project should include an ontology that supports your annotations. To ensure feature schema matches, the tool names and classification names should match the `name` field in your annotations.

<CodeGroup>
  ```python Python expandable theme={"theme":"css-variables"}
  ontology_builder = lb.OntologyBuilder(
    classifications=[ # List of Classification objects
      lb.Classification(
        class_type=lb.Classification.Type.RADIO,
        name="radio_question",  # name matching the tool used in the annotation
        options=[lb.Option(value="first_radio_answer")]
      ),
      lb.Classification(
        class_type=lb.Classification.Type.RADIO,
        name="nested_radio_question",
        options=[
          lb.Option(value="first_radio_answer",
            options=[
                lb.Classification(
                  class_type=lb.Classification.Type.RADIO,
                  name="sub_radio_question",
                  options=[
                    lb.Option(value="first_sub_radio_answer")
                  ]
              ),
            ]
          )
        ],
      ),
      lb.Classification(
        class_type=lb.Classification.Type.CHECKLIST,
        name="checklist_question",
        options=[
          lb.Option(value="first_checklist_answer"),
          lb.Option(value="second_checklist_answer"),
          lb.Option(value="third_checklist_answer")
        ]
      ),
       lb.Classification(
        class_type=lb.Classification.Type.TEXT,
        name="free_text"
      ),
      lb.Classification(
        class_type=lb.Classification.Type.CHECKLIST,
        name="nested_checklist_question",
        options=[
            lb.Option("first_checklist_answer",
              options=[
                lb.Classification(
                    class_type=lb.Classification.Type.CHECKLIST,
                    name="sub_checklist_question",
                    options=[lb.Option("first_sub_checklist_answer")]
                )
            ]
          )
        ]
      )
    ]
  )

  ontology = client.create_ontology("Ontology HTML Predictions",
                                    ontology_builder.asdict(),
                                    media_type=lb.MediaType.Html)
  ```
</CodeGroup>

### Step 3: Create model and model run

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  #create Model
  model = client.create_model(name="HTML_model_run_" + str(uuid.uuid4()),
                              ontology_id=ontology.uid)
  #create Model Run
  model_run = model.create_model_run("iteration 1")
  ```
</CodeGroup>

### Step 4: Send data rows to model run

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  model_run.upsert_data_rows(global_keys=[global_key])
  ```
</CodeGroup>

### Step 5: Create prediction payload

Use the examples in [Supported annotations](#supported-annotations) to create your annotation payloads; you can use declare them as Python dictionaries or NDJSON objects. Examples of each type are shown here; they also show how to compose annotations into labels attached to the data rows.

The resulting `label_prediction` and `label_prediction_ndjson` from each approach demonstrates each supported annotation type.

<CodeGroup>
  ```python Python annotation payload theme={"theme":"css-variables"}
  label_prediction = []
  label_prediction.append(
    lb_types.Label(
      data={"global_key" : global_key },
      annotations = [
        radio_prediction, 
        checklist_prediction,
        text_prediction,
        nested_checklist_prediction,
        nested_radio_prediction
      ]
    )
  )
  ```

  ```python NDJSON payload theme={"theme":"css-variables"}
  label_prediction_ndjson = []
  for annot in [
      radio_prediction_ndjson,
      nested_radio_prediction_ndjson,
      checklist_prediction_ndjson,
      text_prediction_ndjson,
      nested_checklist_prediction_ndjson
  ]:
    annot.update({
        "dataRow": {"globalKey": global_key},
    })
    label_prediction_ndjson.append(annot)
  ```
</CodeGroup>

### Step 6: Upload predictions payload to model run

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Upload the prediction label to the Model Run
  upload_job_prediction = model_run.add_predictions(
      name="prediction_upload_job"+str(uuid.uuid4()),
      predictions=label_prediction)

  # Errors will appear for annotation uploads that failed.

  print("Errors:", upload_job_prediction.errors)
  print("Status of uploads: ", upload_job_prediction.statuses)

  ```
</CodeGroup>

### Step 7: Send annotations to model run

*(Optional)* To send annotations to a model run:

1. Import them into a project
2. Create a label payload
3. Send Send them to the model run

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  #7.1. Create a labelbox project
  project = client.create_project(name="HTML prediction import demo",
                                      queue_mode=lb.QueueMode.Batch,
                                      media_type=lb.MediaType.Html)
  project.connect_ontology(ontology)

  # 7.2. Create a batch to send to the project

  project.create_batch(
    "batch_prediction_html", # Each batch in a project must have a unique name
    global_keys=[global_key], # Paginated collection of data row objects, list of data row ids or global keys
    priority=5 # priority between 1(Highest) - 5(lowest)
  )

  # 7.3 Create the annotations payload
  radio_annotation...
  nested_radio_annotation...
  nested_checklist_annotation...
  checklist_annotation...
  text_annotation...

  # 7.4 Create the label object
  label.append(
    lb_types.Label(
      data={"global_key" : global_key },
      annotations=[
        text_annotation,
        checklist_annotation,
        radio_annotation,
        nested_checklist_annotation,
        nested_radio_annotation
      ]
    )
  )

  #7.5. Upload annotations to the project using Label Import
  upload_job_annotation = lb.LabelImport.create_from_objects(
      client = client,
      project_id = project.uid,
      name="html_annotation_import" + str(uuid.uuid4()),
      labels=label)

  upload_job_annotation.wait_until_done()
  # Errors will appear for annotation uploads that failed.
  print("Errors:", upload_job_annotation.errors)
  print("Status of uploads: ", upload_job_annotation.statuses)

  # 7.6 Send the annotations to the Model Run
  model_run.upsert_labels(project_id=project.uid)
  ```
</CodeGroup>


# Upload image predictions
Source: https://docs.labelbox.com/horizon/developer-guides/upload-image-predictions

How to upload predictions on image data in a model run and sample upload formats.

<CardGroup>
  <Card title="Open In Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/prediction_upload/image_predictions.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/prediction_upload/image_predictions.ipynb" />
</CardGroup>

## Overview

To upload predictions in Labelbox, you need to create a prediction payload. In this section, we provide this payload for every supported annotation type.

### Predictions Payload Types

Labelbox supports two formats for the annotations payload:

* Python annotation types (recommended)

  * Provides a seamless transition between third-party platforms, machine learning pipelines, and Labelbox.
  * Allows you to build annotations locally with local file paths, numpy arrays, or URLs
  * Easily convert Python Annotation Type format to NDJSON format to quickly import annotations to Labelbox
  * Supports one-level nested classification (radio, checklist, or free-form text) under a tool or classification annotation.

* JSON

  * Skips formatting annotation payload in the Labelbox Python annotation type
  * Supports any levels of nested classification (radio, checklist, or free-form text) under a tool or classification annotation.

### Confidence Score

You can include confidence scores and custom metrics when you upload your model predictions to a model run. However, given the predictions and annotations in a model run, Labelbox will automatically calculate some [auto-generated metrics](/horizon/guides/model-runs#auto-generated-metrics) upon upload.

<Info>
  ### Uploading confidence scores is optional

  If you do not specify a confidence score, the prediction will be treated as if it had a confidence score of 1.
</Info>

### Supported Prediction

The following predictions are supported for an image data row:

* Radio
* Checklist
* Free-form text
* Bounding box
* Point
* Polyline
* Polygon
* Segmentation masks

## Classification

### Radio (single-choice)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  radio_prediction = lb_types.ClassificationAnnotation(
      name="radio_question",
      value=lb_types.Radio(answer = lb_types.ClassificationAnswer(
        name = "second_radio_answer", confidence=0.5
        )
      )
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  radio_prediction_ndjson = {
    "name": "radio_question",
    "answer": {"name": "second_radio_answer", "confidence": 0.5}
  }
  ```
</CodeGroup>

### Classification: Nested radio

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  nested_radio_prediction = lb_types.ClassificationAnnotation(
    name="nested_radio_question",
    value=lb_types.Radio(
      answer=lb_types.ClassificationAnswer(
        name="first_radio_answer",
        confidence=0.5,
        classifications=[
          lb_types.ClassificationAnnotation(
            name="sub_radio_question",
            value=lb_types.Radio(
              answer=lb_types.ClassificationAnswer(
                name="first_sub_radio_answer",
                confidence=0.5
              )
            )
          )
        ]
      )
    )
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  nested_radio_prediction_ndjson = {
      "name": "nested_radio_question",
      "answer": {
          "name": "first_radio_answer",
          "confidence": 0.5,
          "classifications": [{
              "name": "sub_radio_question",
              "answer": {
                  "name": "first_sub_radio_answer",
                  "confidence": 0.5
              }
          }]
      }
  }
  ```
</CodeGroup>

### Classification: Nested checklist

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}

  nested_checklist_prediction = lb_types.ClassificationAnnotation(
    name="nested_checklist_question",
    value=lb_types.Checklist(
      answer=[lb_types.ClassificationAnswer(
        name="first_checklist_answer",
        confidence=0.5,
        classifications=[
          lb_types.ClassificationAnnotation(
            name="sub_checklist_question",
            value=lb_types.Checklist(
              answer=[lb_types.ClassificationAnswer(
              name="first_sub_checklist_answer",
              confidence=0.5
            )]
          ))
        ]
      )]
    )
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  nested_checklist_prediction_ndjson = {
      "name": "nested_checklist_question",
      "answer": [{
          "name": "first_checklist_answer",
          "confidence": 0.5,
          "classifications": [{
              "name": "sub_checklist_question",
              "answer": {
                  "name": "first_sub_checklist_answer",
                  "confidence": 0.5
              }
          }]
      }]
  }
  ```
</CodeGroup>

### Checklist (multiple choice)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  checklist_prediction = lb_types.ClassificationAnnotation(
    name="checklist_question",
    value=lb_types.Checklist(
        answer = [
          lb_types.ClassificationAnswer(
              name = "first_checklist_answer",
              confidence=0.5
          ),
          lb_types.ClassificationAnswer(
              name = "second_checklist_answer",
              confidence=0.5
          )
        ]
      )
    )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  checklist_prediction_ndjson = {
    "name": "checklist_question",
    "answer": [
      {"name": "first_checklist_answer" , "confidence": 0.5},
      {"name": "second_checklist_answer", "confidence": 0.5}
    ]
  }
  ```
</CodeGroup>

### Bounding box

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  bbox_prediction = lb_types.ObjectAnnotation(
    name = "bounding_box",
    confidence=0.5,
    value=lb_types.Rectangle(
      start=lb_types.Point(x=1690, y=977),  #  x = left, y = top
      end=lb_types.Point(x=1915, y=1307),  # x= left + width , y = top + height
    )
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  bbox_prediction_ndjson = {
    "name": "bounding_box",
    "confidence": 0.5,
    "bbox": {
            "top": 977,
            "left": 1690,
            "height": 330,
            "width": 225
        }
  }
  ```
</CodeGroup>

### Bounding box with nested classification

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  bbox_with_radio_subclass_prediction = lb_types.ObjectAnnotation(
      name="bbox_with_radio_subclass",
      confidence=0.5,
      value=lb_types.Rectangle(
          start=lb_types.Point(x=541, y=933),  #  x = left, y = top
          end=lb_types.Point(x=871, y=1124),  # x= left + width , y = top + height
      ),
      classifications=[
      	lb_types.ClassificationAnnotation(
          	name="sub_radio_question",
        		value=lb_types.Radio(answer=lb_types.ClassificationAnswer(name="first_sub_radio_answer", confidence=0.5))
      )
    ]
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  bbox_with_radio_subclass_prediction_ndjson = {
      "name": "bbox_with_radio_subclass",
    	"confidence": 0.5
      "classifications": [{
          "name": "sub_radio_question",
          "answer":
              { "name":"first_sub_radio_answer", "confidence": 0.5}

      }],
      "bbox": {
            "top": 933,
            "left": 541,
            "height": 191,
            "width": 330
          }
  }
  ```
</CodeGroup>

### Polygon

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  polygon_prediction = lb_types.ObjectAnnotation(
    name = "polygon",
    confidence = 0.5,
    value=lb_types.Polygon(
          points=[lb_types.Point(x=1489.581,y=183.934), lb_types.Point(x=2278.306,y=256.885), lb_types.Point(x=2428.197,y=200.437), lb_types.Point(x=2560.0,y=335.419),
                  lb_types.Point(x=2557.386,y=503.165), lb_types.Point(x=2320.596,y=503.103), lb_types.Point(x=2156.083, y=628.943), lb_types.Point(x=2161.111,y=785.519),
                  lb_types.Point(x=2002.115, y=894.647), lb_types.Point(x=1838.456,y=877.874), lb_types.Point(x=1436.53,y=874.636), lb_types.Point(x=1411.403,y=758.579),
                  lb_types.Point(x=1353.853,y=751.74), lb_types.Point(x=1345.264, y=453.461), lb_types.Point(x=1426.011,y=421.129)]
      ),
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  polygon_prediction_ndjson = {
    "name": "polygon",
    "confidence": 0.5,
    "polygon": [
      {"x": 1489.581, "y": 183.934},
      {"x": 2278.306, "y": 256.885},
      {"x": 2428.197, "y": 200.437},
      {"x": 2560.0, "y": 335.419},
      {"x": 2557.386, "y": 503.165},
      {"x": 2320.596, "y": 503.103},
      {"x": 2156.083, "y": 628.943},
      {"x": 2161.111, "y": 785.519},
      {"x": 2002.115, "y": 894.647},
      {"x": 1838.456, "y": 877.874},
      {"x": 1436.53, "y": 874.636},
      {"x": 1411.403, "y": 758.579},
      {"x": 1353.853, "y": 751.74},
      {"x": 1345.264, "y": 453.461},
      {"x": 1426.011, "y": 421.129},
      {"x": 1489.581, "y": 183.934}
    ]
  }
  ```
</CodeGroup>

### Classification: free-form text

<CodeGroup>
  ```python Python Annotation theme={"theme":"css-variables"}
  text_annotation = lb_types.ClassificationAnnotation(
    name="free_text", 
    value=lb_types.Text(answer="sample text", confidence=0.5)
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  text_annotation_ndjson = {
    "name": "free_text",
    "confidence": 0.5,
    "answer": "sample text",
  }
  ```
</CodeGroup>

### Segmentation mask

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  # We recommend to use byte array
  url = "https://storage.googleapis.com/labelbox-datasets/image_sample_data/raster_seg.png"
  response = requests.get(url)
  mask_data = lb.types.MaskData(im_bytes=response.content) # You can also use "url" instead of img_bytes to pass the PNG mask url.

  mask_prediction = lb_types.ObjectAnnotation(
      name="mask",
      confidence=0.5,
      value=lb_types.Mask(
          mask=mask_data,
          color=(255, 255, 255))
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  # using instanceURI, bytes array is not fully supported.
  mask_prediction_ndjson = {
    "name": "mask",
    "confidence": 0.5,
    "classifications": [],
    "mask": {"instanceURI": url,
    "colorRGB": (255, 255, 255)}
  }
  ```
</CodeGroup>

### Segmentation mask with nested classification

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  url_2 = "https://storage.googleapis.com/labelbox-datasets/image_sample_data/raster_seg_with_subclass.png"
  response_2 = requests.get(url_2)
  mask_data_2 = lb_types.MaskData(im_bytes=response_2.content)

  # Python annotation
  mask_with_text_subclass_prediction = lb_types.ObjectAnnotation(
    name = "mask_with_text_subclass", # must match your ontology feature"s name
    value=lb_types.Mask(
      mask=mask_data_2,
      confidence=0.5,
      color=(255, 255, 255)),
    classifications=[
      lb_types.ClassificationAnnotation(
        name="sub_free_text",
        value=lb_types.Text(answer="free text answer")
      )]
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  mask_with_text_subclass_prediction_ndjson = {
    "name": "mask_with_text_subclass",
    "mask": {"instanceURI": url_2,
    "colorRGB": (255, 255, 255)},
    "classifications":[{
      "name": "sub_free_text",
      "answer": "free text answer"
    }]
  }
  ```
</CodeGroup>

### Point

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  point_prediction = lb_types.ObjectAnnotation(
    name = "point",
    confidence=0.5,
    value = lb_types.Point(x=1166.606, y=1441.768),
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  point_prediction_ndjson = {
    "name": "point",
    "confidence": 0.5,
    "classifications": [],
    "point": {"x": 1166.606, "y": 1441.768}
  }
  ```
</CodeGroup>

### Polyline

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  polyline_prediction = lb_types.ObjectAnnotation(
    name = "polyline",
    confidence=0.5,
    value=lb_types.Line(
          points=[lb_types.Point(x=2534.353, y=249.471), lb_types.Point(x=2429.492, y=182.092), lb_types.Point(x=2294.322, y=221.962), lb_types.Point(x=2224.491, y=180.463), lb_types.Point(x=2136.123, y=204.716),
                  lb_types.Point(x=1712.247, y=173.949), lb_types.Point(x=1703.838, y=84.438), lb_types.Point(x=1579.772, y=82.61), lb_types.Point(x=1583.442, y=167.552),
                  lb_types.Point(x=1478.869, y=164.903), lb_types.Point(x=1418.941, y=318.149), lb_types.Point(x=1243.128, y=400.815), lb_types.Point(x=1022.067, y=319.007),
                  lb_types.Point(x=892.367, y=379.216), lb_types.Point(x=670.273, y=364.408), lb_types.Point(x=613.114, y=288.16), lb_types.Point(x=377.559, y=238.251),
                  lb_types.Point(x=368.087, y=185.064), lb_types.Point(x=246.557, y=167.286), lb_types.Point(x=236.648, y=285.61), lb_types.Point(x=90.929, y=326.412)]
      ),
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}

  polyline_prediction_ndjson = {
    "name": "polyline",
    "confidence":0.5,
    "classifications": [],
    "line": [
      {"x": 2534.353, "y": 249.471},
      {"x": 2429.492, "y": 182.092},
      {"x": 2294.322, "y": 221.962},
      {"x": 2224.491, "y": 180.463},
      {"x": 2136.123, "y": 204.716},
      {"x": 1712.247, "y": 173.949},
      {"x": 1703.838, "y": 84.438},
      {"x": 1579.772, "y": 82.61},
      {"x": 1583.442, "y": 167.552},
      {"x": 1478.869, "y": 164.903},
      {"x": 1418.941, "y": 318.149},
      {"x": 1243.128, "y": 400.815},
      {"x": 1022.067, "y": 319.007},
      {"x": 892.367, "y": 379.216},
      {"x": 670.273, "y": 364.408},
      {"x": 613.114, "y": 288.16},
      {"x": 377.559, "y": 238.251},
      {"x": 368.087, "y": 185.064},
      {"x": 246.557, "y": 167.286},
      {"x": 236.648, "y": 285.61},
      {"x": 90.929, "y": 326.412}
    ]
  }
  ```
</CodeGroup>

## Example: Upload predictions to model run

To upload predictions to a model run:

### Before you start

These examples require the following libraries:

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  import labelbox as lb
  import labelbox.types as lb_types
  import uuid
  import requests
  ```
</CodeGroup>

Replace the value of `API_KEY` with a valid [API key](/horizon/developer-guides/create-api-key) to connect to the Labelbox client.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  API_KEY = None
  client = lb.Client(API_KEY)
  ```
</CodeGroup>

### Step 1: Import data rows into Catalog

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # send a sample image as batch to the project
  global_key = "2560px-Kitano_Street_Kobe01s5s4110.jpeg"
  test_img_url = {
      "row_data": "https://storage.googleapis.com/labelbox-datasets/image_sample_data/2560px-Kitano_Street_Kobe01s5s4110.jpeg",
      "global_key": global_key
  }
  dataset = client.create_dataset(name="image_prediction_demo")
  task = dataset.create_data_rows([test_img_url])
  print("Errors:",task.errors)
  print("Failed data rows:", task.failed_data_rows)
  ```
</CodeGroup>

### Step 2: Set up ontology for predictions

Your model run ontology should support all tools and classifications used in your predictions.

This example shows how to create an ontology with all supported [prediction types](#supported-predictions) .

<CodeGroup>
  ```python Python expandable theme={"theme":"css-variables"}
  ontology_builder = lb.OntologyBuilder(
    classifications=[ # List of Classification objects
      lb.Classification(
        class_type=lb.Classification.Type.RADIO,
        name="radio_question",
        options=[
          lb.Option(value="first_radio_answer"),
          lb.Option(value="second_radio_answer")
        ]
      ),
      lb.Classification(
        class_type=lb.Classification.Type.CHECKLIST,
        name="checklist_question",
        options=[
          lb.Option(value="first_checklist_answer"),
          lb.Option(value="second_checklist_answer")
        ]
      ),
      lb.Classification(
        class_type=lb.Classification.Type.TEXT,
        name="free_text"
      ),
      lb.Classification(
          class_type=lb.Classification.Type.RADIO,
          name="nested_radio_question",
          options=[
              lb.Option("first_radio_answer",
                  options=[
                      lb.Classification(
                          class_type=lb.Classification.Type.RADIO,
                          name="sub_radio_question",
                          options=[lb.Option("first_sub_radio_answer")]
                      )
                  ]
              )
            ]
          ),
      lb.Classification(
        class_type=lb.Classification.Type.CHECKLIST,
        name="nested_checklist_question",
        options=[
            lb.Option("first_checklist_answer",
              options=[
                lb.Classification(
                    class_type=lb.Classification.Type.CHECKLIST,
                    name="sub_checklist_question",
                    options=[lb.Option("first_sub_checklist_answer")]
                )
            ]
          )
        ]
      ),
    ],
    tools=[ # List of tools
      lb.Tool(
        tool=lb.Tool.Type.BBOX,
        name="bounding_box"),
      lb.Tool(
        tool=lb.Tool.Type.BBOX,
        name="bbox_with_radio_subclass",
        classifications=[
              lb.Classification(
                  class_type=lb.Classification.Type.RADIO,
                  name="sub_radio_question",
                  options=[
                    lb.Option(value="first_sub_radio_answer")
                  ]
                ),
          ]
        ),
      lb.Tool(
        tool=lb.Tool.Type.POLYGON,
        name="polygon"),
      lb.Tool(
        tool=lb.Tool.Type.RASTER_SEGMENTATION,
        name="mask"),
      lb.Tool(
        tool=lb.Tool.Type.RASTER_SEGMENTATION,
        name="mask_with_text_subclass",
        classifications=[
            lb.Classification(
                class_type=lb.Classification.Type.TEXT,
                name="sub_free_text")
            ]
        ),
   	  lb.Tool(
        tool=lb.Tool.Type.POINT,
        name="point"),
      lb.Tool(
        tool=lb.Tool.Type.LINE,
        name="polyline")]
  )

  ontology = client.create_ontology("Image Prediction Import Demo",
  ontology_builder.asdict(),
  media_type=lb.MediaType.Image
  )

  ```
</CodeGroup>

### Step 3: Create model and model run

Create a Model using the ontology and a model run.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # create Model
  model = client.create_model(name="image_model_run_" + str(uuid.uuid4()),
                              ontology_id=ontology.uid)
  # create Model Run
  model_run = model.create_model_run("iteration 1")
  ```
</CodeGroup>

### Step 4: Send data rows to model run

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  model_run.upsert_data_rows(global_keys=[global_key])
  ```
</CodeGroup>

### Step 5: Create prediction payloads

For help creating prediction payloads, see [supported predictions](#supported-predictions). You can declare payloads as Python annotation types (*preferred*) or as NDJSON objects. This example demonstrates each format and shows how to compose annotations into labels attached to the data rows.

The resulting `label_prediction_ndjson` and `label_prediction` payloads should have exactly the same prediction content (except for the value of generated `uuid` string values).

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  label_prediction = []
  label_prediction.append(lb_types.Label(
      data={"global_key" : global_key },
      annotations = [
        radio_prediction,
        nested_radio_prediction,
        checklist_prediction,
        nested_checklist_prediction,
        bbox_prediction,
        bbox_with_radio_subclass_prediction,
        polyline_prediction,
        polygon_prediction,
        mask_prediction,
        mask_with_text_subclass_prediction,
        point_prediction,
        text_annotation
        ]
    )
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  label_prediction_ndjson = []

  for annot in [
      radio_prediction_ndjson,
      checklist_prediction_ndjson,
      bbox_prediction_ndjson,
      bbox_with_radio_subclass_prediction_ndjson,
      polygon_prediction_ndjson,
      mask_prediction_ndjson,
      mask_with_text_subclass_prediction_ndjson,
      point_prediction_ndjson,
      polyline_prediction_ndjson,
      text_annotation_ndjson,
      nested_radio_prediction_ndjson,
      nested_checklist_prediction_ndjson
  ]:
    annot.update({
        "dataRow": {"globalKey": global_key}
    })
    label_prediction_ndjson.append(annot)
  ```
</CodeGroup>

### Step 6: Upload prediction payloads to model run

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Upload the prediction label to the Model Run
  upload_job_prediction = model_run.add_predictions(
      name="prediction_upload_job"+str(uuid.uuid4()),
      predictions=label_prediction)

  # Errors will appear for prediction uploads that failed.

  print("Errors:", upload_job_prediction.errors)
  print("Status of uploads: ", upload_job_prediction.statuses)
  ```
</CodeGroup>

### Step 7: Send annotations to a model run

This step is optional.

This example creates a project with ground truth annotations to visualize both annotations and predictions in the model run.

To send annotations to a model run:

<Steps>
  <Step title="Import them into a project" />

  <Step title="Create a label payload" />

  <Step title="Send them to the model run." />
</Steps>

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # 7.1 Create a labelbox project 
  project = client.create_project(name="Image Prediction Demo",
                                      media_type=lb.MediaType.Image)
  project.connect_ontology(ontology)

  # 7.2 Create a batch 
  project.create_batch(
    "batch_predictions_demo", # Each batch in a project must have a unique name
    global_keys=[global_key], # Paginated collection of data row objects, list of data row ids or global keys
    priority=5 # priority between 1(Highest) - 5(lowest)
  )

  # 7.3 Create your annotation payload as explained in:
  # https://docs.labelbox.com/horizon/developer-guides/import-image-annotations#supported-annotations
  radio_annotation ...
  nested_radio_annotation ... 
  nested_checklist_annotation ...
  checklist_annotation  ... 
  bbox_annotation ...
  bbox_with_radio_subclass_annotation ...
  polygon_annotation ... 
  text_annotation ...
  mask_annotation  ...
  mask_with_text_subclass_annotation ...
  point_annotation ...
  polyline_annotation ...

  # 7.4  Create a Label object by identifying the applicable data row in Labelbox 
  # and providing a list of annotations
  labels = []
  annotations = [
      radio_annotation,
      nested_radio_annotation,
      checklist_annotation,
      nested_checklist_annotation,
      text_annotation,
      bbox_annotation,
      bbox_with_radio_subclass_annotation,
      polygon_annotation,
      mask_annotation,
      mask_with_text_subclass_annotation,
      point_annotation,
      polyline_annotation
  ]
  labels.append(
      lb_types.Label(
          data={"global_key" : global_key },
          annotations=annotations))

  # 7.5 Upload annotations to the project using Label import 
  upload_job_annotation = lb.LabelImport.create_from_objects(
      client = client,
      project_id = project.uid,
      name="annotation_import_" + str(uuid.uuid4()),
      labels=labels)

  upload_job_annotation.wait_until_done()

  # Errors will appear for annotation uploads that failed.
  print("Errors:", upload_job_annotation.errors)
  print("Status of uploads: ", upload_job_annotation.statuses)


  # 7.6 Send the annotations to the model run
  model_run.upsert_labels(project_id=project.uid)
  ```
</CodeGroup>


# Upload LLM response evaluation
Source: https://docs.labelbox.com/horizon/developer-guides/upload-llm-response-evaluation-predictions

How to upload predictions on LLM response evaluation data in a model run and sample upload formats.

<CardGroup>
  <Card title="Open In Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/prediction_upload/conversational_LLM_predictions.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/prediction_upload/conversational_LLM_predictions.ipynb" />
</CardGroup>

## Supported predictions

To upload predictions in Labelbox, you need to create a prediction payload. This section shows how to create a payload for each supported prediction type.

Samples are show for the following supported supported payload formats:

* Python annotation types (*recommended*)
* NDJSON

Both are described below.

### Entity (Message-based)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  ner_prediction = lb_types.ObjectAnnotation(
      name="ner",
      confidence=0.5,
      value=lb_types.ConversationEntity(
          start=0,
          end=8,
          message_id="message-1"
      )
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  ner_prediction_ndjson = {
          "name": "ner",
          "confidence": 0.5,
          "location": {
              "start": 0,
              "end": 8
          },
          "messageId": "message-1"
      }
  ```
</CodeGroup>

### Classification: Radio (Single-choice)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  radio_prediction = lb_types.ClassificationAnnotation(
      name="Choose the best response",
      value=lb_types.Radio(answer=lb_types.ClassificationAnswer(
          name="Response B", confidence=0.5)))
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  radio_prediction_ndjson = {
      "name": "Choose the best response",
      "answer": {
        "name": "Response B",
        "confidence": 0.5
      }
  }
  ```
</CodeGroup>

### Classification: Checklist (Multi-choice, Message-based)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  checklist_prediction= lb_types.ClassificationAnnotation(
    name="checklist_convo", # must match your ontology feature"s name
    value=lb_types.Checklist(
        answer = [
          lb_types.ClassificationAnswer(
              name = "first_checklist_answer",
              confidence=0.5
          ),
          lb_types.ClassificationAnswer(
              name = "second_checklist_answer",
              confidence=0.5
          )
        ]
      ),
    message_id="message-1" # Message specific annotation
   )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  checklist_prediction_ndjson = {
      "name": "checklist_convo",
      "answers": [
          {"name": "first_checklist_answer","confidence":0.5},
          {"name": "second_checklist_answer","confidence":0.5}
      ],
      "messageId": "message-1"
  }
  ```
</CodeGroup>

### Classification: Free-form text

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  text_prediction = lb_types.ClassificationAnnotation(
      name="Provide a reason for your choice",
      value=lb_types.Text(answer="the answer to the text questions right here", confidence=0.5)
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  text_prediction_ndjson = {
      "name": "Provide a reason for your choice",
      "answer": "This is the more concise answer",
      "confidence": 0.5

  }
  ```
</CodeGroup>

### Classification: Nested checklist (Message-based)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  nested_checklist_prediction = lb_types.ClassificationAnnotation(
    name="nested_checklist_question",
    message_id="message-1",
    value=lb_types.Checklist(
      answer=[lb_types.ClassificationAnswer(
        name="first_checklist_answer",
        confidence=0.5, # Confidence scores should be added to the answer
        classifications=[
          lb_types.ClassificationAnnotation(
            name="sub_checklist_question",
            value=lb_types.Checklist(
              answer=[lb_types.ClassificationAnswer(
              name="first_sub_checklist_answer",
              confidence=0.5 # Confidence scores should be added to the answer
            )]
          ))
        ]
      )]
    )
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  nested_checklist_prediction_ndjson = {
    "name": "nested_checklist_question",
    "messageId": "message-1",
    "answer": [{
        "name": "first_checklist_answer",
        "confidence": 0.5, # Confidence scores should be added to the answer
        "classifications" : [
          {
            "name": "sub_checklist_question",
            "answer": {
              "name": "first_sub_checklist_answer",
              "confidence": 0.5, # Confidence scores should be added to the answer
            }
          }
        ]
    }]
  }
  ```
</CodeGroup>

### Classification: Nested radio

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  nested_radio_prediction = lb_types.ClassificationAnnotation(
    name="nested_radio_question",
    value=lb_types.Radio(
      answer=lb_types.ClassificationAnswer(
        name="first_radio_answer",
        confidence=0.5, # Confidence scores should be added to the answer
        classifications=[
          lb_types.ClassificationAnnotation(
            name="sub_radio_question",
            value=lb_types.Radio(
              answer=lb_types.ClassificationAnswer(
                name="first_sub_radio_answer",
                confidence=0.5 # Confidence scores should be added to the answer
              )
            )
          )
        ]
      )
    )
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  nested_radio_prediction_ndjson = {
    "name": "nested_radio_question",
    "answer": {
        "name": "first_radio_answer",
        "confidence": 0.5,
        "classifications": [{
            "name":"sub_radio_question",
            "answer": { "name" : "first_sub_radio_answer",
                       "confidence": 0.5}
          }]
      }
  }
  ```
</CodeGroup>

## Example: Upload predictions to a model run

Here are the steps to upload predictions to a model run:

### Before you start

You must import these libraries to use the code examples in this section.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  import labelbox as lb
  import uuid
  ```
</CodeGroup>

Replace the value of `API_KEY` with a valid [API key](/horizon/developer-guides/create-api-key) to connect to the Labelbox client.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  API_KEY = None
  client = lb.Client(API_KEY)
  ```
</CodeGroup>

### Step 1: Import data rows

Upload your data rows to Catalog. To learn how to format your import JSON files, see [Import conversation & model response data](/horizon/developer-guides/import-conversation-model-response-data).

Note the additional `modelOutputs` object required. For descriptions, see [Model outputs object definitions](/horizon/developer-guides/import-conversation-model-response-data#model-outputs-object).

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  global_key = "pairwise_shooping_asset"
  convo_data =  {
          "row_data": "https://storage.googleapis.com/labelbox-datasets/conversational-sample-data/pairwise_shopping_2.json",
          "global_key": global_key
  }
  # Create a dataset
  dataset = client.create_dataset(name="pairwise_prediction_demo")
  # Create datarows
  task = dataset.create_data_rows([convo_data])
  print("Errors:",task.errors)
  print("Failed data rows:", task.failed_data_rows)
  ```

  ```python Sample row_data theme={"theme":"css-variables"}
  pairwise_shopping_2 =  """
   {
    "type": "application/vnd.labelbox.conversational",
    "version": 1,
    "messages": [
      {
        "messageId": "message-0",
        "timestampUsec": 1530718491,
        "content": "Hi! How can I help?",
        "user": {
          "userId": "Bot 002",
          "name": "Bot"
        },
        "align": "left",
        "canLabel": false
      },
      {
        "messageId": "message-1",
        "timestampUsec": 1530718503,
        "content": "I just bought a vacuum cleaner from you guys like a week ago and it's already broken!!",
        "user": {
          "userId": "User 00686",
          "name": "User"
        },
        "align": "right",
        "canLabel": true
      }

    ],
    "modelOutputs": [
      {
        "title": "Response A",
        "content": "I'm really sorry to hear that your vacuum cleaner is not working as expected. We certainly understand how frustrating this can be, especially with a recent purchase. I assure you, we're here to help!\n\n To proceed with resolving this issue, could you please provide some details about the problem you're experiencing with the vacuum cleaner? Additionally, if you could share your order number or receipt details, that would be very helpful. We will definitely look into the available options for you, which may include a replacement or a refund, depending on the specific circumstances and our return policy.\n\n Your satisfaction is our priority, and we'll do our best to resolve this matter promptly and to your satisfaction.",
        "modelConfigName": "GPT-3.5 with temperature 0"
      },
      {
        "title": "Response B",
        "content": "I'm sorry about the issue with your vacuum cleaner. Please send us the order number or receipt details so we can quickly assist you with a replacement. Your satisfaction is important to us!",
        "modelConfigName": "Fine Tuned GPT-3.5 with demo data"
      }
    ]
  }
  """
  ```
</CodeGroup>

### Step 2: Create/select an ontology for your model predictions

Your model run should have the correct ontology set up with all the tools and classifications supported for your predictions.

Here is an example of creating an ontology programmatically for all the example predictions above:

<CodeGroup>
  ```python Python expandable theme={"theme":"css-variables"}
  ontology_builder = lb.OntologyBuilder(
    tools=[
      lb.Tool(tool=lb.Tool.Type.NER,name="ner"),
    ],
    classifications=[
      lb.Classification(
        class_type=lb.Classification.Type.RADIO,
        scope=lb.Classification.Scope.GLOBAL,
        name="Choose the best response",
        options=[lb.Option(value="Response A"), lb.Option(value="Response B"), lb.Option(value="Tie")]
      ),
      lb.Classification(
        class_type=lb.Classification.Type.TEXT,
        name="Provide a reason for your choice"
      ),
      lb.Classification(
        class_type=lb.Classification.Type.CHECKLIST,
        scope=lb.Classification.Scope.INDEX,
        name="checklist_convo",
        options=[
          lb.Option(value="first_checklist_answer"),
          lb.Option(value="second_checklist_answer")
        ]
      ),
      lb.Classification(
        class_type=lb.Classification.Type.CHECKLIST,
        name="nested_checklist_question",
        scope = lb.Classification.Scope.INDEX,
        options=[
            lb.Option("first_checklist_answer",
              options=[
                lb.Classification(
                    class_type=lb.Classification.Type.CHECKLIST,
                    name="sub_checklist_question",
                    options=[lb.Option("first_sub_checklist_answer")]
                )
            ])
        ]
      ),
      lb.Classification(
          class_type=lb.Classification.Type.RADIO,
          name="nested_radio_question",
          scope = lb.Classification.Scope.GLOBAL,
          options=[
              lb.Option("first_radio_answer",
                  options=[
                      lb.Classification(
                          class_type=lb.Classification.Type.RADIO,
                          name="sub_radio_question",
                          options=[lb.Option("first_sub_radio_answer")]
                      )
                  ])
            ]
      )
    ]
  )

  ontology = client.create_ontology("Pairwise comparison ontology", ontology_builder.asdict(), media_type=lb.MediaType.Conversational)

  ```
</CodeGroup>

### Step 3: Create a model and a model run

Create a Model using the ontology and a model run.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # create model
  model = client.create_model(name="Comparison_model_run_"+ str(uuid.uuid4()),
                              ontology_id=ontology.uid)
  # create model run
  model_run = model.create_model_run("iteration 1")
  ```
</CodeGroup>

### Step 4: Send data rows to the model run

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  model_run.upsert_data_rows(global_keys=[global_key])
  ```
</CodeGroup>

### Step 5: Create the predictions payload

Create the predictions payload using the snippets of code shown [above](#supported-predictions).

Labelbox supports two formats for the annotations payload: NDJSON and Python annotation types. Both approaches are described below with instructions to compose annotations into Labels attached to the data rows.

The resulting `label_ndjson` and `label_prediction `payloads should have exactly the same prediction content.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  label_prediction = []
  label_prediction.append(lb_types.Label(
    data= {"global_key": global_key},
    annotations= [
      ner_prediction,
      text_prediction,
      checklist_prediction,
      radio_prediction,
      nested_radio_prediction,
      nested_checklist_prediction
    ]
  ))
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  label_ndjson = []
  for annotations in [
      ner_prediction_ndjson,
      text_prediction_ndjson,
      checklist_prediction_ndjson,
      radio_prediction_ndjson,
      nested_checklist_prediction_ndjson,
      nested_radio_prediction_ndjson
      ]:
    annotations.update({
        "dataRow": {
            "globalKey": global_key
        }
    })
    label_ndjson.append(annotations)
  ```
</CodeGroup>

### Step 6: Upload predictions payload to the model run

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Upload the prediction label to the Model Run
  upload_job_prediction = model_run.add_predictions(
      name="prediction_upload_job"+str(uuid.uuid4()),
      predictions=label_prediction)

  # Errors will appear for annotation uploads that failed.

  print("Errors:", upload_job_prediction.errors)
  print("Status of uploads: ", upload_job_prediction.statuses)

  ```
</CodeGroup>

### Step 7: Send annotations to the model run (optional)

We will create a project with ground truth annotations to visualize both annotations and predictions in the model run.

To send annotations to a model run, we must first import them into a project, create a label payload, and then send them to the model run.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # 7.1 Create a labelbox project
  project = client.create_project(name="Conversational Human Evaluation Demo",
                                      media_type=lb.MediaType.Conversational)
  project.connect_ontology(ontology)

  # 7.2 Create a batch to send to the project
  project.create_batch(
    "batch_convo_prediction_demo", # Each batch in a project must have a unique name
    global_keys=[global_key], # Paginated collection of data row objects, list of data row ids or global keys
    priority=5 # priority between 1(Highest) - 5(lowest)
  )

  # 7.3 Create the annotations payload (
  # See here for more details: 
  # https://labelbox-group.readme.io/reference/import-conversational-text-annotations#supported-annotations
  ner_annotation ...
  text_annotation ...
  checklist_annotation ...
  radio_annotation ...
  nested_radio_annotation ...
  nested_checklist_annotation ...

  # 7.4 Create the label object 
  label_annotation = []
  label_annotation.append(lb_types.Label(
    data= {"global_key": global_key},
    annotations= [
      ner_annotation,
      text_annotation,
      checklist_annotation,
      radio_annotation,
      nested_radio_annotation,
      nested_checklist_annotation
    ]
  ))

  #7.5 Upload annotations to the project using label import 
  upload_job_annotation = lb.LabelImport.create_from_objects(
      client = client,
      project_id = project.uid,
      name="label_import_job"+ str(uuid.uuid4()),
      labels=label_annotation)

  upload_job_annotation.wait_until_done()
  # Errors will appear for annotation uploads that failed.
  print("Errors:", upload_job_annotation.errors)
  print("Status of uploads: ", upload_job_annotation.statuses)

  # 7.6 Send the annotations to the model run 
  model_run.upsert_labels(project_id=project.uid)
  ```
</CodeGroup>


# Upload text predictions
Source: https://docs.labelbox.com/horizon/developer-guides/upload-text-predictions

How to upload predictions on text data in a model run and sample upload formats.

<CardGroup>
  <Card title="Open In Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/prediction_upload/text_predictions.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/prediction_upload/text_predictions.ipynb" />
</CardGroup>

## Supported prediction types

To upload predictions in Labelbox, you need to create a predictions payload. This section shows how to declare payloads for each supported prediction type. You can declare payloads using Python annotation types (*preferred*) or as NDJSON objects.

Confidence scores are optional. If you do not include confidence scores in your prediction payloads, the prediction is treated as if it had a confidence value of one (`1`).

### Entity

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  named_entity = lb_types.TextEntity(start=10, end=20)
  entities_prediction = lb_types.ObjectAnnotation(value=named_entity, name = "named_entity", confidence=0.5)
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  entities_prediction_ndjson = {
      "name": "named_entity",
      "confidence": 0.5,
      "location": {
          "start": 10,
          "end": 20
      }
  }
  ```
</CodeGroup>

### Classification: radio (single choice)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  radio_prediction = lb_types.ClassificationAnnotation(
      name="radio_question",
      value=lb_types.Radio(answer =
          lb_types.ClassificationAnswer(name = "first_radio_answer", confidence=0.5)
      )
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  radio_prediction_ndjson = {
    "name": "radio_question",
    "answer": {"name": "first_radio_answer", "confidence": 0.5}
  }
  ```
</CodeGroup>

### Classification: radio nested

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  nested_radio_prediction = lb_types.ClassificationAnnotation(
    name="nested_radio_question",
    value=lb_types.Radio(
      answer=lb_types.ClassificationAnswer(
        name="first_radio_answer",
        confidence=0.5, # Confidence scores should be added to the answer
        classifications=[
          lb_types.ClassificationAnnotation(
            name="sub_radio_question",
            value=lb_types.Radio(
              answer=lb_types.ClassificationAnswer(
                name="first_sub_radio_answer",
                confidence=0.5
              )
            )
          )
        ]
      )
    )
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  nested_radio_prediction_ndjson= {
    "name": "nested_radio_question",
    "answer": {
        "name": "first_radio_answer",
        "confidence": 0.5, # Confidence scores should be added to the answer
        "classifications": [{
            "name":"sub_radio_question",
            "answer": { "name" : "first_sub_radio_answer",
                       "confidence": 0.5}
          }]
      }
  }
  ```
</CodeGroup>

### Classification: checklist nested

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  nested_checklist_prediction = lb_types.ClassificationAnnotation(
    name="nested_checklist_question",
    value=lb_types.Checklist(
      answer=[lb_types.ClassificationAnswer(
        name="first_checklist_answer",
        confidence=0.5, # Confidence scores should be added to the answer
        classifications=[
          lb_types.ClassificationAnnotation(
            name="sub_checklist_question",
            value=lb_types.Checklist(
              answer=[lb_types.ClassificationAnswer(
              name="first_sub_checklist_answer",
              confidence=0.5,
            )]
          ))
        ]
      )]
    )
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  nested_checklist_prediction_ndjson = {
    "name": "nested_checklist_question",
    "answer": [{
        "name": "first_checklist_answer",
        "confidence": 0.5, # Confidence scores should be added to the answer
        "classifications" : [
          {
            "name": "sub_checklist_question",
            "answer": {"name": "first_sub_checklist_answer",
                       "confidence": 0.5}
          }
        ]
    }]
  }
  ```
</CodeGroup>

### Classification: checklist (multiple choice)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  checklist_prediction = lb_types.ClassificationAnnotation(
      name="checklist_question",
      value=lb_types.Checklist(
          answer = [
              lb_types.ClassificationAnswer(
                  name = "first_checklist_answer",
                  confidence=0.5
              ),
              lb_types.ClassificationAnswer(
                  name = "second_checklist_answer",
                  confidence=0.5
              ),
              lb_types.ClassificationAnswer(
                  name = "third_checklist_answer",
                  confidence=0.5
              )
      ])
    )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  checklist_prediction_ndjson = {
    "name": "checklist_question",
    "answer": [
      {"name": "first_checklist_answer", "confidence": 0.5}
    ]
  }
  ```
</CodeGroup>

### Classification: free-form text

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  text_prediction = lb_types.ClassificationAnnotation(
      name = "free_text",
      value = lb_types.Text(answer="sample text", confidence=0.5)
  )
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  text_prediction_ndjson = {
    "name": "free_text",
    "answer": "sample text",
    "confidence": 0.5
  }
  ```
</CodeGroup>

## Example: Upload predictions to model run

To upload predictions to a model run:

### Before you start

These examples require the following libraries:

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  import labelbox as lb
  import labelbox.data.annotation_types as lb_types
  import labelbox.data.serialization as lb_serializers
  import uuid
  ```
</CodeGroup>

Replace the value of `API_KEY` with a valid [API key](/horizon/developer-guides/create-api-key) to connect to the Labelbox client.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  API_KEY = None
  client = lb.Client(API_KEY)
  ```
</CodeGroup>

### Step 1: Import data rows into Catalog

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  global_key = "lorem-ipsum.txt"
  test_img_url = {
      "row_data": "https://storage.googleapis.com/labelbox-sample-datasets/nlp/lorem-ipsum.txt",
      "global_key": global_key
  }
  dataset = client.create_dataset(
      name="text prediction demo dataset",
      iam_integration=None # Removing this argument will default to the organziation's default iam integration
      )
  task = dataset.create_data_rows([test_img_url])
  task.wait_till_done()
  print("Errors:",task.errors)
  print("Failed data rows:", task.failed_data_rows)
  ```
</CodeGroup>

### Step 2: Set up ontology for predictions

Your model run ontology should support all tools and classifications required by your predictions.

this example shows how to create an ontology containing all [supported prediction types](#supported-prediction-types).

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  ## Set up the ontology and link the tools created above.

  ontology_builder = lb.OntologyBuilder(
    classifications=[ # List of Classification objects
      lb.Classification(
        class_type=lb.Classification.Type.RADIO,
        name="radio_question", 
        options=[lb.Option(value="first_radio_answer")]
      ),
      lb.Classification(
        class_type=lb.Classification.Type.RADIO,
        name="nested_radio_question", 
        options=[
          lb.Option(value="first_radio_answer",
            options=[
                lb.Classification(
                  class_type=lb.Classification.Type.RADIO,
                  name="sub_radio_question",
                  options=[
                    lb.Option(value="first_sub_radio_answer")
                  ]
              ),
            ]
          )
        ],
      ),
      lb.Classification(
        class_type=lb.Classification.Type.CHECKLIST,
        name="checklist_question", 
        options=[
          lb.Option(value="first_checklist_answer"),
          lb.Option(value="second_checklist_answer"),
          lb.Option(value="third_checklist_answer")
        ]
      ), 
       lb.Classification(
        class_type=lb.Classification.Type.TEXT,
        name="free_text"
      ),
      lb.Classification(
        class_type=lb.Classification.Type.CHECKLIST,
        name="nested_checklist_question",
        options=[
            lb.Option("first_checklist_answer",
              options=[
                lb.Classification(
                    class_type=lb.Classification.Type.CHECKLIST,
                    name="sub_checklist_question", 
                    options=[lb.Option("first_sub_checklist_answer")]
                )
            ]
          )
        ]
      )
    ],
    tools=[ # List of Tool objects
           lb.Tool(tool=lb.Tool.Type.NER,
                name="named_entity")
      ]
  )

  ontology = client.create_ontology("Ontology Text Predictions",
   ontology_builder.asdict() ,
   media_type=lb.MediaType.Text)
  ```
</CodeGroup>

### Step 3: Create model and model run

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  model = client.create_model(name="text_model_run_"+ str(uuid.uuid4()),
                              ontology_id=ontology.uid)
  model_run = model.create_model_run("iteration 1")
  ```
</CodeGroup>

### Step 4: Send data rows to model run

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  model_run.upsert_data_rows(global_keys=[global_key])
  ```
</CodeGroup>

### Step 5: Create prediction payloads

See [supported prediction types](#supported-prediction-types) for help creating prediction payloads. You can declare predictions as Python annotation types (*preferred*) or NDJSON objects. These examples show each type and describe how to compose predictions into labels attached to the data rows.

The resulting `label_ndjson_predictions` and `label_predictions` payloads should have exactly the same prediction content (except for the `uuid` string values).

<CodeGroup>
  ```python Python annotation payload theme={"theme":"css-variables"}
  label_predictions = []
  label_predictions.append(
    lb_types.Label(
      data={"global_key" : global_key },
      annotations = [
        entities_prediction,
        nested_radio_prediction,
        radio_prediction,
        checklist_prediction,
        nested_checklist_prediction,
        text_prediction,
      ]
    )
  )
  ```

  ```python NDJSON payload theme={"theme":"css-variables"}
  label_ndjson_predictions= []
  for annot in [
      entities_prediction_ndjson,
      radio_prediction_ndjson,
      checklist_prediction_ndjson,
      text_prediction_ndjson,
      nested_radio_prediction_ndjson,
      nested_checklist_prediction_ndjson
    ]:
    annot.update({
        "dataRow": {"globalKey": global_key}
    })
    label_ndjson_predictions.append(annot)
  ```
</CodeGroup>

### Step 6: Upload payload to model run

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Upload the prediction label to the Model Run
  upload_job_prediction = model_run.add_predictions(
      name="prediction_upload_job"+str(uuid.uuid4()),
      predictions=label_predictions)

  # Errors will appear for annotation uploads that failed.

  print("Errors:", upload_job_prediction.errors)
  print("Status of uploads: ", upload_job_prediction.statuses)
  ```
</CodeGroup>

### Step 7: Send annotations to model run

This step is optional.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # 7.1. Create a labelbox project
  project = client.create_project(name="Text Prediction Import Demo",
                                      media_type=lb.MediaType.Text)
  project.connect_ontology(ontology)

  # 7.2. Create a batch to send to the project
  project.create_batch(
    "batch_text_prediction_demo", # Each batch in a project must have a unique name
    global_keys=[global_key], # Paginated collection of data row objects, list of data row ids or global keys
    priority=5 # priority between 1(Highest) - 5(lowest)
  )

  # 7.3 Create your annotation payload as explained in:
  # /horizon/developer-guides/import-text-annotations#supported-annotations
  entities_annotation ...
  nested_radio_annotation ...
  radio_annotation ...
  checklist_annotation ...
  nested_checklist_annotation ...
  text_annotation ...

  # 7.4. Create the label object
  label = []
  label.append(
    lb_types.Label(
      data={"global_key" : global_key },
      annotations = [
        entities_annotation,
        nested_radio_annotation,
        radio_annotation,
        checklist_annotation,
        nested_checklist_annotation,
        text_annotation,
      ]
    )
  )

  # 7.5 Upload annotations to the project using Label Import
  upload_job_annotation = lb.LabelImport.create_from_objects(
      client = client,
      project_id = project.uid,
      name="text_label_import_job"+ str(uuid.uuid4()),
      labels=label)

  upload_job_annotation.wait_until_done()
  # Errors will appear for annotation uploads that failed.
  print("Errors:", upload_job_annotation.errors)
  print("Status of uploads: ", upload_job_annotation.statuses)

  # 7.6 Send the annotations to the model run

  # get the labels id from the project
  model_run.upsert_labels(project_id=project.uid)
  ```
</CodeGroup>


# Upload video predictions
Source: https://docs.labelbox.com/horizon/developer-guides/upload-video-predictions

How to upload predictions on video data in a model run and sample upload formats.

<CardGroup>
  <Card title="Open In Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/prediction_upload/video_predictions.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/prediction_upload/video_predictions.ipynb" />
</CardGroup>

## Supported annotations

To upload predictions in Labelbox, you need to create a predictions payload. In this section, we provide this payload for every supported prediction type.

<Info>
  ### Uploading confidence scores is optional

  If you do not specify a confidence score, the prediction will be treated as if it had a confidence score of 1. Note that confidence scores are not supported for `VideoObjectAnnotation` objects.

  Nested `ClassificationAnnotation` objects inside of a `VideoObjectAnnotation` can still have a confidence score.
</Info>

### Bounding box

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  # bbox dimensions 
  bbox_dm = {
    "top":617,
    "left":1371,
    "height":419,
    "width":505
  }

  bbox_prediction = [
    lb_types.VideoObjectAnnotation(
      name = "bbox_video",  
      keyframe=True,
      frame=13,
      segment_index=0,
      value = lb_types.Rectangle(
            start=lb_types.Point(x=bbox_dm["left"], y=bbox_dm["top"]), # x = left, y = top 
            end=lb_types.Point(x=bbox_dm["left"] + bbox_dm["width"], y=bbox_dm["top"] + bbox_dm["height"]), # x= left + width , y = top + height
        )
    ),
    lb_types.VideoObjectAnnotation(
      name = "bbox_video",  
      keyframe=True,
      frame=15,
      segment_index=0,
      value = lb_types.Rectangle(
            start=lb_types.Point(x=bbox_dm["left"], y=bbox_dm["top"]),
            end=lb_types.Point(x=bbox_dm["left"] + bbox_dm["width"], y=bbox_dm["top"] + bbox_dm["height"]),
        )
    ),
    lb_types.VideoObjectAnnotation(
      name = "bbox_video",  
      keyframe=True,
      frame=19,
      segment_index=0,
      value = lb_types.Rectangle(
            start=lb_types.Point(x=bbox_dm["left"], y=bbox_dm["top"]), 
            end=lb_types.Point(x=bbox_dm["left"] + bbox_dm["width"], y=bbox_dm["top"] + bbox_dm["height"]),
        )
    )
  ]
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  # bbox dimensions
  bbox_dm = {
    "top":617,
    "left":1371,
    "height":419,
    "width":505
  }

  bbox_prediction_ndjson = {
      "name" : "bbox_video",
      "segments" : [{
          "keyframes" : [
              {
                "frame": 13,
                "bbox" : bbox_dm
             },
             {
                "frame": 15,
                "bbox" : bbox_dm
             },
             {
                "frame": 19,
                "bbox" : bbox_dm
             }
          ]
        }
      ]
  }
  ```
</CodeGroup>

### Point

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  point_prediction = [
      lb_types.VideoObjectAnnotation(
          name = "point_video",
          keyframe=True,
          frame=17,
          value = lb_types.Point(x=660.134, y=407.926),
          )
  ]
  ```

  ```python NDJSON theme={"theme":"css-variables"}

  point_prediction_ndjson = {
      "name": "point_video",
      "confidence": 0.5,
      "segments": [{
          "keyframes": [{
              "frame": 17,
              "point" : {
                  "x": 660.134 ,
                  "y": 407.926
              }
          }]
      }]
  }
  ```
</CodeGroup>

### Polyline

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  polyline_prediction = [
    lb_types.VideoObjectAnnotation(
      name = "line_video_frame",
      keyframe=True, 
      frame=5,
      segment_index=0,
      value=lb_types.Line( 
            points=[lb_types.Point(x=680, y=100), lb_types.Point(x=100, y=190)]
        )
    ),
    lb_types.VideoObjectAnnotation(
      name = "line_video_frame",
      keyframe=True, 
      frame=12,
      segment_index=0,
      value=lb_types.Line( 
            points=[lb_types.Point(x=680, y=100), lb_types.Point(x=100, y=190)]
        )
    ),
    lb_types.VideoObjectAnnotation(
      name = "line_video_frame",
      keyframe=True, 
      frame=20,
      segment_index=0,
      value=lb_types.Line( 
            points=[lb_types.Point(x=680, y=100), lb_types.Point(x=100, y=190)]
        )
    ),
    lb_types.VideoObjectAnnotation(
      name = "line_video_frame",
      keyframe=True, 
      frame=24,
      segment_index=1,
      value=lb_types.Line( 
            points=[lb_types.Point(x=680, y=100), lb_types.Point(x=100, y=190)]
        )
    ),
    lb_types.VideoObjectAnnotation(
      name = "line_video_frame",
      keyframe=True, 
      frame=45,
      segment_index=1,
      value=lb_types.Line( 
            points=[lb_types.Point(x=680, y=100), lb_types.Point(x=100, y=190)]
        )
    )  
  ]
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  polyline_prediction_ndjson = {
    "name": "line_video_frame",
    "segments": [
        {
          "keyframes": [
            {
              "frame": 5,
              "line": [{
                "x": 680,
                "y": 100
              },{
                "x": 100,
                "y": 190
              },{
                "x": 190,
                "y": 220
              }]
            },
            {
              "frame": 12,
              "line": [{
                "x": 680,
                "y": 280
              },{
                "x": 300,
                "y": 380
              },{
                "x": 400,
                "y": 460
              }]
            },
            {
              "frame": 20,
              "line": [{
                "x": 680,
                "y": 180
              },{
                "x": 100,
                "y": 200
              },{
                "x": 200,
                "y": 260
              }]
            }
          ]
        },
        {
          "keyframes": [
            {
              "frame": 24,
              "line": [{
                "x": 300,
                "y": 310
              },{
                "x": 330,
                "y": 430
              }]
            },
            {
              "frame": 45,
              "line": [{
                "x": 600,
                "y": 810
              },{
                "x": 900,
                "y": 930
              }]
            }
          ]
        }
      ]
  }
  ```
</CodeGroup>

### Classification: Radio (global)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  global_radio_prediction = [lb_types.ClassificationAnnotation(
      name="radio_class_global",
      value=lb_types.Radio(answer = lb_types.ClassificationAnswer(
        name = "first_radio_answer",
        confidence=0.5
        ))
  )]
  ```

  ```python NDJSON theme={"theme":"css-variables"}

  global_radio_classification_ndjson = {
      "name": "radio_class_global",
      "answer": { "name": "first_radio_answer", "confidence": 0.5}
  }
  ```
</CodeGroup>

### Classification: Radio (frame-based)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  radio_prediction = [
      lb_types.VideoClassificationAnnotation(
          name="radio_class",
          frame=9,
          segment_index=0,
          value=lb_types.Radio(answer = lb_types.ClassificationAnswer(
            name = "first_radio_answer",
            confidence=0.5
            ))
      ),
      lb_types.VideoClassificationAnnotation(
          name="radio_class",
          frame=15,
          segment_index=0,
          value=lb_types.Radio(answer = lb_types.ClassificationAnswer(
            name = "first_radio_answer",
            confidence=0.5
            ))
      )
  ]
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  frame_radio_classification_prediction_ndjson = {
      "name": "radio_class",
      "answer": { "name": "first_radio_answer", "frames": [{"start": 9, "end": 15}]}
  }
  ```
</CodeGroup>

### Classification: Checklist (global)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  global_checklist_prediction=[lb_types.ClassificationAnnotation(
    name="checklist_class_global",
    value=lb_types.Checklist(
        answer = [
          lb_types.ClassificationAnswer(
              name = "first_checklist_answer",
              confidence=0.5
          ),
          lb_types.ClassificationAnswer(
              name = "second_checklist_answer",
              confidence=0.5
          )
        ]
      )
   )]
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  global_checklist_classification_ndjson = {
      "name": "checklist_class_global",
      "answer": [
          { "name": "first_checklist_answer" , "confidence": 0.5},
          { "name": "second_checklist_answer", "confidence": 0.5}
    ]
  }
  ```
</CodeGroup>

### Classification: Checklist (frame-based)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  checklist_prediction= [
      lb_types.VideoClassificationAnnotation(
          name="checklist_class",
          frame=29,
          segment_index=0,
          value=lb_types.Checklist(
              answer = [
                  lb_types.ClassificationAnswer(
                      name = "first_checklist_answer",
                      confidence=0.5 
                  )
              ]
              )
          ),
      lb_types.VideoClassificationAnnotation(
          name="checklist_class", 
          frame=35,
          segment_index=0,
          value=lb_types.Checklist(
              answer = [
                  lb_types.ClassificationAnswer(
                      name = "first_checklist_answer",
                      confidence=0.5 
                  )
              ]
              )
          ),
      lb_types.VideoClassificationAnnotation(
          name="checklist_class", 
          frame=39, 
          segment_index=1,
          value=lb_types.Checklist(
              answer = [
                  lb_types.ClassificationAnswer(
                      name = "second_checklist_answer",
                      confidence=0.5 
                  )
              ]
              )
          ),
      lb_types.VideoClassificationAnnotation(
          name="checklist_class", 
          frame=45, 
          segment_index=1,
          value=lb_types.Checklist(
              answer = [
                  
                  lb_types.ClassificationAnswer(
                      name = "second_checklist_answer",
                      confidence=0.5 
                  )
              ]
              )
          )
  ]
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  frame_checklist_classification_prediction_ndjson = {
      "name": "checklist_class",
      "answer": [
          { "name": "first_checklist_answer" , "frames": [{"start": 29, "end": 35 }]},
          { "name": "second_checklist_answer", "frames": [{"start": 39, "end": 45 }]}
    ]
  }
  ```
</CodeGroup>

### Classification: Nested radio (global)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  nested_radio_prediction =[lb_types.ClassificationAnnotation(
    name="nested_radio_question",
    value=lb_types.Radio(
      answer=lb_types.ClassificationAnswer(
        name="first_radio_answer",
        confidence=0.5 ,
        classifications=[
          lb_types.ClassificationAnnotation(
            name="sub_radio_question",
            value=lb_types.Radio(
              answer=lb_types.ClassificationAnswer(
                name="first_sub_radio_answer",
                confidence=0.5
              )
            )
          )
        ]
      )
    )
  )]
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  nested_radio_prediction_ndjson = {
    "name": "nested_radio_question",
    "answer": {"name": "first_radio_answer", "confidence": 0.5,
    "classifications" : [
      {"name": "sub_radio_question", "answer": {"name": "first_sub_radio_answer", "confidence": 0.5}}
     ]
    }
  }
  ```
</CodeGroup>

### Classification: Nested checklist (global)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  nested_checklist_prediction = [lb_types.ClassificationAnnotation(
    name="nested_checklist_question",
    value=lb_types.Checklist(
      answer=[lb_types.ClassificationAnswer(
        name="first_checklist_answer",
        confidence=0.5 ,
        classifications=[
          lb_types.ClassificationAnnotation(
            name="sub_checklist_question",
            value=lb_types.Checklist(
              answer=[lb_types.ClassificationAnswer(
              name="first_sub_checklist_answer",
              confidence=0.5
            )]
          ))
        ]
      )]
    )
  )]
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  nested_checklist_prediction_ndjson = {
    "name": "nested_checklist_question",
    "answer": [{
        "name": "first_checklist_answer",
        "confidence": 0.5,
        "classifications" : [
          {
            "name": "sub_checklist_question",
            "answer": {"name": "first_sub_checklist_answer", "confidence": 0.5}
          }
        ]
    }]
  }
  ```
</CodeGroup>

### Bounding box with sub-classifications (frame-based)

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  bbox_dm2 = {
    "top": 146.0,
    "left": 98.0,
    "height": 382.0,
    "width": 341.0
  }

  frame_bbox_with_checklist_subclass_prediction = [
    lb_types.VideoObjectAnnotation(
      name = "bbox_class",  
      keyframe=True,
      frame=10,
      segment_index=0,
      value = lb_types.Rectangle(
            start=lb_types.Point(x=bbox_dm2["left"], y=bbox_dm2["top"]), # x = left, y = top 
            end=lb_types.Point(x=bbox_dm2["left"] + bbox_dm2["width"], y=bbox_dm2["top"] + bbox_dm2["height"]), # x= left + width , y = top + height
        )
    ),
    lb_types.VideoObjectAnnotation(
      name = "bbox_class",  
      keyframe=True,
      frame=11,
      segment_index=0,
      value = lb_types.Rectangle(
            start=lb_types.Point(x=bbox_dm2["left"], y=bbox_dm2["top"]), 
            end=lb_types.Point(x=bbox_dm2["left"] + bbox_dm2["width"], y=bbox_dm2["top"] + bbox_dm2["height"]),
        ),
      classifications=[
                  lb_types.ClassificationAnnotation(
                      name='checklist_class',
                      value=lb_types.Checklist(answer=[lb_types.ClassificationAnswer(
                          name="first_checklist_answer",
                          confidence=0.5 
                      )])
                  )
              ]
    ),
    lb_types.VideoObjectAnnotation(
      name = "bbox_class",  
      keyframe=True,
      frame=13,
      segment_index=0,
      value = lb_types.Rectangle(
            start=lb_types.Point(x=bbox_dm2["left"], y=bbox_dm2["top"]), 
            end=lb_types.Point(x=bbox_dm2["left"] + bbox_dm2["width"], y=bbox_dm2["top"] + bbox_dm2["height"]),
        ),
      classifications=[
                lb_types.ClassificationAnnotation(
                    name='checklist_class',
                    value=lb_types.Checklist(answer=[lb_types.ClassificationAnswer(
                        name="second_checklist_answer",
                        confidence=0.5 
                    )])
                )
            ]
    )
  ]
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  frame_bbox_with_checklist_subclass_prediction_ndjson = {
      "name": "bbox_class",
      "segments": [{
          "keyframes": [
              {
              "frame": 10,
              "bbox": bbox_dm2
            },
            {
            "frame": 11,
              "bbox": bbox_dm2,
              "classifications": [
                {
                  "name": "bbox_radio",
                  "answer": [{"name": "first_checklist_answer", "confidence": 0.5}]
                }
              ]
            },
            {
            "frame": 13,
              "bbox": bbox_dm2,
              "classifications": [
                {
                  "name": "bbox_radio",
                  "answer": [{"name": "second_checklist_answer", "confidence": 0.5}]
                }
              ]
            }
          ]
        }
      ]
  }
  ```
</CodeGroup>

### Text

<CodeGroup>
  ```python Python annotation theme={"theme":"css-variables"}
  text_prediction = [lb_types.ClassificationAnnotation(
    name="free_text",  # must match your ontology feature's name
    value=lb_types.Text(answer="sample text", confidence=0.5)
  )]
  ```

  ```python NDJSON theme={"theme":"css-variables"}
  text_prediction_ndjson = {
    'name': 'free_text',
    'confidence': 0.5,
    'answer': 'sample text',
  }
  ```
</CodeGroup>

### Masks

Raster segmentation masks are not yet supported in the Model product.

## End-to-end example: Upload predictions to a Model Run

Follow the steps below to upload predictions to a model run.

### Before you start

You will need to import these libraries to use the code examples in this section:

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  import labelbox as lb
  import uuid
  ```
</CodeGroup>

Replace the value of `API_KEY` with a valid [API key](/horizon/developer-guides/create-api-key) to connect to the Labelbox client.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  API_KEY = None
  client = lb.Client(API_KEY)
  ```
</CodeGroup>

### Step 1: Import data rows into Catalog

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  global_key = "sample-video-2.mp4"
  test_img_url = {
      "row_data": "https://storage.googleapis.com/labelbox-datasets/video-sample-data/sample-video-2.mp4",
      "global_key": global_key
  }
  dataset = client.create_dataset(
      name="Video prediction demo",
      iam_integration=None # Removing this argument will default to the organziation's default iam integration
  )
  task = dataset.create_data_rows([test_img_url])
  task.wait_till_done()
  print("Errors: ",task.errors)
  print("Failed data rows: ",task.failed_data_rows)
  ```
</CodeGroup>

### Step 2: Create/select an ontology for your model predictions

Your project should have the correct ontology setup with all the tools and classifications supported for your predictions and the tool and classifications `name` should match the `name` fields in your predictions to ensure the correct feature schemas are matched.

<CodeGroup>
  ```python Python expandable theme={"theme":"css-variables"}
  ontology_builder = lb.OntologyBuilder(
      tools=[
          lb.Tool(tool=lb.Tool.Type.BBOX, name="bbox_video"),
          lb.Tool(tool=lb.Tool.Type.POINT, name="point_video"),
          lb.Tool(tool=lb.Tool.Type.LINE, name="line_video_frame"),
          lb.Tool(
            tool=lb.Tool.Type.BBOX, name="bbox_class",
            classifications=[
              lb.Classification(
              class_type=lb.Classification.Type.CHECKLIST,
              name="checklist_class",
              scope = lb.Classification.Scope.INDEX, ## defined scope for frame classifications
              options=[
                  lb.Option(value="first_checklist_answer"),
                  lb.Option(value="second_checklist_answer")
              ]
              )
            ]
          )
      ],
      classifications=[
          lb.Classification(
              class_type=lb.Classification.Type.CHECKLIST,
              name="checklist_class",
              scope = lb.Classification.Scope.INDEX, ## defined scope for frame classifications
              options=[
                  lb.Option(value="first_checklist_answer"),
                  lb.Option(value="second_checklist_answer")
              ]
          ),
          lb.Classification(
              class_type=lb.Classification.Type.RADIO,
              name="radio_class",
              scope = lb.Classification.Scope.INDEX,
              options=[
                  lb.Option(value="first_radio_answer"),
                  lb.Option(value="second_radio_answer")
              ]
          ),
           lb.Classification(
                class_type=lb.Classification.Type.RADIO,
                name="nested_radio_question",
                options=[
                    lb.Option("first_radio_answer",
                          options=[
                              lb.Classification(
                                  class_type=lb.Classification.Type.RADIO,
                                  name="sub_radio_question",
                                  options=[lb.Option("first_sub_radio_answer")]
                              )
                          ]
                    )
                ]
          ),
          lb.Classification(
            class_type=lb.Classification.Type.CHECKLIST,
            name="nested_checklist_question",
            options=[
                lb.Option("first_checklist_answer",
                  options=[
                    lb.Classification(
                        class_type=lb.Classification.Type.CHECKLIST,
                        name="sub_checklist_question",
                        options=[lb.Option("first_sub_checklist_answer")]
                    )
                ]
              )
            ]
          ),
          lb.Classification(
            class_type=lb.Classification.Type.RADIO,
            name="radio_class_global",
            options=[
                  lb.Option(value="first_radio_answer"),
                  lb.Option(value="second_radio_answer")
              ]
          ),
          lb.Classification(
            class_type=lb.Classification.Type.CHECKLIST,
            name="checklist_class_global",
            options=[
                  lb.Option(value="first_checklist_answer"),
                  lb.Option(value="second_checklist_answer")
            ]
          ),
          lb.Classification(
            class_type=lb.Classification.Type.TEXT,
            name="free_text"
          )
      ]
  )

  ontology = client.create_ontology("Ontology Video Annotations",
  ontology_builder.asdict(),
  media_type=lb.MediaType.Video
  )

  ```
</CodeGroup>

### Step 3: Create a Model and model run

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # create Model
  model = client.create_model(name="video_model_run_" + str(uuid.uuid4()),
                              ontology_id=ontology.uid)
  # create model run
  model_run = model.create_model_run("iteration 1")
  ```
</CodeGroup>

### Step 4: Send data rows to the model run

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  model_run.upsert_data_rows(global_keys=[global_key])
  ```
</CodeGroup>

### Step 5: Create the predictions payload

Create the annotations payload using the snippets of code here.

<CodeGroup>
  ```python Python annotation payload theme={"theme":"css-variables"}
  label_predictions = []
  annotations_list = [
      point_prediction,
      bbox_prediction,
      polyline_prediction, 
      checklist_prediction, 
      radio_prediction,
      nested_radio_prediction,
      nested_checklist_prediction,
      frame_bbox_with_checklist_subclass_prediction,
      global_radio_prediction,
      global_checklist_prediction,
      text_prediction
        ]

  flatten_list_annotations = [ann for ann_sublist in annotations_list for ann in ann_sublist] 

  label_predictions.append(
      lb_types.Label(
          data={"global_key" : global_key },
          annotations = flatten_list_annotations
      )
  )
  ```

  ```python NDJSON payload theme={"theme":"css-variables"}
  # Create a Label object by identifying the applicable data row in Labelbox and providing a list of annotations

  label_prediction_ndjson = []

  for annotation in [
      point_prediction_ndjson,
      bbox_prediction_ndjson,
      polyline_prediction_ndjson,
      frame_checklist_classification_prediction_ndjson,
      frame_radio_classification_prediction_ndjson,
      nested_radio_prediction_ndjson,
      nested_checklist_prediction_ndjson,
      frame_bbox_with_checklist_subclass_prediction_ndjson,
      global_radio_classification_ndjson,
      global_checklist_classification_ndjson,
      video_mask_prediction_ndjson,
      text_prediction_ndjson
  ]:
    annotation.update({
        "dataRow": {
            "globalKey": global_key
        }
    })
    label_prediction_ndjson.append(annotation)
  ```
</CodeGroup>

### Step 6: Upload the predictions payload to the model run

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Upload the prediction label to the Model Run
  upload_job_prediction = model_run.add_predictions(
      name="prediction_upload_job"+str(uuid.uuid4()),
      predictions=label_predictions)

  # Errors will appear for annotation uploads that failed.

  print("Errors:", upload_job_prediction.errors)
  print("Status of uploads: ", upload_job_prediction.statuses)

  ```
</CodeGroup>

### Step 7: Send annotations to the model run (optional)

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # 7.1 Create a labelbox project
  # Create a Labelbox project
  project = client.create_project(name="video_prediction_demo",
                                      media_type=lb.MediaType.Video)
  project.connect_ontology(ontology)

  # 7.2 Create a batch to send to the project
  project.create_batch(
    "batch_video_prediction_demo", # Each batch in a project must have a unique name
    global_keys=[global_key], # A list of data rows, data row ids or global keys
    priority=5 # priority between 1(Highest) - 5(lowest)
  )

  # 7.3 create the annotations payload
  # See here for more details:
  # /horizon/developer-guides/import-video-annotations#supported-annotations
  checklist_annotation ...
  radio_annotation ...
  bbox_annotation ...
  frame_bbox_with_checklist_subclass ...
  point_annotation ...
  polyline_annotation ...
  global_checklist_annotation ...
  global_radio_annotation ...
  nested_checklist_annotation ...
  nested_radio_annotation ...
  text_annotation ...

  # 7.4 Create a Label object by identifying the applicable data row in Labelbox and providing a list of annotations
  labels = []
  annotations_list = [
            checklist_annotation,
            radio_annotation,
            bbox_annotation,
            frame_bbox_with_checklist_subclass,
            point_annotation,
            polyline_annotation,
            global_checklist_annotation,
            global_radio_annotation,
            nested_checklist_annotation,
            nested_radio_annotation,
            text_annotation
        ]

  flatten_list_annotations = [ann for ann_sublist in annotations_list for ann in ann_sublist]

  labels.append(
      lb_types.Label(
          data={"global_key" : global_key },
          annotations = flatten_list_annotations
      )
  )

  # 7.5 Upload annotations to the project using Label Import

  upload_job_annotation = lb.LabelImport.create_from_objects(
      client = client,
      project_id = project.uid,
      name="video_annotations_import_" + str(uuid.uuid4()),
      labels=labels)

  upload_job_annotation.wait_until_done()
  # Errors will appear for annotation uploads that failed.
  print("Errors:", upload_job_annotation.errors)
  print("Status of uploads: ", upload_job_annotation.statuses)

  # Errors will appear for annotation uploads that failed.
  print("Errors:", upload_job_annotation.errors)

  # 7.6 Send the annotations to the Model Run

  # get the labels id from the project
  model_run.upsert_labels(project_id=project.uid)
  ```
</CodeGroup>


# User
Source: https://docs.labelbox.com/horizon/developer-guides/user

Developer guide for managing team members using the Python SDK.

# Client

```python theme={"theme":"css-variables"}
import labelbox as lb
client = lb.Client(api_key="<YOUR_API_KEY>")
```

# Get the current user

```python theme={"theme":"css-variables"}
user = client.get_user()
```

# Methods

## Update an organization role

```python theme={"theme":"css-variables"}
# get the available roles
roles = client.get_roles()

# update the role (in this case, the user is granted labeler permissions)
user_to_update.update_org_role(roles["LABELER"])
```

## Upsert a project-based role

```python theme={"theme":"css-variables"}
# get the available roles
roles = client.get_roles()

# get the project
project = client.get_project("<project_id>")

# update the project-based role (in this case, the user is granted reviewer permissions)
user_to_update.upsert_project_role(
  project=project,
  role=roles["REVIEWER"]
)
```

## Remove from a project

```python theme={"theme":"css-variables"}
# get the project
project = client.get_project("<project_id>")

# remove from the project
user_to_remove.remove_from_project(project)
```

# Attributes

## Get the basics

```python theme={"theme":"css-variables"}
# name (str)
user.name

# nickname (str)
user.nickname

# email (str)
user.email

# created at (datetime)
user.created_at

# updated at (datetime)
user.updated_at

# role (relationship to OrgRole object)
role = user.org_role()

# organization (relationship to Organization object)
organization = user.organization()
```

## Get the projects

```python theme={"theme":"css-variables"}
# get the projects (relationship to Project objects)
projects = user.projects()

# inspect one project
next(projects)

# inspect all projects
for project in projects:
  print(project)

# for ease of use, you can convert the paginated collection to a list
list(projects)
```

## Get the created tasks

```python theme={"theme":"css-variables"}
# get the tasks (relationship to Task objects)
tasks = user.created_tasks()

# get the tasks in progress
user_task_in_progress = user.created_tasks(where=Task.status_as_enum == TaskStatus.In_Progress)

# inspect one project
next(tasks)

# inspect all projects
for task in tasks:
  print(task)

# for ease of use, you can convert the paginated collection to a list
list(tasks)
```

# Manage user groups

<Note>
  `users` replaced by `members`

  Starting with version 7.0.0 of the SDK, the `users` attribute is now deprecated. Please use `members` instead in order to specify a user account and a role.
</Note>

Managing user groups requires you to import additional classes:

```python theme={"theme":"css-variables"}
from labelbox.schema.user_group import UserGroup, UserGroupColor, UserGroupMember
```

## Create user groups

```python theme={"theme":"css-variables"}
# Retrieve roles
roles = client.get_roles()


# Define a user group with individual roles (members)
user_group = UserGroup(
    client=client,
    name="User Group with Explicit Roles",
    color=UserGroupColor.GREEN,
    members={
        UserGroupMember(user=user1, role=roles["LABELER"]),
        UserGroupMember(user=user2, role=roles["REVIEWER"]),
        UserGroupMember(user=user3, role=roles["LABELER"])
    },
    projects={project1},
    description="Basic test"
)

# Create the defined user group
created_group = user_group.create()
```

## Update a user group

```python theme={"theme":"css-variables"}
# Define the user group properties to be updated
user_group.name = "Updated User Group Name"
user_group.color = UserGroupColor.GREEN
user_group.description = "New group"


# Add new projects to the group
user_group.projects.add(project1)
## or
user_group.projects.update({project1, project2})

# Add new users to the group
user_group.members.add(UserGroupMember(user, roles["TEAM_MANAGER"]))
## or
ugm = {
    UserGroupMember(user1, roles["LABELER"]),
    UserGroupMember(user2, roles["TEAM_MANAGER"])
}
user_group.members.update(ugm)

# Push the changes to the group
user_group.update()
```

## Search for and get info on user groups

<CodeGroup>
  ```python From name theme={"theme":"css-variables"}
  group_name = "example_name"

  # Get all user groups in your workspace
  user_groups = UserGroup.get_user_groups(client)

  # Search for a user group by its name
  example_group = next(
      (group for group in user_groups if group.name == group_name), 
      None
  )

  if example_group:
      print(f"Found user group 'example_name' with ID: {example_group.id}")
  else:
      print("No user group named 'example_name' found.")

  # Get info of the user group found
  example_group.get()
  ```

  ```python From id theme={"theme":"css-variables"}
  # Search for a user group by its id
  group_id = "<group id>" # Example af4ff010-cb4c-11f0-a929-8300a55d380
  group = UserGroup(client, id=group_id).get()
  ```
</CodeGroup>


# Webhooks
Source: https://docs.labelbox.com/horizon/developer-guides/webhook

Developer guide on how to set up Labelbox webhooks through Python SDK

<CardGroup>
  <Card title="Open In Colab" icon="infinity" href="https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/project_configuration/project_setup.ipynb" />

  <Card title="GitHub" icon="github" href="https://github.com/Labelbox/labelbox-notebooks/blob/main/project_configuration/project_setup.ipynb" />
</CardGroup>

See [Webhooks](/horizon/guides/webhooks) guide page for details on how to use webhooks to receive notifications from activities in Labelbox.

## Example configurations to receive requests

<CodeGroup>
  ```python Python (Flask) theme={"theme":"css-variables"}
  from flask import Flask, request
  import json
  import hmac
  import hashlib
  import threading
  from werkzeug.serving import run_simple

  # This can be any secret that matches your webhook config (we will set later)
  secret = b"<replace with secret>"

  # Example for server-side code to receive webhook events
  app = Flask(__name__)


  @app.route("/webhook-endpoint", methods=["POST"])
  def print_webhook_info():
      payload = request.data
      computed_signature = hmac.new(
          secret, msg=payload, digestmod=hashlib.sha1
      ).hexdigest()
      if request.headers["X-Hub-Signature"] != "sha1=" + computed_signature:
          print(
              "Error: computed_signature does not match signature provided in the headers"
          )
          return "Error", 500

      print("=========== New Webhook Delivery ============")
      print("Delivery ID: %s" % request.headers["X-Labelbox-Id"])
      print("Event: %s" % request.headers["X-Labelbox-Event"])
      print(
          "Payload: %s"
          % json.dumps(json.loads(payload.decode("utf8")), indent=4)
      )
      return "Success"


  thread = threading.Thread(
      target=lambda: run_simple("0.0.0.0", 3001, app)
  )
  thread.start()
  ```

  ```python Python (AWS Lambda) theme={"theme":"css-variables"}
  import hmac
  import hashlib
  import re

  secret = "<replace with secret>"

  # Example of a AWS Lambda to receive webhook events
  def lambda_handler(event, context):
      webhook = re.sub(
          r"^sha1=",
          "",
          event["headers"]["x-hub-signature"],
      )

      digest = hmac.new(
          key=bytes(secret, "utf-8"),
          msg=event["body"].encode("utf-8"),
          digestmod=hashlib.sha1,
      )

      if webhook != digest.hexdigest():
          return {
              "statusCode": 400,
              "body": "Error: computed_signature does not match",
          }

      print("=========== New Webhook Delivery ============")
      print("Delivery ID: %s" % event["headers"]["x-labelbox-id"])
      print("Event: %s" % event["headers"]["x-labelbox-event"])
      print("=========== Payload =============")
      print(event["body"])
      return "Success"
  ```

  ```python Python (GC Function) theme={"theme":"css-variables"}
  import functions_framework
  import json
  import hmac
  import hashlib

  secret = b"<replace with secret>"

  # Example of a Google Cloud Function to receive webhook events
  @functions_framework.http
  def hello_http(request):
      payload = request.data
      computed_signature = hmac.new(
          secret, msg=payload, digestmod=hashlib.sha1
      ).hexdigest()
      if request.headers["X-Hub-Signature"] != "sha1=" + computed_signature:
          print(
              "Error: computed_signature does not match signature provided in the headers"
          )
          return "Error", 500

      print("=========== New Webhook Delivery ============")
      print("Delivery ID: %s" % request.headers["X-Labelbox-Id"])
      print("Event: %s" % request.headers["X-Labelbox-Event"])
      print(
          "Payload: %s"
          % json.dumps(json.loads(payload.decode("utf8")), indent=4)
      )
      return "Success"
  ```

  ```python Python (Azure Function) theme={"theme":"css-variables"}
  import azure.functions as func
  import json
  import hmac
  import logging
  import hashlib

  secret = b"<replace with secret>"

  # Example of an Azure Function to receive webhook events
  app = func.FunctionApp(http_auth_level=func.AuthLevel.FUNCTION)

  @app.route(route="http_trigger")
  def http_trigger(req: func.HttpRequest) -> func.HttpResponse:
      payload = req.get_body()
      computed_signature = hmac.new(
          secret, msg=payload, digestmod=hashlib.sha1
      ).hexdigest()
      if req.headers["X-Hub-Signature"] != "sha1=" + computed_signature:
          logging.warning(
              "Error: computed_signature does not match signature provided in the headers"
          )
          return func.HttpResponse("Error", status_code=500)

      logging.info("=========== New Webhook Delivery ============")
      logging.info("Delivery ID: %s" % req.headers["X-Labelbox-Id"])
      logging.info("Event: %s" % req.headers["X-Labelbox-Event"])
      logging.info(
          "Payload: %s"
          % json.dumps(json.loads(payload.decode("utf8")), indent=4)
      )
      return func.HttpResponse("Success", status_code=200)
  ```
</CodeGroup>

## Create webhook

### Webhook v2 setup

You can create a v2 webhook through the [Labelbox UI](/horizon/guides/webhooks#manage-webhooks-in-the-ui). The v2 webhook mimics the [export](/horizon/developer-guides/export-overview) payload.

### Webhook v1 setup

As shown below, you can create a v1 webhook with the Python SDK or the Labelbox UI, which is similar to v2 webhooks.

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  from labelbox import Client, Webhook

  public_url = "https://example.com/webhook-endpoint" # Your server's public url - where the messages will be sent to
  secret = b"CHANGE-ME" # Use to verify Labelbox is sending the message

  client = Client(api_key="<YOUR_API_KEY>")
  project = client.get_project("<project_id>")
  print([topic.value for topic in Webhook.Topic]) # See all available topics
  secret = b"example_secret" # This can be any secret that matches your webhook config (we will set later)

  webhook = Webhook.create(client,
  topics=["LABEL_CREATED"],
  url="public_url",
  secret=secret.decode(),
  project=project)

  ```
</CodeGroup>

## Get webhooks

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # Fetch all webhooks
  org = client.get_organization()
  webhooks = org.webhooks() #paginated

  # Fetch project webhooks
  project = client.get_project("<project_id>")
  webhooks = project.webhooks()  #paginated

  webhook = next(webhooks)
  status = webhook.status
  server_url = webhook.url
  topics = webhook.topics
  project = webhook.project()
  ```
</CodeGroup>

## Update webhook

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  # url, topics, and status can all be updated
  updated_url = f"{public_url}/webhook-endpoint"
  webhook.update(url=updated_url, topics=[Topic.LABEL_DELETED], status=Webhook.Status.INACTIVE.value)
  ```
</CodeGroup>

## Delete webhook

<CodeGroup>
  ```python Python theme={"theme":"css-variables"}
  webhook.delete()
  ```
</CodeGroup>

### Sample response

#### v2

<CodeGroup>
  ```json Label created expandable theme={"theme":"css-variables"}
  {
    "data_row": {
      "id": "clvmpaetz76ct0706730rfcq9",
      "external_id": null,
      "global_key": "TEST-ID-157128604429964085398241585654637789196d",
      "row_data": "https://storage.googleapis.com/labelbox-datasets/People_Clothing_Segmentation/jpeg_images/IMAGES/img_0001.jpeg",
      "metadata_fields": [],
      "details": {
        "dataset_id": "clvmpa9et010j0750rlgs8qdi",
        "created_at": "2024-04-30T18:07:10.82Z",
        "updated_at": "2024-04-30T18:07:14.199Z",
        "created_by": null
      }
    },
    "projects": {
      "clvo7ajpc014907zxagiw9hry": {
        "project_name": "test_lambda",
        "labels": [
          {
            "label_kind": "Default",
            "id": "clvo7b6h501ic07fdbg62c32w",
            "label_details": {
              "created_at": "2024-05-06T17:18:48Z",
              "updated_at": "2024-05-06T17:18:48Z",
              "created_by": "[email protected]"
            },
            "performance_details": {
              "seconds_to_create": 5759.536,
              "seconds_to_review": 0,
              "skipped": false,
              "benchmark_reference_label": null,
              "benchmark_score": null,
              "consensus_score": null
            },
            "annotations": {
              "classifications": [],
              "objects": [
                {
                  "bbox": {
                    "height": 359,
                    "left": 24,
                    "top": 85,
                    "width": 480
                  },
                  "color": "#1976d2",
                  "feature_id": "clvv876mw00023b6qr6mk2o4s",
                  "instance_uri": "https://api.labelbox.com/masks/feature/clvv876mw00023b6qr6mk2o4s",
                  "schema_id": "clvcnmppk0aq507z11haa4x66",
                  "title": "person",
                  "value": "person"
                }
              ],
              "relationships": []
            }
          }
        ],
        "project_details": {
          "ontology_id": "clvcnmpor0aq207z14jesh8w7",
          "batch_id": "11EF07D4F9E60FC08820734F207E70B1",
          "priority": 5,
          "consensus_expected_label_count": {
            "Int32": 1,
            "Valid": true
          }
        }
      }
    },
    "media_attributes": {
      "asset_type": "image",
      "content_length": 122898,
      "exif_rotation": 1,
      "height": 825,
      "mime_type": "image/jpeg",
      "sub_type": "jpeg",
      "super_type": "image",
      "width": 550
    }
  }
  ```

  ```json Label updated expandable theme={"theme":"css-variables"}
  {
    "data_row": {
      "id": "clvmpaetz76ct0706730rfcq9",
      "external_id": null,
      "global_key": "TEST-ID-157128604429964085398241585654637789196d",
      "row_data": "https://storage.googleapis.com/labelbox-datasets/People_Clothing_Segmentation/jpeg_images/IMAGES/img_0001.jpeg",
      "metadata_fields": [],
      "details": {
        "dataset_id": "clvmpa9et010j0750rlgs8qdi",
        "created_at": "2024-04-30T18:07:10.82Z",
        "updated_at": "2024-04-30T18:07:14.199Z",
        "created_by": null
      }
    },
    "projects": {
      "clvo7ajpc014907zxagiw9hry": {
        "project_name": "test_lambda",
        "labels": [
          {
            "label_kind": "Default",
            "id": "clvo7b6h501ic07fdbg62c32w",
            "label_details": {
              "created_at": "2024-05-06T17:18:48Z",
              "updated_at": "2024-05-06T17:18:48Z",
              "created_by": "[email protected]"
            },
            "performance_details": {
              "seconds_to_create": 5759.536,
              "seconds_to_review": 0,
              "skipped": false,
              "benchmark_reference_label": null,
              "benchmark_score": null,
              "consensus_score": null
            },
            "annotations": {
              "classifications": [],
              "objects": [
                {
                  "bbox": {
                    "height": 359,
                    "left": 24,
                    "top": 85,
                    "width": 480
                  },
                  "color": "#1976d2",
                  "feature_id": "clvv876mw00023b6qr6mk2o4s",
                  "instance_uri": "https://api.labelbox.com/masks/feature/clvv876mw00023b6qr6mk2o4s",
                  "schema_id": "clvcnmppk0aq507z11haa4x66",
                  "title": "person",
                  "value": "person"
                }
              ],
              "relationships": []
            }
          }
        ],
        "project_details": {
          "ontology_id": "clvcnmpor0aq207z14jesh8w7",
          "batch_id": "11EF07D4F9E60FC08820734F207E70B1",
          "priority": 5,
          "consensus_expected_label_count": {
            "Int32": 1,
            "Valid": true
          }
        }
      }
    },
    "media_attributes": {
      "asset_type": "image",
      "content_length": 122898,
      "exif_rotation": 1,
      "height": 825,
      "mime_type": "image/jpeg",
      "sub_type": "jpeg",
      "super_type": "image",
      "width": 550
    }
  }
  ```

  ```json Label deleted expandable theme={"theme":"css-variables"}
  {
    "data_row": {
      "id": "clvmpaetz76cp0706m3jw8ww4",
      "external_id": null,
      "global_key": "TEST-ID-157128695542350976802229818230180675596d",
      "row_data": "https://storage.googleapis.com/labelbox-datasets/People_Clothing_Segmentation/jpeg_images/IMAGES/img_0004.jpeg",
      "metadata_fields": [],
      "details": {
        "dataset_id": "clvmpa9et010j0750rlgs8qdi",
        "created_at": "2024-04-30T18:07:10.819Z",
        "updated_at": "2024-04-30T18:07:14.196Z",
        "created_by": null
      }
    },
    "projects": {
      "clvo7ajpc014907zxagiw9hry": {
        "project_name": "test_lambda",
        "labels": [
          {
            "label_kind": "Default",
            "id": "clvo7b02m01p807znd5bwfr2v",
            "label_details": {
              "created_at": "2024-05-01T19:19:31Z",
              "updated_at": "2024-05-06T17:22:06Z",
              "created_by": "[email protected]"
            },
            "performance_details": {
              "seconds_to_create": 3.75,
              "seconds_to_review": 0,
              "skipped": false,
              "benchmark_reference_label": null,
              "benchmark_score": null,
              "consensus_score": null
            },
            "annotations": {
              "classifications": [],
              "objects": [
                {
                  "bbox": {
                    "height": 540,
                    "left": 130,
                    "top": 82,
                    "width": 311
                  },
                  "color": "#1976d2",
                  "feature_id": "clvo7b6qr00023b6q4xe4nnff",
                  "instance_uri": "https://api.labelbox.com/masks/feature/clvo7b6qr00023b6q4xe4nnff",
                  "schema_id": "clvcnmppk0aq507z11haa4x66",
                  "title": "person",
                  "value": "person"
                }
              ],
              "relationships": []
            }
          }
        ],
        "project_details": {
          "ontology_id": "clvcnmpor0aq207z14jesh8w7",
          "batch_id": "11EF07D4F9E60FC08820734F207E70B1",
          "priority": 5,
          "consensus_expected_label_count": {
            "Int32": 1,
            "Valid": true
          }
        }
      }
    },
    "media_attributes": {
      "asset_type": "image",
      "content_length": 167155,
      "exif_rotation": 1,
      "height": 825,
      "mime_type": "image/jpeg",
      "sub_type": "jpeg",
      "super_type": "image",
      "width": 550
    }
  }
  ```

  ```json Workflow expandable theme={"theme":"css-variables"}
  {
    "organizationId": "cloordigw0091073kc6pohyab",
    "projectId": "clvo7ajpc014907zxagiw9hry",
    "action": "MOVE",
    "dataRowIds": ["clvmpaetz76ct0706730rfcq9"],
    "originTaskId": "b363420e-1ea9-09dd-914c-d105efc5785a",
    "originTaskName": "Initial labeling task",
    "destinationTaskId": "c7a8a45d-c37d-4dba-bde7-792581597b10",
    "destinationTaskName": "Initial review task",
    "actorId": "cloordihe0092073kcorncj7g",
    "timestamp": "2024-05-06T17:18:48.797Z"
  }
  ```
</CodeGroup>

#### v1

<CodeGroup>
  ```json Label created expandable theme={"theme":"css-variables"}
  {
    "agreement": null,
    "benchmarkAgreement": null,
    "createdAt": "2023-02-13T18:22:46Z",
    "dataRow": {
      "createdAt": "2023-01-25T17:54:20Z",
      "deletedAt": null,
      "externalId": "multi_thread_update_4cfdcb29-be33-4e07-b586-849829385210",
      "id": "cldbyu7c7n4jp072scuqt66ll",
      "labelCountInProject": 1,
      "rowData": "https://labelbox.s3-us-west-2.amazonaws.com/datasets/mapillary_traffic/images/F1xqHzDSi1qP3qSLBXVyJQ.jpg",
      "updatedAt": "2023-01-25T20:42:57Z"
    },
    "dataset": {
      "createdAt": "2023-01-25T17:53:58Z",
      "deleted": false,
      "description": "",
      "id": "cldbytq0w2uw8074hgl3hbxr2",
      "name": "Datarow Update Test - 5K",
      "updatedAt": "2023-01-25T17:54:21Z"
    },
    "deleted": false,
    "id": "cle357uex0g5307z996y28ic0",
    "label": "{\"objects\":[{\"featureId\":\"cle357vrp00013b6jsi6dfei0\",\"schemaId\":\"cldan3u0701sc07yi6xyw6up6\",\"color\":\"#ff0000\",\"title\":\"bounding_box\",\"value\":\"bounding_box\",\"bbox\":{\"top\":453,\"left\":951,\"height\":670,\"width\":952},\"instanceURI\":\"https://api.labelbox.com/masks/feature/cle357vrp00013b6jsi6dfei0\"}],\"classifications\":[],\"relationships\":[]}",
    "project": {
      "createdAt": "2023-02-13T18:20:52Z",
      "deleted": false,
      "description": "",
      "id": "cle355i370h86070hf1v7ata1",
      "name": "Webhooks",
      "updatedAt": "2023-02-13T18:22:17Z"
    },
    "secondsToLabel": 0,
    "skipped": false,
    "updatedAt": "2023-02-13T18:22:46Z",
    "user": { "email": "[email protected]", "id": "clbpgxfaehww6076tedo5ejwx" }
  }
  ```

  ```json Label updated expandable theme={"theme":"css-variables"}
  {
    "id": "cl4g555g81ltr07cka1a2b1wv",
    "createdAt": "2022-06-16T19:41:49Z",
    "updatedAt": "2022-06-16T19:58:25Z",
    "secondsToLabel": 6.002,
    "label": "{\"objects\":[{\"featureId\":\"cl4hg2phm00013b6j2thsko0u\",\"schemaId\":\"cl2i6hbju1o2y10a321w3cusv\",\"title\":\"grape\",\"value\":\"grape\",\"color\":\"#1CE6FF\",\"bbox\":{\"top\":298,\"left\":445,\"height\":520,\"width\":930},\"instanceURI\":\"https://api.labelbox.com/masks/feature/cl4hg2phm00013b6j2thsko0u\"}],\"classifications\":[],\"relationships\":[]}",
    "agreement": null,
    "benchmarkAgreement": null,
    "deleted": false,
    "skipped": false,
    "labelCountInProject": 2,
    "project": {
      "id": "cl2i6gssn1o2210a3gq549fq0",
      "createdAt": "2022-04-27T22:57:43Z",
      "updatedAt": "2022-06-15T22:19:36Z",
      "name": "grapes",
      "description": "",
      "deleted": false
    },
    "dataRow": {
      "id": "cl2hzfg8g18kw0ztwbtnq0b6q",
      "createdAt": "2022-04-27T19:40:43Z",
      "updatedAt": "2022-04-27T20:13:51Z",
      "deletedAt": null,
      "externalId": "CFR_1620.jpg",
      "rowData": "https://storage.labelbox.com/ckhmnux5zeutm0825bt0kj34y%2F5375dc13-6442-22d7-f523-e9d252e715ea-CFR_1620.jpg?Expires=1656619105&KeyName=labelbox-assets-key-3&Signature=iOg3tbKe4j7xWZHw1r5rQD4iJIk="
    },
    "dataset": {
      "id": "cl2hz9qay0unl10a3f4wmcsfi",
      "createdAt": "2022-04-27T19:36:16Z",
      "updatedAt": "2022-04-27T19:40:43Z",
      "name": "data",
      "description": "",
      "deleted": false
    },
    "user": {
      "id": "ckhmnux6hi0p907898jext199",
      "email": "[email protected]"
    }
  }
  ```

  ```json Label deleted expandable theme={"theme":"css-variables"}
  {
    "id": "cl2i6hegh1x660z9i9fzz01a0",
    "createdAt": "2022-04-27T22:58:35Z",
    "updatedAt": "2022-06-16T20:12:41Z",
    "secondsToLabel": 31.448,
    "label": "{\"objects\":[{\"featureId\":\"cl2i6hhgd00013f6ghfnrcn6h\",\"schemaId\":\"cl2i6hbju1o2y10a321w3cusv\",\"title\":\"grape\",\"value\":\"grape\",\"color\":\"#1CE6FF\",\"bbox\":{\"top\":701,\"left\":402,\"height\":316,\"width\":238},\"instanceURI\":\"https://api.labelbox.com/masks/feature/cl2i6hhgd00013f6ghfnrcn6h\"},{\"featureId\":\"cl2i6hp3h00033f6gaofl29lx\",\"schemaId\":\"cl2i6hbju1o3010a30ksn2yiy\",\"title\":\"grape mask\",\"value\":\"grape_mask\",\"color\":\"#FF34FF\",\"instanceURI\":\"https://api.labelbox.com/masks/feature/cl2i6hp3h00033f6gaofl29lx\"}],\"classifications\":[],\"relationships\":[]}",
    "agreement": null,
    "benchmarkAgreement": null,
    "deleted": true,
    "skipped": false,
    "labelCountInProject": 1,
    "project": {
      "id": "cl2i6gssn1o2210a3gq549fq0",
      "createdAt": "2022-04-27T22:57:43Z",
      "updatedAt": "2022-06-16T20:06:25Z",
      "name": "grapes",
      "description": "",
      "deleted": false
    },
    "dataRow": {
      "id": "cl2hzfg8g18kk0ztwg22ke4vz",
      "createdAt": "2022-04-27T19:40:43Z",
      "updatedAt": "2022-04-27T20:13:52Z",
      "deletedAt": null,
      "externalId": "SYH_2017-04-27_1324.jpg",
      "rowData": "https://storage.labelbox.com/ckhmnux5zeutm0825bt0kj34y%2F1d883928-046f-97d5-e3f7-e6c296967c7e-SYH_2017-04-27_1324.jpg?Expires=1656619961&KeyName=labelbox-assets-key-3&Signature=0ma5Yw5qxdJE6_x7ACR7aPEHePM="
    },
    "dataset": {
      "id": "cl2hz9qay0unl10a3f4wmcsfi",
      "createdAt": "2022-04-27T19:36:16Z",
      "updatedAt": "2022-04-27T19:40:43Z",
      "name": "data",
      "description": "",
      "deleted": false
    },
    "user": {
      "id": "ckhmnux6hi0p907898jext199",
      "email": "[email protected]"
    }
  }
  ```

  ```json Workflow expandable theme={"theme":"css-variables"}
  {
    "organizationId": "cl81imdtk0cie0yx4g39i0du8",
    "projectId": "clbxofp0h05d2074475g14f9v",
    "action": "APPROVE",
    "dataRowIds": ["cl9elx44f001i076k2pz84tw6"],
    "originTaskId": "92665e1f-a507-4308-bdfc-d6be3c629ec7",
    "originTaskName": "My custom review step",
    "destinationTaskId": null,
    "destinationTaskName": "Done",
    "actorId": "cl81imdtx0cif0yx4ak0ndq8a",
    "timestamp": "2023-01-20T12:06:01.421Z"
  }
  ```
</CodeGroup>


# What's new
Source: https://docs.labelbox.com/horizon/developer-guides/whats-new

See the latest version updates of the Labelbox SDK

<Update label="7.13.0 · 2026-10-12" description="2026-10-12">
  ## Added

  * Add performance report methods that return the numbers behind a project's Performance dashboard and the workspace Monitor: `Client.get_performance_metrics()`, `Client.get_workspace_performance_metric()`, `Client.get_workspace_labeler_performance()`, `Project.get_performance_metric()`, `Project.get_labeler_performance()`, `Project.get_reviewer_performance()`, and `Project.get_performance_report_download()`. See the [Performance API](/horizon/developer-guides/performance-api) guide
  * Add `allowed_ip_cidrs` to `Client.create_api_key()` and `ApiKey.create_api_key()` to restrict an API key to a list of IP addresses or CIDR ranges, and expose `ApiKey.allowed_ip_cidrs`

  ## Changed

  * `Client.is_feature_schema_archived()` now includes the HTTP status and response body in the error it raises when the archived status cannot be read
</Update>

<Update label="7.12.0 · 2026-08-25" description="2026-08-25">
  ## Fixed

  * Route embedding metadata operations (`Client.create_embedding()`, `Client.get_embeddings()`, `Client.get_embedding_by_id()`, `Embedding.delete()`) through GraphQL so they work with opaque `lbx_` API keys instead of raising `JSONDecodeError`
</Update>

<Update label="7.11.0 · 2026-07-28" description="2026-07-28">
  ## Added

  * Add `User.last_login_at` to expose GraphQL `lastLoginAt` (nullable datetime of last session in the organization)
</Update>

<Update label="7.10.0 · 2026-07-22" description="2026-07-22">
  ## Added

  * Add `Tool.Type.MARKER` for video marker tool ontologies — fetching an ontology containing a marker tool no longer raises `ValueError`

  ## Removed

  * Remove LLM prompt/response and response-creation project support: `Client.create_prompt_response_generation_project()`, `Client.create_response_creation_project()`, `MediaType.LLMPromptCreation`, `MediaType.LLMPromptResponseCreation`, `OntologyKind.ResponseCreation`, `EditorTaskType.ResponseCreation`, and `Project.is_prompt_response()`

  ## Fixed

  * Fix Foundry app lookups to raise `ResourceNotFoundError` for apps that do not exist, instead of a generic `LabelboxError`
</Update>

<Update label="7.9.0 · 2026-07-13" description="2026-07-13">
  ## Added

  * Optional `batch_ids` parameter to `Project.get_overview()` to return workflow state counts scoped to one or more batches
</Update>

<Update label="7.8.0 · 2026-06-15" description="2026-06-15">
  ## Added

  * `ModelRun.total_cost` and `ModelRun.total_data_rows` properties to retrieve inference cost and data row count for a model run
  * `ModelRun.refresh_cost_and_usage()` to re-fetch live cost and usage data
</Update>

<Update label="7.7.0 · 2026-04-29" description="2026-04-29">
  ## Added

  * Add `Project.bulk_assign_data_rows()` method for bulk assigning data rows to a user

  ## Fixed

  * Reorder `create_api_key` validation to check input params before making API calls
  * Fix mypy type error in `DataRowUpsertItem.build`
</Update>

<Update label="7.6.0 · 2026-03-18" description="2026-03-18">
  ## Added

  * Add CRUD support for `Issues`, `Comments` and `Issue Categories`
</Update>

<Update label="7.5.0 · 2026-01-30" description="2026-01-30">
  ## Added

  * Add support for text subclasses under global text subclasses
</Update>

<Update label="7.4.0 · 2026-01-22" description="2026-01-22">
  ## Added

  * Add support of groups to [invite\_user()](/horizon/developer-guides/organization#invite-users)
  * Add group classification type
  * Add model config response count support

  ## Fixed

  * Workflow Management allow multiple inputs
</Update>

<Update label="7.3.0 · 2025-10-27" description="2025-10-27">
  ## Added

  * Add support for Audio Temporal Annotations
</Update>

<Update label="7.2.0 · 2025-08-28" description="2025-08-28">
  ## Updated

  * DataRowMetadata: Remove enforcement of character limit
</Update>

<Update label="7.1.2 · 2025-08-14" description="2025-08-14">
  ## Fixed

  * Project Setup: Remove project setup complete restriction
  * API key creation: fix support of role with a space in the name
  * Workflow cloning: Fix validation for some nodes
</Update>

<Update label="7.1.1 · 2025-07-09" description="2025-07-09">
  ## Fixed

  * `reset_to_initial_nodes()` and `clone_workflow_from()` will not update initial node IDs anymore
</Update>

<Update label="7.1.0 · 2025-07-08" description="2025-07-08">
  ## Added

  * Add the ability to specify relationship constraints

  ## Fixed

  * Minor fixes to CRUD operations when specifying user groups
</Update>

<Update label="7.0.0 · 2025-06-17" description="2025-06-17">
  ## Added

  * Add support for Likert scale
  * Add creation of IAM integrations
  * Add workflow management
  * Add model app ID to model run creation and return it
  * Add support for project ontology change
  * Remove `mypy` from dependencies

  ## Updated

  * **BREAKING CHANGE** `UserGroup` no longer supports the `users` field - it's been replaced by the `members` field.
  * Update `UserGroup` to fix project membership

  ## Fixed

  * `ApiKey.created_by` now correctly displays the `user` for old API keys
</Update>

<Update label="6.10.0 · 2025-04-04" description="2025-04-04">
  ## Added

  * [API key management](/horizon/developer-guides/create-api-key#manage-api-keys-from-the-sdk) using the SDK
  * Methods for [getting](/horizon/developer-guides/organization#get-pending-invites) and [canceling](/horizon/developer-guides/organization#cancel-invite) invitations to workspaces or projects
  * The `organization.tasks()` method for getting organization tasks and the `TaskStatus` enum for filtering both [organization](/horizon/developer-guides/organization#get-tasks) and [user](/horizon/developer-guides/user#get-the-created-tasks) tasks
</Update>

<Update label="6.9.0 · 2025-03-17" description="2025-03-17">
  ## Added

  * [Task cancellation](/horizon/developer-guides/tasks#cancel-tasks) for supported task types
  * `AUDIO` as a [supported attachment type](/horizon/developer-guides/attachments#supported-attachment-types)
</Update>

<Update label="6.8.0 · 2025-02-20" description="2025-02-20">
  ## Added

  * An optional `source_ontology_name` field to `Relationship` class. You can now specify relationships using `source_ontology_name` instead of a concrete source annotation. This approach is especially useful for [PDF document annotations](/horizon/developer-guides/import-document-annotations#relationships-page-specific) because it allows you to reference ontology classes directly when you create relationships.
  * Validation ensures that either `source` or `source_ontology_name` is provided, but not both
  * Support for [searching by name](/horizon/developer-guides/slice#get-catalog-slice) for `get_catalog_slice`
  * `get_catalog_slices` to [get all slices](/horizon/developer-guides/slice#get-catalog-slice)
</Update>

<Update label="6.7.0 · 2025-02-06" description="2025-02-06">
  ## Added

  * (Beta) MAL support for PDF [relationships](/horizon/developer-guides/import-document-annotations#relationships-between-classifications-and-objects)
  * (Beta) `readonly` [setting](/horizon/developer-guides/import-document-annotations#read-only-relationships) for relationships
</Update>

<Update label="6.6.0 · 2025-01-14" description="2025-01-14">
  ## Added

  * Support for python 3.13
    * Note: minimum numpy version that supports python 3.13 is 2.1
  * Include predictions in slice export

  ## Fixed

  * GraphQL upload SSL configuration
</Update>

<Update label="6.5.0 · 2025-01-06" description="2025-01-06">
  ## Added

  * Support including `predictions` in catalog exports
</Update>

<Update label="6.4.0 · 2024-12-19" description="2024-12-19">
  * Add feature schema attributes to support the auto OCR tool in ontologies
  * `task_queue_id` is now optional and defaults to `None` for better type support when [moving data rows to a workflow task](/horizon/developer-guides/project#move-data-rows-to-a-workflow-task)
</Update>

<Update label="6.3.0 · 2024-12-12" description="2024-12-12">
  ## Added

  * Group member upload
  * Group member export

  <Info>
    - For security reasons, the `Invite` class doesn't contain actual invite id (uid) any more. This change affects all versions.
  </Info>
</Update>

<Update label="6.2.0 · 2024-11-26" description="2024-11-26">
  ## Added

  * Multimodal chat evaluation [ontology support](/horizon/developer-guides/multimodal-chat-evaluation-project#option-a-create_ontology) for two new tools:
    * Prompt rating: [Example export payload](/horizon/developer-guides/export-multimodal-chat-annotations#prompt-rating)
    * Fact-checking: [Example export payload](/horizon/developer-guides/export-multimodal-chat-annotations#fact-checking)
  * Added the following new import methods that replace the deprecated `BulkImportRequest` class:
    * `project.get_mal_prediction_imports()` to retrieve the list of MAL import jobs.
    * `project.get_label_imports()` to to retrieve the list of ground-truth import jobs.
    * `MALPredictionImport.delete()` to delete a MAL import.
  * Deprecation warning to `wait_until_done` methods
  * Deprecation warning to `get_data_row_ids`

  ## Fixed

  * Fixed relationship label bug
  * Fixed the `Import lb-client` exceptions for labelbox
  * Fixed `get_roles` return type to `Dict[str, Role]` (instead of `List[Role]`)
  * Adjusted deprecation message to a new standard
  * Improved support for `Client.enable_experimental`
</Update>

<Update label="6.1.0 · 2024-11-12" description="2024-11-12">
  ## Added

  * Added the [step reasoning ontology tool](/horizon/developer-guides/multimodal-chat-evaluation-project#option-a-create_ontology) .

  ## Fixed

  * Use valid `lbox.exception` in client IAMIntegration validation
  * Improve SDK method name detection for Datadog dashboard
  * Limit labeling front-end visibility and mentions of a custom editor
  * Remove all unused python imports
  * Remove bulk import requests
</Update>

<Update label="6.0.1 · 2024-10-30" description="2024-10-30">
  ## Fixed

  * Updated `labelbox.exceptions` to `lbox.exceptions` for IAM Integration check
</Update>

<Update label="6.0.0 · 2024-10-15" description="2024-10-15">
  ## Added

  * Added deprecation messages to all `export_v2` methods
  * Added logic to extract SDK methods to pass to Datadog dashboard
  * Added the `lbox.request_client` Graphql client as a separate component

  ## Updated

  * Updated the minimum requirement of `numpy` version from 1.24 to 1.25 and cleaned up `_TypedArray` validations
  * Refactored client `create_project` inputs to be named parameters instead of \*\*kwargs
  * Updated pydantic v2 deprecation warnings
  * Removed experimental from `UserGroup`
  * Improved labeling service error reporting
  * Switched to ruff linter

  ## Fixed

  * Labeling dashboard serialization
  * Handling of global key in `ModelEvaluationTemplate`

  ## Removed

  * Discontinued the support for Python 3.8
  * Removed various deprecated classes, methods and attributes, including:
    * `QueueMode` for project
    * LabelGenerator `assign_feature_schema_ids`
    * Label `assign_feature_schema_ids`
    * AttachmentType `attachment_type.TEXT`
    * DataRowMetadata bulk\_delete `deletes` parameter - support for string for data row ids or global keys, use `UniqueId` or `GlobalKey` class instead
    * Dataset `create_data_rows_sync`. Use create\_data\_rows instead
    * Slice `get_data_row_ids`. Use get\_data\_row\_identifiers instead
    * LabelingParameterOverrideInput remove instance of `DataRow` as a type of input
    * Project `_update_queue_mode` and `get_queue_mode`
    * `Project.setup` and `Project.setup_editor`. Use `project.connect_ontology` instead
  * Removed deprecated get stream - related classes from ExportTask
    * `JsonConverter`, `JsonConverterOutput`, `FileConverter`, `FileConverterOutput`
  * Removed label `Data` classes as Label data attribute. Use `GenericDataRowData` instead
    <Info>
      `MaskData` is NOT removed
    </Info>
  * Removed NDJsonConverter `deserialize`
  * Removed the COCO package and `BulkImportRequest` package, including:
    * `BulkImportRequest`
    * `BulkImportRequestState`
    * `project.upload_annotations`
    * `project.bulk_import_requests`
</Update>

<Update label="5.2.1 · 2024-10-09" description="2024-10-09">
  ## Fixed

  * Exporter encoding
</Update>

<Update label="5.2.0 · 2024-10-09" description="2024-10-09">
  ## Added

  * Support data row / batch for live mmc projects([#1856](https://github.com/Labelbox/labelbox-python/pull/1856))
</Update>

<Update label="5.1.0 · 2024-09-27" description="2024-09-27">
  ## Added

  * Support self-signed SSL certs([#1811](https://github.com/Labelbox/labelbox-python/pull/1811))
  * Rectangle units now correctly support percent inputs([#1848](https://github.com/Labelbox/labelbox-python/pull/1848))
</Update>

<Update label="5.0.0 · 2024-09-17" description="2024-09-17">
  ## Updated

  * Set `tasks_remaining_count`attribute of [labeling service dashboard](/horizon/developer-guides/labeling-workforce#get-labeling-service-details) to `None` if labeling has not started.
  * Improve error messaging when creating LLM project with invalid dataset id parameter.

  ## Removed

  * **BREAKING CHANGE**: Export v1 SDK methods are all removed. Use the latest [export methods](/horizon/developer-guides/export-overview) instead.
  * **BREAKING CHANGE**: Unused labelbox\_v1 serialization package is removed.

  ## Fixed

  * Fixed CUID dependencies that cause a crash if numpy is not installed.
</Update>

<Update label="4.0.0 · 2024-09-10" description="2024-09-10">
  ## Added

  * **BREAKING CHANGE** for Pydantic v1 users: Labelbox Python SDK now uses Pydantic v2.
  * Added automation test support for multiple SDK versions.

  ## Fixed

  * Fixed flaky tests.
</Update>

<Update label="3.78.1 · 2024-09-10" description="2024-09-10">
  ## Fixed

  * Fixed labeling dashboard query for tags.
</Update>

<Update label="3.78.0 · 2024-09-03" description="2024-09-03">
  ## Added

  * Added `project.get_labeling_service_dashboard()` and `client.get_labeling_service_dashboards()`, which can retrieve information of projects with labeling service requested. See [Get labeling service details](/horizon/developer-guides/labeling-workforce#get-labeling-service-details).
  * Added `client.get_task_by_id()`, which fetches tasks by their IDs. See [Get tasks](/horizon/developer-guides/tasks#get-tasks) .
  * Added support for [importing multimodal chat annotations](/horizon/developer-guides/import-multimodal-chat-annotations) .
</Update>

<Update label="3.77.1 · 2024-08-28" description="2024-08-28">
  ## Fixed

  * Restored `client.headers`.
</Update>

<Update label="3.77.0 · 2024-08-09" description="2024-08-09">
  ## Added

  * Added `labeling_service.request()`, which validates all project requirements and requests a labeling service. See [Labeling Workforce](/horizon/developer-guides/labeling-workforce).
  * Added the `is_benchmark_reference` flag that supports importing ground truth labels as benchmarks. See import annotations guides for example payloads.

  ## Updated

  * `get_labeling_service()` now automatically requests labeling service if none has been used on a project. See [Labeling Workforce](/horizon/developer-guides/labeling-workforce).

  ## Removed

  * **BREAKING CHANGE**: Removed `Project.labeling_frontend.disconnect()`. Now you can only have one default labeling front end per project and can't disconnect it.
  * **BREAKING CHANGE**: Removed the experimental method `project.request_labeling_service()`. See [Labeling Workforce](/horizon/developer-guides/labeling-workforce) for the latest approach.
</Update>

<Update label="3.76.0 · 2024-07-29" description="2024-07-29">
  ## Added

  * Added experimental methods `Project.get_labeling_service()`, `request_labeling_service()` and `get_labeling_service_status()` that support requesting and checking the status of labeling service. See [Labeling Workforce](/horizon/developer-guides/labeling-workforce).
  * Added `Client.create_prompt_response_generation_project()` and `create_response_creation_project()` methods that support project and ontology creation for prompt response projects. See [Import prompt and response annotations](/horizon/developer-guides/import-prompt-and-response-annotations) and [Prompt and response projects](/horizon/developer-guides/prompt-and-response-projects) to learn how to use these methods to set up prompt and response projects.
  * Added `is_benchmark_enabled` and `is_consensus_enabled` flags that specify whether a project supports [benchmarks](/horizon/guides/benchmark) and [consensus](/horizon/guides/consensus) .

  ## Updated

  * Made Project quality modes a list to allow combining more than 1 quality mode per project.

  ## Notebooks

  * Added back the [python notebook](https://colab.research.google.com/github/Labelbox/labelbox-python/blob/develop/examples/exports/export_v1_to_v2_migration_support.ipynb) on migrating from Export V1 to the latest `export()`.
  * Added correct data param to the video notebooks.

  ## Other

  * Use connection pool for all http and graphql requests
</Update>

<Update label="3.75.1 · 2024-07-16" description="2024-07-16">
  ## Removed

  * Project` media_type` no longer supports JSON
</Update>

<Update label="3.75.0 · 2024-07-10" description="2024-07-10">
  ## Added

  * Added `project.set_project_model_setup_complete()` method
  * Added [user group management](/horizon/developer-guides/user#manage-user-groups) methods.
  * Refactored `dataset.create_data_rows_sync` to `upsert`
  * Added `upload_type` to Project
  * Added prompt classification for python object
  * Alias `wait_xxx` functions

  ## Fixed

  * Predictions missing during Catalog slice Export
  * Prevented adding batches to live chat evaluation projects
  * Added missing media types
  * Deprecate `project.setup_editor` in favor of `project.connect_ontology`
  * Bumped dateutil max version
  * Bumped version rye
  * Updated create ontology for project setup
</Update>

<Update label="3.74.0 · 2024-06-24" description="2024-06-24">
  ## Added

  * Include predictions in export
  * Adds upsert label feedback method `client upsert_label_feedback()`

  ## Removed

  * Removed deprecated class `LabelList`
</Update>

<Update label="3.73.0 · 2024-06-20" description="2024-06-20">
  ## Added

  * Conversational data row checks
  * UI ontology mode support
  * Empty data row validation

  ## Fixed

  * Numpy semver locked to \< 2.0.0
</Update>

<Update label="3.72.2 · 2024-06-10" description="2024-06-10">
  ## Added

  * SLSA provenance generation
</Update>

<Update label="3.72.1 · 2024-06-06" description="2024-06-06">
  ## Fixed

  * Fix `client.get_project()` for LLM projects
  * Throw user-friendly errors when creating a custom embedding
</Update>

<Update label="3.72.0 · 2024-06-04" description="2024-06-04">
  ## Added

  * Update Dataset `create_data_rows` to allow upload of unlimited number of data rows
  * New Dataset methods for iam\_integration: `add_iam_integration`, `remove_iam_integration`

  ## Notebooks

  * Added model evaluation SDK method notebook
  * Added quick start notebook geared towards new users
</Update>

<Update label="3.71.0 · 2024-05-28" description="2024-05-28">
  ## Added

  * `project.get_overview()` to be able to retrieve project details
  * `project.clone()` to be able to clone projects
  * Support for Rye 0.34
  * `Requirements.lock`, `Requirements-dev.lock` to latest dependencies
  * `ExportTask.get_buffered_stream` to replace `ExportTask.get_stream`

  ## Fixed

  * `ExportTask.result` / `ExportTask.errors` parsing content incorrectly
  * Lack of exceptions related to updating model config
</Update>

<Update label="3.70.0 · 2024-05-20" description="2024-05-20">
  ## Added

  * Added [chat model evaluation](/horizon/developer-guides/multimodal-chat-evaluation-project) support:
    * `client.create_model_config()`
    * `project.project_model_configs()`
    * `project.add_model_config()`
    * `project.delete_project_model_config()`
    * `ProjectModelConfig.delete()`
    * `client.create_model_evaluation_project()`
  * Update existing methods to support chat model evaluation project
    * `client.create_ontology()`
    * `client.create_ontology_from_feature_schemas()`
  * Coco deprecation message

  ## Fixed

  * Fixed error reporting for client.create\_project()
  * Do not retry http 422 errors

  ## Notebooks

  * `Send_to_annotate_from_catalog` functionalities outside Foundry

  ## Fixed in Notebooks

  * Fixed meta notebook
  * Modified queue\_management.ipynb to remove some parameters
  * Update\_huggingface.ipynb
  * Corrected\_HF.ipynb
</Update>


# Workflow
Source: https://docs.labelbox.com/horizon/developer-guides/workflow

Developer guide for creating and modifying workflows via the Python SDK.

# Client

```python theme={"theme":"css-variables"}
import labelbox as lb
client = lb.Client(api_key="<YOUR_API_KEY>")
```

# Fundamentals

Workflows are connected to the `Project` class and are generated automatically during project creation. Workflows help organize and control the flow of labeling tasks through different stages.

Key concepts:

* Workflows are composed of **nodes** and **edges**.
* Each node can have only one input connection, except when both `Initial labeling task` and `Rework (All Rejected)` nodes serve as inputs to a single downstream node.
* No changes are pushed to the platform until you call `update_config()`.
* All nodes must be connected for the workflow to be valid.

## Access a workflow

```python theme={"theme":"css-variables"}
project.get_workflow()
```

## Clone a workflow from a different project

```python theme={"theme":"css-variables"}
project_source_id = "<project_source_id>"
project_target_id = "<project_target_id>"
project_source = client.get_project(project_source_id)
project_target = client.get_project(project_target_id)

project_target.clone_workflow_from(project_source.uid)
```

## Reset a workflow

Use `reset_to_initial_nodes()` to only have the InitialLabeling and InitialRework nodes and start from scratch. The method will return an object `InitialNodes` with two attributes `labeling` and `rework`. You can also pass parameters to configure the initial nodes.

```python theme={"theme":"css-variables"}
# Get initial nodes with basic status
initial_nodes = workflow.reset_to_initial_nodes()

initial_labeling = initial_nodes.labeling
initial_rework = initial_nodes.rework

# Configure initial nodes
# All parameters are optional 
from labelbox.schema.workflow.config import LabelingConfig, ReworkConfig
from labelbox.schema.workflow.enums import IndividualAssignment

initial_nodes = workflow.reset_to_initial_nodes(
    labeling_config=LabelingConfig(
        instructions="Instructions for labeling",
        max_contributions_per_user=10
    ),
    rework_config=ReworkConfig(
        instructions="Instructions for reworking",
        individual_assignment=[IndividualAssignment.LabelCreator],
        max_contributions_per_user=5
    )
)
```

## Commit changes

To push changes made to a workflow, you must use `update_config()`. In case of validation issue, it will throw a `ValueError` exception.

```python theme={"theme":"css-variables"}
# Commit changes without changing node locations
workflow.update_config()

# Commit changes and attempt to realign nodes
workflow.update_config(reposition=True)
```

# Nodes

## Add a node

Types of nodes are listed in the enum `NodeType`:

**Initial nodes:**

* `NodeType.InitialLabeling` - Entry point for new labeling tasks
* `NodeType.InitialRework` - Entry point for tasks that need to be reworked

**Step nodes:**

* `NodeType.Review` - Review completed labels
* `NodeType.Logic` - Apply filters to route tasks conditionally
* `NodeType.CustomRework` - Custom rework step with configurable settings

**Terminal nodes:**

* `NodeType.Done` - Marks tasks as completed
* `NodeType.Rework` - Sends tasks back to the rework queue
* `NodeType.CustomRework` can be used as a terminal node or be connected to another node.

`NodeType.CustomRework` can be used as a terminal node or be connected to another node.

```python theme={"theme":"css-variables"}
from labelbox.schema.workflow import NodeType

review_node = workflow.add_node(type=NodeType.Review)
```

## Delete a node

This automatically removes connected edges. Initial nodes can't be deleted.

```python theme={"theme":"css-variables"}
# Get nodes to delete
nodes_to_delete = [
    node
    for node in workflow.get_nodes()
    if node.name == "NodeToDelete"
]

workflow.delete_nodes(nodes_to_delete)
```

# Edges

## Add an edge

Edges connect the output of a source node to the input of a target node. All nodes must be connected in the workflow.

Types of outputs are listed in the enum `NodeOutput`:

* `NodeOutput.If` (default value, can be omitted)
* `NodeOutput.Else`
* `NodeOutput.Approved`
* `NodeOutput.Rejected`

### Outputs per node

| Node | Available outputs |
| - | - |
| `InitialLabeling` | `NodeOutput.If` |
| `InitialRework` | `NodeOutput.If` |
| `Review` | `NodeOutput.Approved`, `NodeOutput.Rejected` |
| `Logic` | `NodeOutput.If`, `NodeOutput.Else` |
| `CustomRework` | Optional `NodeOutput.If` |
| `Done` | |
| `Rework` | |

```python theme={"theme":"css-variables"}
from labelbox.schema.workflow import NodeOutput

# Connect nodes with appropriate outputs
workflow.add_edge(initial_labeling, initial_review) # Default NodeOutput.If
workflow.add_edge(initial_rework, initial_review)
workflow.add_edge(initial_review, logic, NodeOutput.Approved)
workflow.add_edge(initial_review, rework_node, NodeOutput.Rejected)
workflow.add_edge(logic, done, NodeOutput.If) # NodeOutput.If can be omitted
workflow.add_edge(logic, custom_rework_1, NodeOutput.Else)
```

## Node attributes

The following attributes can be configured for each node type:

| **Node** | **Configuratble attributes** |
| :- | :- |
| `InitialLabeling` | `instructions`, `max_contributions_per_user` |
| `InitialRework` | `instructions`, `individual_assignment`, `max_contributions_per_user` |
| `Review` | `instructions`, `group_assignment`, `max_contributions_per_user` |
| `Logic` | `name`, `match_filters`, `filters` |
| `CustomRework` | `name`, `instructions`, `group_assignment`, `individual_assignment`, `max_contributions_per_user` |
| `Done` | `name` |
| `Rework` | `name` |

**Common attributes:**

* `max_contributions_per_user`: Positive integer representing the maximum number of labels per task queue (empty for no limit)
* `instructions`: Custom instructions for labelers working on this node
* `group_assignment`: List of user group IDs assigned to this node
* `individual_assignment`: Individual assignment strategy (you can use the `IndividualAssignment` enum)

# Filters

## Logic node

The Logic node contains filters that determine how tasks flow through the workflow. The `match_filters` attribute controls how multiple filters are evaluated:

* `MatchFilters.Any`: Match any of the filters (OR logic)
* `MatchFilters.All`: Match all of the filters (AND logic)

## Add and remove filters

```python theme={"theme":"css-variables"}
from labelbox.schema.workflow.enums import WorkflowDefinitionId
from labelbox.schema.workflow.enums import FilterField

workflow = project.get_workflow()


logic = next(
    node for node in workflow.get_nodes()
    if node.definition_id == WorkflowDefinitionId.Logic
)
# logic = workflow.get_node_by_id("0359113a-6081-4f48-83d1-175062a0259b")

# Remove a filter based on its type 
logic.remove_filter(FilterField.ModelPrediction)

# Apply changes
#workflow.update_config()

# Add a filter
logic.add_filter(
    model_prediction([
        mp_condition.is_none()
    ])
)

# Apply changes
#workflow.update_config()
```

## Manage filters on Logic nodes

```python theme={"theme":"css-variables"}
from labelbox.schema.workflow.enums import WorkflowDefinitionId, FilterField
from labelbox.schema.workflow import mp_condition, model_prediction

workflow = project.get_workflow()

# Get the Logic node
logic = next(
    node for node in workflow.get_nodes()
    if node.definition_id == WorkflowDefinitionId.Logic
)
# Alternative: get by node ID
# logic = workflow.get_node_by_id("0359113a-6081-4f48-83d1-175062a0259b")

# Remove a filter based on its type 
logic.remove_filter(FilterField.ModelPrediction)

# Add a filter
logic.add_filter(
    model_prediction([
        mp_condition.is_none()
    ])
)

# Apply changes
workflow.update_config()
```

## Available filters

Each filter type can be used at most once per Logic node. The enum `FilterField` provides available filters for search or deletion:

* `FilterField.Annotation`
* `FilterField.Batch`
* `FilterField.ConsensusAverage`
* `FilterField.Dataset`
* `FilterField.FeatureConsensusAverage`
* `FilterField.IssueCategory`
* `FilterField.LabelingTime`
* `FilterField.LabeledAt`
* `FilterField.LabeledBy`
* `FilterField.Metadata`
* `FilterField.ModelPrediction`
* `FilterField.NlSearch`
* `FilterField.ReviewTime`
* `FilterField.Sample`

### annotation

Filter by the presence of specific annotations. `schema_node_ids` is a list of schema node IDs that correspond to tools or classifications defined in the project's ontology schema.

**Operators:** None (direct list filter)

```python theme={"theme":"css-variables"}
from labelbox.schema.workflow import annotation

# Using named parameter
annotation(schema_node_ids=["<schema_node_id>"])

# Using positional parameter
annotation(["<schema_node_id>"])
```

### batch

Filter by batch membership.

\*\*Operators: \*\*

* `is_one_of`
* `is_not_one_of`

```python theme={"theme":"css-variables"}
from labelbox.schema.workflow import batch

# Using named parameter
batch.is_one_of(values=["<batch_id>"])

# Using positional parameter
batch.is_one_of(["<batch_id>"])
```

### consensus\_average

Filter by overall consensus score.

**Operators:** None (range filter with min/max)

```python theme={"theme":"css-variables"}
from labelbox.schema.workflow import consensus_average

# Using named parameters
consensus_average(min=0.17, max=0.61)

# Using positional parameters
consensus_average(0.17, 0.61)
```

### dataset

Filter by dataset membership.

**Operators:** None (direct list filter)

```python theme={"theme":"css-variables"}
from labelbox.schema.workflow import dataset

# Using named parameter
dataset(dataset_ids=["<dataset_id>"])

# Using positional parameter
dataset(["<dataset_id>"])
```

### feature\_consensus\_average

Filter by consensus score for specific features. `annotations` is a list of schema node IDs that correspond to tools or classifications defined in the project's ontology schema.

**Operators:** None (range filter with min/max and annotation list)

```python theme={"theme":"css-variables"}
from labelbox.schema.workflow import feature_consensus_average

# Using named parameters
feature_consensus_average(min=0.17, max=0.67, annotations=["<schema_node_id>"])

# Using positional parameters
feature_consensus_average(0.17, 0.67, ["<schema_node_id>"])
```

### issue\_category

Filter by issue categories flagged during review.

**Operators:**

* `is_one_of`

```python theme={"theme":"css-variables"}
from labelbox.schema.workflow import issue_category

# Using named parameter
issue_category.is_one_of(category_ids=["<issue_category_id>"])

# Using positional parameter
issue_category.is_one_off(["<issue_category_id>"])
```

### labeling\_time

Filter by how long it took to create the label. The value represents a number of seconds.

**Operators:**

* `greater_than`
* `less_than`
* `greater_than_or_equal`
* `less_than_or_equal`
* `between`

```python theme={"theme":"css-variables"}
from labelbox.schema.workflow import labeling_time

# Using named parameter
labeling_time.greater_than(seconds=1000)

# Using positional parameter
labeling_time.greater_than(1000)
```

### labeled\_at

Filter by when the label was created.

**Operators:**

* `between`

```python theme={"theme":"css-variables"}
from labelbox.schema.workflow import labeled_at
from datetime import datetime

# Using named parameters
labeled_at.between(
    start=datetime(2024, 3, 9, 5, 5, 42), 
    end=datetime(2025, 4, 28, 13, 5, 42)
)

# Using positional parameters
labeled_at.between(
    datetime(2024, 3, 9, 5, 5, 42), 
    datetime(2025, 4, 28, 13, 5, 42)
)
```

### labeled\_by

Filter by the user who created the label.

**Operators:**

* `is_one_of`

```python theme={"theme":"css-variables"}
from labelbox.schema.workflow import labeled_by

# Using named parameter
labeled_by.is_one_of(user_ids=["<user_1_id>", "<user_2_id>"])

# Using positional parameter
labeled_by.is_one_of(["<user_1_id>", "<user_2_id>"])
```

### metadata

Filter by data row metadata values.

Each condition `m_condition`) takes a metadata schema ID `key`) and list of strings as parameters `value`).

**Operators:**

* `contains`
* `starts_with`
* `ends_with`
* `does_not_contain`
* `is_any`
* `is_not_any`

```python theme={"theme":"css-variables"}
from labelbox.schema.workflow import metadata, m_condition

# Using named parameters
metadata(conditions=[m_condition.contains(key="<metadata_schema_id>", value=["test"])])

# Using positional parameters
metadata([m_condition.contains("<metadata_schema_id>", ["test"])])
```

### model\_prediction

Filter by model predictions. Model predictions use a list of conditions named `mp_condition`.

Each condition takes a list of model IDs, followed by an integer for the minimum score `min_score`) and an integer for the optional maxim score `max_score`).

`is_none` takes precedence over other operators.

**Operators:**

* `is_one_of`
* `is_not_one_of`
* `is_none`

```python theme={"theme":"css-variables"}
from labelbox.schema.workflow import model_prediction, mp_condition

# Using named parameter
model_prediction(conditions=[
    mp_condition.is_one_of(models=["<model_id>"], min_score=1),
    mp_condition.is_not_one_of(models=["<model_id>"], min_score=2, max_score=6),
    mp_condition.is_none()
])

# Using positional parameter
model_prediction([
    mp_condition.is_one_of(["<model_id>"], 1),
    mp_condition.is_not_one_of(["<model_id>"], 2, 6),
    mp_condition.is_none()
])
```

### natural\_language

Filter using semantic search. The `content` (or prompt) follows this format:

`"Find this / more of this / not this / bias_value"`

where `bias_value` is a decimal between 0 and 1.

**Operators**: None (semantic search with score range)

```python theme={"theme":"css-variables"}
from labelbox.schema.workflow import natural_language

# Using named parameters
natural_language(
    content="Birds in the sky/Blue sky/clouds/0.5", 
    min_score=0.178, 
    max_score=0.768
)

# Using positional parameters
natural_language("Birds in the sky/Blue sky/clouds/0.5", 0.178, 0.768)
```

### review\_time

Filter by how long it took to review the label.

**Operators:**

* `greater_than`
* `less_than`
* `greater_than_or_equal`
* `less_than_or_equal`
* `between`

```python theme={"theme":"css-variables"}
from labelbox.schema.workflow import review_time

# Using named parameter
review_time.less_than_or_equal(seconds=100)

# Using positional parameter
review_time.less_than_or_equal(100)
```

### sample

Filter by percentage sampling. The percentage is entered as an integer.

**Operators:** None (percentage value)

```python theme={"theme":"css-variables"}
from labelbox.schema.workflow import sample

# Using named parameter
sample(percentage=23)

# Using positional parameter
sample(23)
```

# Example: Create a minimal workflow

The following example will create a basic workflow with three nodes:

* Initial labeling task
* Rework (all rejected)
* Done

```python theme={"theme":"css-variables"}
import labelbox as lb
from labelbox.schema.workflow import NodeType

# Initialize client and project
client = lb.Client(api_key="<YOUR_API_KEY>")
project_id = "<project_id>"
project = client.get_project(project_id)

# Get workflow
workflow = project.get_workflow()

# Create nodes
initial_nodes = workflow.reset_to_initial_nodes()
done = workflow.add_node(type=NodeType.Done)

# Connect nodes
workflow.add_edge(initial_nodes.labeling, done)
workflow.add_edge(initial_nodes.rework, done)


# Validate and commit changes
try:
    updated_workflow = workflow.update_config(reposition=True)
    print("Workflow updated successfully!")
except ValueError as e:
    print(f"Validation failed: {e}")
```

# Example: Showcase complete workflow

The following example shows all the nodes and filters.

```python theme={"theme":"css-variables"}
import labelbox as lb
from labelbox.schema.workflow import (
    NodeType, 
    NodeOutput, 
    ProjectWorkflowFilter,
    LabelingConfig,
    ReworkConfig,
    labeled_by,
    metadata,
    sample,
    labeled_at,
    mp_condition,
    m_condition,
    labeling_time,
    review_time,
    issue_category,
    batch,
    dataset,
    annotation,
    consensus_average,
    model_prediction,
    natural_language,
    feature_consensus_average
)
from labelbox.schema.workflow.enums import IndividualAssignment, MatchFilters
from datetime import datetime

# Initialize client and project
client = lb.Client(api_key="<YOUR_API_KEY>")
project_id = "<project_id>"
project = client.get_project(project_id)

# Get workflow
workflow = project.get_workflow()

# Reset config to start a fresh workflow
initial_nodes = workflow.reset_to_initial_nodes(
    labeling_config=LabelingConfig(
        instructions="This is the entry point",
        max_contributions_per_user=10
    ),
    rework_config=ReworkConfig(
        individual_assignment=[IndividualAssignment.LabelCreator]
    )
)
initial_labeling = initial_nodes.labeling
initial_rework = initial_nodes.rework

initial_review = workflow.add_node(
    type=NodeType.Review,
    name="Initial review task",
    group_assignment=["<user_group_id_1>", "<user_group_id_2>"]
)

logic = workflow.add_node(
    type=NodeType.Logic,
    name="Logic node",
    match_filters=MatchFilters.Any,
    filters=ProjectWorkflowFilter([
        labeled_by.is_one_of(["<user_id_1>", "<user_id_2>", "<user_id_3>"]),
        metadata([m_condition.contains("<metadata_schema_id>", ["test"])]),
        sample(23),
        labeled_at.between(
            datetime(2024, 3, 9, 5, 5, 42), 
            datetime(2025, 4, 28, 13, 5, 42)
        ),
        labeling_time.greater_than(1000),
        review_time.less_than_or_equal(100),
        issue_category.is_one_of(["<issue_category_id>"]),
        batch.is_one_of(["<batch_id>"]),
        dataset.is_one_of(["<dataset_id>"]),
        annotation.is_one_of(["<schema_node_id>"]),
        consensus_average(0.17, 0.61),
        model_prediction([
            mp_condition.is_one_of(["<model_id_1>"], 1),
            mp_condition.is_not_one_of(["<model_id_2>"], 2, 6),
            mp_condition.is_none()
        ]),
        natural_language("Birds in the sky/Blue sky/clouds/0.5", 0.178, 0.768),
        feature_consensus_average(0.17, 0.67, ["<schema_node_id>"])
    ])
)

# Terminal and step nodes
done = workflow.add_node(type=NodeType.Done)
rework = workflow.add_node(type=NodeType.Rework, name="To rework")

custom_rework_1 = workflow.add_node(
    type=NodeType.CustomRework,
    name="Custom Rework 1",
    individual_assignment=IndividualAssignment.LabelCreator,
    group_assignment=["<user_group_id_1>", "<user_group_id_2>"]
)

review_2 = workflow.add_node(
    type=NodeType.Review,
    name="Review 2"
)

custom_rework_2 = workflow.add_node(
    type=NodeType.CustomRework,
    name="Custom Rework 2",
    instructions="Additional rework instructions"
)

done_2 = workflow.add_node(
    type=NodeType.Done,
    name="Ready for final review"
)

# Create edges between nodes
workflow.add_edge(initial_labeling, initial_review)
workflow.add_edge(initial_rework, initial_review)
workflow.add_edge(initial_review, logic, NodeOutput.Approved)
workflow.add_edge(initial_review, rework, NodeOutput.Rejected)
workflow.add_edge(logic, review_2, NodeOutput.If)
workflow.add_edge(logic, custom_rework_1, NodeOutput.Else)
workflow.add_edge(review_2, done, NodeOutput.Approved)
workflow.add_edge(review_2, custom_rework_2, NodeOutput.Rejected)
workflow.add_edge(custom_rework_2, done_2)

# Commit the workflow
try:
    updated_workflow = workflow.update_config(reposition=True)
    print("Workflow updated successfully!")
except ValueError as e:
    print(f"Validation failed: {e}")
```


# Access, storage & security
Source: https://docs.labelbox.com/horizon/guides/access-storage

Information on account and data accessibility, storage, and security.

This guide provides information on your options for storing data when using Labelbox, as well as details on our security and privacy measures.

## Data storage configurations

You have three options for connecting your data to Labelbox. The method you choose determines where your assets are stored.

* **IAM delegated access (recommended)**: This is the recommended method for connecting your data to Labelbox. It allows you to host your data in your own cloud storage ([AWS S3](/horizon/guides/import-aws-s3-data), [Google Cloud Storage](/horizon/guides/using-google-cloud-storage), or [Azure Blob Storage](/horizon/guides/microsoft-azure-blob-storage)) and grant Labelbox access using native Identity and Access Management (IAM) roles.
* **Pre-signed URLs**: You can create a JSON file containing pre-signed or public URLs that point to files in your own cloud storage. You are responsible for [generating these signed URLs](/horizon/guides/signed-urls). Once you have the JSON file, you can upload it to Labelbox.
* **Direct file upload**: If you choose to upload your data directly, it will be stored in private buckets on Labelbox's Google Cloud Services. Labelbox will have access to your data to generate signed URLs for rendering assets in the browser. The standard expiration value for these signed URLs is one day.

<Tip>
  **A note on data location and performance**

  To ensure the best performance and fastest loading times for your labeling team, we recommend that you host your cloud data in a location that is geographically close to your team members.
</Tip>

Labelbox uses Google Cloud Services for cloud storage and all data is stored in the US. If your data is hosted on our servers, we use a CDN that provides geo-loading as close to the location as possible. Labelbox does not store data in the EU and will not do so on request.

## Security and compliance

We are committed to ensuring that your data is secure and that we meet our obligations under privacy and security laws.

**Data encryption**\
All data hosted by Labelbox, including labeled data, assets, and private user information, is encrypted at rest using AES-256. We use Google Cloud for storage, which means your data is also encrypted on the server-side using GCP's default encryption keys. Data is automatically decrypted when read by an authorized user.

**User authentication**\
To ensure that only authorized users can access your data, we use Auth0 for authentication. We also support [multi-factor authentication (MFA)](/horizon/guides/multifactor-authentication) and [Single Sign-On (SSO)](/horizon/guides/single-sign-on) for Enterprise accounts.

**Regulatory compliance**\
Labelbox has a comprehensive privacy program to meet our obligations under regulations such as [CCPA](https://en.wikipedia.org/wiki/California_Consumer_Privacy_Act), [GDPR](https://en.wikipedia.org/wiki/General_Data_Protection_Regulation), [SOC 2](https://en.wikipedia.org/wiki/System_and_Organization_Controls) Type II, and [HIPAA](https://en.wikipedia.org/wiki/Health_Insurance_Portability_and_Accountability_Act). For more details, please see our [Privacy FAQ](https://labelbox.com/company/privacy-faq).

## Data privacy and portability

**Data usage**\
Labelbox does not sell customer or end-user personal data. We share personal data with third-party service providers for business purposes only, but we do not share this information for monetary value or other valuable consideration. For more information, please see our [Privacy Notice](/page/privacy-notice).

**Data portability**\
Upon request, we can export your data and provide it to you. We can also permanently delete all of your data from our servers.

## More information

<CardGroup>
  <Card title="Labelbox: Privacy & Security FAQ" icon="shield-check" href="https://labelbox.com/company/privacy-faq" />
</CardGroup>


# AI critic
Source: https://docs.labelbox.com/horizon/guides/ai-critic



AI Critic helps you ensure the quality of your data by allowing you to define validation rules using natural language. The AI is contextually aware of all elements on the data row—including messages, features, and rubrics—enabling you to create highly specific critiques.

<Note>
  AI critic is supported for the Multi-modal chat editor, the Audio editor, and the Video editor.
</Note>

## How to create a new critic

1. Go to your **Project settings** → **Advanced** and enable **AI critic.** You can also choose which Foundry model to power the AI critic by setting the **AI critic model**.
2. In the editor, select the **AI critic setup** icon in the top navbar. Then, click **Add new critic**.
3. Add a short descriptive **title** for your critic. Then, in the text box, **describe your critic** using natural language. Define which elements you want to critique and the specific criteria they must meet. The more specific your description, the better the critic performs. For the best results, make sure your prompt references these element names:

| Element | Description |
| - | - |
| Actor | A participant in the conversation - either a "human" (the prompter) or a "model" (an AI responder). Each actor has metadata like a display name or model configuration name. |
| Checklist classification | A multi-select classification where labelers choose one or more options from a list. Like radios, options can contain nested sub-questions. |
| Display rules | Conditional visibility logic that controls when a classification appears. A criterion with display rules only shows when other criteria have specific values, enabling branching evaluation workflows. |
| Feature | The actual value a labeler selected or entered for a classification. Contains the answer data, including which option was chosen (for radio/checklist) or what text was entered. |
| Feature schema | The definition or blueprint for a classification. Describes its type (radio, checkbox, text), available options, scope, and display rules. Think of it as the question template. |
| Index data | Binds a scoped classification to a specific element in the conversation. For example, a response-message-scoped classification's index data contains the ID of the response it applies to. |
| Prompt message | A message sent by the human participant that initiates a conversation turn. In multi-turn conversations, each new prompt builds on the previous exchange. |
| Radio classification | A single-select classification where labelers choose exactly one option from a list. Options can have nested sub-questions that appear when selected. |
| Response message | A message generated by an AI model in reply to a prompt. In multi-model evaluation, a single prompt can have multiple responses from different models. |
| Rubric criterion | A structured evaluation question used to assess model responses. Can be a radio (single-select), checkbox (multi-select), or text (free-form) question. Each criterion is associated with a specific prompt message and evaluates the responses to that prompt. |
| Rubric group | A container that organizes related rubric criteria under a common header. Groups can have min/max constraints on how many criteria they contain. |
| Scope | Determines where a classification applies. "Global" applies to the entire conversation. Other scopes target a specific prompt message, response message, rubric criterion, or turn. |
| Text classification | A free-form text input annotation where labelers enter arbitrary text. |
| Turn | A complete prompt-response exchange in the conversation. Each turn contains one prompt message and one or more response messages from model actors. |

5. Set the **Enforcement**. This setting determines what happens when a critique fails. Select from the following options.
   * **Block**: Prevents the user from submitting the label until the failing critique is resolved. This is the strictest quality-control option.
   * **Acknowledge**: When the user clicks submit, a notification appears detailing the quality failures. The user must acknowledge the message to proceed, but they are not blocked from submitting.
   * **None**: Displays the critique result as a visual aid directly in the editor but does not interfere with the submission process.
6. Click **Update Critic** to save it. The AI will immediately begin critiquing elements based on your new rule.

## How to write effective critiques

You can create critiques for various elements within the editor. The AI understands the context and relationships between different elements, such as a classification and the message it is attached to.

To verify that a classification's value is correct based on its label and the message it's associated with, you can use a prompt like this:

`Look at the instructions or label for each classification. Make sure the value for the classification is correct based on its label and all of the contextual information. If it's a scoped classification, make sure to read the associated message in order to make your critique.`

The AI critic will then:

1. Target all classifications in the editor.
2. Read the classification's label.
3. If the classification is attached to a message, it will read the message content.
4. Determine if the selected value is appropriate based on the context.

### Example prompts

* *"All responses must be at least 3 sentences long and written in formal English. Responses that are too short or use casual language should fail."*
* *"Each response must directly address the question asked in the prompt message. If the response is off-topic or does not answer the question, it should fail."*
* *"For all checklist classifications, ensure that no contradictory options have been selected together."*
* *"For all radio classifications scoped to response messages, verify the selected option accurately reflects the tone of the response."*
* *"Review the entire conversation holistically. The overall exchange must feel natural and coherent. Flag data rows where the conversation feels disjointed or artificially constructed."*
* *Ensure that no personally identifiable information (PII) appears anywhere in the conversation — in prompts, responses, or classifications."*

### Important considerations

**Be specific**, vague prompts produce inconsistent results. **Use standard vocabulary** — terms like "turn," "rubric criterion," "global scope," "response message scope" help the AI target the right elements. **Experiment**  with your prompts — writing a good critic prompt is an art; test different phrasings and check results in the editor to find what works.


# Alignerr
Source: https://docs.labelbox.com/horizon/guides/alignerr





# Overview
Source: https://docs.labelbox.com/horizon/guides/annotate-overview

Collaboratively annotate data with the internal team, your own vendor, or Labelbox data labeling service.

Annotate is the data labeling platform within Labelbox. It allows your organization to label data with any human workforce at any scale.

## How Annotate works

When it comes to deciding how to label your data, you have the following options:

* Outsource this task to a labeling service — these external teams receive training on the specific labeling tasks required and quickly proceed to label large datasets (see [Workforce](/horizon/guides/labeling-services)).
* Import your model predictions as pre-labels to speed up the labeling process (see [Model-assisted labeling](/horizon/guides/model-assisted-labeling).
* Rely on your own internal team of labelers to label your dataset (see [Labeling editors](/horizon/guides/label-data)).

Regardless of your labeling method, Labelbox Annotate is a central place where you can manage all your labeling projects, customize your labeling & quality workflows, and monitor your labeling team's performance.

## Customizable labeling editor

The editor is the labeling interface purposefully designed to be highly configurable. The editor is the primary tool for creating, viewing, and editing annotations. The labeling editor supports the following media types out of the box:

<CardGroup>
  <Card title="Images" icon="image" href="/horizon/guides/image-editor" />

  <Card title="Video" icon="video" href="/horizon/guides/video-editor" />

  <Card title="Text" icon="file-lines" href="/horizon/guides/text-editor" />

  <Card title="Documents" icon="file-pdf" href="/horizon/guides/document-editor" />

  <Card title="Geospatial" icon="map" href="/horizon/guides/tiled-editor" />

  <Card title="Audio" icon="microphone" href="/horizon/guides/audio-editor" />

  <Card title="Conversational text" icon="comment" href="/horizon/guides/conversational-editor" />

  <Card title="HTML" icon="code" href="/horizon/guides/html-editor" />

  <Card title="LLM human preference" icon="user" href="/horizon/guides/llm-human-preference-editor" />

  <Card title="Prompt and response generation" icon="pen-to-square" href="/horizon/guides/prompt-and-response-generation-editor" />

  <Card title="Multimodal chat evaluation" icon="messages" href="/horizon/guides/multimodal-chat-evaluation-editor" />
</CardGroup>

## Model Assisted Labeling with Foundry

Users can use Foundry to add tools or classifications onto data rows as an auto-labeling feature. This is set up when a project has data to label and an editor with a set ontology. Once completed, users can enable the Model Assisted Labeling tool and configure an LLM to create annotations.

## Customizable labeling & review settings

You can set up customized review steps based on your decided quality strategy in your project's [Workflow](/horizon/guides/workflows) tab. As you work with large, complex projects, having to review all labeled data rows becomes increasingly time-consuming and expensive.

You can leverage workflows to create a highly-customizable, step-by-step review pipeline to drive efficiency and automation into your review process. To learn more, read [Workflows](/horizon/guides/workflows).

Additionally, you can leverage the following quality tools:

<CardGroup>
  <Card title="Issues & comments" icon="comment-dots" href="/horizon/guides/issues-comments" />

  <Card title="Benchmarks" icon="tag" href="/horizon/guides/benchmark" />

  <Card title="Consensus" icon="users" href="/horizon/guides/consensus" />

  <Card title="Performance dashboard" icon="chart-line" href="/horizon/guides/performance-dashboard" />
</CardGroup>

## Labeling workforce

Powered by Labelbox’s data engine, you can leverage Labelbox labeling services to collaborate with an external labeling workforce in real time and produce high-quality data while leveraging AI and automation techniques to keep human labeling costs at a minimum.

To learn more, see [Labeling services](/horizon/guides/labeling-services)

## Getting started with Annotate

<CardGroup>
  <Card title="Create a project" icon="square-1" href="/horizon/guides/create-a-project" />

  <Card title="Label some data" icon="square-2" href="/horizon/guides/label-data" />

  <Card title="Set up a review workflow" icon="square-3" href="/horizon/guides/workflows" />

  <Card title="Monitor your team performance" icon="square-4" href="/horizon/guides/performance-dashboard" />
</CardGroup>


# Send project updates
Source: https://docs.labelbox.com/horizon/guides/annotate-project-updates

Learn how to send mass communication to members in your organization or external orgs with project updates.

Project updates in Labelbox are a great way to keep everyone on your team informed. This feature allows you to communicate and collaborate directly within the Labelbox platform, which is especially helpful when working with an external workforce.

You can find the **Updates** section in the **Notifications** tab. This is where you can coordinate with all stakeholders, including your team, a labeling service admin, and the labeling workforce.

To keep things organized, Labelbox allows you to create different types of updates for common scenarios. This makes it easier to find important information later.

## How to create an update

Follow these steps to create a project update:

1. **Open a new update**: Select a project and go to **Notifications** → **Updates**. Click **+ New update**.
2. **Select the update type**: Select the type of update from the dropdown menu and add a message to provide more details. The text field for creating and editing updates supports markdown, allowing you to easily format your text. Here are the types of updates you can choose from:

| Update type | What it does |
| - | - |
| Progress update | Shares updates on the project's progress. |
| ETA updated | Communicates any changes to the project's expected completion date. |
| Instructions updated | Notifies the labeling team of any changes to the project instructions. |
| Issue | Informs the labeling team about any labeling issues you find in the project. |
| General | For any communication that doesn't fit into the other categories. |

3. **Add a message**: You may optionally add a message to make your update more descriptive. Markdown formatting is supported in your message description.
4. **Share with**: You can specify which organizations will receive this update.
   * **All**: This will make it available for all organizations added to the project (it will enable this update for external organizations added in the future)
   * **Internal only**: This will always make it private for your organization, meaning no other organizations (even those added later) will have access to this update.
   * **Specific organization**: You can select specific organizations that will have access to it (your organization is always included). This option is only available if your project is shared with external orgs.
5. **Notify with emails**: To support engagement, you may select to notify users of the update via email. Check the box to enable email notifications.
6. **Add attachments**: You may optionally include local files in your update. Click **Add** under **Attachments** to do so.
7. When you are done, click **Create update**.

For easy retrieval, you can filter your updates in the **Notifications** tab with two filters:

* **Author:** filters by the email address of the party creating the update
* **Type:** filters by the update type

Once a new update is created, you can reply to the same update. This will add it as an additional reply to the thread. You can reply to updates to close the thread or ask additional clarifying questions.

## Automatic email notifications

Once you are added as a project admin, you will automatically receive email notifications for every new update. These emails will come from [support@labelbox.com](mailto:support@labelbox.com), and the subject line will include the project name, making it easy to sort notifications from different projects into unique email threads.


# Audio
Source: https://docs.labelbox.com/horizon/guides/audio-editor

Guide for labeling audio data.

With the audio editor, you can add annotations to audio files, like classifying natural language conversations and music, to train conversational AI and audio-based ML models. The editor supports automatic speech-to-text recognition with the Whisper model, enabling you to transcribe any audio segment.

## Set up audio annotation projects

To set up an audio annotation project:

1. [Create an audio dataset](/horizon/guides/datasets-datarows).
2. On the [Annotate projects page](https://app.labelbox.com/projects), click the **+ New project** button.
3. Select **Audio**. Provide a **name** and an optional **description** for your project.
4. Click **Save**. The system then creates the project and redirects you to the project overview page.
5. Click **Add data**. Then select your audio dataset. Click **Sample** to sample your dataset, or you can manually select data rows and click **Queue batch**.

<Info>
  ### Data row size limit

  To view the maximum size allowed for a data row, see [limits](/horizon/guides/limits).
</Info>

## Set up ontologies

After setting up an audio annotation project, you can [add an ontology](/horizon/guides/labelbox-ontology#create-a-new-ontology) based on how you want to label the data. The audio editor supports the following annotation types that you can include in your ontology:

| Feature | Import annotations | Export annotations |
| - | - | - |
| **Radio classification** | [See payload](/horizon/developer-guides/import-audio-annotations#radio-single-choice) | [See payload](/horizon/developer-guides/export-audio-annotations#classification---radio) |
| **Checklist classification** | [See payload](/horizon/developer-guides/import-audio-annotations#checklist-multiple-choice) | [See payload](/horizon/developer-guides/export-audio-annotations#classification---checklist) |
| **Free-form text classification** | [See payload](/horizon/developer-guides/import-audio-annotations#free-form-text) | [See payload](/horizon/developer-guides/export-audio-annotations#classification---free-form-text) |

<Info>
  ### Classification scopes

  You can apply classifications as **global classifications** at the file level, **temporal classifications** at the frame level, or **nested classifications** under other annotations.
</Info>

## Use the audio editor

After adding data and setting up an ontology for your audio annotation project, you can add labels to data rows using the audio editor. Each data row displays in the editor with:

* **A waveform** visualizing pattern of sound pressure variation.
* **A spectrogram** showing the range of sound frequencies and their strengths over time.
* **A timeline** of audio split into 500 millisecond intervals by default and a **Timeline Resolution** slider that allows you to adjust the time intervals on the timeline.
* **Basic player controls**, such as the play/pause button, back/forward 10-second buttons, and the playback speed. You can also click anywhere on the waveform to instantly move to your desired location.

To add a global classification. select the classification and enter the value.

To add a temporal classification, select the classification, choose the interval on the timeline or waveform for when the classification starts, and add the classification value. You will see a circle representing the classification value on the timeline.

<Info>
  ### Timeline resolution differences for labels

  If you set a lower resolution with the timeline resolution slider, the classification label you add may not align exactly with the current timeline resolution. This indicates that the classification was placed at a timestamp with a higher resolution than the one currently being used. You can adjust the timeline resolution to a higher resolution to see the exact position of the classification.
</Info>

### Enable speech recognition

The in-editor automatic speech-to-text support allows you to recognize and extract text from audio segments using free-form text classifications. To enable it:

1. When [creating the ontology](/horizon/guides/labelbox-ontology#step-1-create), add a temporal **Text** classification feature.
2. Select the temporal text classification, and then select a starting frame on the timeline.
3. Click **START TRANSCRIBING**.
4. Click **END TRANSCRIBING** at your desired ending frame.

If the system detects speech, it automatically generates a transcript in the text annotation.

### Keyboard shortcuts

| Function | Hotkey | Description |
| - | - | - |
| Play/Pause | `Space` | Play or pause the audio playback |
| Move backward one frame | `←` | Move backward one frame |
| Move forward one frame | `→` | Move forward one frame |
| Select frames | `Shift` + `Mouse` | Select frames for adding temporal classifications |
| Advance to the previous keyframe | `⇧` + `←` | Advance to the previous keyframe |
| Advance to the next keyframe | `⇧` + `→` | Advance to the next keyframe |
| Jump to objects | `Down` | Jump to objects |
| Next object | `Down` | Move to the next object |
| Previous object | `Up` | Move to the previous object |
| Toggle | `⌘` + `/` | Toggle the keyboard shortcuts menu |


# Monitor activity with cloud audit logs
Source: https://docs.labelbox.com/horizon/guides/audit-logs

How to configure data access audit logs.

This guide demonstrates how to use your cloud provider's native logging tools to get full transparency into how and when Labelbox accesses your data via IAM Delegated Access. By monitoring these logs, you can maintain a complete, unchangeable record of every interaction, which is essential for security compliance and internal audits.

## Gain visibility through logs

When you grant Labelbox access using the IAM Delegated Access method, every action our platform takes is authenticated by your cloud provider. Every time Labelbox temporarily assumes the IAM role to access a file for labeling, your cloud provider records that event in an audit log.

These logs answer critical security questions:

* **Who** made the request? (The specific Labelbox IAM role)
* **What** resource was accessed? (The specific file/object in your bucket)
* **When** did the access occur? (A precise timestamp)
* **From where** was the request made? (The source IP address, which will be one of Labelbox's servers)

By analyzing these logs, you can have complete confidence that Labelbox is only accessing data as needed for your users to perform their labeling tasks.

## Step-by-step monitoring guides

One of the benefits of using IAM delegated access is that security-conscious teams can choose to access audit logs of when their raw data is read by Labelbox through their cloud provider.

<CardGroup>
  <Card title="AWS S3" icon="aws" href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-s3-access-logs-to-identify-requests.html" />

  <Card title="Google GCS" icon="google" href="https://cloud.google.com/logging/docs/audit/configure-data-access" />

  <Card title="Azure Blob Storage" icon="microsoft" href="https://docs.microsoft.com/en-us/azure/storage/blobs/monitor-blob-storage?tabs=azure-portal" />
</CardGroup>


# Send a batch to a labeling project
Source: https://docs.labelbox.com/horizon/guides/batches

Instructions for sending a batch of data rows from Catalog to a labeling project via the app UI.

In Labelbox, a "batch" is a group of data rows that you send from your Catalog to a project for labeling. Using batches gives you more control and flexibility over your labeling workflow.

## How to send a batch to a labeling project

Follow these steps to send a group of data rows to a labeling project:

1. Navigate to the **Catalog** and use the filters to select the data rows you want to include in your batch.
2. Once you have selected the data rows, click the **Manage selection** dropdown menu and select **Send to Annotate**.
3. In the **Configure batch** window, you will need to:
   1. Choose the project you want to send the batch to.
   2. Set a priority for the batch. A priority of 1 is the highest and 5 is the lowest. This will determine the order the data rows are placed in the labeling queue.
   3. Enable or disable **Consensus** for the batch. If you enable Consensus, you will need to set a **% Coverage** to indicate the percentage of data rows that will be queued for labeling by multiple labelers, and a **# Labels** value to indicate how many labels can be added to the data row.
   4. Click **Submit**. Any data rows that have already been submitted to the labeling project will be excluded from the batch.

<Info>
  **Limits**

  See [this page](/horizon/guides/limits) to learn the limits for sending batches to a project.
</Info>

## How to create a batch by sampling

Sampling is a useful technique for creating a batch when you are working with a large amount of data. Instead of manually selecting data rows, you can use sampling to create a batch of a specific size.

1. In the **Catalog**, use the filters to find the data rows you want to sample from.
2. Click the **Sample** button in the top right corner.
3. In the batch creation window, you can choose how many data rows to sample and the sampling method. Labelbox supports two sampling methods:
   * **Random**: This method will randomly select the specified number of data rows from your filtered results.
   * **Ordered**: This method will select the specified number of data rows based on the sorted order of the results. You can sort the results by the **Created At** timestamp in ascending or descending order.
     <Note>
       The sequence generated by ordered sampling does not influence the sequence of the data rows in the labeling queue.
     </Note>
4. Fill out the project, batch name, and priority details.
5. Click **Submit batch**.

## How to send predictions as pre-labels

To speed up the labeling process, you can include model predictions as pre-labels in your batch.

1. In the batch creation window, toggle the **Include predictions** option on.
2. From the dropdown menu, select the model run and the specific predictions you want to include in the batch.
3. Confirm that the model run is compatible with the labeling project. For the predictions to be compatible, the features in the model run ontology and the labeling project ontology must be the same. If the model run and the labeling project do not share any features, you will not be able to send the predictions as pre-labels.
4. Click **Submit batch**.

## Important considerations

* **Media type compatibility**: You can only send a batch to a project that has the same media type. For example, you cannot send a batch of text data to a project that is set up for labeling images.
* **Appending to batches**: Once a batch has been sent to a project, you cannot add more data rows to it.
* **Data rows in multiple batches**: A data row cannot be in more than one batch in the same project at the same time.
* **Sharing batches**: A batch cannot be shared between projects. However, you can create a new batch with the same data rows and send it to a different project.


# Benchmark
Source: https://docs.labelbox.com/horizon/guides/benchmark

Learn how to set up benchmarks to analyze the quality of your labels.

Benchmarks serve as the gold standard for other labels. You can designate specific data rows with annotations as benchmarks, and all other annotations on these data rows are automatically compared to these benchmark reference labels to calculate benchmark agreement scores.

<Info>
  ### Benchmarks in the queue

  Labelbox ensures that the first five data rows in your labeling queue are benchmark labels. After these initial five, the order of the remaining benchmarks is not guaranteed. Beyond the first five, there is a 10% chance that any subsequent data row you encounter will be a benchmark label. If you have fewer than five benchmarks, they are the first data rows in your queue, and no additional benchmarks appear unless you add more.
</Info>

## Supported types

Currently, benchmarking is supported for the following asset and annotation types:

| Asset type | Bounding box | Polygon | Polyline | Point | Segmentation mask | Entity | Radio | Checklist |
| - | - | - | - | - | - | - | - | - |
| Images | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | N/A | <Icon icon="check" /> | <Icon icon="check" /> |
| Videos | <Icon icon="check" /> | - | <Icon icon="check" /> | <Icon icon="check" /> | - | N/A | <Icon icon="check" /> | <Icon icon="check" /> |
| Audio | N/A | N/A | N/A | N/A | N/A | N/A | <Icon icon="check" /> | <Icon icon="check" /> |
| Text | N/A | N/A | N/A | N/A | N/A | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> |
| Tiled imagery | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | - | N/A | <Icon icon="check" /> | <Icon icon="check" /> |
| Documents | <Icon icon="check" /> | N/A | N/A | N/A | N/A | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> |
| HTML | N/A | N/A | N/A | N/A | N/A | N/A | <Icon icon="check" /> | <Icon icon="check" /> |
| Conversational text | N/A | N/A | N/A | N/A | N/A | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> |
| Human-generated responses | N/A | N/A | N/A | N/A | N/A | N/A | <Icon icon="check" /> | <Icon icon="check" /> |

## Set up benchmarks

To set an individual data row as a benchmark reference:

1. In the editor, label the data row and click **Submit**.
2. Navigate back to the project homepage.
3. Go to the **Data rows** tab.
4. Select the labeled data row from the list. This will open the Data row browser.
5. Click on the three dots next to the data row and select **Add as benchmark** from the dropdown.

To bulk-assign labeled data rows as benchmark references:

1. On the **Data rows** tab, select the data rows that you want to set as benchmarks.
2. Click the **selected** dropdown and select **Assign labels as benchmarks**.
3. If you have permissions to access benchmark scores, you can select from the following two options:
   * **Infer automatically**: Set the labels created within the benchmark quality mode as benchmark references. If the project also uses consensus as the quality mode, this option also sets consensus winners as benchmark references.
   * **Target label by score**: Select or set a range of benchmark scores to set labeled data rows matching the filter as benchmark references. If you don't have permissions to access benchmark scores or if no benchmark scores are available, you can only see and use the **Infer automatically** option.
4. Click **Submit**.

Once a label is designated as a benchmark, the data row is automatically moved to **Done**. Benchmarked data rows will be served to all labelers in a project. Benchmarked data rows can't be moved to any other step unless the benchmark is removed.

For a benchmark agreement to be calculated, one benchmark-reference label and at least one non-benchmark label need to be on the data row. Whenever annotations are added or updated, the benchmark agreement is recalculated as long as at least one non-benchmark label exists on the data row.

## Search and filter data using benchmark scores

The **Benchmark agreement** filter helps you find qualified data rows based on benchmark scores. You can apply this filter in the following locations:

* The [Data Rows](/horizon/guides/data-rows-activity) tab
* The [Workflow](/horizon/guides/workflows) tab
* The [Catalog](/horizon/guides/search) page

When using the filter, you can configure the following options:

* **Scope**: Specify the type of agreement to measure:
  * **Feature-level** measures the alignment between annotators' labels and the predefined benchmark reference labels for each data row. If you select this option, further specify one or more feature schemas in the ontology using the dropdown menu.
  * **Label-level** evaluates the overall agreement of all annotations within a single data row compared to the benchmark reference label.
* **Calculation**: Choose whether to calculate the agreement as an absolute or average score.
* **Range (0-1)**: Set the score range from 0 to 1, 0 indicates no agreement with the benchmark reference label and 1 represents complete agreement.


# A guide to Labelbox's LBUs
Source: https://docs.labelbox.com/horizon/guides/billing

Learn how to perform common billing tasks and how Labelbox charges works.

## What are Labelbox (LBUs)?

A Labelbox Unit (LBU) is a normalized unit of data in the Labelbox platform. Think of it as a credit that you consume as you use Labelbox's products.

Each LBU represents a specific amount of work done, which can involve one or more data rows (e.g., images, text files, videos). Since each of Labelbox's products—Catalog, Annotate, and Model—is designed for different tasks, the way you consume LBUs will vary depending on the product you're using, the type and amount of data you're working with, and the specific actions you take.

## How LBU consumption is calculated

The table below shows how many LBUs are consumed for different types of data across the three Labelbox products.

| Data row asset type | Catalog (monthly) | Annotate (one-time) | Model (one-time) |
| - | - | - | - |
| Image | 1 LBU per 60 data rows | 1 LBU per data row | 1 LBU per 5 data rows |
| Text | 1 LBU per 60 data rows | 1 LBU per data row | 1 LBU per 5 data rows |
| Chat (Conversation) & Offline Multi Modal | 1 LBU per 60 data rows | 1 LBU per data row | 1 LBU per 5 data rows |
| Audio | 1 LBU per 60 data rows | 1 LBU per data row | 1 LBU per 5 data rows |
| Document (PDF), per page | 1 LBU per 60 data rows + 1 LBU per 60 pages | 1 LBU per page | 1 LBU per 5 pages |
| Video | 1 LBU per 60 data rows + 1 LBU per 5,000 frames | 1 LBU per data row + 1 LBU per 150 frames | 1 LBU per 5 data rows + 1 LBU per 1,500 frames |
| Geospatial & Medical Tiled Imagery | 1 LBU per 6 data rows | 4 LBU per data row | 1 LBU per data row |
| Live Multi Modal/LLM | 1 LBU per 60 data rows | 20 LBU per data row | 1 LBU per 5 data rows |

Free accounts have 500 free LBU credits each month. If you reach this limit, you can access and export your data. However, you cannot add data rows, labels, or predictions until the next billing period. (You can [**upgrade your account**](https://docs.labelbox.com/horizon/guides/update-account-plan#upgrade-current-plan) at any time.)

## How LBUs are consumed by each product

Different actions within Labelbox consume LBUs. Here's how consumption works for each Labelbox product:

### Catalog consumption

In **Catalog**, you consume LBUs when you upload and store data, regardless of the data type.

To help you avoid accidental charges, there is a *seven-day grace period* that starts when you first upload data. If you remove any unused data rows before this grace period ends, you won't be charged any LBUs for them.

However, if you take any action that adds a label to a data row (like annotating it), that row will consume LBUs, even if it's within the grace period. Once the seven-day grace period is over, all data rows in your Catalog will start consuming LBUs.

Here’s an example:

* You upload 600 images to a dataset.
* For the first 7 days, no LBUs are consumed. You can delete any of these images without being charged.
* If you label 60 of those images during the grace period, you will consume 1 Catalog LBU for the current billing period, even if you delete them later.
* Once the grace period ends, the remaining 540 data rows will consume 9 LBUs for the current billing period.
* If you then decide to delete 300 rows after the grace period, you will still be charged a total of 10 Catalog LBUs for the current billing period. In the next billing period, your LBU consumption will drop to 5 LBUs to reflect the new data row count.

### Annotate consumption

In **Annotate**, you consume LBUs when a data row is first submitted to a project and labeled. This is a *one-time charge per data row*; submitting the same data row to additional projects does not consume additional LBUs.

For videos, LBUs are consumed for the entire video as a single unit, no matter how many frames you label. If you label just one frame, the entire video is charged once. If you go back and label more frames in the same video later, you won't be charged any additional LBUs.

<Warning>
  **Skipped data rows**

  For Annotate projects, "skipped" data rows are considered labeled and will contribute to LBU consumption.
</Warning>

### Model consumption

In **Model**, you consume LBUs when you do any of the following:

* Add data rows to a model run.
* Use Foundry to create and submit model predictions.
* Use data rows for model-assisted labeling (MAL), either through the app or the SDK.

These are *one-time charges* for each data row. Once a data row has been used in Model and consumed LBUs, you can use it again for other Model tasks without incurring additional LBU charges. However, new data rows will consume more Model LBUs.

Be aware that Foundry may incur additional inference costs based on the number of rows processed, the complexity of the task, the model used, and other factors. To learn more, see the **Add-on charges** section below.

## Examples of LBU consumption

Here are a couple of examples to illustrate how LBU consumption works in common workflows.

**Example 1: Data labeling with model assisted pre-labels**

Let's say you need to label 1,000 images, and you use Foundry to generate pre-labels for half of them (500 images). Here's how you would be charged:

* **Catalog**: 17 LBUs per month to store the 1,000 data rows.
* **Model**: 100 LBUs (one-time charge) for generating predictions on 500 of the data rows.
* **Annotate**: 1,000 LBUs (one-time charge) for the 1,000 labeled data rows.

If this activity occurred within a single billing period, you would be charged 1,117 LBU once and 1.67 LBU each month the data rows remain in Catalog.

**Example 2: Data curation and labeling with enrichment**

Now, imagine you have 60,000 images that all have Foundry predictions, and you end up labeling 10,000 of them. Here’s the breakdown:

* **Catalog**: 1,000 LBUs per month to store the 60,000 data rows (or 12,000 LBU annually).
* **Model**: 12,000 LBUs (one-time charge) for the 60,000 data rows with predictions.
* **Annotate**: 10,000 LBUs (one-time charge) for the 10,000 labeled data rows.

If this activity occurred within the same billing period, you would be charged 23,000 LBU once and 1,000 LBU each month the data remains in **Catalog**.

## Charges for add-on services

Your account may also be billed for add-on services. These charges are billed separately from your subscriptions and may require you to add a credit card to your account.

Available add-ons include:

* **Foundry**: This service uses foundational models to predict labels and enrich your data. When you use Foundry, you will incur *inference costs*, which are based on the amount of data processed, the model used, and the complexity of the task.
* **Labeling services**: You can connect with a team of professional labelers with specialized knowledge to annotate and review large amounts of data for you.
* **Single sign-on (SSO)**: This allows Enterprise teams to use their company's single sign-on service to sign in to Labelbox.
* **HIPAA compliance**: This helps Enterprise subscribers comply with the Health Insurance Portability and Accountability Act of 1996.

If you are an Enterprise customer, you should contact your technical account manager to verify the terms, payment details, and other information about your add-ons.

## Where can I find more information?

See our [Limits](/horizon/guides/limits) page to learn more about the platform limitations per tier.


# Bulk classification for zero-shot learning
Source: https://docs.labelbox.com/horizon/guides/bulk-classification



You can use bulk classification in Labelbox to perform zero-shot learning. After curating a subset of data that shares common traits, you can classify them all at once, directly from the Catalog.

## What is zero-shot learning?

Zero-shot learning (ZSL) is a machine learning technique that allows an AI model to recognize and categorize objects or concepts it has never seen before.

In a typical supervised learning model, an AI is trained on a large dataset of labeled examples. For instance, a model might be trained on thousands of images of cats and dogs, with each image labeled as either "cat" or "dog." The model then learns to distinguish between cats and dogs based on the patterns it observes in the training data.

Zero-shot learning, on the other hand, does not require any labeled examples of the categories it is being asked to classify.

Instead, it relies on auxiliary information, such as textual descriptions or attributes, to make predictions. For example, a zero-shot learning model could be told that a "zebra" is a "striped horse." Even if the model has never seen a zebra before, it can use its existing knowledge of horses and stripes to recognize a zebra in an image.

Labelbox's bulk classification using zero-shot learning is a great way to speed up your classification projects, as you can quickly generate classifications without having to label each asset one-by-one. This allows you to integrate off-the-shelf neural networks as zero-shot classifiers.

## How to bulk classify data

Here is a step-by-step guide on how to bulk classify data in Labelbox.

1. **Select a subset of data to classify:** Use the search features in Labelbox to select a subset of data. You can select the top results of a natural language search or all assets that look similar to each other using similarity search.
2. **Add a classification:** Click on the number of items selected in the top right corner and choose **Add classification**.
3. **Pick the destination labeling project:** Select the destination labeling project from the dropdown menu. Only projects with a global classification question in their ontology will appear.
4. **Provide classification values:** Answer the classification questions that appear. These values will apply to all data rows in the bulk classification job. You must answer all required classifications and subclassifications. You can search for a classification question by typing its name.
5. **Specify the workflow step:** Choose which step of the labeling and review workflow the data rows should be sent to. For example, if you pick the **Initial labeling task**, then the classifications will be sent as [*pre-labels*](/horizon/guides/model-assisted-labeling). If you pick any other task - such as **Rework**, **Initial review task**, or **Done** - then the classifications will be sent as [*labels*](/horizon/guides/import-ground-truth).
6. **Include or exclude data rows that already have labels:** If a data row already has a label in the destination project, you can either *overwrite* the existing label or *exclude* the data row from the job.
   * **Overwrite previous value with new value**: Only classification questions that you answer -- as part of the bulk classification job -- will overwrite already-existing classifications.
   * **Exclude from job**: Excludes these data rows from the bulk classification job to preserve the already-existing labels.
7. **Submit the bulk classification:** Click **Submit batch** to launch the job. It may take a few moments for the data to be sent to the labeling project.
8. **Track the progress:** You can track the progress of the job in the notification panel and will be notified by a pop-up message when it's complete.

## Limitations

There are a few limitations to keep in mind when using bulk classification:

* There are limits to the number of data rows you can classify at once. You can work around this by classifying data in chunks. For example, you can leverage the **Annotation** filter with the attributes **is not one of** and \[your classification question] to filter out already classified data rows and proceed with the next bulk classification job.
* You cannot use bulk classification with a consensus project.
* Benchmark data rows are excluded from bulk classification jobs. Since benchmark labels are a gold standard label, they should not be overwritten by bulk classifications.

The data rows are sent to the labeling project as a [batch](/horizon/guides/batches). You can control the name of the batch in the top-left corner, as well as the data row priority in the **Advanced settings**.


# Search and view compatibility
Source: https://docs.labelbox.com/horizon/guides/catalog-search-compatibility

A summary of search and view capabilities available in Catalog by data type.

Below are the supported search filters that Catalog supports out of the box. You can extend the usability of these core offerings by uploading your own [custom metadata](/horizon/guides/datarow-metadata#custom-fields) and [embeddings](/horizon/guides/similarity#supported-embeddings), then combining them with the supported filters below to accomplish your search and curation objectives.

## Supported search filters (basic)

To learn more about the supported filters, see [Filters](/horizon/guides/search).

| Asset type | Annotation | Dataset | Metadata | Project | Media attribute | Batch | Data row |
| - | - | - | - | - | - | - | - |
| **Image** | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> |
| **Video** | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> |
| **Text** | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | Only mime type | <Icon icon="check" /> | <Icon icon="check" /> |
| **HTML** | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | Only mime type | <Icon icon="check" /> | <Icon icon="check" /> |
| **Document** | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | Only mime type | <Icon icon="check" /> | <Icon icon="check" /> |
| **Tiled imagery** | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | Only mime type | <Icon icon="check" /> | <Icon icon="check" /> |
| **Audio** | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> |
| **Conversational** | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | Only mime type | <Icon icon="check" /> | <Icon icon="check" /> |

### Enrich your data with custom metadata

As noted above, Labelbox supports metadata on any data type. To learn how to use custom metadata and these Catalog filters to enrich your data, read this blog post on [how to make your data queryable using foundation models](https://labelbox.com/blog/make-videos-queryable-using-foundation-models/).

## Supported search filters (advanced)

| Asset type | [Find text](/horizon/guides/find-text) | [Natural language](/horizon/guides/natural-language-search) |
| - | - | - |
| **Image** | - | <Icon icon="check" />\* |
| **Video** | - | <Icon icon="check" /> (beta, add-on feature) |
| **Text** | <Icon icon="check" /> | <Icon icon="check" />\*\* |
| **HTML** | <Icon icon="check" /> | <Icon icon="check" />\*\* |
| **Document** | <Icon icon="check" /> | <Icon icon="check" />\*\*\* |
| **Tiled imagery** | - | <Icon icon="check" />\* |
| **Audio** | - | - |
| **Conversational** | <Icon icon="check" /> | <Icon icon="check" />\*\* |

\* Uses the off-the-shelf [CLIP-ViT-B-32](https://huggingface.co/sentence-transformers/clip-ViT-B-32) vision model (512 dimensions).

\*\* Uses the off-the-shelf [all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) text model (768 dimensions), based on the first 64k characters.

\*\*\* Users can pick between off-the-shelf [CLIP-ViT-B-32](https://huggingface.co/sentence-transformers/clip-ViT-B-32) vision model (512 dimensions) and [all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) text model (768 dimensions, based on the first 64k characters).

## Similarity (embeddings)

| Asset type | Off-the-shelf embeddings | Custom embeddings |
| - | - | - |
| **Image** | [CLIP-ViT-B-32](https://huggingface.co/sentence-transformers/clip-ViT-B-32) (512 dimensions) | Up to 2048 dimensions per embedding; up to 100 custom embeddings per workspace. |
| **Video** | [Google Gemini Pro Vision](https://ai.google.dev/gemini-api/docs/models/gemini) . First two (2) minutes of content is embedded. Audio signal is not used currently. This is a paid add-on feature available upon request. | Up to 2048 dimensions per embedding; up to 100 custom embeddings per workspace. |
| **Text** | [all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) (768 dimensions) | Up to 2048 dimensions per embedding; up to 100 custom embeddings per workspace. |
| **HTML** | [all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) (768 dimensions) | Up to 2048 dimensions per embedding; up to 100 custom embeddings per workspace. |
| **Document** | [CLIP-ViT-B-32](https://huggingface.co/sentence-transformers/clip-ViT-B-32) (512 dimensions) and[all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) (768 dimensions) | Up to 2048 dimensions per embedding; up to 100 custom embeddings per workspace. |
| **Tiled imagery** | [CLIP-ViT-B-32](https://huggingface.co/sentence-transformers/clip-ViT-B-32) (512 dimensions) | Up to 2048 dimensions per embedding; up to 100 custom embeddings per workspace. |
| **Audio** | Audio is transcribed to text.[all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) (768 dimensions) | Up to 2048 dimensions per embedding; up to 100 custom embeddings per workspace. |
| **Conversational** | [all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) (768 dimensions) | Up to 2048 dimensions per embedding; up to 100 custom embeddings per workspace. |

### Enhance similarity search with custom embeddings

As noted above, Labelbox supports custom embeddings on any data type. Powerful embeddings can be generated using foundational models and easily uploaded to Labelbox. You can then use these embeddings in combination with any of the above filters to accomplish your data search goals.

For an example of how to get started, check out this guide on [how to generate custom embeddings using foundational models and upload them to Labelbox](https://labelbox.com/guides/using-labelbox-and-hugging-face-to-generate-custom-embeddings-and-curate-impactful-data/).

## Options for viewing your data

| Asset type | Thumbnail view | Detail view | Annotations overlay (thumbnail) | Annotations overlay (detail) |
| - | - | - | - | - |
| **Image** | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> |
| **Video** | <Icon icon="check" /> | <Icon icon="check" /> | - | Classifications only |
| **Text** | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> |
| **HTML** | <Icon icon="check" /> | <Icon icon="check" /> | - | <Icon icon="check" /> |
| **Document** | <Icon icon="check" /> | <Icon icon="check" /> | Classifications only | Classifications only |
| **Tiled imagery** | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> |
| **Audio** | - | - | - | <Icon icon="check" /> |
| **Conversational** | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> |


# Import data
Source: https://docs.labelbox.com/horizon/guides/connect-to-cloud-storage

Learn how to import your data by connecting your cloud storage to Labelbox.

## Overview

To start labeling your data, you first need to grant our platform secure access to the files stored in your private cloud (AWS, GCP, or Azure). This guide explains the two methods for connecting your data, helping you choose the best one for your project's security and workflow needs.

The two methods are

* **IAM Delegated Access**: A robust, long-term connection method.
* **Signed URLs**: A flexible method using temporary, secure links to your data.

**Our Recommendation**: For most use cases, especially long-term projects, we recommend **IAM Delegated Access** for its superior security and lower maintenance.

| Feature | IAM Delegated Access | Signed URLs |
| - | - | - |
| Setup complexity | **High.** Requires a one-time configuration within your cloud provider's IAM console. | **Low.** No initial cloud configuration is needed in Labelbox. |
| Maintenance | **Low.** "Set it and forget it." Works for all data in the configured location. | **High.** Requires a continuously running service on your end to generate new URLs. |
| Data Freshness | **Real-time.** New data added to your bucket is immediately available for labeling. | **Delayed.** New data requires new signed URLs to be generated and uploaded to Labelbox. |
| Ideal for | Long-term projects, enterprise-scale data operations, and stringent security environments. | Quick-start projects, proof-of-concepts, or when you cannot create IAM roles. |

***

## IAM delegated access

IAM (Identity and Access Management) Delegated Access is the most secure and scalable method for connecting your data. You create a trust relationship by setting up a dedicated role within your own cloud account that Labelbox is permitted to assume. This gives Labelbox temporary, read-only credentials to access your data when your users are labeling.

### How it works

1. **You**: Create an IAM role in your AWS, GCP, or Azure account that has read-only permissions to your data bucket.
2. **You**: Provide Labelbox with the unique identifier (ARN/ID) of that role.
3. **Labelbox**: When a user needs to view an image or document, Labelbox uses the provided identifier to request temporary access credentials from your cloud provider.
4. **Your Cloud Provider**: Validates the request and grants Labelbox a short-lived token to access only the specified data.

### Key advantages

* **Superior Security**: Your secret keys are never shared with Labelbox. Access is easily auditable and can be revoked at any time from your cloud console.
* **Low Maintenance**: After the initial setup, you never have to worry about managing credentials or access again. It just works.
* **Simplified Workflow**: Data scientists and annotators can easily browse and import data without needing to handle URLs.

### Step-by-step guides

<Card title="Connect to AWS S3" icon="sparkles" href="/horizon/guides/import-aws-s3-data" />

<Card title="Connect to Google Cloud Storage" icon="sparkles" href="/horizon/guides/using-google-cloud-storage" />

<Card title="Connect to Azure Blob Storage" icon="sparkles" href="/horizon/guides/microsoft-azure-blob-storage" />

***

## Signed URLs

A signed URL is a web link that provides temporary access to a specific file in your storage bucket. Each URL is "signed" with cryptographic keys that validate the request and expire after a set time (e.g., 7 days). You are responsible for generating these URLs and providing them to Labelbox.

### How it works

1. **You**: Write and run a script or service that generates a unique signed URL for each data asset you want to label.
2. **You**: Create a JSON file containing these URLs and upload it to Labelbox.
3. **Labelbox**: When a user accesses a task, Labelbox uses the corresponding signed URL from your JSON file to fetch and display the data.
4. **Your Cloud Provider**: Validates the signature on the URL and serves the file. Access is denied if the URL has expired.

### Key advantages

* **Fast to Start**: Bypasses the need for complex IAM configuration, making it ideal for quick tests or proof-of-concepts.
* **Granular Control**: You have explicit, file-level control over what data is accessible and for how long.

### Step-by-step guide

<Card title="Generate signed URLs" icon="sparkles" href="/horizon/guides/signed-urls" />

***

## Automate imports with the Python SDK

Labelbox provides a [Python SDK](/horizon/developer-guides/install-python-sdk) to help automate data setup.

You can download sample code from the app or use the online docs to learn more.

To download samples from the app:

1. From the dataset default screen, select **Use Python SDK**.
2. From the **Create data rows** prompt, select the tab appropriate for your data type.
3. Use the **Copy** button to copy the code to the Clipboard or the **Download** button to save it locally.

Once you have a copy of the sample script, you can customize for your needs.

More information is available for each supported data type, including:

<CardGroup>
  <Card title="Image" icon="image" href="/horizon/developer-guides/import-image-data" />

  <Card title="Video" icon="video" href="/horizon/developer-guides/import-video-data" />

  <Card title="Text" icon="text" href="/horizon/developer-guides/import-text-data" />

  <Card title="Geospatial" icon="map-pin" href="/horizon/developer-guides/import-geospatial-data" />

  <Card title="Documents" icon="file-pdf" href="/horizon/developer-guides/import-document-data" />

  <Card title="Conversational text" icon="messages" href="/horizon/developer-guides/import-conversational-text-data" />

  <Card title="Audio" icon="microphone" href="/horizon/developer-guides/import-audio-data" />

  <Card title="HTML" icon="code" href="/horizon/developer-guides/import-html-data" />
</CardGroup>

***

## Upload local files

To upload local files directly to the Labelbox platform, go to **Catalog**, click **+New**, then select **Choose files to upload**.

<Warning>
  **Direct upload not recommended**

  Uploading your files to Labelbox is NOT recommended. We recommend using IAM delegated access or Signed URLs instead (see sections above).
</Warning>


# Consensus
Source: https://docs.labelbox.com/horizon/guides/consensus

Learn how to set up consensus scoring to analyze the quality of your labels.

Consensus represents the agreement between your labeling workforce. Consensus agreement scores are calculated in real-time for features and labels with multiple annotations by different labelers. Whenever an annotation is created, updated, or deleted, the consensus score is recalculated for data rows with two or more labels.

## Supported types

Currently, consensus scoring is supported for the following asset and annotation types:

| Asset type | Bounding box | Polygon | Polyline | Point | Segmentation mask | Entity | Relationship | Radio | Checklist | Free-form text |
| - | - | - | - | - | - | - | - | - | - | - |
| Image | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | N/A | N/A | <Icon icon="check" /> | <Icon icon="check" /> | - |
| Video | <Icon icon="check" /> | - | <Icon icon="check" /> | <Icon icon="check" /> | - | N/A | N/A | <Icon icon="check" /> | <Icon icon="check" /> | - |
| Text | N/A | N/A | N/A | N/A | N/A | <Icon icon="check" /> | - | <Icon icon="check" /> | <Icon icon="check" /> | - |
| Chat | N/A | N/A | N/A | N/A | N/A | <Icon icon="check" /> | - | <Icon icon="check" /> | <Icon icon="check" /> | - |
| Audio | N/A | N/A | N/A | N/A | N/A | N/A | N/A | <Icon icon="check" /> | <Icon icon="check" /> | - |
| Geospatial | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | N/A | <Icon icon="check" /> | N/A | <Icon icon="check" /> | <Icon icon="check" /> | - |
| Documents | <Icon icon="check" /> | N/A | N/A | N/A | N/A | N/A | N/A | <Icon icon="check" /> | N/A | - |
| HTML | N/A | N/A | N/A | N/A | N/A | N/A | <Icon icon="check" /> | <Icon icon="check" /> | N/A | - |
| Human-generated responses | N/A | N/A | N/A | N/A | N/A | N/A | <Icon icon="check" /> | <Icon icon="check" /> | N/A | <Icon icon="check" /> |

## Set up consensus scoring

When adding data rows to an **Annotate** project, use the **Queue batch** option to enable consensus scoring and configure additional settings. You can't change these settings after submission.

| Consensus setting | Description |
| - | - |
| **Data row priority** | The position in the labeling queue these data rows will be slotted based on priority. |
| **% coverage** | The percentage of the data rows in the batch that will enter the labeling queue as consensus data rows for multi-labeling. Defaults to 0. |
| **# labels** | The number of labels that each consensus data row will be added. Defaults to 2. Must be less than or equal to the number of labelers on the project. |

<Info>
  ### Consensus calculation can take up to five minutes
</Info>

## Select consensus winners

After a data row is labeled and enters the review stage, the first set of annotations entered for a data row represents consensus by default. Reviewers can reassign consensus to another set of annotations once the data row has more than one label.

If your data row has been labeled more than once, you'll view all of the label entries on that data row in the data row browser. The following example shows a data row with two sets of labels. The green trophy icon indicates that the first set of annotations is considered "consensus."

To change consensus, click the trophy icon next to the preferred annotations.

<Warning>
  ### Recalculation of consensus agreement scores

  The consensus score reflects agreement among labelers, so changing the winning label might lead to a recalculation of the score based on the new consensus.
</Warning>

<Info>
  ### Set consensus winners as benchmark references

  You can designate consensus winners as benchmarks. See [Set up benchmarks](/horizon/guides/benchmark#set-up-benchmarks).
</Info>

## Search and filter data using consensus scores

The **Consensus agreement** filter helps you find qualified data rows based on consensus scores. You can apply this filter in the following locations:

* The [Data Rows](/horizon/guides/data-rows-activity) tab
* The [Workflow](/horizon/guides/workflows) tab
* The [Catalog](/horizon/guides/search) page

When using the filter, you can configure the following options:

* **Scope**: Specify the type of agreement to measure:
  * **Feature-level** measures the agreement on a specific feature schema in the ontology for each data row. If you select this option, further specify one or more feature schemas in the ontology using the dropdown menu.
  * **Label-level** evaluates the overall agreement across all annotations within a single data row.
* **Calculation**: Choose whether to calculate the agreement as an absolute or average score.
* **Range (0-1)**: Set the score range from 0 to 1, where 0 indicates no agreement among annotators and 1 indicates complete agreement.


# Customer support
Source: https://docs.labelbox.com/horizon/guides/contacting-customer-support

How to work with Labelbox support to report issues and feedback.

The Labelbox Support Team is available to help you troubleshoot issues and get the most out of the platform.

## Support availability

Our support team is available from 9:00 AM to 8:00 PM Eastern Time, Monday through Friday, excluding major U.S. holidays.

## Status page

Before submitting a support ticket, please check our [Status page](https://status.labelbox.com/ "https://status.labelbox.com/") to see if there are any ongoing incidents or scheduled maintenance that might be affecting the platform. This can often provide immediate answers and updates regarding widespread issues.

## Labelbox support bot

Our AI-powered support bot in the Labelbox platform is your first stop for quick questions and information retrieval. We recommend using the bot to scan our documentation, look up SDK functions, or learn about different data export methods.

To get the most accurate answers, it's best to ask clear and specific questions. For example, instead of asking a general question like "How do I script?", try asking something more precise, such as "What is the SDK function to create a new project?" Well-formulated questions will help the bot give you the most insightful response.

Additionally, you can use the **Ask a question** chat bot from anywhere in the *Labelbox docs*. This AI-powered chat bot allows you quickly retrieve specific content from our docs (e.g., SDK functions, export methods, etc).

## How to file a support ticket

To file a support ticket, you'll need to use the Labelbox support portal.

<Note>
  Access to the portal is by invitation only. If you don't have an account, please contact your sales representative.
</Note>

To help our support team resolve your issue as quickly as possible, please include the following information in your ticket:

| Field | Description |
| - | - |
| On behalf of | The email address entered in this field should belong to the primary stakeholder of the concern being reported. The ticket will be associated with the Jira portal identity for this user and the email address should be associated with a Labelbox account. Other users can be added as `Request participants` after the creation of the ticket in order to view and participate in the thread. |
| Affected email address | The email address(es) entered in this field should represent the Labelbox account(s) in which the relevant issue is occurring. Typically, the entry here will be the same as the above field, but this is not always the case when reporting a team-related concern. |
| Issue urgency | Select the urgency level that best describes your issue. <ul><li>**Critical**: *Labelbox will not load or I cannot log in.* (Example: Labelbox will not load for any team members; team members are unable to log in).</li><li>**High**: *I'm mostly blocked from using Labelbox.* (Example: I cannot access a specific feature or functionality of Labelbox).</li><li>**Medium**: *I'm finding it difficult to use Labelbox.* (Example: The pen tool is not working properly for a large number of team members).</li><li>**Low**: *I'm running into minor issues using Labelbox.* (Example: The usage report on the Account page is not working properly).</li><li>**No Urgency**: Utilize this option for general questions or any concern that is not a bug.</li></ul> |

Additionally, you will need to include the following information in your ticket:

### Workspace ID

Click your initials in the left panel, select **Workspace settings**, and the copy the **Workspace ID**.

### Project ID

Open a project and copy the ID from the URL. The Project ID will be the string of characters after `projects/` and before the next `/`.

### Dataset ID

Go to **Catalog**, select your dataset, and copy the ID from the URL. The Dataset ID will be the string of characters after `catalog/dataset/`.

### Data Row ID and Label ID

Go to **Catalog**, click on a data row to open the detailed view, and find the Data row ID on the right side of the panel.

You can also find the data row ID and label ID by selecting a project and navigating the **Data Rows** tab. Click on any data row to open the **Data browser**.

* In the URL, the data row ID will be located immediately after `data-rows/` and span until the next `/` in the URL ( *in red*), followed by the label ID (*in green*).
* The label ID will appear in the left panel (*in green*). Data rows that have not been labeled will not have a label ID.

Asset ID and Data row ID are used synonymously and refer to the Labelbox-generated IDs assigned to each data row in an organization.

A Label ID is only generated once annotations have been submitted on a data row. Keep in mind, when utilizing Benchmarks or Consensus, there may be more than one Label ID associated with a single data row.

## How to view your support tickets

You can view your support tickets in two ways:

1. **From your email:** After you create a ticket, the email address entered in the **on behalf of** field will receive an email from Labelbox's Customer Help Portal. The subject of the email will follow the format of `LS-#### <Summary>`, where `Summary` is the summary you entered when filing the ticket. When you receive the email, click on **View request** to open the ticket in the Customer Help Portal. If someone else raised the ticket on your behalf and you have never used the portal, you may need to complete a brief sign-up procedure.
2. **From the support portal:** Log in to the customer support portal and click **Requests** in the top right corner to see all tickets created by you and your team. Once here, you can filter your tickets, monitor their status, and select a ticket directly from the portal.

## Best practices for getting a fast resolution

Here are some tips to help us resolve your tickets as quickly as possible:

* **Include attachments:** Screenshots, screen recordings, JSON files, Python snippets, or any other resources that provide context are always helpful.
* **Create separate tickets for separate issues:** If you have multiple, unrelated questions and concerns, please open separate tickets. When possible, please refrain from raising a new issue on an existing ticket. Keeping issues separate helps us reach resolutions faster as a team and allows us to better monitor trends so that we can continue to improve our Support overall.
* **Comment to re-open a ticket:** If a ticket has been closed by the Support team and the same issue emerges -- or perhaps you have more feedback to add -- you can always comment on the ticket and it will automatically re-open.
* **Use the customer support portal:** Communication on tickets will be clearer, more efficient, and easier to digest on the Jira portal compared to over email. In addition, you can use the portal to view all tickets raised by your team, which can illuminate trends and enable the consolidation of your efforts.


# Conversational text
Source: https://docs.labelbox.com/horizon/guides/conversational-editor

Guide for conversation or thread-based text data.

When creating a project, select **Conversations**.

<br />

<br />

## Import conversational text

To learn how to import conversational text data, visit our documentation on [importing conversational text](/horizon/developer-guides/import-conversational-text-data).

<Info>
  ### Data row size limit

  To view the maximum size allowed for a data row, visit our [limits](/horizon/guides/limits) page.
</Info>

## Supported annotation types

Below are all of the annotation types you may include in your ontology when you are labeling conversational or thread data. Classification-type annotations can be applied globally and/or nested within an object-type annotation.

| Feature | Import annotations | Export annotations |
| - | - | - |
| **Entity** | [See payload](/horizon/developer-guides/import-conversational-text-annotations#entity-message-based) | [See payload](/horizon/developer-guides/export-conversational-text-annotations#text-entity-named-entity) |
| **Relationships** | [See payload](/horizon/developer-guides/import-conversational-text-annotations#relationship-with-entity-message-based) | [See payload](/horizon/developer-guides/export-conversational-text-annotations#relationship) |
| **Radio classification** (Global or message-based) | [See payload](/horizon/developer-guides/import-conversational-text-annotations#classification-radio-single-choice-message-based) | [See payload](/horizon/developer-guides/export-conversational-text-annotations#classification---radio) |
| **Checklist classification** (Global) | [See payload](/horizon/developer-guides/import-conversational-text-annotations#classification-checklist-multi-choice-message-based) | [See payload](/horizon/developer-guides/export-conversational-text-annotations#classification---checklist) |
| **Free-form text classification** (Global) | [See payload](/horizon/developer-guides/import-conversational-text-annotations#classification-free-form-text-message-based) | [See payload](/horizon/developer-guides/export-conversational-text-annotations#classification---free-form-text) |

### Entity

To create an entity, simply choose the entity tool in your ontology and select the text string by clicking the desired starting character and dragging to select a sequence of characters in the unstructured text.

Please note that in both the conversational and thread-based UI, the annotator will only be able to label messages that have been marked as `canLabel` in the import file. The messages that can be annotated will have a white background while those that cannot be annotated will have a grey background.

### Message-based radio classifications

One unique feature of our conversation editor is the ability to label specific messages in a conversation with a radio classification value. This enables you to annotate messages with values such as intent or user sentiment.

In order to configure your radio classification as a message-based classification, you must configure your radio classification task as a **Frame/pixel-based** classification value during ontology configuration. If this value is not configured correctly, the radio classification will apply to the entire conversation rather than a single message.

### Relationship

With annotation relationships, you can create and define relationships between entity annotations in the conversational text editor. You can then use these annotation relationships to consolidate labeling workflows and potentially reduce the number of language models needed.

Follow these steps to create a relationship between two entity annotations:

1. In the editor, create two entity annotations.

2. Select the relationship annotation from the Tools menu, then click on one of the entity annotations. To create the relationship, move your cursor to another entity and click to create the relationship.

3. Add an optional subclassification to the relationship.

## Text-specific hotkeys

| Function | Hotkey | Description |
| - | - | - |
| Create a relationship | `Option` + Click (source) + Click (target) | Select an entity to be the source of a relationship, then connect it to another entity to be the target. |


# Create a project
Source: https://docs.labelbox.com/horizon/guides/create-a-project

Instructions for creating and modifying a labeling project in Labelbox.

This guide will walk you through the essential steps of creating a new project, as well as managing its lifecycle through duplication and deletion.

## How to create a new project

1. Navigate to the **Projects** tab in the main Labelbox application.
2. Click the **"New Project"** button in the top right corner.
3. **Choose the data modality or task type**. Note that you will only be able to attach data rows that match the data type you set here. For example, you cannot select video as the data type upon project creation and then send image data rows to this project later.
4. Give your project a clear, descriptive **Name**. You may optionally add a **Description** and project **tags**. Then, click **Save**.
5. From the project overview, click **Add data** to select the data rows to include in this project. Refer to our [Catalog docs](/horizon/guides/catalog-overview) to learn how to import data.
6. In this step, you can [filter and search](/horizon/guides/search) your data rows to curate a batch. Select the data rows you wish to send to the project, then click **Queue batch of *n*** to add the data rows to the labeling queue.
7. In the **Advanced batch settings**, you can optionally set the **Data row priority** and toggle on **Consensus** mode. Select **Submit** to send this batch to your labeling project.
8. The final step is to **Set up ontology and labeling experience**. You may either create a new ontology or reuse an existing one.
9. Once you complete these steps, the **Start labeling** button will be activated in your project overview.

## How to duplicate a project

Duplicating a project is a great way to reuse a complex setup without starting from scratch. When you duplicate a project, you copy its ontology, labeling interface settings, and all other configurations.

1. From the **Projects** tab, find the project you wish to duplicate. Open the project.
2. Click on the project name at the top. From the dropdown menu, select **Duplicate project**.
3. A new project will be created with the same settings.

Below are the settings that can get copied when you duplicate a project:

| Setting | Copied to duplicate version |
| - | - |
| Project Name | <Icon icon="square-check" /> |
| Project Tags | <Icon icon="square-check" /> |
| Project Description | <Icon icon="square-check" /> |
| Ontology | <Icon icon="square-check" /> |
| Quality | <Icon icon="square-check" /> |
| Batches | <Icon icon="xmark" /> |
| Data rows | <Icon icon="xmark" /> |
| Workflow instructions/setup | <Icon icon="square-check" /> |
| Performance | <Icon icon="xmark" /> |
| Issues | <Icon icon="xmark" /> |
| Issue Categories | <Icon icon="square-check" /> |
| Notifications | <Icon icon="xmark" /> |
| Import labels | <Icon icon="xmark" /> |
| Export | <Icon icon="xmark" /> |
| Members (settings) | <Icon icon="square-check" /> |
| External workforces | <Icon icon="xmark" /> |
| Models (for Live multimodal chat projects) | <Icon icon="xmark" /> |

## How to delete a project

If a project is no longer needed, you can delete it to keep your workspace organized.

1. From the **Projects** tab, find the project you wish to delete. Open the project.
2. Click on the project name at the top. From the dropdown menu, select **Delete project**.
3. A confirmation dialog will appear. Type the name of the project to confirm and click **"Delete"**.

<Warning>
  Deleting a project is a permanent action and cannot be undone.
</Warning>

## How to create project tags

As you create more projects in Labelbox, keeping them organized becomes crucial. Tags are a simple yet powerful way to categorize, filter, and manage your projects.

Benefits of using project tags:

* **Categorization:** Group projects by team (e.g., `team-alpha`, `team-beta`), status (`in-progress`, `completed`), or data type (`images-night`, `text-legal`).
* **Easy filtering:** Quickly find all projects associated with a specific tag, saving you time from manually searching.
* **Improved visibility:** Get a clear overview of your labeling initiatives by filtering your project list by tag.

**To create and apply tags**

1. From the **Projects** tab, select a project.
2. From the project overview, hover over the **Tags** section and click the **Edit** icon.
3. You can either *select an existing tag* from the dropdown list or *type a new tag* and press **Enter** to create it. A project can have multiple tags.

**Filtering by tags**

Use the filter bar at the top of the **Projects** page to filter by tag. Click on the tag you want to filter by, and the view will instantly update to show only the projects with that tag. You can select multiple tags to further refine your search.

**To delete a tag**

To manage existing tags, select a project, click the **Edit** icon next to **Tags** in the project overview. Click **Manage tags** and click the **Delete** icon next to the tag name.

<Warning>
  Deleting a tag removes it from all associated projects, and modifying a tag updates it across all associated projects.
</Warning>


# Configure CORS
Source: https://docs.labelbox.com/horizon/guides/create-cors-headers

Instructions for configuring Cross-Origin Resource Sharing (CORS) for AWS S3, GCS, and Microsoft Azure.

This guide explains what a CORS policy is and how to configure it correctly in AWS, GCP, and Azure. Completing this step ensures that your data assets will render correctly within the Labelbox application.

### What is CORS and why is it necessary?

* **The Problem:** Modern web browsers enforce a "Same-Origin Policy" to protect you from malicious websites. This policy prevents a web page (like the Labelbox app at `https://app.labelbox.com`) from making requests to a different domain (like your storage bucket at `your-bucket.s3.amazonaws.com`).
* **The Solution:** A CORS policy is a small configuration file you add to your storage bucket. It acts as a permission slip, telling the browser, "It's okay to allow requests from `https://app.labelbox.com` to access files in this bucket."

When configuring CORS for your cloud storage bucket, you will need to include both of these Labelbox origins: `https://app.labelbox.com` and `https://editor.labelbox.com`.

## AWS S3

1. Navigate to your S3 bucket in the AWS Console.
2. Click the bucket name.
3. Go to the **Permissions** tab.
4. Scroll to the **Cross-origin resource sharing (CORS)** section, click **Edit**.
5. Paste the provided policy into the editor and save.

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  [
      {
          "AllowedHeaders": [
              "*"
          ],
          "AllowedMethods": [
              "GET"
          ],
          "AllowedOrigins": [
              "https://app.labelbox.com",
              "https://editor.labelbox.com"
          ],
          "ExposeHeaders": []
      }
  ]
  ```
</CodeGroup>

7. For more details on setting up CORS for your AWS S3 bucket, see [these AWS docs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/enabling-cors-examples.html).

## Google Cloud Storage (GCS)

The configuration must allow requests from both `https://app.labelbox.com` and `https://editor.labelbox.com` to ensure data loads correctly across the entire platform.

You can set this policy using one of the two methods below.

### Option 1: Using the gcloud CLI on your local machine

This method is ideal if you have the `gcloud` command-line tool installed on your computer.

1. Create a new file on your computer named `cors-config.json`.
2. Copy and paste the following JSON content into the file. Note: The `"*"` for `responseHeader` allows all necessary headers and is recommended for robust compatibility.
   ```json theme={"theme":"css-variables"}
   [ { "origin": ["https://app.labelbox.com", "https://editor.labelbox.com"], "method": ["GET"], "responseHeader": ["*"], "maxAgeSeconds": 3600 } ]
   ```
3. Open your terminal or command prompt and run the following command. Be sure to replace `[BUCKET_NAME]` with the actual name of your GCS bucket.
   ```shellscript theme={"theme":"css-variables"}
   gcloud storage buckets update gs://[BUCKET_NAME] --cors-file=cors-config.json
   ```

### Option 2: Using the Google Cloud Shell

This method is convenient if you prefer to work directly within the Google Cloud Console without creating local files.

1. Open the [Google Cloud Shell](https://shell.cloud.google.com/ "https://shell.cloud.google.com/") from your GCP console.
2. Run the following command in the Cloud Shell terminal. This command creates the `cors-config.json` file and writes the correct configuration to it in a single step.
   ```shellscript theme={"theme":"css-variables"}
   echo '[{"origin": ["https://app.labelbox.com", "https://editor.labelbox.com"], "method": ["GET"], "responseHeader": ["*"], "maxAgeSeconds": 3600}]' > cors-config.json
   ```
3. Next, run the command below to apply the configuration to your bucket. Remember to replace `[BUCKET_NAME]` with your bucket's name.
   ```shellscript theme={"theme":"css-variables"}
   gcloud storage buckets update gs://[BUCKET_NAME] --cors-file=cors-config.json
   ```

## Microsoft Azure

1. Navigate to the your Storage Account in your Azure Portal.
2. Under **Settings**, find **Resource sharing (CORS)**.
3. Select the **Blob Service** tab.
4. Fill in the fields: Allowed origins (`https'//app.labelbox.com` and `https://editor.labelbox.com`). Allowed methods (`GET`), etc. and save.

## Troubleshooting

| Error message | Troubleshooting |
| - | - |
| Unable to detect proper CORS configuration | Ensure your cloud storage bucket has CORS configured with the following origins: `https://app.labelbox.com` and `https://editor.labelbox.com`. For more troubleshooting help, see: - [AWS troubleshooting CORS](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cors-troubleshooting.html) - [GCS troubleshooting CORS requests](https://cloud.google.com/storage/docs/configuring-cors#troubleshooting) - [Cross-Origin Resource Sharing (CORS) support for Azure Storage](https://docs.microsoft.com/en-us/rest/api/storageservices/cross-origin-resource-sharing--cors--support-for-the-azure-storage-services) |


# Custom model integration
Source: https://docs.labelbox.com/horizon/guides/custom-model-integration

Describes how to set up and integrate a custom model so that it can be used with Foundry.

If you are on an enterprise plan, you can integrate custom models with [Foundry](/horizon/guides/foundry) to use them to predict labels, enrich data, and generate responses for evaluation purposes. To upgrade to the enterprise plan, please [contact sales](https://labelbox.com/sales/).

## Host custom models

Before integrating your custom model, you need to deploy it on an HTTP endpoint accessible via the Internet that accepts HTTP POST calls with a JSON payload. You can host it either on your own infrastructure or through any model hosting vendor, such as [Vertex AI](https://cloud.google.com/vertex-ai/docs/general/deployment), [Databricks](https://docs.databricks.com/en/machine-learning/model-serving/create-manage-serving-endpoints.html), [Huggingface](https://huggingface.co/inference-endpoints), [Replicate](https://replicate.com/docs/how-does-replicate-work#private-models), [OpenAI](https://platform.openai.com/docs/guides/fine-tuning/use-a-fine-tuned-model).

## Create model integrations

Once you have a public HTTP endpoint for your custom model, you can create the integration:

1. On the [Models](https://app.labelbox.com/mea) page, click **Create** and select **Custom Model**.
2. Select the data type for the model.
3. Add custom model information, including:

* **Name**: A unique identifier for the model.
* **HTTP endpoint**: The URL of the HTTP endpoint hosting your model.
* **Secret** (optional): The authentication token for secret-secured endpoints only.
* **Description** (optional): The descriptive context of the model.

4. Click **Create model**.

On the **Settings** tab, you can review and edit the model information. You can add a rate limit and a Readme. To send data to your model for label prediction, click **+ Model run**. From there, you can define and [preview your model run](/horizon/guides/foundry-define-model-run), [view prediction and details](/horizon/guides/foundry-view-predictions), and [send predictions to Annotate](/horizon/guides/foundry-annotate-predictions).

<Warning>
  ### Bounding box and mask tasks not supported

  Currently, this model integration flow doesn't support tasks involving bounding box and mask annotations. To integrate a custom model for these tasks, see [Create model integrations for bounding box and mask tasks](#create-model-integrations-for-bounding-box-and-mask-tasks).
</Warning>

## Create model integrations for bounding box and mask tasks

For a custom model predicting bounding box and mask labels, you need to create a model manifest file and [contact customer solutions](https://labelbox.atlassian.net/servicedesk/customer/portal/2/group/3/create/214) to manually establish the integration. The Labelbox solutions team can help you manage the job queuing, track status, and process predictions using the Labelbox platform.

### Create manifest files

To integrate your model into the Foundry workflow, you need to specify and provide a `model.yaml` manifest file. This file stores metadata about the model, including its name, description, inference parameters, model output ontology, API endpoint, and other details. You need to create the `model.yaml` file in the following format:

<CodeGroup>
  ```yaml Example YAML manifest file theme={"theme":"css-variables"}
  name: My custom model
  inference_endpoint: my_inference_endpoint # Deploy your service to an API endpoint that can be accessed
  secrets: my_secret # Your secret, API keys to be authenticated with your endpoint
  requests_per_second: 0.1 # Your estimate of requests per second
  description: My awesome custom model for object recognition
  readme: | # optional readme in markdown format
    ### Intended Use
    Object recognition model on my custom classes.
    ### Limitations
    My custom model has limitations, such as ...
    ### Citation
    ...

  allowed_asset_types: [image] # list of allowed asset types, one or more of [image, "text", "video", "html", "conversational"]
  allowed_feature_kinds: [text, radio, checklist] # list of allowed feature kinds. One or more of [text, radio, checklist, rectangle, raster-segmentation, named-entity, polygon, point, edge]

  # Only needed if your model has a predefined set of classes for classification or object detection. If your model is an LLM or takes any text input, you can remove this section.
  ontology:
    media_type: IMAGE # This example ontology has two classification classes and two object detection classes.
    classifications:
      - instructions: label
          name: label
          type: radio
          options:
          - label: tench
            value: tench
            position: 0
          - label: goldfish
            value: goldfish
            position: 1
    tools:
      - name: person
        tool: rectangle
      - name: bicycle
        tool: rectangle

  inference_params_json_schema: # hyperparmeters configured in the app and passed to your API endpoint.
    properties: # Examples follow, each with different types and defaults.
      prompt:
        description: "Prompt to use for text generation"
        type: string
        default: ""
      confidence:
        description: object confidence threshold for detection
        type: number
        default: 0.25
        minimum: 0.0
        maximum: 1.0
      max_new_tokens:
        description: Maximum number of tokens to generate. Each word is generally 2-3 tokens.
        type: integer
        default: 1024
        minimum: 100
        maximum: 4096
      use_image_attachments:
        description: Set to true if model should also process datarow attachments.
        type: boolean
        default: False
    required: # Use to specify hyperparameters that must have values for each model run.
      - prompt

  max_tokens: 1024 # only relevant for LLM to control maximum token size
  ```
</CodeGroup>

## Endpoint requests for model tasks

Every time you use your integrated custom model to predict labels or run other tasks, it sends a JSON request to your model's endpoint. The request payload provides the data row for prediction and includes the ontology and inference parameter values you selected. Here's an example request body:

<CodeGroup>
  ```json Example JSON request theme={"theme":"css-variables"}
  {
   "prompt": [
     {
       "role": "system",
       "parts": [
         {
           "text": "Start each sentence with three equal signs ==="
         }
       ]
     },
     {
       "role": "user",
       "parts": [
         {
           "text": "what is in this text and image?"
         },
         {
           "text": "Hello. This is a user-provided txt file content."
         },
         {
           "image": "base64_encoded_image_string"
         }
       ]
     }
   ]
  }
  ```
</CodeGroup>

Here are descriptions of fields in the request body:

* `prompt`: Contains the current conversation with the model. For single-turn queries, it’s a single instance. For multi-turn queries, it includes conversation history and the latest request. Each `prompt` has a message structure with two properties: `role` and `parts`.
* `role`: A string indicating the individual producing the message content. Possible values include:
  * `system`: Instructions to the model.
  * `user`: User-generated message sent by a real person.
  * `assistant`: Model-generated message, used to insert responses from the model during multi-turn conversations.
* `parts`: A list of ordered parts that make up a multi-part message content. It can contain the following segments of data:
  * `text`: Text prompt or code snippet.
  * `image`: Base64 encoded image.

### Response

Responses are expected to match the format of the labels predicted by the custom model, such as a string containing the raw model response or a JSON object for NER and classifications. Here's an example JSON response with keys corresponding to feature names in the ontology:

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  // Object Detection
  {
    "cat": {
  	// coordinate order: left, top, width, height
        "boxes": [[0, 0, 10, 10], [40, 40, 8, 10]],
        "scores": [0.9, 0.7],
    },
    "dog": {
        "boxes": [[20, 20, 5, 5]],
        "scores": [0.8],
    },
  }
  // Classification
  {
    "summary": "Tom and Bob are happy to work at IBM", // Free Text
    "sentiment": "positive",  // Radio classification
    "emotion": ["joy", "fear"], // Checklist classification
  }
  // Segmentation
  {
    "cat": {
  	// Can use pycocotools.mask.encode for RLE encoding
  	"masks": [
  		{
  			"size": [<height>, <width>],
  			"counts": "<run-length-encoded-boolean-mask>"
  }
  ]
    }
  }
  // Named Entity
  {
    "person": [
      {"start": 0, "end": 3, "text": "Tom"},
      {"start": 5, "end": 8, "text": "Bob"},
    ]
  }
  ```
</CodeGroup>


# Manage your project's data rows
Source: https://docs.labelbox.com/horizon/guides/data-rows-activity

The Data Rows tab enables you to easily view, filter, and manage the data rows in your project.

The **Data rows** tab in your project dashboard is your central hub for viewing, filtering, and managing all the data rows within a specific project. Whether a data row is labeled, waiting in the queue, or undergoing review, you can find it here.

## Filtering your data rows

Each data row displays key information, including how much time has been spent on labeling and review, which dataset it belongs to, and whether there are any open issues.

There are several ways you can filter your data.

### Use search filters

To find specific data rows, you can use a powerful and flexible filtering system. Click on **Search your data** to get started. You can combine multiple filters using AND/OR conditions to create highly specific queries.

For example, you could search for all data rows in a specific `batch` AND that have been `reworked by` a particular labeler. This allows you to precisely target the data you need.

Here are the available filters:

| Filter | What it does |
| - | - |
| Dataset | Find data rows belonging to a specific dataset. |
| Metadata | Filter data rows based on their metadata fields and values. |
| Media attribute | Query data rows based on attributes of the asset itself (e.g., image height or video duration). |
| Batch | Find data rows that are part of a particular batch. |
| Data row | Search for data rows by their ID, Global Key, creation date, or last activity date. |
| Find text | Perform a text search within your data rows. |
| Natural language | Use natural language to search for data rows (e.g., "show me all the pictures of cats"). |
| Task | Find data rows that are currently in a specific workflow task. |
| Issue | Filter data rows based on any issues associated with their labels. |
| Label actions | Find data rows based on actions taken on their labels, such as who labeled them, when they were labeled, or if they were skipped. **Note:** Bulk moving a data row to a different step is not considered a review or rework, so it won't appear in filters like `Reworked at`, `Reworked by`, `Reviewed by`, or `Reviewed at`. |
| Annotation | Filter data rows based on the number of annotations on their labels. |
| Benchmark agreement | Find data rows based on benchmark agreement metrics. |
| Consensus agreement | Find data rows based on consensus agreement metrics. |

### Filter data rows by status

The status of a data row tells you where it is in the labeling and review process. The following statuses are available by default for every project.

You can see a count of data rows for each status on the left panel. Clicking on a status will filter the table to show only the data rows with that status.

| Status | What it means |
| :- | :- |
| To label | These data rows need to be labeled. For a standard data row, this means it has no labels. For a consensus or benchmark data row, this means it has fewer than the required number of labels (e.g., it has two labels but needs three). |
| In review | These data rows are currently in a review task within the workflow. |
| In rework | These data rows have been sent back for corrections and are in the rework task. |
| Done | These data rows have all the required labels and have successfully passed all review steps in the workflow. |

## Updating your data rows

You have several options to update your data rows in bulk or at the individual level

### Task browser: Update individual data rows

To view or edit the labels on a single data row, click on the data row in the table. This will open the **Task browser**.

From here, you can:

* **Edit the label**: Click the **Edit** button to create or edit annotations. Press **Save** to save your changes.
* **View all labels (for consensus or benchmark)**: If you are using consensus or benchmarks, you can view all the different labels created for that data row. Use the `>` icon in the left-side panel to see all labels, view their IDs, and select a "winner" label.

Click on the options menu next to a data row to access more actions you can take at the data row level.

From this menu, you can:

* **Copy link:** Copy the link to the data row. You can send the link to the data row to other members in your organization.
* **Delete labels & requeue:** Remove all existing labels from a selected group of data rows and then send those data rows back to the beginning of the labeling queue to be worked on again.
* **Move to step:** Choose to move the selected data row to **Initial review task**, **Rework**, or **Done**.
* **Add as benchmark:** Select this label as a winning benchmark label.

### Bulk update multiple data rows

If you need to update the configurations for many data rows at one time, you can use the Bulk actions feature.

Select multiple data rows to perform the following actions in bulk:

| Action | What it does |
| - | - |
| **Change priority** | Assign a new priority to the selected data rows to change their order in the labeling queue. |
| **Delete and requeue** | Allows you to remove all existing labels from a selected group of data rows and then send those data rows back to the beginning of the labeling queue to be worked on again. You can select to preserve the existing labels or delete them. |
| **Export data** | Export the selected data rows and their labels. |
| **Hide/unhide from labelers** | Control which data rows are visible to labelers. Hidden data rows will have an "Authorized only" tag. Admins can still view the row details, but labelers will only be able to copy the external ID and will see an "Unauthorized to view this data row" message if they try to access it. |
| **Move to step** | Choose to move the selected data rows to: **Initial review task**, **Rework**, or **Done**. |
| **Assign labels as benchmarks** | Designate the labels on the selected data rows as benchmark reference labels. |

<Info>
  Bulk **Move to step** actions are not considered a review or rework. Therefore, these actions will not show up when applying **Reworked at**, **Reworked by**, **Reviewed by**, or **Reviewed at** filters.
</Info>

## Updating your batches

From the Data Rows tab, you can also update the batches of data rows in your project.

### View and update batches

Click on **Batches** → **View batches** to view and update all the existing batches in your project. From here, you can:

* **Rename the batch**.
* **Copy batch ID**.
* **Change priority**: When you initially create the batch, you have the option to select the data row priority (value between 1 - 5). Leaving it as-is will default to batch value to 5. From the **View batches** panel you can update the priority for the batch.
* **Remove queued data rows**: This will remove any data rows from the batch that have not yet been labeled.
* **Delete labels**: This will delete the labels from the data rows in the batch and give you the option to requeue them, either with or without the old label as a template.
* **View history**: See a log of when data rows were added or removed from the batch.

### Add more data to batch

Click on **Batches** → **Add data from Catalog** to select more data rows to add to the batch.


# Metadata
Source: https://docs.labelbox.com/horizon/guides/datarow-metadata

Instructions for adding, filtering, and modifying metadata in the app UI.

**Developer guide**: [Metadata](/horizon/developer-guides/metadata)

Metadata is non-annotation information about an asset. You can use metadata to search and filter your data rows in Labelbox. The metadata schema lives at the organization level. This allows you to apply the same metadata fields across multiple datasets. You can use reserved metadata fields and add custom metadata fields.

## Data types

All metadata needs to be one of the following types:

| Type | Descriptions | Filtering |
| - | - | - |
| `DateTime` | An ISO 8601 datetime field. All times must be in UTC timezone | Equals, greater than, less than, between |
| `Number` | Floating-point value (max: 64-bit float) | Greater than & less than |
| `String` | free text field. Max 4,096 characters. | Equals & prefix matching |
| `Enum` | Enum field with options. Multiple options can be imported. | Equals |
| `Option (Enum)` | Option of an enum. Max 64 options can be created per Enum type. 128 for enterprise customers and can be further increased upon request. | Equals |
| `Embedding` | 128 float 32 vector used for similarity | |

## Reserved fields

By default, Labelbox defines several metadata fields on your data rows. You don't have to use these fields and can change the field values, but you can't delete or rename these fields.

Each metadata field has a unique schema ID used to upload data to Labelbox.

| Name | Type | Description |
| - | - | - |
| `tag` | `String` | The tags of the data row |
| `split` | `Enum` | The split of the dataset that the data row belongs, including`train`, `valid`, and `test` |
| `captureDateTime` | `DateTime` | The timestamp when the data is captured |
| `skipNFrames` | `Number` | (Video data only) The number of frames to skip |
| `turnInstructions` | `String` | JSON string that contains instructions for each turn in a Multi-modal chat conversation. |

## Custom fields

To create custom metadata fields via the UI, go to the [**Schema**](https://app.labelbox.com/schema/metadata) tab. Select **Create**.

Each metadata field must have a unique name and a type. The max number of fields per organization is determined by account tier.

To view your metadata schema, go to the [**Schema tab**](https://app.labelbox.com/schema/metadata) and then select the **Metadata** subtab.

<Info>
  ### Metadata field limits

  You can modify the names of custom metadata schema by clicking the **Edit** button in the detail view of each schema, but you can't change the type once you create a field,. You also can't modify the names of reserved fields. See [limits](/horizon/guides/limits) to learn the maximum number of allowed metadata fields of your account.
</Info>

## Update custom metadata schema via the UI

Go to the Schema tab to update custom metadata fields via the UI. Then, find the metadata you want to modify and click the **Edit** button. Only the following can be updated:

* The name of custom metadata for [non-reserved](#reserved_fields) metadata
* The name of options for Enum metadata

## Bulk add metadata

Follow these steps to bulk add metadata to your data rows in **Catalog**. See [limits](/horizon/guides/limits) to learn the limits for bulk adding metadata to data rows in Catalog.

### Step 1: Select data rows

You will need to select a curated subset of data.

For example, you may select a cluster of data from the [projector view](/horizon/guides/model-runs). Another option could be to select the top results of a [natural language search](/horizon/guides/natural-language-search). This way, you can use neural networks like CLIP as zero-shot classifiers. A third possibility is to select all assets that look similar to each other - thanks to [Labelbox similarity search](/horizon/guides/similarity). Similarity search powered by embeddings allows you to leverage any off-the-shelf neural network as a zero-shot classifier.

There are three ways to select data rows:

* Option 1: Select all filtered data rows by choosing **Select all**.
* Option 2: Use the selection icon (checkbox) to manually select individual data rows.
* Option 3: Bulk select data rows by selecting the first data row, press and hold `Shift`, and then select the last data row. All data rows between the first and last ones are selected.

### Step 2: Add metadata

After you select your data rows, select **Add metadata** from the selection menu.

### Step 3: Pick a metadata field

From the selection menu, select the metadata field you want to apply.

You can search for metadata fields by typing their name if you don't see them in the dropdown.

Metadata fields must exist in Labelbox before they appear in the menu.

### Step 4: Provide metadata values

Enter a metadata value. This metadata field and value will apply to all selected data rows.

Choose **Save** to apply the metadata in bulk. Changes can take time to be reflected in the data row metadata.

## View metadata

You can surface data rows with the new metadata values, by [searching on metadata in Labelbox](/horizon/guides/search#supported-attributes-for-search-and-filter).

The newly created metadata tag will also show up in the detailed view.

## Filter metadata

Once you upload your metadata, you can easily filter and view metadata in Catalog. If you want to label this set of data rows, you can filter by that metadata field and send them to a labeling project as a [batch](/horizon/guides/batches).

<Info>
  ### Metadata viewing access

  Labelers cannot filter and view metadata in Catalog. However, labelers will be able to view metadata information in the data row information panel.
</Info>

Select a data row to open its detailed view. The **Metadata** panel shows all current metadata.

## Export metadata

When you export your metadata, you can sync the newly created metadata with any outside system such as a cloud bucket, data lake, data warehouse, or database. To learn how to export data rows containing metadata, see [Export from catalog](/horizon/guides/export-from-catalog).

## Delete metadata

You can delete metadata for a data row through the SDK. Visit [this guide](/horizon/developer-guides/metadata) for instructions.


# Manage datasets
Source: https://docs.labelbox.com/horizon/guides/datasets-datarows

Instructions for managing datasets in the Labelbox UI.

This guide covers how to create, modify, and manage your datasets directly within the Labelbox user interface.

## Before you start

Before you can label data, you need to import it into Labelbox. Take a look at our guides for connecting your cloud data to Labelbox, then come back to this page.

<Card title="Connect to cloud storage" icon="sparkles" href="/horizon/guides/connect-to-cloud-storage" />

## Key definitions

The following are fundamental concepts for organizing your data in Labelbox. Understanding these terms is the first step to building powerful ML models.

All data in Labelbox is organized into **Datasets**, which are made up of individual **Data Rows**.

| Term | Definition |
| :- | :- |
| Asset | The file you want to label (e.g., an image, a video, a text file). This is your core, raw data. |
| Data row | The container for a single asset and all its related information. This includes the asset itself, its metadata, attachments, and any annotations (labels) created for it. |
| Dataset | A collection of data rows, typically from a single source or domain. For example, you might create one dataset for "Medical Images from Device X" and another for "Customer Support Transcripts." |
| Attachment | Supplementary information you can add to an asset to provide extra context for your labelers. For example, you could attach a PDF with instructions or a reference image. |
| Global key | An optional but highly recommended unique ID for each data row. Using global keys helps prevent duplicate data uploads and makes it easy to map data rows in Labelbox back to your own external databases or file systems. |

## Create a new dataset

1. Navigate to the **Catalog** tab.
2. Click the **+ New** button to open the "Create a new dataset" dialog.
3. Give your dataset a clear, descriptive name (e.g., `q1-2026-night-driving-images`).
4. Optionally, add a description for more context.
5. Choose an import method:
   * [Connect to your cloud storage (RECOMMENDED)](/horizon/guides/connect-to-cloud-storage)
   * [Signed URLs](/horizon/guides/signed-urls)
   * [Direct upload](/horizon/guides/upload-local-files)
6. Click **Create dataset**.

<Note>
  Best practices

  * Naming: Dataset names can be up to 256 characters and include letters, numbers, spaces, and `_-.,()/`. Use names that clearly explain the data's source and purpose.
  * Organization: Group data from a single domain or source into its own dataset. This simplifies labeling workflow setup. You can use Metadata to further organize and filter data rows within a dataset.
</Note>

## Append data to an existing dataset

You can add more data to a dataset at any time.

1. Go to **Catalog** and select your dataset from the list on the left.
2. Click the **Append to dataset** button.
3. You will be prompted to choose an import method to add your new data rows.

## Delete a dataset

1. In **Catalog**, select the dataset you wish to delete.
2. Click the three-dot menu and select **Delete dataset**.
3. A confirmation dialog will appear. Type `delete` and click the **Delete dataset** button to confirm.

<Warning>
  Deleting a dataset is a permanent, irreversible action. All associated data rows, annotations, metadata, and classifications will be lost.
</Warning>


# Deprecations
Source: https://docs.labelbox.com/horizon/guides/deprecations

A list of active and completed deprecations

## Active deprecations

This section lists all products and features which are currently in a deprecation period.

***

## Completed deprecations

This section lists all products and features which have been decommissioned. These products and features are no longer available.

### Model experiments

As of July 15th, the Model experiments feature is no longer available.

Note: This update does ***not*** impact any of the “Foundry” models or capabilities.

### Code runner

Code Runner was a Labelbox feature that allowed users to execute small code snippets directly within MMC conversations via Cloud Run functions. Code runner was sunset on July 7, 2026 due to lack of usage.

### Prompt and response editor

The Prompt and response editor in the Annotate product was sunset on June 30, 2026.

### Public Demo organization

The public demo workspace in the Labelbox platform was created to provide sample projects for users to explore the platform's features and capabilities. It was sunset on April 3, 2026.

### Census integration

As of June 30, 2025, we no longer support the Census integration.

### Export v2 nonstreamable methods

In May 2025, we introduced changes to the Python SDK that broke compatibility with the following Export v2 non-streamable methods in version 3.67 and earlier:

* `catalog.export_v2()`
* `data_row.export_v2()`
* `dataset.export_v2()`
* `model_run.export_v2()`
* `project.export_v2()`
* `slice.export_v2()`

If you're using any of these methods with SDK version 3.67 or earlier, please update to a newer version or migrate to the `export()` method, which provides a more scalable and performant way to export data across the Labelbox platform. To learn more, see [Export overview](/horizon/developer-guides/export-overview).

### Automation efficiency score

On April 16, 2025, we removed Automation efficiency score from the Labelbox platform.

### Catalog cluster view

On February 28, 2025, we deprecated Catalog cluster view for all customers due to a lack of usage.

### Smart select

In February 2025, we deprecated the **Smart select** button for selecting datasets on the Catalog page.

### Reporting (Enterprise Dashboard)

In February 2025, we deprecated the Reporting page (Enterprise Dashboard) for all customers. The [Monitor tab](/horizon/guides/monitor) offers all of the same metrics and filters (plus more). Unlike the Reporting page, the Monitor page is a native solution that is built into the Labelbox platform, so it offers a more seamless and robust user experience. To learn more, read [this guide](/horizon/guides/migration-guide-reporting-page-to-monitor).

### DICOM editor

Due to a lack of usage, we sunsetted the DICOM editor on November 25, 2024.

### YOLO models in Foundry

On November 7th 2024, Labelbox disabled YOLO models in Foundry due to a lack of usage. Foundry still supports other object detection and segmentation model alternatives (OWL-ViT, Rekognition, GroundingDINO, GroundingDINO + SAM). If you would like to set up your own YOLO model for inferencing, refer to our [custom model integration](/horizon/guides/custom-model-integration#create-model-integrations-for-bounding-box-and-mask-tasks) docs.

### Fine-tuning

On November 7th 2024, Labelbox disabled the model fine-tuning feature. This means that image fine-tuning is no longer usable. We may re-enable the fine-tuning feature in the future.

### Export V1

In early September, we disabled Export v1 for all remaining customers. All users should use the `export()` method instead.

### Data connector libraries

All data connector libraries, including `labelbase`, `labelspark`, `labelpandas`, `labelsnow`, and `labelbox-bigquery` libraries, have been publicly archived and are no longer maintained.

### Custom editor

We sunsetted the custom editor for all customers on June 30, 2024. Custom editor projects and their associated labels are no longer accessible. Please use one of our native labeling editors instead.

### Dropdown schema

Dropdown is a deprecated classification type (the other classification types are radio, checklist, and free-form text). Labelbox implemented a UI solution that provides a dropdown view for radio and checklist classifications. Therefore, we removed the capability to add the dropdown schema to new ontologies.

We recommend using radio or checklist classifications and selecting the dropdown toggle instead.

### Labels tab

Prior to the rollout of the [Data Rows tab](/horizon/guides/data-rows-activity), the Labels tab was a dashboard view in Annotate. The Labels tab was replaced by the Data Rows tab as part of the shift to our new data row-centric paradigm.

Effective August 2023, no more customers are using the Labels tab.

### Review step

Prior to the rollout of [Workflows](/horizon/guides/workflows), the review step was the mechanism for reviewing labels in Annotate. The review step was replaced by Workflows in order to offer more flexibility in customizing the review flow for labeling tasks.

Effective August 2023, no more customers are using the Review step.

### Dataset-based queueing

Prior to the rollout of [batch-based queueing](/horizon/guides/batches), the only way to send data rows to a labeling project was by sending the entire dataset. This was replaced by batch-based queueing to enable customers to send a curated subset of data rows across multiple data rows to a labeling project.

Effective August 2023, no more customers are using dataset-based queueing.

### On-prem

At the end of 2022, Labelbox decided to sunset our on-premise offering. All customers are encouraged to use our SaaS product instead.

As of August 2023, no more customers are using the Labelbox on-prem offering.

### Superpixel

In 2023, Labelbox released the [auto-segment tool](/horizon/guides/label-data#auto-segment), which replaced the Superpixel feature. Auto-segment is embedded into the segmentation mask tool. You can use it to auto-generate segmentation masks.

As of January 2023, no more customers are using the Superpixel feature.


# Connect to Discord
Source: https://docs.labelbox.com/horizon/guides/discord

Follow these instructions to set up your Alignerr account with Discord.

As part of our onboarding and communication process, all Alignerrs are required to sign up for Discord. Going forward, Discord will be our official platform for project updates, team communications, and key announcements.

# Refresh membership

If you *already have* an Alignerr Discord account, follow these steps:

1. Log onto [app.alignerr.com](http://app.alignerr.com).

2. Go to **Settings** → **Discord** → click **Refresh Discord membership**.

3. Sign into Discord using your existing credentials.

# Create/link account

If you *do not have* an Alignerr Discord account, follow these steps:

1. Log onto [app.alignerr.com](http://app.alignerr.com).

2. Go to **Settings** → **Discord** → select **Link Discord account**.

3. Select **Authorize** in the Alignerr Manager modal.

4. Create your Discord account using the email address you use for Alignerr projects.

5. You will receive an email from Discord. Open the email and click **Verify email**.

6. When you enter Discord, you’ll see a modal that says **Create Your First Discord Server**. Exit out of that modal. Do not select anything.


# Documents
Source: https://docs.labelbox.com/horizon/guides/document-editor

Guide for labeling document (PDF) data.

The **Documents** editor lets you annotate and review PDF documents for tasks such as document review, text extraction, and data labeling for machine learning. You can navigate multi-page documents, manage annotation layers, and track progress efficiently. The editor also supports automatic optical character recognition (OCR) with the ChatGPT o1 model to detect and extract text from selected areas.

## Set up document annotation projects

To set up a document annotation project:

1. [Create a document (PDF) dataset](/horizon/guides/datasets-datarows).
2. On the [Annotate projects page](https://app.labelbox.com/projects), click the **+ New project** button.
3. Select **Documents**. Provide a **name** and an optional **description** for your project.
4. Click **Save**. The system then creates the project and redirects you to the project overview page.
5. Click **Add data**. Then select your audio dataset. Click **Sample** to sample your dataset, or you can manually select data rows and click **Queue batch**. To learn how to import documents using the SDK, see [importing document data](/horizon/developer-guides/import-document-data).

<Info>
  ### Data row size limit

  To view the maximum size allowed for a data row, see [limits](/horizon/guides/limits).
</Info>

<Warning>
  ### Image encoding

  If your PDF files contain images, use JPEG encoding and RGB colorspace for color images.
</Warning>

## Supported annotation types

Below are the annotation types you may include in your ontology for labeling document data. Classification-type annotations can be applied globally or nested within a bounding box or entity annotation.

| Feature | Import annotation | Export annotation |
| - | - | - |
| **Bounding box** | [See payload](/horizon/developer-guides/import-document-annotations#bounding-box) | [See payload](/horizon/developer-guides/export-document-annotations#bounding-box) |
| **Entity** | [See payload](/horizon/developer-guides/import-document-annotations#entity) | [See payload](/horizon/developer-guides/export-document-annotations#text-entity-named-entity) |
| **Relationship** | [See payload](/horizon/developer-guides/import-document-annotations#relationship) | [See payload](/horizon/developer-guides/export-document-annotations#relationship) |
| **Radio classification** | [See payload](/horizon/developer-guides/import-document-annotations#classification---radio-single-choice) | [See payload](/horizon/developer-guides/export-document-annotations#classification---radio) |
| **Checklist classification** | [See payload](/horizon/developer-guides/import-document-annotations#classification---checklist-multi-choice) | [See payload](/horizon/developer-guides/export-document-annotations#classification---checklist) |
| **Free-form text classification** | [See payload](/horizon/developer-guides/import-document-annotations#classification---free-form-text) | [See payload](/horizon/developer-guides/export-document-annotations#classification---free-form-text) |

All PDF documents support bounding box annotations. To create other annotations, PDF documents *must* have text boxes before you upload them to Catalog. For best results, verify text layers *before* uploading PDFs.

### Bounding box

To create a bounding box, use your cursor to create the shape around a character, word(s), or section in the document. To reposition the bounding box on the document, click + hold, then use your mouse or trackpad to reposition the annotation on the document. You can also click + drag the corners to resize the bounding box.

### Entity

To create an entity annotation, click the desired starting character and drag to select a sequence of characters in the text. Characters are not restricted to a single class; entity annotations may overlap completely or partially. Entities may also span multiple pages. To edit an entity's class, right-click the entity and select **Change class**.

Shortcut: In the **Tools** panel, you will see a numerical hotkey next to the name of the annotation. Use the specified number hotkey (e.g., `1`, `2`, `3`) to activate the entity tool.

To create another entity, press the number hotkey again to activate the tool, then create another entity. Once all entities have been created, press `E` to submit your label.

#### Token selection

We also support tokenization, so you can create and highlight entities at both word and character levels, which is determined by the data in your JSON upload.

Clicking on a specific word will highlight the entire word. This is helpful when labeling text, as it can be easy to accidentally miss certain characters or words when highlighting.

### Relationships

To create a relationship between annotations:

1. Select a relationship tool and hover over the source annotation of the relationship to reveal the annotation's anchor points.
2. Click an anchor point to create the starting point of the relationship, then bring your mouse over to the annotation you want to relate it to, hovering over it to reveal its anchor points.
3. Click one of the anchor points to complete the relationship.

Right-click a relationship to change its direction, make it bi-directional, or delete it altogether.

#### Relationships for annotations across pages

If you want to create an annotation relationship for annotations that exist on different pages, you will need to follow a slightly different workflow:

1. Select the relationship tool.
2. Go to the annotation where you want to start the relationship, right-click, and click **Select relationship start**.
3. Scroll to your destination annotation, right-click, and click **Select relationship end**.

After you have selected both the starting and end annotation of the relationship, your relationship will be established.

### Radio classification

Create a radio classification by activating the classification question and inputting the answer value. In the below example, press `8`, `k`, and `esc` to complete the radio classification.

Once all classifications have been completed, press `e` to submit your label.

### Checklist classification

Create a checklist classification by activating the classification question and inputting the answer value(s). In the below example, pressing `7` and pressing `Down` + `Enter` on the answer values completes the checklist classification.

Once all classifications have been completed, press `e` to submit your label.

### Free text classification

Create a free text classification by activating the classification question and inputting the answer value. In the below example, pressing `6`, typing the answer value, and pressing `Enter` completes the free text classification.

Once all classifications have been completed, press `e` to submit your label.

## Enable auto-OCR

The in-editor automatic OCR support allows you to recognize and extract text from PDFs using bounding boxes. To enable it, add a bounding box feature and toggle on the Automatic OCR option. When you draw a bounding box on the document, OCR automatically detects and extracts text within the selected area. The extracted text is stored as a text subclassification.

## Custom text layers

A unique aspect of our document editor is the ability to view text layers. You can toggle the text layer on and it will appear whenever you want to highlight an entity.

## Navigate the document

Use your mouse scroll wheel or trackpad to move forward and backward through the pages of the document. To jump to a specific page, highlight the current page number in the top navigation bar, type your desired page number, and press `Enter`.

To zoom in, press `Z` and click on the section of the page you want to zoom in on. To zoom out, press `Opt` + `Z` and click on the page, or press `Shift` + `Z` to return the page to its original zoom level.

## Document-specific hotkeys

| Function | Hotkey | Description |
| - | - | - |
| Show Text Layer | `Shift` + `T` | Show or hide the text layer. |


# Similarity search with embeddings
Source: https://docs.labelbox.com/horizon/guides/embeddings

Learn how to use embeddings to perform a similarity search in Labelbox.

Embeddings are numerical representations of your data that make it possible to find items that are visually or contextually similar. This is especially useful for unstructured data like images, text, and video, where you can't use traditional search methods. In Labelbox, this is called a similarity search.

By using similarity search, you can quickly find more examples of rare edge cases, identify and remove bad data, and ultimately build a higher-quality dataset for training your models.

## How to use similarity search

The easiest way to get started with similarity search is to use the *pre-computed embeddings* that Labelbox provides for common data types.

**To perform a similarity search:**

1. **Find an anchor**: In the Catalog, hover over a data row you want to find more examples of and click the **Find similar data** icon. This initial data row will be your "anchor".
2. **Add more anchors**: You can optionally select multiple data rows to use as anchors and then select **Add selection as anchors**.
3. **Refine your results**:
   * **Add or remove anchors**: To add more examples and refine the search, select more data rows and click "Add selection to anchors". To remove an anchor, click on "Anchors (n)" to view all anchors, and click the "—" icon on the one you want to remove.
   * **Adjust the similarity score**: You can adjust the range of the similarity score (from 0 to 1) to broaden or narrow your search. A higher score means the results will be more similar to your anchors.

| Asset type | Pre-computed embedding |
| - | - |
| Image | [CLIP-ViT-B-32](https://huggingface.co/sentence-transformers/clip-ViT-B-32) (512 dimensions) |
| Video | [Google Gemini Pro Vision](https://ai.google.dev/gemini-api/docs/models). First two (2) minutes of content is embedded. Audio signal is not used currently. This is a paid add-on feature available upon request. |
| Text | [all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) (768 dimensions) |
| HTML | [all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) (768 dimensions) |
| Document | [CLIP-ViT-B-32](https://huggingface.co/sentence-transformers/clip-ViT-B-32) (512 dimensions) and [all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) (768 dimensions) |
| Tiled imagery | [CLIP-ViT-B-32](https://huggingface.co/sentence-transformers/clip-ViT-B-32) (512 dimensions) |
| Audio | [all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) (768 dimensions). Audio is transcribed to text. |
| Conversational | [all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) (768 dimensions) |

## How to use your own embeddings

If you have your own embeddings, you can upload them to Labelbox to use in similarity searches.

<Note>
  View the [limits](https://docs.labelbox.com/horizon/guides/limits) page to learn the custom embedding limits per workspace and the maximum dimensions allowed per custom embedding.
</Note>

**To create and upload a custom embedding:**

1. Create an [API key](https://docs.labelbox.com/horizon/developer-guides/create-api-key).
2. Navigate to **Schema > Embeddings**.
3. Click **+ Create** and give your embedding a name and specify the number of dimensions.
4. Once created, you can use this [Google Colab notebook](https://colab.research.google.com/drive/159lWZzY3wtGacLjwfPuiqdz7eaQ8TfXj?usp=sharing) to upload your custom embeddings. You can view all of the embedding fields in JSON format by clicking the **\</>** button.

To delete a custom embedding, you can do so from the same **Schema > Embeddings** page by selecting the embedding and clicking the gear icon.

To learn more about custom embeddings, see our [Custom embeddings developer guide](/horizon/developer-guides/custom-embeddings).

## Advanced search techniques

To further refine your data curation workflow, you can combine a similarity search with other filters in the Catalog, such as metadata, annotations, or datasets. Once you have a set of filters that you want to reuse, you can save them as a "slice". Slices are dynamic, so any new data that matches your filter criteria will automatically be added to the slice.


# Explore your data in Catalog
Source: https://docs.labelbox.com/horizon/guides/explore-your-data-in-catalog



Welcome to Labelbox Catalog, your command center for understanding, curating, and preparing unstructured data for your machine learning workflows. Before you can build a high-performing model, you need to deeply understand the data you're working with. Catalog is designed to move you from raw data to curated, label-ready datasets with confidence and speed.

Think of Catalog as an interactive, searchable index of all your training data. It provides the tools to not just see your data, but to interact with it, ask questions of it, and organize it in powerful ways.

## What can you do with Catalog?

* **Visualize and explore your data to uncover insights:** Instead of guessing what’s in your dataset, you can directly visualize it. Spot imbalances, identify outliers, find rare edge cases, and understand the distribution of your data before it ever touches a model. Use the gallery view for a visual survey, the list view for metadata analysis, and the analytics view to see statistical breakdowns.
* **Find specific data with powerful search and filtering:** Move beyond simple filename searches. Catalog allows you to build complex queries to find the exact data you need. You can filter by a rich set of attributes including metadata, annotation-class, dataset, project, and even the content of the data itself using AI-powered search methods.
* **Curate and organize datasets for any workflow:** Your raw data is just the beginning. Catalog helps you organize it for specific tasks. You can create static **batches** of data to send to a labeling project or define dynamic **slices** that automatically track specific subsets of your data over time, like "all images flagged for review."
* **Take targeted action on your data:** Finding data is only half the battle. Catalog is fully integrated with the rest of the Labelbox platform, allowing you to take immediate action on your findings. Select a group of data rows and, with a few clicks, you can add metadata, export them for analysis, or send them directly to a labeling project.

By providing a single, unified interface to explore and manage all your unstructured data, Catalog empowers you to make smarter, data-driven decisions throughout the entire model development lifecycle.

## Key concepts

To effectively navigate and use Catalog, it's important to understand its core components. These are the fundamental building blocks you'll encounter as you explore and manage your data.

| Term | Definition |
| - | - |
| Data rows | **What it is:** The most basic unit in Labelbox, representing a single item of your data. A *data row* is a pointer to a single data asset (e.g., an image, a video, a text file, a medical image) along with all its associated information, including metadata, attachments, and annotations.<br />**Why it's important:** Every action in Catalog—from filtering to labeling—is performed on one or more data rows. |
| Datasets | **What it is:** A *dataset* is a top-level container that holds a collection of data rows, often grouped by a specific project, data source, or collection period. You organize your data into datasets when you upload it to Labelbox.<br />Why it's important: Datasets provide the initial organization for your data and serve as the starting point for exploration in Catalog. |
| Slices | **What it is:** A *slice* is a dynamic, saved query that represents a subset of your data. Think of it as a "smart folder" or a saved search that always stays up-to-date.<br />**Why it's important:** Slices let you continuously monitor specific subsets of your data without re-applying filters. When new data is added to the dataset that matches the slice's filters, it automatically appears in the slice. This is perfect for tracking data quality issues or monitoring for specific edge cases. |
| Batches | **What it is:** A *batch* is a static, fixed group of data rows that you can send to a labeling project.<br />**Why it's important:** Batches are the primary mechanism for queuing up work for human labelers. Once created, a batch does not change unless you manually add or remove items, providing a stable workload for your labeling projects. |
| Embeddings | \*\*What it is: \*\**Embeddings* are powerful numerical representations (vectors) of your data that capture its semantic meaning. They are the engine behind Catalog's AI-powered search features. Labelbox can generate these for you, or you can provide your own.<br />**Why it's important:** Embeddings allow you to search for data based on meaning and similarity, not just metadata. They power features like **similarity search** ("find more images like this one") and **natural language search** ("find images of a dog playing in a park"), making it possible to find relevant data in a more intuitive and powerful way. |

## Gallery view

The gallery view is the default, providing a visual grid of your data. It's designed for rapid visual scanning and exploration.

* **Best for:** Getting a high-level visual sense of your dataset, spotting visual outliers, discovering trends, and making quick selections based on what the data looks like.
* **How to use it:** Simply scroll through the grid to explore. You can click on any data row to open the detailed view or use `Shift + Click` to select a range of items.

## List view

The list view organizes your data rows in a familiar table format, emphasizing metadata over visual appearance.

* **Best for:** Analyzing and sorting your data by specific metadata attributes, comparing values across data rows, and finding data with specific metadata characteristics.
* **How to use it:**
  1. Switch to the list view using the view-selector icon.
  2. Click on any column header to sort the data by that attribute.
  3. Use the **Manage columns** button to customize which metadata fields are displayed in the table, tailoring the view to your specific task.

## Analytics view

The analytics view provides interactive charts and graphs that summarize the distribution of data in your current selection.

* **Best for:** Understanding the composition of your dataset, identifying class imbalances, and seeing how your data is distributed across different metadata values.
* **How to use it:**
  1. Switch to the analytics view using the view-selector icon.
  2. The view will display histograms and charts for your data's attributes.
  3. You can click on a bar in any chart (e.g., a specific annotation class) to automatically filter your dataset down to only the data rows with that attribute.

## Detailed view

The detailed view allows you to perform a deep dive into a single data row.

* **Best for:** Inspecting an individual data asset, viewing all of its associated metadata, examining its annotations, and accessing all related information in one place.
* **How to use it:** Simply click on any data row from the gallery, list, or cluster view to open the detailed view. Here you can see the asset itself, edit metadata, view annotation history, and more.

The display panel allows you to customize these display settings:

| Display setting | Description |
| - | - |
| Pin data row details | Pins key details in the gallery view. |
| Black color fill image | Replace the data row with black pixels. This is useful to see annotations and predictions stand out |
| Show objects | Show the objects in the preview. Supported for predictions and ground truth. |
| Show segmentation masks | Show segmentation masks in the preview. Supported for predictions and ground truth. |
| Show classifications | Show classifications in the preview. Supported for predictions and ground truth. |
| Conversational text formatting - Raw | Display conversational text data rows in raw, unformatted version. |
| Conversational text formatting - Markdown | Display conversational text data rows in markdown. |
| Segmentation colors: Agree, Disagree | When **Show segmentation masks** is toggled ON, display the agree/disagree in different colors. |


# Export model run data
Source: https://docs.labelbox.com/horizon/guides/export-data-for-model-training

Instructions for exporting model run data from the app UI to train a model in your desired computing environment.

Labelbox offers three primary methods for exporting your data: from the Catalog, from a labeling project, or directly from a model run. *This guide specifically focuses on how to export data from a model run.*

Exporting model run data is typically done when you want to train a machine learning model in your own custom computing environment outside of Labelbox. By exporting this data, you extract your assets (such as images, video, text, documents, audio, geospatial imagery, or HTML) along with their corresponding annotations so your external models can learn from them.

## How to export your model run data

To export annotations from a model run via the Labelbox user interface (UI), follow these initial navigation steps:

1. Go to the **Model** tab and open the **Custom** subtab.
2. Select your desired model from this view.
3. Click on the **Model runs** tab and select a specific model run.

<Tip>
  Tip: You can use filters to query data rows based on their label status, metadata, batch, annotations, and workflow history. Excluding unneeded fields from your export will make the export process faster and the resulting file much smaller.
</Tip>

Once you have opened your model run, you have four options for choosing which data rows to export:

### Option 1: Export all data rows

If you want to export the entirety of the data in the current model run, locate and click on **All data** (this will also display the total number of data rows). From the drop-down menu that appears, click **Export data**.

### Option 2: Export a specific split or slice

In machine learning, data is often divided into "splits" (**Training**, **Validation**, **Test**, or **Unlabeled**) to train and evaluate models, or categorized into custom "slices."

To export one of these specific groups, look to the left side of the Model Run page and click on your desired **Slice** or **Split**. Next, click on the slice or split name at the top of the data view, and select **Export data** from the drop-down menu.

### Option 3: Export filtered data rows

If you only need data that meets highly specific criteria, you can build custom queries using the **Model filters**. Once your filtered results appear:

1. Click **Select all** on the right side of the screen.
2. Click the **Manage selection** dropdown.
3. Select **Export data** to export those specific rows.

### Option 4: Export hand-selected data rows

For the most granular control, you can manually select specific data rows using the checkboxes next to each row. After you have checked the boxes for your desired data:

1. Click the **Manage selection** dropdown.
2. Select **Export data**.

## Downloading your exported file

After you trigger the export (for example, by clicking **Export JSON**), a notification banner will appear letting you know the job has started.

You can track the progress of your export in the Labelbox **Notifications center**. Once the system finishes processing the job, simply click the **Download** link in your notifications to save the export file to your computer.

## Export specifications

Click through these links to view the export specifications based on data type.

<CardGroup>
  <Card title="Image" icon="image" href="/horizon/developer-guides/export-image-annotations" />

  <Card title="Video" icon="video" href="/horizon/developer-guides/export-video-annotations" />

  <Card title="Text" icon="file-lines" href="/horizon/developer-guides/export-text-annotations" />

  <Card title="Documents" icon="file-pdf" href="/horizon/developer-guides/export-document-annotations" />

  <Card title="Geospatial/tiled imagery" icon="map" href="/horizon/developer-guides/export-geospatial-annotations" />

  <Card title="Audio" icon="microphone" href="/horizon/developer-guides/export-audio-annotations" />

  <Card title="Conversational text" icon="comment" href="/horizon/developer-guides/export-conversational-text-annotations" />

  <Card title="HTML" icon="code" href="/horizon/developer-guides/export-html-annotations" />
</CardGroup>


# Export data from Catalog
Source: https://docs.labelbox.com/horizon/guides/export-from-catalog

Instructions for exporting your data from the Labelbox Catalog UI.

Labelbox offers three main ways to export your data: from the Catalog, from a model run, or from a labeling project. *This guide focuses specifically on how to select and export data directly from your Catalog.*

Exporting from the Catalog is highly flexible. It allows you to use various filters (like similarity, natural language search, annotations, and metadata) to pinpoint exactly the data rows you need. Additionally, Catalog exports give you the powerful option to bundle in associated project labels and model run predictions alongside your data.

## How to export data from Catalog

Follow these steps to export your data rows from the Catalog interface.

<Tip>
  Tip: Only include the optional fields you truly need. Excluding unnecessary fields will make your export process significantly faster and keep your resulting file size smaller.
</Tip>

### Step 1: Select data rows

1. Navigate to the **Catalog** tab in Labelbox.
2. Apply one or more filters to narrow down the specific data rows you want to export.
3. Click **Select all** in the top-right corner of your screen.
4. Open the dropdown menu under **Manage selection** (this will display the number of data rows you selected) and click **Export data**.

### Step 2: Select fields to include in your export

Choose any optional data fields you wish to include in your final export file. If you are unsure what a specific field means, you can refer to the [Labelbox Export Glossary](/horizon/developer-guides/export-glossary) for definitions.

### Step 3: Include project data (optional)

If you need project-specific labels included in your export file:

1. Select the **Export labels from project** option.
2. A dropdown will appear, allowing you to **select projects**. *(Note: Only projects where at least one of your selected data rows has been labeled will appear in this list).*
3. For the projects you select, all associated labels will be included in the resulting NDJSON export file for each respective data row.

### Step 4: Include model run data (optional)

If you need model predictions and labels included in your export:

1. Select the **Export labels and predictions from model runs** option.
2. A dropdown will appear for you to **select model runs**. *(Note: Only model runs that contain your selected data rows will appear here).*
3. For the selected model runs, all labels and predictions will be included in the resulting NDJSON export file for each respective data row.

## How to specify your data row source

Before you export, you need to decide where your data rows are coming from. Depending on how you want to curate your data, you can choose from one of the four source options below:

### Option 1: Export from all datasets

1. Select **All datasets** in the top-left corner of the Catalog.
2. Apply your desired filter or combination of filters to search across everything.
3. Click **Select all** in the top-right corner.
4. Click **Manage selection**, choose **Export data** from the dropdown menu, and select your optional fields.

### Option 2: Export from a single dataset

1. Select a specific dataset from the list of datasets on the left-side menu.
2. Apply your desired filters to narrow the data down further within that dataset.
3. Click **Select all** in the top-right corner.
4. Click **Manage selection**, choose **Export data** from the dropdown menu, and select your optional fields.

### Option 3: Export from a slice

1. Toggle to **Slices** on the left-side menu.
2. Select an existing slice you want to work with.
3. Modify or add to the filters that make up the slice, if needed.
4. Click **Select all** in the top-right corner.
5. Click **Manage selection**, choose **Export data** from the dropdown menu, and select your optional fields.

### Option 4: Export specific, hand-picked data rows

1. Manually select the specific data rows you want to export by checking the boxes in the top-left corner of each data row's thumbnail.
2. Click **Manage selection** in the top-right corner.
3. Choose **Export data** from the dropdown menu, and select any optional fields you wish to include.

## Export specifications

Click through these links to view the export specifications based on data type.

<CardGroup>
  <Card title="Image" icon="image" href="/horizon/developer-guides/export-image-annotations" />

  <Card title="Video" icon="video" href="/horizon/developer-guides/export-video-annotations" />

  <Card title="Text" icon="file-lines" href="/horizon/developer-guides/export-text-annotations" />

  <Card title="Documents" icon="file-pdf" href="/horizon/developer-guides/export-document-annotations" />

  <Card title="Geospatial/tiled imagery" icon="map" href="/horizon/developer-guides/export-geospatial-annotations" />

  <Card title="Audio" icon="microphone" href="/horizon/developer-guides/export-audio-annotations" />

  <Card title="Conversational text" icon="comment" href="/horizon/developer-guides/export-conversational-text-annotations" />

  <Card title="HTML" icon="code" href="/horizon/developer-guides/export-html-annotations" />
</CardGroup>

## Developer guide

To learn how to export data from Catalog programmatically, visit this developer guide.

<CardGroup>
  <Card title="Export data from catalog" icon="file-export" href="/horizon/developer-guides/export-overview#export-data-rows-and-labels-from-catalog" />
</CardGroup>


# Export labels from project
Source: https://docs.labelbox.com/horizon/guides/export-labels

Instructions for exporting your annotations from a labeling project via the UI.

Once you and your team have labeled your data in Labelbox, you will likely need to export those labels to train your machine learning models, run analytics, or simply keep secure backups.

While Labelbox allows you to export data from multiple places (like the Catalog or a model run), this guide focuses specifically on how to export your annotations directly from a **labeling project** using the Labelbox user interface.

## Step 1: Select data rows for export

To get started, you must select a project that has at least one label in the "Done" column. Navigate to **Annotate**, open your desired project, and go to the **Data Rows** tab.

From here, you have three flexible options for deciding exactly which data rows you want to include in your export file:

### Option 1: Export all data rows

If you want to download your entire project's dataset at once, you have two quick ways to do this:

* **The quick button:** Click **All (X data rows)**, then simply select **Export data** from the dropdown menu.
* **The checkbox method:** Click **All (X data rows)**, then check the box in the top-left corner of the table header to select all visible rows. Finally, click the **"X selected"** dropdown menu and choose **Export data**.

### Option 2: Quick filter by label status

Sometimes you only need labels that are in a specific stage of your workflow, such as ones that are fully "Done" or currently "In review."

1. Locate the **"Search your data"** field. Right next to it, select a specific label status. *Note: You can only filter by one status at a time.*
2. Once the table updates, manually check the boxes next to the rows you want to export, or check the box in the top-left of the table header to select them all.
3. Click the **"X selected"** menu and choose **Export data**.

### Option 3: Filter by specific data row attributes

For a highly customized export, you can build specific queries to find rows matching certain attributes—like a specific dataset, batch, or metadata value.

1. Click the **Search your data** field to reveal the advanced filter options.
2. Add your desired filters (e.g., *Dataset, Batch, Find text, Annotation, Metadata*) to narrow down your results.
3. Select the data rows from the filtered list that you want to include.
4. Click the **"X selected"** menu and choose **Export data**.

## Step 2: Choose your export fields and download

After narrowing down your data rows and clicking **Export data**, you get to customize exactly what information is packaged into your final file.

1. **Select your fields:** A panel will appear featuring date range queries and various data fields. Choose exactly which fields you want to include or exclude, or click **Select all** if you need everything. *(Tip: If you are unsure what a specific field means, consult the Labelbox export specifications).*
2. **Generate the export:** Once your fields are selected, click the **Export JSON** button. A notification banner will pop up to let you know your export job is processing in the background.
3. **Open notifications:** Hover over the bell icon at the bottom of your left navigation bar and click on **Notifications**.
4. **Download your file:** In the Notifications center, locate your export task (you can use filters if you have a lot of alerts). Click the **Download icon** to save the JSON file directly to your computer.

## Export specifications

Click through the links below to view the export specifications for each data type.

<CardGroup>
  <Card title="Image" icon="image" href="/horizon/developer-guides/export-image-annotations" />

  <Card title="Video" icon="video" href="/horizon/developer-guides/export-video-annotations" />

  <Card title="Text" icon="file-lines" href="/horizon/developer-guides/export-text-annotations" />

  <Card title="Documents" icon="file-pdf" href="/horizon/developer-guides/export-document-annotations" />

  <Card title="Geospatial/tiled imagery" icon="map" href="/horizon/developer-guides/export-geospatial-annotations" />

  <Card title="Audio" icon="microphone" href="/horizon/developer-guides/export-audio-annotations" />

  <Card title="Conversational text" icon="comment" href="/horizon/developer-guides/export-conversational-text-annotations" />

  <Card title="HTML" icon="code" href="/horizon/developer-guides/export-html-annotations" />
</CardGroup>


# Export model predictions
Source: https://docs.labelbox.com/horizon/guides/export-predictions

Describes how to export Foundry predictions and the options you can choose while doing so.

Exporting predictions from Foundry allows you to use your model's output outside of the Labelbox platform. This is useful for a variety of tasks, such as:

* **Offline analysis:** Analyze model performance in a Jupyter notebook, Excel, or other data analysis tools.
* **Integration with other systems:** Feed predictions into downstream applications or custom MLOps pipelines.
* **Custom reporting:** Create detailed reports and visualizations on model accuracy and behavior.

You can start an export from three main areas in the Labelbox platform, depending on what you need:

| Location | Use case |
| - | - |
| Catalog | Best for when you want to search for and select specific data rows that have predictions you want to export. |
| Model | Ideal for when you need to export predictions from a particular model run. |
| Annotate | Use this when you want to export predictions for data rows that are part of a specific project. |

## Step-by-step instructions

Follow these simple steps to export your predictions:

### Step 1: Select your data

First, navigate to the **Catalog**, **Model**, or **Annotate** section and select the data rows with the predictions you want to export.

### Step 2: Start the export

From the **Manage selection** menu that appears, choose **Export data**. This will open the **Export** panel where you can customize your export.

### Step 3: Choose your export options

In the **Export** panel, you can select what information you want to include in your export file. Depending on where you export your data from (i.e., Catalog, Annotate, Model), you can choose to include these details in your export:

* Data row details
* Metadata
* Attachments
* MMC code executions
* Project details
* Performance details
* Label details
* Interpolated frames
* **Model run details**
* **Predictions**
* Embeddings
* Model type override
* Export labels from projects
* **Export labels and predictions from model runs**

### Step 4: Generate the export file

Once you've selected your options, click **Export JSON**. Labelbox will then start building your export file.

### Step 5: Download your results

When your export is ready, you'll see a **Download** link in the **Notifications Center**. Clicking this will give you two options for downloading your results:

| Download method | Description |
| - | - |
| Browser download | This is the quickest way to get your file. <br />Click the **Download** button to save the results directly to your computer. **Important:** Don't close your browser until the download is complete, or the file will be incomplete. |
| Python SDK script | For large datasets, it's better to download using a Python script. The download panel will provide you with a sample script you can copy. This allows the download to happen in the background, so you don't have to worry about closing your browser. You'll need a Labelbox API key to run the script.<br />Click the **Code sample** button to view a sample Python script to download your export results based on the selected options. |


# Find text
Source: https://docs.labelbox.com/horizon/guides/find-text

Search for text and uncover assets in Catalog.

When working with text data such as [text](/horizon/developer-guides/import-text-annotations), [conversational text](/horizon/developer-guides/export-conversational-text-annotations), [documents](/horizon/developer-guides/import-document-annotations), or [HTML](/horizon/guides/html-editor), you can use the **find text** filter in Catalog to surface data rows that contain specific keywords.

## How the find text filter works

When you use the **find text** filter, you are allowed to specify multiple keywords in the expression. Labelbox will then surface data rows that contain those words.

<Info>
  ### Character/word limits

  To view the minimum & maximum characters/words for the *find text* filter, visit our [limits](/horizon/guides/limits) page.
</Info>

If multiple words are provided in the expression, Labelbox will surface data rows in this order:

1. Data rows containing all words
2. Data rows containing all words except one
3. Data rows containing all words except two

... and so on until all results containing at least one of the provided words are returned.

## Supported media types

The filter is supported for the following data types:

| Media type | Limit |
| - | - |
| [Text](/horizon/guides/text-editor) | First 64k characters |
| [Conversational text](/horizon/guides/conversational-editor) | First 64k characters |
| [HTML](/horizon/guides/html-editor) | First 64k characters |
| [Documents](/horizon/guides/document-editor) | First 64k characters |

The limits above indicate the last character in the media file that the filter will reach. For example, with the current limits, if the keyword you are searching for appears after the 64,000th character in the media file, the data row will not appear in the filter results.

## Combine find text with other filters

You can combine the **find text** filter with other filters in Catalog. Some filters are best used for targeting *unstructured* data and others are best for targeting *structured* data.

Combine the **find text** with the following filters to target data rows by structured data:

<CardGroup>
  <Card title="Metadata" href="/horizon/developer-guides/metadata" icon="square-1" />

  <Card title="Annotations" href="/horizon/guides/annotate-overview" icon="square-2" />

  <Card title="Datasets" href="/horizon/developer-guides/dataset" icon="square-3" />

  <Card title="Projects" href="/horizon/developer-guides/project" icon="square-4" />
</CardGroup>

Combine **find text** with the following filters to target data rows by unstructured data:

<CardGroup>
  <Card title="Similarity search" href="/horizon/guides/similarity" icon="square-1" />

  <Card title="Natural language search" href="/horizon/guides/natural-language-search" icon="square-2" />
</CardGroup>

## Automate data curation with slices

After you populate the Catalog filters, you can save these filters as a [slice](/horizon/guides/slices) of data. When you save a filter as a slice, you will not need to populate the same filters over and over again. Also, slices are dynamic, so any new incoming data rows will automatically appear in the relevant slices.

Read through the following resources to learn how to take action on the filtered data.

<CardGroup>
  <Card title="Refine the similarity search" href="/horizon/guides/similarity#adding-anchors" icon="square-1" />

  <Card title="Send filtered data rows to a labeling project as a batch" href="/horizon/guides/batches" icon="square-2" />

  <Card title="Add metadata to the filtered data rows" href="/horizon/guides/datarow-metadata" icon="square-3" />
</CardGroup>


# Labelbox Foundry
Source: https://docs.labelbox.com/horizon/guides/foundry



Welcome to Labelbox Foundry, your integrated environment for leveraging foundation models and custom AI to accelerate your data labeling and model development workflows. Foundry is designed to help you move from raw, unlabeled data to high-quality training data with speed and efficiency.

With Foundry, you can:

* **Generate high-quality predictions:** Use state-of-the-art foundation models to automatically generate labels for your data.
* **Compare model performance:** Run multiple models on the same data to evaluate and compare their performance, helping you choose the best model for your use case.
* **Automate labeling workflows:** Set up Foundry Apps to create repeatable, automated labeling pipelines that can be triggered from the Catalog or via API.
* **Active learning workflows:** Identify the most valuable data to label by finding where your model is least confident, and send those assets directly to a human labeling project.
* **Integrate custom models:** Bring your own models into the Labelbox ecosystem to take advantage of Foundry's powerful workflow tools.

Foundry helps you close the gap between your model's capabilities and your production needs, streamlining the path to building better AI.

## Key concepts

| Concept | Description |
| - | - |
| Model run | The core process in Foundry. A model run is a single execution of a model on a selected batch of data rows. It generates predictions for each data row. |
| Foundry app | A saved model run configuration. Foundry Apps allow you to save a specific model and its settings so you can easily and repeatedly run it on new batches of data, automating your workflows. |
| Predictions | The output generated by a model run. Predictions are the labels or values that the model "predicts" for your data (e.g., bounding boxes, classifications). |
| Ontology | The schema that defines the features your model can predict. You will map the model's output to your project's ontology to ensure consistency. |

## Before you begin

To ensure a smooth experience with Foundry, please complete the following setup steps:

1. [**Connect your cloud data**](/horizon/guides/connect-to-cloud-storage)**:** Foundry works directly with your data stored in the cloud. Make sure you have successfully connected your AWS, GCP, or Azure cloud storage to Labelbox.
2. [**Create a project**](/horizon/guides/create-a-project)**:** You'll need a Labelbox project set up with a defined ontology. This project will be the destination for predictions that need human review.
3. **Select your data:** In the Labelbox Catalog, identify and select the data rows you wish to use for your model run.

## Add Foundry to your workspace

Foundry is available for all plans except for Educational subscriptions.

* **Self-Service (Free, Starter):** Go to your workspace settings to enable Foundry as an add-on. You will be prompted to agree to the terms and add a payment method if needed.
* **Enterprise:** Please contact your account manager to enable Foundry.

### How billing works

Foundry billing has two components: Inference Costs and Labelbox Units (LBUs).

| Cost type | Description |
| - | - |
| Inference costs | This is a direct cost for using a Labelbox-hosted model, charged in USD. It varies by model and the amount of data processed. These fees are passed on to you and charged immediately after a model run. |
| Labelbox units (LBUs) | Your Labelbox subscription includes a set number of LBUs. Foundry consumes LBUs when you run a model and when you send predictions to a labeling project. This usage is deducted from your subscription's LBU balance. |

For example:

1. Suppose you import 1,000 images into Catalog; this consumes 17 LBUs.
2. You select 500 images and use a Foundry model run to generate predictions. Based on the model you selected and the parameters of your model run, this generates a \$2.00 inference fee.
3. When the Foundry model run is complete, 500 images and their predictions are now available in Model. This consumes 100 LBUs.
4. To verify the predictions, you send them to Annotate for human review. This consumes another 500 LBUs.

Overall, you've generated \$2 in inference costs (Step 2); this is charged immediately. You've also used 600 LBUs as one time charges (Model and Annotate) and generate a 17 LBU charge each month your data remains in **Catalog**. The LBU consumption is charged against the terms of your subscription at the end of the current billing cycle.

Compute fees depend on the specific model used, the amount of data processed, and other factors. For details, consult the model card. To learn more about LBUs, see [Labelbox Units (LBUs)](https://docs.labelbox.com/horizon/guides/billing#labelbox-units-lbus).

You can review the specific costs for any model by selecting it in the Model gallery and viewing the **Pricing** details on its overview tab. The total cost of any completed model run can be found in the run's details.

## Remove Foundry from workspace

You can unsubscribe to Foundry and remove it from your Labelbox subscription at any time.

To do so:

1. Select **Workspace Settings** from the Labelbox main menu to open the **Organization Settings**.
2. From the **Billing tab**, locate the **Add-ons** section and then select **Remove**.
3. A confirmation prompt asks you to confirm your request. Select **Unsubscribe** to do so.

Removing Foundry from your subscription doesn't affect your data. Any predictions created during the subscription remain.


# Create a Foundry app
Source: https://docs.labelbox.com/horizon/guides/foundry-apps

Foundry apps help you automate Foundry workflows. Learn how to create and manage Foundry apps.

As you move from experimenting with models to building production-grade AI systems, you need tools that provide consistency, scalability, and automation. **Foundry Apps** are the key to achieving this in Labelbox.

A Foundry App is more than just a saved model; it's a *saved, reusable, and executable workflow configuration*. It captures the entire logic of a model run—the specific model, the ontology mapping, the confidence threshold, and naming conventions—and packages it into a single, on-demand tool.

## Create a Foundry app

There are two main ways to create a Foundry app in Labelbox:

* **From the Model section**: This is the most direct way to create a new app.
* **From a model run configuration**: If you've already configured a model run that you'd like to reuse, you can save it as an app.

### Option 1: Create an app from the Model tab

1. Sign in to Labelbox and select **Model** from the main menu.
2. In the Model gallery, click the **Create** menu and select **App**.
3. Choose the foundational model you want to use for your app.
4. Give your app a name and an optional description, then click **Proceed**.
5. Define the parameters for your model run. If you have a previous model run with settings you'd like to reuse, you can use the **Load model run config** option.
6. Optionally, you can select up to ten data rows to preview how the app will work. To do this, click **Select data for preview**, choose your data rows, and then click **Import selected**.
7. When you're finished, click **Save & Create**.

### Option 2: Save a model run configuration as an app

1. Start a regular Foundry model run by selecting your data rows and a model.
2. Configure your model run settings as you normally would.
3. Before submitting the model run, click the **Save as App** button, which appears near the model's name.
4. You'll be prompted to enter a name and description for your new app.
5. Click **Proceed** to create the app.

<Note>
  The **Save as App** button will not be displayed for models that are not compatible with Foundry apps.
</Note>

## Update a Foundry app

To make changes to an existing Foundry app:

1. Sign in to Labelbox and go to the **Model** section.
2. Click on the **Apps** tab and select the app you want to update.
3. Adjust the app's settings as needed.
4. When you're done, click **Save & Update**.

## Delete a Foundry app

If you no longer need a Foundry app, you can delete it. Please be careful, as this action cannot be undone.

1. Sign in to the Labelbox app and go to the **Model** section.
2. From the **Apps** tab, select the app you want to delete.
3. Go to the **Settings** tab and click the **Delete app** button.
4. A confirmation prompt will appear. Type the word "delete" and then click **Delete**.

<Warning>
  Use care when deleting apps. Deleted apps cannot be recovered or restored.
</Warning>


# Create pre-labels with Foundry model-assisted labeling
Source: https://docs.labelbox.com/horizon/guides/foundry-model-assisted-labeling

Use an off-the-shelf Foundry model to pre-label your data rows in your project.

Model-assisted labeling is a feature in Labelbox that uses AI to automatically generate labels on your data. This can help you label your data faster and more accurately.

Here's how it works:

* **You provide a model**: You can choose from a list of pre-trained models in Labelbox or bring your own.
* **The model predicts labels**: The model will analyze your data and suggest labels based on what it has learned.
* **You review and approve**: You can then review the suggested labels and make any necessary corrections.

This process can significantly speed up your labeling workflow, especially for large datasets.

## How to enable model assisted labeling

Here's a step-by-step guide to using model-assisted labeling in your Labelbox project.

1. **Set up a new project**: To get started, create a new project in Labelbox. Make sure your project has an ontology and a batch of data that is compatible with Foundry. Once your project is set up, the **Model assisted labeling** button next to **Start labeling** will be enabled.
2. **Choose your model**: Click on the **Model assisted labeling** button. You will be taken to a page where you can choose the model you want to use. Labelbox offers a variety of models to choose from. The models at the top of the list are the easiest to set up, but you can also use one of the other models if you prefer.
3. **Configure your model**: After you have selected a model, you will need to configure it. This involves two main steps:
   * **Align the model with your ontology**: This means mapping the labels that the model can predict to the labels in your project's ontology.
   * **Set the confidence threshold:** This is a value between 0 and 1 that tells the model how certain it needs to be before it suggests a label. A higher confidence threshold will result in fewer, but more accurate, suggestions.

<Note>
  If your ontology is open-ended, you can also provide your own criteria to the model, similar to prompt engineering.
</Note>

4. **Preview the selected model**: Before you apply the model to your entire dataset, you can generate a preview to see how it performs on a small sample of your data. This is a great way to test your model and make sure it is performing as you expect. During the preview, you can:
   * **Adjust the confidence threshold** to find the optimal setting for your use case.
   * **Highlight annotations** to see which tool was used and the model's confidence for each annotation. If the model is not performing well, you can go back and choose a different model. If you are happy with the results, click **Submit**.
5. **Submit the model run**: After you submit the model, it will start running on your data. You can track the progress of the model in the notification tab at the bottom left of the page. When the model has finished running, you will see a confirmation in the Notifications tab and the results will be shown in your project.
6. **Review the pre-labels**: Once the model has finished running, the suggested annotations will be superimposed on your data rows. You can then review the suggestions and either submit them as they are or remove them and create your own annotations.


# HTML
Source: https://docs.labelbox.com/horizon/guides/html-editor

A guide for using the native HTML editor to label data.

When creating a project, select HTML.

<br />

You can use the native HTML editor to customize the way your data rows render in the labeling interface. You can use this guide to understand how the HTML editor works, how to set up an HTML labeling project, and common configurations for the HTML editor.

## Import HTML data

To learn how to prepare and format your HTML data for importing to Labelbox, see [Import HTML data](/horizon/developer-guides/import-html-annotations). You have two options for uploading to Labelbox:

1. Store your HTML files in your cloud storage and send Labelbox a JSON file containing URL links to the HTML files (recommended)
2. Upload the HTML files directly to Labelbox

<Note>
  Note: To label a webpage, you must save it as an HTML file. Direct linking to web pages via a URL is not supported.
</Note>

Please take a look at the Common HTML editor configurations section below to explore ways to configure your import file to render your data in the editor.

## Supported annotation types

Create (or reuse) an ontology for your labeling project. The table below shows the annotation types that are supported for labeling HTML data. Click through the links in the table to view the import and export formats for each classification type. Note: You can use [Model-assisted labeling](/horizon/guides/model-assisted-labeling) to import classifications as prelabels to your HTML data rows.

| Feature | Import format | Export format |
| - | - | - |
| **Radio classification** | [See payload](/horizon/developer-guides/import-html-annotations#classification-radio-single-choice) | [See payload](/horizon/developer-guides/export-html-annotations#classification---radio) |
| **Checklist classification** | [See payload](/horizon/developer-guides/import-html-annotations#classification-checklist-multi-choice) | [See payload](/horizon/developer-guides/export-html-annotations#classification---checklist) |
| **Free-form text classification** | [See payload](/horizon/developer-guides/import-html-annotations#classification-free-form-text) | [See payload](/horizon/developer-guides/export-html-annotations#classification---free-form-text) |

Note: HTML files are rendered in Labelbox as an iframe with the sandbox attribute. This means that scripts, pop-ups, and third-party API calls are blocked.

## Custom HTML editor configurations

You can leverage the HTML format to customize the way your data appears in the labeling interface. Here are some common configurations our customers set up using the HTML editor.

### Compare and rank products

Add CSS to your HTML file to display objects side-by-side in a grid container. This is helpful for ranking or comparing multiple products in the labeling interface. To do this, ensure your CSS display property is set to `grid`.


# Hubstaff timer
Source: https://docs.labelbox.com/horizon/guides/hubstaff-timer

This guide explains how to configure your Hubstaff app for labeling.

Use the instructions below as a guide for navigating the Hubstaff timer for your labeling tasks.

## Step 1:  Create or join your Hubstaff account

Follow these steps to set up your Hubstaff account and install the desktop app.

1. You should have received an email containing an invitation to Hubstaff. Click the link in the email to create your Hubstaff account (You can also set up your Hubstaff account via the onboarding checklist in your Alignerr app dashboard).
2. If you already have a Hubstaff account, the Labelbox team will add you to our organization. You'll receive an email inviting you to join the Alignerr organization in Hubstaff.

## Step 2: Install the Hubstaff desktop app

1. Download and install the Hubstaff desktop app on your computer.
2. Sign in using the credentials from the invitation email.

## Step 3: Open your project in Labelbox

1. Go to your assigned project in Labelbox.
2. Review the labeling instructions.
3. In the project overview tab, click the **Start timer** button. This will open the Hubstaff browser extension (you will need to install the Hubstaff browser extension, if you haven't already).
4. In the pop-up window, select **Always allow** and click **Open Hubstaff**.
5. When the Labelbox app has detected that the Hubstaff timer is enabled, you will be able to start labeling

## Step 4: Start labeling

1. Click **Start working** to open the labeling task.

### Case 1: Labeling in an external platform

1. When you click **Start working**, you may be directed to another platform to complete your labeling task.
2. Labelbox will automatically detect that you have opened the labeling task and will start the Hubstaff timer.
3. Complete the labeling tasks.
4. When you are done labeling, return to the Labelbox app and click **Stop timer**.

### Case 2: Labeling in the Labelbox platform

1. When you click **Start working**, you may be directed to the Labelbox editor to complete the Labeling task.
2. Before you can begin labeling, you’ll be prompted to acknowledge that two timers are running (Labelbox & Hubstaff) and that all work will be paid according to the Labelbox timer.
3. If the Hubstaff timer is not running, the submit button in the Labelbox app will be disabled. You will see a prompt asking you to confirm when the Hubstaff timer is enabled.
4. Complete the labeling tasks.
5. When you are done labeling, you do not need to explicitly start or stop any timer if you are labeling in the Labelbox editor. The app will start and stop your timer automatically.
6. When you have completed the task, submit your work in the Labelbox editor.


# Images
Source: https://docs.labelbox.com/horizon/guides/image-editor

Guide for labeling image data.

The image editor lets you label images with various supported annotation types, attachments, and editor settings.

## Set up image annotation projects

To set up an image annotation project:

1. [Create an image dataset](/horizon/guides/datasets-datarows) and load the images you want to label. Alternatively, use the SDK to [import images](/horizon/developer-guides/image).
2. On the [Annotate projects page](https://app.labelbox.com/projects), click the **+ New project** button.
3. Select **Image**. Add a **Name** and an optional **Description** for your project.
4. Click **Save**. The system then creates the project and redirects you to the project overview page.
5. Click **Add data**. Then select your image dataset. Click **Sample** to sample your dataset, or you can manually select data rows and click **Queue batch**.

<Info>
  ### Data row size limit

  To view the maximum size allowed for a data row, see [limits](/horizon/guides/limits).
</Info>

## Set up an ontology

After setting up an audio annotation project, you can [add an ontology](/horizon/guides/labelbox-ontology#create-a-new-ontology) based on how you want to label the data. The image editor supports the following annotation types that you can include in your ontology:

| Feature | Import annotation | Export annotation |
| - | - | - |
| **Bounding box** | [See payload](/horizon/developer-guides/import-image-annotations#bounding-box) | [See payload](/horizon/developer-guides/export-image-annotations#bounding-box) |
| **Segmentation mask** | [See payload](/horizon/developer-guides/import-image-annotations#segmentation-mask) | [See payload](/horizon/developer-guides/export-image-annotations#mask) |
| **Polygon** | [See payload](/horizon/developer-guides/import-image-annotations#polygon) | [See payload](/horizon/developer-guides/export-image-annotations#polygon) |
| **Polyline** | [See payload](/horizon/developer-guides/import-image-annotations#polyline) | [See payload](/horizon/developer-guides/export-image-annotations#polyline) |
| **Point** | [See payload](/horizon/developer-guides/import-image-annotations#point) | [See payload](/horizon/developer-guides/export-image-annotations#point) |
| **Relationship** | [See payload](/horizon/developer-guides/import-image-annotations#relationship-with-bounding-box) | [See payload](/horizon/developer-guides/export-image-annotations#relationship) |
| **Radio classification** | [See payload](/horizon/developer-guides/import-image-annotations#classification-radio-single-choice) | [See payload](/horizon/developer-guides/export-image-annotations#classification---radio) |
| **Checklist classification** | [See payload](/horizon/developer-guides/import-image-annotations#classification-checklist-multi-choice) | [See payload](/horizon/developer-guides/export-image-annotations#classification---checklist) |
| **Free-form text classification** | [See payload](/horizon/developer-guides/import-image-annotations#classification-free-form-text) | [See payload](/horizon/developer-guides/export-image-annotations#classification---free-form-text) |

<Info>
  ### Classification scopes

  Classification annotations can be applied globally or nested within an object-type annotation.
</Info>

### Audio-to-text for text classification

You can enable free-text classification to allow labelers to record audio instead of typing text manually. The system transcribes the recorded audio into text using the integrated OpenAI Whisper model. To set up and use audio-to-text for text classification:

1. When [creating the ontology](/horizon/guides/labelbox-ontology#step-1-create), add a **Text** classification feature and enable **Record audio for transcription**.
2. In the editor, select the text classification annotation and click **Record audio**. Each recording can last up to 30 minutes and automatically stops when it reaches the limit.
3. Click **Stop recording** when you finish speaking. The system transcribes the audio into text and adds the recording as an attachment to the annotation.
4. Edit the transcribed text, remove the attached audio file, or click **Record audio** to re-record if needed.
5. Click **Submit** to save the annotation.

### Bounding box

Create a bounding box by starting at one corner and dragging your cursor to create the shape around an object in the image.

You can also click and drag to reposition the bounding box on the image.

### Cuboid

Create a cuboid by starting at one corner and dragging your cursor to create a box shape around an object in the image. Once you release it, it will automatically become a cuboid.

You can use the various levers on the tool to adjust its rotation along the x, y, and z axes. Along the center of the top bar, you will find buttons to switch to **Rotate** mode,**Move** mode, or **Scale** mode.

You can also directly input the rotation into the modal that appears in the ontology pane once the cuboid is initially drawn or selected for editing.

The fields available to edit are:

* **Scale:** This controls the cuboid size in each dimension. The unit is in pixels.
* **Rotate:** This controls the rotation of the cuboid along the three axes.

### Segmentation mask

The segmentation mask annotation supports different tools based on the ontology creation time.

<Warning>
  ### Empty mask error

  If you receive an error message that reads, “The object is missing selected pixels. You should relabel or delete this object”, it means that the mask is considered empty.

  By definition, a segmentation mask cannot have overlapping pixels. Thus, drawing a second mask fully within the first creates an empty one, resulting in this warning message.
</Warning>

#### AutoSegment 2.0

Ontologies created after June 13, 2023, by default, use a new *AutoSegment 2.0* tool powered by Meta's Segment Anything model.

You can use the AutoSegment 2.0 tool to generate mask predictions for individual objects in your image. The Auto Segment 2.0 tool is designed for instance segmentation, meaning you can quickly label individual objects as it predicts one class at a time. Adjusting the contrast and brightness of the image **does not** affect the accuracy of the SAM model.

When selecting a segmentation tool, choose the AutoSegment option from the top navigation bar. There are two modes available:

* **Box mode**: allows you to draw a box around an object to generate a mask on it.

* **Hover and Click mode**: allows you to visualize masks as you move the cursor around, click to generate, and further modify by adding positive and negative points.

#### AutoSegment (Legacy)

Only ontologies created before June 13, 2023 use this is the older version of AutoSegment.

When a segmentation tool is selected, choose the AutoSegment option from the top navigation bar, and draw a box around the object you want to label. For best results, draw the boundary of the box close to the object edges. You can also adjust the brightness and contrast settings in the **Adjustments** menu in the top navigation bar to further improve results. Creating more contrast between the object you are labeling and its background will help the model detect the object more accurately.

#### Brush tool

Ontologies created after June 13, 2023 can access the brush tool. Use the brush tool to draw a freehand mask as though you are painting the canvas. You can choose from a circle or square-shaped brush and adjust the size in pixels. Click the brush icon with the minus sign to use the eraser tool. It behaves similarly but removes mask pixels as you move the mouse.

#### Pen tool

Use the pen tool to outline the item in the image. Hold the cursor down to draw freehand or let the cursor go to draw straight lines between points. The pen tool is only available when creating segmentation annotations. Click the pen icon with the plus sign to use the pen tool.

#### Erase tool

You can use the erase tool to clean up the edges of a segmentation mask. Click the pen icon with the minus sign to use the erase tool.

#### Fill tool

You can use the fill tool to easily label backgrounds or, otherwise, assign a segmentation annotation to all other pixels in the image that have not already been labeled. The fill tool is only enabled for segmentation annotations.

Click the droplet icon to use the fill tool.

### Polygon

Create a polygon annotation by clicking to create each point in the shape. Click the first point to close the polygon.

### Polyline

Use the polyline tool to label lines in an image. Click on the last point to complete the shape.

### Point

Use the point tool to label precise locations on the image.

### Relationship

To create a relationship between annotations, select a relationship tool and hover over the annotation where you want the relationship to start to reveal the annotation's anchor points. Click an anchor point to create the starting point of the relationship, then bring your mouse over to the annotation you want to relate it to, hovering over it to reveal its anchor points. Finally, click one of the anchor points to complete the relationship.

Right-click a relationship to change its direction, make it bi-directional, or delete it from the asset.

## Editor settings

The image editor has the following settings for object-type annotations:

* **Overlay object titles**: Displays the names of annotation objects on the image.
* **Show annotations**: Displays created annotation objects on the image.
* **Enable annotations occlusion**: Allows annotation objects to overlap based on z-order (front objects occlude those behind them).
* **Polygon snapping**: Allows points on lines, polygons, and bounding boxes to align or attach to polygon edges.

### Setting object positions

When the **Enable annotations occlusion** setting is active, you can adjust the z-order of annotation objects to control how they stack visually. The **OBJECTS** panel provides two modes for managing object positions:

* **Universal arrangement**: Adjust the stacking order incrementally by selecting and repositioning individual objects.

* **List-based arrangement**: Use a drag-and-drop interface to reorder objects directly in a list.

<Warning>
  ### Z-order settings limit

  All object-type annotations have a z-order value that you can [export](/horizon/developer-guides/export-image-annotations#sample-project-export), but you can't change the z-order value for cuboids and segmentation masks because they can't occlude other objects.
</Warning>


# Connect AWS S3 to Labelbox via IAM Delegated Access
Source: https://docs.labelbox.com/horizon/guides/import-aws-s3-data

Step-by-step guide for importing your S3 bucket data to Labelbox via IAM delegated access.

This guide provides a complete set of step-by-step instructions for securely connecting your Amazon S3 bucket data to Labelbox using IAM delegated access.

## Prerequisites

Before you begin, ensure you have the following:

* You have permissions to create IAM roles and policies in your AWS account.
* You know the name of the S3 bucket you want to connect.
* You have configured [Cross-Origin Resource Sharing (CORS)](/horizon/guides/create-cors-headers) on your S3 bucket to allow Labelbox to request resources from your cloud storage.

## Step 1: Begin integration in Labelbox

First, you'll start the integration process in the Labelbox UI to obtain the necessary credentials.

1. In Labelbox, navigate to **Settings** > **Integrations**.
2. Under **Add integrations**, select **Cloud provider**.
3. Select **AWS** as your source.
4. From the **Create AWS integration page**, copy the Labelbox AWS account ID and the External ID.
5. Leave this page open in your browser: you will return to it in a later step.

## Step 2: Create a role for Labelbox in AWS

Next, you will create a role in your AWS account that Labelbox can assume to access your S3 bucket.

### Part A: Create a permission policy

1. In your AWS account, navigate to the **IAM Management Console > Policies** page.
2. Click **Create policy** and select the **JSON** policy editor.
3. Paste the following JSON policy, which grants read-only access to a specific S3 bucket.

```json theme={"theme":"css-variables"}
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::CustomerBucketARN/*",
                "arn:aws:s3:::CustomerBucketARN"
            ]
        }
    ]
}
```

<Note>
  Remember to replace `CustomerBucketARN`with the actual ARN of your S3 bucket.
</Note>

4. Add Add a name for the policy (for example, `LabelboxReadAccess`) and click **Create policy**.

### Part B: Create a role

1. From the **Roles** page in the IAM Management Console, click **Create role**.
2. Select **Custom trust policy** and paste the following policy. Be sure to enter the **External ID** you obtained from Labelbox in Step 1.

```json theme={"theme":"css-variables"}
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::340636424752:role/lb-aws-delegated-access-role"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": ""
                }
            }
        }
    ]
}
```

3. In the **Add Permissions** step, attach the permission policy you created in Part A (e.g., `LabelboxReadAccess`).
4. Add a name for the role (e.g., `LabelboxS3Access`) and click **Create role**.
5. Click on the role you just created and copy the **Role ARN** from the **Summary** tab.

## Step 3: Complete the integration setup in Labelbox

Now, you will add the Role ARN to the new integration you added in Labelbox in Step 1.

1. Go back to the **Create AWS integration** page in Labelbox.
2. In the **Provider ARN and name** section:
   * Set the **integration name**.
   * Enter the **AWS bucket name**.
   * Paste the **AWS Role ARN**.
3. Click **Save integration**.

## Step 4: Validate the integration

After you complete the setup in Labelbox, the system will automatically run a validation check on the integration. You can check the status on the **Integrations > Manage integrations** page. If the integration fails, you can click the refresh icon to view error messages and troubleshoot your setup.

Here are possible error messages and our suggestions for troubleshooting your integration setup.

| Error | Troubleshooting |
| :- | :- |
| *Role cannot be assumed* | Ensure that the integration’s role ARN is correct and that the Labelbox External ID is properly configured in your AWS account. Additionally, your AWS account admin must [activate STS](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_enable-regions.html) in the `us-east-2` region using the IAM console |
| *External ID configured insecurely* | Ensure that the Labelbox External ID is properly configured in your AWS account. |

## Step 5: Create, upload, and validate the dataset

Finally, you need to create and validate your dataset.

1. When creating your import file, use virtual-hosted-style URLs that follow this format: `https://<bucket-name>.s3.<region>.amazonaws.com/<key>`. To learn how to format your import file, visit these guides:

<Columns>
  <Column>
    <Card title="Import image data" icon="plus" href="/horizon/developer-guides/import-image-data" />

    <Card title="Import text data" icon="plus" href="/horizon/developer-guides/import-text-data" />

    <Card title="Import document data" icon="plus" href="/horizon/developer-guides/import-document-data" />

    <Card title="Import HTML data" icon="plus" href="/horizon/developer-guides/import-html-data" />
  </Column>

  <Column>
    <Card title="Import video data" icon="plus" href="/horizon/developer-guides/import-video-data" />

    <Card title="Import geospatial data" icon="plus" href="/horizon/developer-guides/import-geospatial-data" />

    <Card title="Import audio data" icon="plus" href="/horizon/developer-guides/import-audio-data" />
  </Column>
</Columns>

2. If you created your integration and imported your dataset using the Labelbox UI, Labelbox automatically runs validation checks to determine whether the CORS setup was configured properly. It also checks if Labelbox can successfully fetch data from your S3 bucket and properly sign the URLs.

Your dataset should now be set up with IAM delegated access. Labelbox will use the AWS role you created to generate temporary signed URLs every time it accesses data in your S3 bucket.


# Import errors & warnings
Source: https://docs.labelbox.com/horizon/guides/import-errors-and-warnings

Learn about the warnings and errors you may encounter when importing a dataset to Labelbox.

This guide explains the processing issues you may encounter when importing data and how to resolve them.

When data is ingested, Labelbox processes it to generate embeddings, extract media attributes, and standardize formats. If an issue occurs, it will be flagged as an error or a warning.

## Processing states

* **Processing**: The initial state when a data row is ingested.
* **Success**: The data row was processed successfully.
* **Failure**: The data row could not be processed due to an error or warning. It will not appear in the Catalog until the issue is fixed.

## View and fix failed data rows

1. **Find failures**: In Catalog, an issue icon will appear next to datasets with processing issues. Click this icon to go to the "Processing Issues" view.
2. **Filter issues**: You can filter the view to see data rows with specific errors or warnings.
3. **Fix the source issue**: Use the error descriptions below to diagnose the problem (e.g., fix a broken URL, correct cloud storage permissions).
4. **Re-process**: Once you've fixed the underlying issue, select the failed data rows in the "Processing Issues" view and click **Re-process**.

<Info>
  ### Large files

  Large files such as TIF, Video, and Audio can take a few seconds to process.
</Info>

## Common errors (failures)

An error is a serious issue that prevents a data row from being used.

| Error | Description & action |
| - | - |
| ConversionFailed | The file could not be converted to a format required for labeling. <br />**Action:** Ensure the file is not corrupt and meets format requirements. |
| FetchFailed | The data row URL could not be reached. <br />**Action:** Check that the URL is correct and that the Labelbox backend has access. |
| FetchTimeout | The data row could not be fetched due to a timeout error.<br />**Action**: Re-process the data rows. |
| Forbidden | Access to the URL was denied. <br />**Action:** Check your cloud storage permissions or pre-signed URL validity. |
| InternalError | An internal error prevented the data row from being processed.<br />**Action**: Re-process the data rows. |
| InprocessableContent | The data row could not be processed due to malformed or invalid content.<br />**Action**: Review the structure of your import file. |
| InvalidEspg | The data row has invalid ESPG metadata. |
| NotFound | The file does not exist at the specified URL. <br />**Action:** Verify the URL is correct. |
| TooManyBands | The geospatial data row has too many bands. |
| TooManyRequests | The data row could not be fetched due to a rate-limiting error. |

## Warnings

A warning indicates a non-critical issue. The data row can be used, but your experience may be degraded.

| Warning | Description & action |
| - | - |
| InvalidCors | The file is served with invalid CORS headers, which may impact the labeling experience. <br />**Action:** Check that the URL is being served with the correct CORS headers. |

<Warning>
  **Processing issues and the Python SDK**

  * If you export data rows from Catalog using the SDK, all data rows are exported, including those with processing issues.
  * If you send data rows to a labeling project using the SDK, all data rows are sent to the labeling project, including those with processing issues.
</Warning>


# Import ground truth annotations into a project
Source: https://docs.labelbox.com/horizon/guides/import-ground-truth

Learn how to import your ground truth data from internal or third-party tools into Labelbox.

This guide will walk you through the process of importing your existing ground truth annotations from other internal or third-party tools into Labelbox.

## What is ground truth?

In machine learning, "ground truth" refers to the data that we consider to be the "true" or accurate labels for a given dataset. This data is often created by human annotators and is used to train and evaluate machine learning models.

By importing your ground truth annotations into Labelbox, you can consolidate all of your labeled data in one place, creating a single source of truth for your machine learning projects. This is particularly useful when you are migrating from another labeling platform to Labelbox.

## How to import your ground truth annotations?

You can generate Python SDK code snippets directly from your project's "Import labels" tab to help you with the import process. These snippets will serve as a great starting point for the steps below.

1. **Import your data rows**: Before you can import your annotations, you need to have the corresponding data rows (the raw data you want to label, such as images, videos, or text) in Labelbox. If you haven't already, you'll need to [create a dataset](/horizon/guides/datasets-datarows) and import your data rows into the Catalog.
2. **Create or select an ontology**: An ontology (also known as a taxonomy) defines the set of classes and attributes that can be used to label your data. For example, an ontology for an image classification project might include classes like "cat," "dog," and "car." You'll need to either [create a new ontology](/horizon/guides/labelbox-ontology) or select an existing one that matches the annotations you want to import.
3. **Create a labeing project**: Your annotations need to be associated with a labeling project in Labelbox. If you don't already have a project set up with the correct ontology, you will need to [create a project](/horizon/guides/create-a-project).
4. **Send a batch of data rows to your project**: Now that your project and ontology are configured, you need to [send a batch of data rows](/horizon/guides/batches) to the project's labeling queue. This will be the specific set of data rows that you will be attaching your imported annotations to.
5. **Create the annotations payload**: This is where you'll prepare the annotations themselves for import. Each annotation in your payload needs to reference a specific annotation class from your ontology and a specific data row ID. Labelbox supports two formats for the annotation payload: **Python Annotation Types** and **NDJSON**. Go to the **Import labels** tab and select **Import ground truth**. There you will find a sample code snippet you can use to import your ground truth annotations.
6. **Import your annotation payload**: Once you have created your annotation payload, the final step is to submit the import job. After the job is complete, your ground truth annotations will appear in your project, and the corresponding data rows will be marked as "Done." Under **Import labels** → **Import ground truth**, copy the code snippet for **Import annotations payload**.

Check out these end-to-end developer guides to learn how to import ground truth annotations based on your data type.

<CardGroup>
  <Card title="Import image annotations" icon="image" href="/horizon/developer-guides/import-image-annotations#example-import-pre-labels-or-ground-truths" />

  <Card title="Import video annotations" icon="video" href="/horizon/developer-guides/import-video-annotations#example-import-pre-labels-or-ground-truths" />

  <Card title="Import text annotations" icon="file-lines" href="/horizon/developer-guides/import-text-annotations#example-import-pre-labels-or-ground-truths" />

  <Card title="Import geospatial annotations" icon="map" href="/horizon/developer-guides/import-geospatial-annotations#example-import-pre-labels-or-ground-truths" />

  <Card title="Import document annotations" icon="file-pdf" href="/horizon/developer-guides/import-document-annotations#example-import-pre-labels-or-ground-truths" />

  <Card title="Import conversational text annotations" icon="message" href="/horizon/developer-guides/import-conversational-text-annotations#example-import-pre-labels-or-ground-truths" />

  <Card title="Import audio annotations" icon="music" href="/horizon/developer-guides/import-audio-annotations#example-import-pre-labels-or-ground-truths" />

  <Card title="Import HTML annotations" icon="code" href="/horizon/developer-guides/import-html-annotations#example-import-pre-labels-or-ground-truths" />
</CardGroup>

## Best practices

* Before starting a new import job, ensure that there are no existing model-assisted labeling (MAL) annotations on the data rows. Duplicate import jobs can overwrite existing labels or cause unexpected behavior.
* The activity page in Labelbox will not show any changes until the entire import job is complete.


# Import annotations as pre-labels into a project
Source: https://docs.labelbox.com/horizon/guides/import-prelabels

Learn how to import your annotations as pre-labels into Labelbox.

This guide will walk you through the process of importing your annotations from other internal or third-party tools as pre-labels into Labelbox.

## What are pre-labels?

Pre-labels are suggested annotations that are programmatically imported into a Labelbox project. Often generated by a machine learning model, these annotations serve as a starting point to accelerate the labeling process. Instead of creating every annotation from scratch, human labelers are presented with these pre-populated suggestions in the labeling editor.

The primary goal of using pre-labels is to boost labeling efficiency. A labeler's task shifts from manual creation to a faster workflow of reviewing, correcting, and confirming the suggested annotations. A pre-label is only converted into a finalized, ground truth annotation after a human labeler has reviewed and submitted the asset, ensuring that the final data maintains a high standard of quality.

## How to import annotations as pre-labels

You can reference Python SDK code snippets directly from the **Import labels** tab in your project dashboard. These snippets will serve as a great starting point for the steps below.

1. **Import your data rows**: To import annotations as pre-labels, you'll need to have a set of data rows to attach the annotations to. If you do not already have a set of data rows, you'll need to [create a dataset](/horizon/guides/datasets-datarows) by importing data rows into Catalog.
2. **Create/select an ontology**: When you import a set of annotations, you'll need to specify the ontology (also called taxonomy) that corresponds to the set of annotations. If the project ontology already exists in Labelbox, you may select the ontology that fits your annotations. If the ontology does not exist in Labelbox yet, you'll need to [create an ontology](/horizon/guides/labelbox-ontology).
3. **Create a labeling project**: Before you can import your annotations, you'll need to make sure you have a project to connect these annotations to. You cannot simply import annotations without specifying which project they'll be associated with. Oftentimes, you will already have a project set up with the correct ontology for your set of annotations. However, If you do not already have a project, you will need to [create a project](/horizon/guides/create-a-project) and attach the ontology that fits your annotations.
4. **Send a batch of data rows to the project**: Now that you have your project and ontology configured, you'll need to send a subset of data Rows (i.e., a [batch](/horizon/guides/batches)) to the project's labeling queue. You will be attaching your annotations to this batch of data rows.
5. **Create the annotations payload**: This is where you’ll prepare the annotations themselves for import. Each annotation in your payload needs to reference a specific annotation class from your ontology and a specific data row ID. Labelbox supports two formats for the annotation payload: **Python Annotation Types** and **NDJSON**. Go to the **Import labels** tab and select **Import pre-labels**. There you will find a sample code snippet you can use to import your ground truth annotations.
6. **Import your annotation payload**: Once you have created your annotation payload, the final step is to submit the import job. After the job is complete, your ground truth annotations will appear in your project, and the corresponding data rows will be marked as “Done.” Under **Import labels** → **Import pre-labels**, copy the code snippet for **Import annotations payload**.

Check out these end-to-end developer guides to learn how to import ground truth annotations based on your data type.

<CardGroup>
  <Card title="Import image annotations" icon="image" href="/horizon/developer-guides/import-image-annotations#example-import-pre-labels-or-ground-truths" />

  <Card title="Import video annotations" icon="video" href="/horizon/developer-guides/import-video-annotations#example-import-pre-labels-or-ground-truths" />

  <Card title="Import text annotations" icon="file-lines" href="/horizon/developer-guides/import-text-annotations#example-import-pre-labels-or-ground-truths" />

  <Card title="Import geospatial annotations" icon="map" href="/horizon/developer-guides/import-geospatial-annotations#example-import-pre-labels-or-ground-truths" />

  <Card title="Import document annotations" icon="file-pdf" href="/horizon/developer-guides/import-document-annotations#example-import-pre-labels-or-ground-truths" />

  <Card title="Import conversational text annotations" icon="message" href="/horizon/developer-guides/import-conversational-text-annotations#example-import-pre-labels-or-ground-truths" />

  <Card title="Import audio annotations" icon="music" href="/horizon/developer-guides/import-audio-annotations#example-import-pre-labels-or-ground-truths" />

  <Card title="Import HTML annotations" icon="code" href="/horizon/developer-guides/import-html-annotations#example-import-pre-labels-or-ground-truths" />
</CardGroup>

## Best practices

* Make sure the annotations are in the proper format. Use the charts above to determine whether an annotation type is supported for the data type you are labeling.
* Before you begin a new import job, make sure there are no existing MAL annotations on the data row.
* To override an imported annotation on a data row, import the annotation again using the same `uuid`.


# Labeling instructions
Source: https://docs.labelbox.com/horizon/guides/instructions-and-quizzes



Your instructions are the single most important resource for your labeling team. This is where you move beyond simple definitions and provide the context, examples, and rules needed to handle real-world complexity. Well-crafted instructions are the first line of defense against inconsistent or inaccurate annotations.

## Create labeling instructions

To add labeling instructions:

1. Go to your project overview.
2. Click on **Labeling instructions**.
3. In the **Instructions** section, you may write the instructions directly in the text box, upload a PDF or HTML file, or add a video link.

What to include in your instructions:

* **Detailed Definitions:** Go beyond the feature name. What exactly constitutes a "partially occluded vehicle" versus a "fully visible vehicle"?
* **Visual Examples (The Good and The Bad):** Show clear examples of correctly labeled data. Just as importantly, show examples of common mistakes or edge cases that should be labeled differently or ignored.
* **Edge Case Guidance:** Your data will never be perfect. Provide rules for how to handle blurry images, rare objects, or situations where multiple interpretations are possible.
* **"When in Doubt" Rules:** Give your team a clear default action to take when they are unsure, such as flagging the asset for review.

Use the following template as a reference (or download this [PDF file](https://storage.googleapis.com/labelbox-sample-datasets/Docs/Labelbox_labeling_instructions_template_v4.pdf)).

<Accordion title="Expand to view labeling instruction template" icon="clipboard-list">
  <Frame>
    <iframe />
  </Frame>
</Accordion>

<Note>
  **Changes to instructions are global**

  Remember that instructions live with the ontology, not the project. A single ontology can be shared across many projects. When you update the instructions for an ontology, those changes will immediately be reflected in **every project** that uses it.
</Note>

If you need different instructions for different projects, the correct approach is to create a separate [copy of your ontology](/horizon/guides/labelbox-ontology). Each copy can then have its own unique instructions tailored to that specific project's needs.

## Use quizzes for quality

Quizzes are a project-level quality check for labelers and Alignerrs. One common use is confirming that contributors understand your instructions; you can also use quizzes to reinforce quality standards, test edge cases, or reassess contributors after guidelines change.

When a quiz is required, labelers must pass it before they can start labeling or start the Hubstaff timer. You can also make a quiz optional so contributors can still begin work if they do not pass.

When both instructions and a quiz are due, the contributor workflow is:

1. **Review instructions:** The labeler is first prompted to carefully read the instructions you've provided.
2. **Take the quiz:** After confirming instructions, they answer the quiz questions.
3. **Receive AI-powered feedback:** Each answer is scored by an AI on a 1–5 scale. A score of **3 or higher** means that individual question is correct.
4. **Pass the quiz:** The quiz passes when the share of correctly answered questions meets the configured **pass threshold (%)**. By default the threshold is **100%** (every question must pass).
5. **Iterate or proceed:** If a labeler fails and retries are still allowed, they can retake the quiz. Pass/fail and scores are always shown; per-question grader feedback is shown when enabled.

Confirming instructions and passing the quiz are independent. Passing the quiz does not confirm instructions, and confirming instructions does not mark the quiz complete.

To create and manage a project quiz—including AI generation, attempt policies, required vs optional gating, force retakes, and lockouts—see [Quiz](/horizon/guides/quiz).


# Using issues to share feedback
Source: https://docs.labelbox.com/horizon/guides/issues-comments

Use issues and comments to provide direct labeling feedback or raise questions.

When you're working on a labeling project, clear communication is key. The **Issues** feature in Labelbox is designed to help you and your team members communicate effectively. You can use it to give feedback, point out errors, or ask questions directly on the data you're labeling.

This guide will walk you through how to use this feature to improve your team's collaboration.

## How it works

Imagine a common scenario between a labeler and a reviewer:

1. A labeler finishes labeling an asset and submits it.
2. The reviewer looks at the label and notices something that needs to be corrected.
3. The reviewer then creates an "issue" on that specific label to flag the problem.
4. The labeler can see all the issues assigned to them in the **Issues** tab.
5. When the labeler clicks on an issue, they are taken directly to the label that needs fixing.
6. The labeler can then either fix the label and "resolve" the issue, or they can reply with a comment to ask for more clarification from the reviewer.

It's important to know that issues and comments are specific to each project, not your entire organization. Also, what you can do with issues and comments depends on your role in the project.

| Role | Permissions |
| - | - |
| Labeler | Can create, update, delete, resolve, and reopen their own issues on assets they have labeled. |
| Reviewer or Admin | Can create and resolve issues on any labeled asset. They can also delete their own issues and manage comments on any asset. |

## How to create an issue

For an issue to be visible to your team, the data row you're working on needs to be submitted or skipped. If the data row is still in the **To label** step, the issue won't appear in the **Issues** tab.

Here’s how to create an issue:

1. First, open the labeling editor by selecting an asset from any of the workflow steps (like “To label” or “To review”).
2. In the top-left corner of the editor, click on the **issues** icon to activate the tool.
3. Click on the specific spot on the asset where you want to place the issue.
4. A panel will appear on the right where you can type your **message**, such as a description of the problem or a question.
5. (Optional) You can add a **category** to the issue. This is helpful for organizing and filtering issues later on. You can either use an existing category or create a new one with a name and description.
6. Finally, click **Create issue** to save it.

## How to reply to an issue

To reply to an issue, go to the **Issues** tab and select an issue, or click on an issue directly in the editor. Type your response in the **Reply** text box and click **Submit**.

This is what a typical flow looks like from a labeler perspective:

1. **Labeler gets the rework task:** From the project overview, the labeler starts the **Rework** task (e.g., “Start reworking”), which opens the editor on a rejected data row. The issue marker is visible on the asset once it’s out of “To label.”
2. **Labeler opens and reads the issue:** In the editor, the labeler clicks the **issue icon/marker** to open the issue thread and read the reviewer’s feedback. Alternatively, they can go to the project’s **Issues tab**, find issues on assets they labeled, and click one to jump directly into that label + issue context.
3. **Labeler responds via comments and/or fixes the label:** If the instructions are clear, the labeler **edits the label** to address the problem, then **submits** the data row again from the editor; this moves it back into review. If they need clarification, they \*\*reply on the issue \*\*by typing a response in the **Reply** box on the issue and click **Submit** to add a comment to the thread.

## How to view all issues for a project

To see all the issues in a project, simply go to the **Issues** tab. From there, you can filter the issues to see which ones are **All**, **Open**, or **Resolved**.

You can also manage your issue categories from this tab. Just select **Issue categories** to create new ones or edit existing ones.

## How to update an issue

Once an issue is created, users on the project can interact with it in a few ways:

* **Leave a comment:** Go to the **Issues** tab in your project dashboard and select an issue from the list. This will open the data row in the browser along with the posted issue. Enter your comment in the **Reply** box and click **Submit** to save your comment.
* **Edit, resolve, or delete an issue:** To resolve an issue, simply click the **Resolve** button at the bottom of the issue. You can also access an options menu to **Edit** the content of the issue, **Resolve** it, or **Delete** it entirely. The actions you can take will depend on your user role.
* **Move an issue:** If you're the one who created the issue, you can also click and drag it to a different location on the asset.


# Key definitions
Source: https://docs.labelbox.com/horizon/guides/key-definitions

A glossary of key terms used throughout the Labelbox platform.

These terms appear throughout the Labelbox platform, including the app, the SDK, and the docs.

## A, B

* Annotation A human-made or computer-generated label on an asset. Annotations can be imported (as ground truth or pre-labels) or they can be created manually in the Labelbox editor. Annotations are categorized as *objects* (such as bounding box or polygon) or *classifications* (e.g. radio, checklist, etc).
* [Attachments](/horizon/guides/label-data#attachments) Supplementary information you can attach to an asset in order to provide contextual information for your labeling team. When viewing data rows in detail view, attachments appear on a separate side panel. For SDK help, see [Attachments (SDK)](/horizon/developer-guides/attachments)
* Asset Assets (or \_data assets\_) are individual files to be labeled, such as an image, a video, or a text file. Can be hosted in a cloud bucket, uploaded from a local file location, or copied from a remote data source.
* [Batch](/horizon/guides/data-rows-activity) A method for selecting data rows from **Catalog** and sending them to a labeling project.
* [Benchmark](/horizon/guides/benchmark) The Benchmark tool lets you designate a labeled asset as a “gold standard” and automatically compare all other labels on that asset to the benchmark label.

## C, D

* [Catalog](/horizon/guides/catalog-overview) An organization-wide platform for curating and exploring your unstructured data. Catalog enables you to easily browse, curate, and develop insights across all labeled and unlabeled data rows in your organization.
* [Consensus](/horizon/guides/consensus) The Consensus tool lets you compare labelers against each other by comparing annotations on a given asset. Consensus works in real-time so you can take immediate and corrective actions toward improving team and model performance.
* [Data row](/horizon/guides/datasets-datarows) Represents an individual data asset, along with associated attributes (such as global ID) and annotations, which can include:
  * URL to your cloud-hosted file
  * Metadata
  * Media attributes (e.g., data type, size, etc.)
  * Attachments (files that provide context for your labelers)
  * Predictions
* [Data split](/horizon/guides/splits) You can split the selected data rows into train, validation, and test splits to prepare for model training and evaluation.
* Data type Type of data row such as image (JPG/PNG), Video (MP4), text (.txt files). For a list of supported data types, see [Data types & import specs](/horizon/guides/data-setup#data-types--import-specs)
* [Dataset](/horizon/guides/datasets-datarows) Datasets are containers for data rows; they collect a set of related data assets.

## E - L

* [Editor](/horizon/guides/label-data) The labeling interface you can use to create, review, and edit annotations. When creating a new project, you're prompted to configure the editor, which defines the data type and the interface used while labeling.
* [Experiment](/horizon/guides/experiments) An experiment is a directory where you can create, manage, and compare a set of Model Runs related to the same machine learning task. Each experiment is specified by an ontology of data: it defines the machine learning task of the Model Runs inside the directory.
* [Feature](/horizon/guides/working-with-features) A feature is the master definition of what you want the model to predict. It is also the blueprint for your ground truth. Ontologies consist of features, which include objects (example: bounding box) and classifications (radio buttons). Features can have multiple, nested classifications.
* [Ground truth](/horizon/guides/import-ground-truth) A *ground truth* is information that is known to be real or true, as supported by direct observation and measurement. Labels made by humans are considered to be empirical ground truths, as opposed to labels added through model inference.
* Label A collection of all annotations on a data row. If a data row had multiple bounding boxes, polylines, and radio classifications, the entire collection would be considered the "label."

## M - O

* [Media attributes](/horizon/developer-guides/media-attributes) When you upload data assets, Labelbox automatically computes media attributes appropriate for the data type and stores their values as part of the data row. Examples include mimeType, width, height, codec, and more. For details, see [Media attributes](/horizon/developer-guides/media-attributes)
* [Model-assisted labeling](/horizon/guides/model-assisted-labeling) Model-assisted labeling (MAL) allows you to import computer-generated predictions (or simply annotations created outside of Labelbox) as pre-labels on a data row queued in a labeling project.
* [Metadata](/horizon/guides/datarow-metadata) Metadata is non-annotation information about the asset to be labeled. There are two types of metadata: reserved keys (which cannot be changed) and custom (user-defined). Metadata helps search and filter data rows.
* [Model run](/horizon/guides/create-a-model-run) A *model run* is a model training experiment within a model directory. Each model run has its data snapshot (data rows, annotations, and data splits) versioned. You can upload predictions to a model run, and compare results and performance against other model runs in the model directory.
* Nested classification A classification-type annotation that is nested within an object-type annotation (as opposed to a global classification).
* [Ontology](/horizon/guides/labelbox-ontology) A collection of features and their relationships (also known as a *taxonomy*). Ontologies can be reused across different projects. Ontologies are essential for data labeling, model training, and evaluation. When you label or review a data asset, the ontology appears in the **Tools** panel.

## P - Z

* [Prediction](/horizon/guides/model-assisted-labeling) Output from your machine learning model that you can add to a data row to serve as a template for faster labeling.
* [Project](/horizon/guides/what-is-a-project) The labeling environment in Labelbox, like a factory assembly line for producing labels. The initial state of the project can start with raw data, pre-existing ground truth, or pre-labeled data.
* [Queue](/horizon/guides/labeling-queue) Labelbox has three queues to help move data rows through the labeling and review workflow: the [batches queue](/horizon/guides/batches), the [labeling queue](/horizon/guides/labeling-queue), and the [review tasks](/horizon/guides/workflows) queue.
* Schema The schema is the master blueprint for your training data and includes [ontologies](/horizon/guides/labelbox-ontology), [features](/horizon/guides/working-with-features), and [metadata](/horizon/guides/datarow-metadata).
* [Template](/horizon/guides/label-data#template) If a data row needs to be relabeled, you can delete the annotations and then select existing annotations to use as a template for the next data row displayed in the editor. This allows you to curate a set of annotations, rather than start from scratch for each data row.
* [Workflow](/horizon/guides/workflows) A *workflow* is a queue for labeling and reviewing assets within a project. Workflows provide granular control over data row reviews. Workflows are highly customizable and help define a step-by-step pipeline leading to an efficient and more accurate process.
* [Workspace](/horizon/guides/workspaces) Enables admins at large organizations to manage multiple instances of Labelbox with the same subscription account.


# Labeling editors
Source: https://docs.labelbox.com/horizon/guides/label-data

A summary of labeling editor controls, enhancements, and supported annotations.

Labelbox automatically adjusts the editor interface based on the asset type selected for the project. However, there are some global editor settings and enhancements shared across all of the editor interfaces. These global editor settings are designed to provide an optimal labeling experience for your team through increased customization and quick accessibility.

Additionally, there are some data enrichment capabilities and supplementary tools available for use in the editor only with certain data modalities. Features such as attachments, instructions, and the data row information panel are available in the editor regardless of the type of asset being labeled.

## Attachments

Attachments can be used to provide supplementary content to any asset to help provide additional context for the labeling team. An attachment applies to an individual asset and may comprise an image, video, text, or HTML content. Markdown rendering is supported for attachments, allowing you to use to format text, add links, and include images within attachments. Multiple attachments can be linked to an singular data row.

For details on how to create attachments or import data rows with attachments using the Python SDK, see the [Attachments developer guide](/horizon/developer-guides/attachments).

When any attachments have been provided for an asset, labelers will see the **Attachment** button appear in the top left of the editor.

You can also use the hotkey `Shift` + `?` to open the attachments panel.

## Data row information panel

Access to additional context and information at the data row level can be extremely helpful to labelers as they label data. Users can click to view a side panel that brings up data row information, providing teams with ample context and easy access to information related to the particular data row being served.

Labelers can view information such as the global key, the dataset, and when the data row was created. Based on data type, labelers can also view specific media attributes, such as an image’s width and height, the number of pages in a PDF document, the frame rate and duration of a video, and more. Additionally, labelers can also see any metadata attached to the data row.

## Image overlay

Image overlay can be used to provide labelers with additional view options for the image being labeled. For example, if you have additional cameras capturing images of your subject matter in different formats (greyscale, thermal, etc.), you may want to provide these images as contextual layers to the primary image.

Layers are applied to individual images, and a maximum of 10 layers may be added per image. Image layers are a visualization tool designed to help you view the asset to be labeled in different ways. You may change visual settings on the image layers (e.g., transparency, brightness) and still create annotations on the data row.

<Info>
  To view the maximum number of images that can be attached as layers, visit our [limits](/horizon/guides/limits) page.
</Info>

When layers have been provided for an image, you will see an **Image overlay** button appear in the top bar of the editor directly to the left of the zoom options.

Clicking on this button opens a dropdown menu where you can select an image to overlay. Alternatively, you can use the keyboard shortcut `Option`/`Alt` + `Layer #` to quickly switch between layers.

For details on how to create image layers or import data rows with image layers using the Python SDK, please view the documentation [here](https://docs.labelbox.com/horizon/guides/datasets-datarows).

## Auto-segment

This tool is embedded into the segmentation mask tool. To use auto-segment, select a segmentation mask tool, toggle on auto-segment by selecting the magic wand icon or using the hotkey `R`, and draw a box around an object.

Labelbox will automatically draw a segmentation mask on the object inside the box. Then, you can make edits, as usual, using the segmentation mask's pen tools. For more details, please view the documentation [here](/horizon/guides/image-editor#auto-segment).

## Bounding box tracking

This tool is embedded into the bounding box tool and is only available when labeling video assets. When you draw a bounding box around an object, click **Track Object** to activate bounding box tracking.

Labelbox will automatically track the object for a specified number of frames. Edit the results by selecting and removing any desired keyframes. For more details, please view the documentation [here](/horizon/guides/video-editor#ai-based-bounding-box-track).

## Supported annotations

The table below indicates which top-level annotation types you can use to label each asset type.

| Annotation type | [Image](/horizon/guides/image-editor) | [Video](/horizon/guides/video-editor) | [Text](/horizon/guides/text-editor) | [Audio](/horizon/guides/audio-editor) | [Documents](/horizon/guides/document-editor) | [Tiled imagery](/horizon/guides/tiled-editor) | [Conversational text](/horizon/guides/conversational-editor) | [HTML](/horizon/guides/html-editor) |
| - | - | - | - | - | - | - | - | - |
| **Segmentation mask** | <Icon icon="check" /> | <Icon icon="check" /> (use video segmentation tool) | N/A | N/A | - | - (Use polygon) | N/A | - |
| **Bounding box** | <Icon icon="check" /> | <Icon icon="check" /> | N/A | N/A | <Icon icon="check" /> | <Icon icon="check" /> | N/A | - |
| **Polygon** | <Icon icon="check" /> | - | N/A | N/A | - | <Icon icon="check" /> | N/A | - |
| **Point** | <Icon icon="check" /> | <Icon icon="check" /> | N/A | N/A | - | <Icon icon="check" /> | N/A | - |
| **Polyline** | <Icon icon="check" /> | <Icon icon="check" /> | N/A | N/A | - | <Icon icon="check" /> | N/A | - |
| **Entity** | N/A | N/A | <Icon icon="check" /> | N/A | <Icon icon="check" /> | N/A | <Icon icon="check" /> | - |
| **Relationship** | <Icon icon="check" /> | - | <Icon icon="check" /> | N/A | <Icon icon="check" /> | - | <Icon icon="check" /> | - |
| **Radio** | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> |
| **Checklist** | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> |
| **Free-form text** | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> |


# Labelbox MCP server
Source: https://docs.labelbox.com/horizon/guides/labelbox-mcp

Connect an AI agent to Labelbox, approve access, and use tools with your current permissions.

The Labelbox platform MCP server connects your AI agent to your workspace. Ask your agent to inspect projects, summarize labeling progress, or set up a project when you approve write access.

Operators and project managers can use it to monitor progress, review quality, and identify work that needs attention. Developers can use the same tools from their coding assistants. You do not need to write code to use Labelbox tools through a compatible AI assistant.

Add this server URL to your MCP client:

```text theme={"theme":"css-variables"}
https://api.labelbox.com/v1/agent-tools/mcp
```

You sign into Labelbox and approve the connection in your browser. You do not need an API key, client secret, or SDK installation.

## Before you connect

You need:

* A Labelbox account with access to a workspace.
* A workspace where external MCP access is available and allowed by your administrator.
* A client that supports remote MCP over Streamable HTTP and OAuth sign-in.

Your existing [roles and permissions](/horizon/guides/roles-and-permissions) determine what the agent can access. Approval does not add permissions to your account.

<Note>
  This URL connects to the Labelbox platform. The [documentation MCP](https://docs.labelbox.com/mcp) searches docs. Recursion Managed Agents has a separate MCP server and authentication flow.
</Note>

## Set up your client

<Tabs>
  <Tab title="Claude">
    1. In Claude, open **Customize → Connectors** and add a custom connector. For a managed account, an organization owner may need to add it first.

    2. Name it `Labelbox` and enter the server URL:

       ```text theme={"theme":"css-variables"}
       https://api.labelbox.com/v1/agent-tools/mcp
       ```

    3. Keep OAuth sign-in enabled. If asked how to identify the client, select automatic registration. Leave static credentials and request headers empty.

    4. Finish adding the connector and start sign-in. Complete [Approve access](#approve-access) in your browser.

    5. Enable the Labelbox connector for your conversation, then [verify the connection](#verify-the-connection).

    Availability depends on your Claude account and organization policy. See [Claude's remote connector instructions](https://support.claude.com/en/articles/11175166-get-started-with-custom-connectors-using-remote-mcp) for current controls.
  </Tab>

  <Tab title="ChatGPT">
    1. Open ChatGPT on the web and go to **Plugins**. Choose **Add custom MCP server** from the add menu.

    2. Name it `Labelbox` and enter this server URL:

       ```text theme={"theme":"css-variables"}
       https://api.labelbox.com/v1/agent-tools/mcp
       ```

    3. Choose **OAuth** authentication and automatic client registration if offered. You do not need a static client ID or secret.

    4. Complete the setup and the [Labelbox approval flow](#approve-access). Find the resulting plugin in **Plugins** and install it before using it in a conversation.

    5. Select Labelbox with `@` in a conversation, then [verify the connection](#verify-the-connection).

    Your ChatGPT account and workspace policy must allow custom MCP connections. See [OpenAI's custom MCP instructions](https://developers.openai.com/api/docs/guides/custom-mcp-server) if the controls differ or are unavailable.
  </Tab>

  <Tab title="Cursor Desktop">
    Add the following entry to `~/.cursor/mcp.json`. Merge it with your existing `mcpServers` entries if the file already exists.

    ```json theme={"theme":"css-variables"}
    {
      "mcpServers": {
        "labelbox": {
          "url": "https://api.labelbox.com/v1/agent-tools/mcp"
        }
      }
    }
    ```

    Open Cursor's MCP controls, find `labelbox`, and start its sign-in flow. Keep Cursor open while you complete authorization in your browser.

    Continue with [Approve access](#approve-access), then return to Cursor's Agent chat to [verify the connection](#verify-the-connection).

    See [Cursor's MCP documentation](https://cursor.com/docs/mcp) for configuration locations and client controls.
  </Tab>

  <Tab title="Codex CLI">
    Add the remote server:

    ```sh theme={"theme":"css-variables"}
    codex mcp add labelbox --url https://api.labelbox.com/v1/agent-tools/mcp
    ```

    If sign-in does not start automatically, run:

    ```sh theme={"theme":"css-variables"}
    codex mcp login labelbox
    ```

    Complete [Approve access](#approve-access) in your browser. Start or reopen your Codex session, then use `/mcp` to check the server and [verify the connection](#verify-the-connection).

    See [Codex's MCP documentation](https://learn.chatgpt.com/docs/extend/mcp) for current client instructions.
  </Tab>

  <Tab title="Claude Code">
    Add the remote server for your user account:

    ```sh theme={"theme":"css-variables"}
    claude mcp add --transport http --scope user labelbox https://api.labelbox.com/v1/agent-tools/mcp
    ```

    Start an interactive Claude Code session and enter `/mcp`. Select the Labelbox server and authenticate.

    Complete [Approve access](#approve-access) in your browser, then return to Claude Code to [verify the connection](#verify-the-connection).

    The `--scope user` option stores the configuration for your user across projects. It does not grant Labelbox permissions.

    See [Claude Code's MCP documentation](https://code.claude.com/docs/en/mcp) for current client instructions.
  </Tab>

  <Tab title="Other clients">
    Use your client's remote MCP setup instructions. Enter the Labelbox server URL and select Streamable HTTP if the client asks for a transport.

    ```text theme={"theme":"css-variables"}
    https://api.labelbox.com/v1/agent-tools/mcp
    ```

    Start the client's OAuth sign-in flow, complete [Approve access](#approve-access), and [verify the connection](#verify-the-connection).

    Compatibility depends on the client's OAuth implementation. Hosted clients and browser connectors can have different account requirements and setup steps.
  </Tab>
</Tabs>

## Approve access

<Steps>
  <Step title="Sign into Labelbox">
    Start authorization from your MCP client. Sign into Labelbox in the browser when prompted. Use your own account rather than an impersonated session.
  </Step>

  <Step title="Select a workspace">
    Choose an eligible workspace for this connection. This choice applies to the MCP client and does not switch your existing browser workspace.

    If no workspace appears, see [Troubleshooting](#troubleshooting).
  </Step>

  <Step title="Review the requested access">
    Check the client name and requested access. Select optional access only if you need it:

    * **Allow project changes** permits supported project setup tools, subject to your current permissions.
    * **Limit access to selected projects** restricts the connection to the projects you select.
    * **Stay connected between sessions** permits connection renewal when the client supports and requests it.

    Write access and renewal start unchecked. These options depend on the access requested by your client and what your workspace permits.
  </Step>

  <Step title="Allow access">
    Choose **Allow access** to approve the connection. You can cancel if you do not want to grant access.
  </Step>

  <Step title="Return to your client">
    Keep the client open until authorization completes. Confirm that the Labelbox server connects and its tools appear.
  </Step>
</Steps>

## Verify the connection

Ask your agent:

```text theme={"theme":"css-variables"}
Use Labelbox to show my connected workspace and access restrictions.
```

The agent should call `get_caller_context` and return the selected workspace and connection restrictions. Confirm that the workspace is the one you intended.

A browser redirect alone does not confirm success. The client must complete authorization and receive a tool response.

You can also ask:

```text theme={"theme":"css-variables"}
Use Labelbox to list projects I can access in this workspace.
```

An empty result can be valid. Projects accessible through user groups may not appear in the list. To check a specific project, give the agent its project ID or Labelbox project URL and ask it to look up that project directly.

## What you can do

Your client discovers the current tools available to your connection from the server. Available actions depend on your permissions and approved access. You can ask:

```text theme={"theme":"css-variables"}
What can you do with my current Labelbox connection?
```

These are representative workflows, rather than a complete list of available tools:

| Workflow | Example prompt |
| - | - |
| Inspect workspace and projects | "Show my workspace and summarize this project's health." |
| Inspect project data | "Which datasets and batches are attached to this project?" |
| Track progress and quality | "Summarize progress, throughput, and benchmarks for this project." |
| Inspect workflow queues | "Show the queues in this project." |
| Set up projects with write access | "Create a project with an ontology and instructions, then add a batch from my existing dataset." |

The project setup workflow requires **Allow project changes**, workspace write availability, and the permissions for each action. A connection limited to selected projects cannot create a new project or use other workspace-level tools.

## Permissions and connection lifetime

Labelbox checks your current permissions on every tool call. The connection's approved access and selected projects can further restrict the agent.

* A connection applies to one workspace.
* Project restrictions limit access; they do not grant a new role or project membership.
* Your client controls its own tool confirmations and automation settings. Labelbox consent does not guarantee a confirmation before each tool call.
* Review how your client handles the data returned by tools.

Clients can request different access. A basic connection may be read-only, and **Stay connected between sessions** may not be offered. Missing options do not necessarily indicate an error.

If you approve renewal, a compatible client can renew the connection between sessions. Connections can still expire or be revoked. Without renewal, you must sign in again when access expires.

To change the approved access, start a new authorization flow from your client. To switch workspaces, follow [I want to connect another workspace](#troubleshooting).

## Manage connections

### Disconnect your client

Open [workspace settings](https://app.labelbox.com/workspace-settings/workspace) and find **Connected MCP clients**. This section lists your connections across workspaces.

Choose **Disconnect** for the client you want to remove and confirm the action. The connection stops working after the disconnect succeeds. Sign out of your client or remove its server entry if you also want to clear its local configuration.

Signing out of the Labelbox website does not disconnect an approved MCP connection.

### Control workspace access

Workspace administrators with permission to manage the workspace can use **Workspace MCP access** in [workspace settings](https://app.labelbox.com/workspace-settings/workspace).

* **Allow external MCP connections** controls whether users can connect external clients. Disabling it disconnects existing clients.
* **Disconnect all clients** invalidates the workspace's existing connections without changing whether new connections are allowed.

Re-enabling access does not restore earlier connections. Users must connect and approve access again.

Workspace access must also be available through Labelbox. If the administrator allows access but the workspace remains unavailable, [contact support](/horizon/guides/contacting-customer-support).

## Troubleshooting

<AccordionGroup>
  <Accordion title="No workspace is available for this connection">
    Confirm that you signed into the correct Labelbox account and belong to the workspace. Ask a workspace administrator to check **Allow external MCP connections**.

    If access is allowed but the workspace remains absent, [contact Labelbox support](/horizon/guides/contacting-customer-support) to check availability.
  </Accordion>

  <Accordion title="The request is invalid or expired">
    Return to your MCP client and start a fresh connection. Do not reuse a bookmarked consent page or a callback from an earlier attempt.
  </Accordion>

  <Accordion title="Localhost refused the connection after approval">
    The client may have stopped listening for the browser callback or the attempt may have timed out. Keep the client open and restart its authorization flow.

    For a local desktop client, complete sign-in in a browser on the same computer. A refused callback does not necessarily mean that your Labelbox session expired.
  </Accordion>

  <Accordion title="The connection was cancelled">
    Start authorization again from your client when you are ready to approve access. A callback from a cancelled attempt cannot complete a new connection.
  </Accordion>

  <Accordion title="Tools do not appear after approval">
    Check the client's MCP connection status and authentication errors. Confirm that the server uses the platform URL at the top of this guide.

    If authorization did not complete, start a fresh sign-in flow. Use `get_caller_context` to verify the workspace after the tools appear.
  </Accordion>

  <Accordion title="My project list is empty or a project is missing">
    Confirm the connected workspace and selected-project restrictions. An empty result is valid when no projects match the connection's access.

    Project lists may omit projects accessible through user groups. Give the agent the project's exact ID or Labelbox project URL and ask it to look up that project directly.

    If that lookup fails, check your [project permissions](/horizon/guides/roles-and-permissions) with an administrator.
  </Accordion>

  <Accordion title="A tool is missing or access is denied">
    Check your role permissions, approved connection access, selected projects, and workspace MCP policy. A restricted connection can have fewer tools than an unrestricted connection.

    Confirm permissions with your administrator. Reconnect if you need a different grant. [Contact support](/horizon/guides/contacting-customer-support) if the expected access still fails.
  </Accordion>

  <Accordion title="Write access or Stay connected is not offered">
    Your client must request the corresponding access. Project changes also require workspace write availability. Check your client's OAuth options and ask an administrator or [support](/horizon/guides/contacting-customer-support) about workspace availability.

    Keep read-only access if it meets your needs. If you change the client's requested access, start a new authorization flow. Do not edit a browser authorization URL.
  </Accordion>

  <Accordion title="My connection expired or stopped working">
    Reconnect from your client. Check that you still belong to the workspace and that external access remains allowed.

    An administrator can revoke connections or disable workspace access. Re-enabling access does not restore old grants; approve a new connection.
  </Accordion>

  <Accordion title="I want to connect another workspace">
    Disconnect or clear the Labelbox MCP authentication in your client, then authorize again and select the other workspace. You do not need to sign out of Labelbox in your browser.

    After reconnecting, ask your agent to call `get_caller_context` and confirm the selected workspace before using other tools.
  </Accordion>

  <Accordion title="The service is temporarily unavailable">
    Retry later. If the problem continues, [contact Labelbox support](/horizon/guides/contacting-customer-support).
  </Accordion>
</AccordionGroup>

For unresolved issues, include your client name, version, and a description of the failed step when you [contact support](/horizon/guides/contacting-customer-support). Do not share tokens, cookies, or full authorization callback URLs.


# Ontologies
Source: https://docs.labelbox.com/horizon/guides/labelbox-ontology

Instructions for creating and managing your ontologies in your Labelbox workspace.

This guide provides a comprehensive overview of how to create, manage, and optimize your ontologies within Labelbox to ensure your team works with precision and your models are built on a foundation of quality.

## What is an ontology?

Think of an ontology as the master blueprint for your data labeling tasks. It defines exactly **what** your team should be labeling and **how** they should label it. An ontology consists of *features* which are:

* **Objects:** The items you want to identify (e.g., *Car*, *Pedestrian*, *Tree*).
* **Classifications:** Questions to categorize the data (e.g., *Is the image blurry?*, *What time of day is it?*).
* **Relationships:** Connections between labeled objects (e.g., a *person* is *walking* a *dog*).

Every project and model in Labelbox requires an ontology. When a labeler works on a task, the ontology appears as their set of tools in the labeling editor.

## Create your first ontology

Follow these steps to create a new, reusable ontology for your projects.

1. Go to the **Schema** tab in your Labelbox workspace and select **Ontologies**.
2. Click the **Create** button to start the setup process.
3. Choose the type of data this ontology will be used for (e.g., Image, Video, Text). *This is a critical step*. The media type determines which annotation tools (bounding box, polygon, etc.) will be available for you to add.
4. In the setup screen, provide the following:

| Item | Description |
| - | - |
| Name | This is the name you will use to look up your ontology later. <br />Give your ontology a descriptive name (e.g., "Vehicle Identification - US Roads"). |
| Objects | *Objects* in this context are specific items you want to identify and annotate in your data (e.g., *car*, *person*, *building*) Labelbox will search for existing features by name to encourage reuse.<br />Click **+ Add** to create or reuse object features. |
| Classifications | *Classifications* are questions or attributes used to categorize or describe objets or the overall data (e.g., *Is the traffic light red or green?*, *What is the weather condition?*).<br />Click **+ Add** to create or reuse classification tasks. |
| Relationships | *Relationships* are connections or interactions that exist between different labeled objects (e.g., a *driver* is *inside* a *car*).<br />Click **+ Add** to define relationships between your objects. |

5. Click **Create ontology** to complete the setup.

<Info>
  ### Ontology size limit

  Visit our [Limits](/horizon/guides/limits) page to view the maximum allowed features per ontology.
</Info>

## Attach an ontology to a project

To use an ontology, you must first attach it to a project.

1. Go to the project's **Settings** tab.
2. Find the **Label editor** section and click **Edit**.
3. Use the dropdown menu to select and attach your desired ontology.

<Warning>
  **Warning: Changing an ontology can invalidate existing labels**

  If you edit an ontology that is already in use, annotations created with the old version will remain, but they may no longer align with the new structure. For example, if you add a new *required* classification, old labels will not have this information, which can impact data exports and quality metrics like consensus.
</Warning>

## Clone an ontology

If you want to build upon an existing ontology without affecting projects that use it, create a copy. This creates a duplicate, independent version that you can freely modify.

1. Create a new project.
2. From the project overview, select **Set up ontology**.
3. Select the ontology you want to copy from the list.
4. Click **Create copy and select**.
5. Name your new ontology.

Your new ontology will automatically be attached to the project you just created.

<Note>
  Adding or archiving features in the copied ontology will **not** impact the original ontology from which the copy was created.

  However, modifications made to a specific feature included in the ontology will cascade through all ontologies containing the feature, including the original ontology.
</Note>

## Edit your ontology

Navigate to Schema > Ontologies and click on the ontology you wish to modify. To edit your ontologies, you can:

**Add a new feature**: Click the **+** button under Objects or Classifications and search for the feature you want to add. Remember, this will not automatically require you to rework existing labels.

**Reorder features**: In the edit screen, simply drag and drop the features into the new order you want them to appear for labelers.

**Archive a feature**: If a feature has already been used to create annotations, you cannot delete it, but you can archive it. Archiving removes the feature from the labeler's view but preserves all historical data.

1. Go to the Schema tab.
2. Click Ontologies and select the ontology you want to edit.
3. Click **Edit**.
4. Select the feature you wish to remove.
5. Click the settings icon in the top right corner to remove or archive the feature. If a feature has **never** been used, you can safely remove it permanently.

You can *unarchive* a feature at any time from the ontology editor to make it available for labeling again.

<Info>
  **Archived features are still included in exports**

  To ensure that all annotation data can be exported from our application, archiving a feature will not remove it from the export file. If you do not plan on using that data in your downstream systems, you will need to ignore those annotations based on feature schema ID.
</Info>

## Delete your ontology

To keep your workspace clean, you can filter for and delete all unused ontologies and features from the Schema tab.

1. Go to the \*\*Schema \*\*tab.
2. Select **Show** : **Unused**.
3. Click **Delete unused ontologies**.

You can do this from the **Schema** tab by adjusting the filter to “Show: Unused”. When this option is selected, you will see a new button to delete all unused features or ontologies.

<Info>
  Deleting an ontology is permanent and cannot be undone.
</Info>


# Configure the labeling queue
Source: https://docs.labelbox.com/horizon/guides/labeling-queue

Learn how to configure the reservations, data row priority, and real-time labeling settings to make your labeling queue more efficient.

The **labeling queue** is the engine that feeds data to your labeling team. It determines which data assets are worked on and in what order. Understanding how to manage this queue is essential for running an efficient labeling operation, ensuring that your most critical data is prioritized, and preventing bottlenecks that can slow down your project.

By default, the queue operates on a **First-In, First-Out (FIFO)** basis—the first data rows you add to the project will be the first ones presented to your labelers. However, Labelbox provides you with powerful tools to override this default behavior and take precise control over the flow of work.

This guide covers the three core pillars of queue management:

1. **Setting data row priority** to control the order of labeling.
2. **Managing reservations** to handle in-progress work.
3. **Re-queuing data** to send assets back for rework.

## Data row priority

The most direct way to manage your queue is by assigning a *priority level* to your data rows. This allows you to push urgent or important data to the front of the queue, ensuring it gets labeled next.

Priority is set on a scale from **1 (highest)** to **5 (lowest)**. A data row with a priority of 1 will be labeled before any data row with a priority of 2, and so on.

There are three ways you can control data row priority:

* **At batch creation:** You'll be prompted to set the priority when you create a batch to be sent to a project. The default priority level is 5.
* **In the Data Rows tab:** Open a project and go to the Data Rows tab to select a set of data rows and update priority.
* **Using the Python SDK:** If you need more granular control (i.e., non-integer values) when setting the priority, reference our [batches developer guide](https://docs.labelbox.com/horizon/developer-guides/batch).

### How to update data row priority

You can update the priority on one or more data rows at once using the **Data Rows** tab.

1. Navigate to the **Data Rows** tab in your project.
2. Use the filters on the left to select the data you want to prioritize. For example, you might filter by metadata, batch name, or upload date.
3. Select the data rows using the checkboxes. You can select individual rows or click the top checkbox to select all rows that match your filter.
4. A **bulk actions menu** will appear at the top of the table.
5. Choose the desired priority level (1-5) from the dropdown menu. Select **Change priority**. You can adjust it later via the SDK (see the [Batches developer guide](/horizon/developer-guides/batch)).
6. Click **Confirm**. A notification banner acknowledges the changes and includes a link to the notification center, where you can follow progress.

The queue will immediately re-order itself. The next labeler to request a task will receive the highest-priority data row that is not currently reserved.

## Reservations

When a labeler begins working on a data row, Labelbox automatically creates a **reservation**. This "locks" the asset to that specific labeler, preventing anyone else from working on it at the same time and avoiding duplicated effort.

To easily view the active reservations per team member, navigate to **Project** → **Settings** → **Queue**.

### How to update reservation count and duration

By default, once a labeler starts labeling, Labelbox reserves 5 data rows for the labeler, and these reservations expire after 4 hours of idle time. The labeling queue replenishes reservations for labelers once they finish labeling a reserved data row.

Once a reservation expires, the associated data rows become available to other labelers, but the reservations still show in the original project settings queue until other labelers claim the data rows.

**To update reservation settings:**

1. Go to the project in Annotate.
2. Go to **Settings → Queue**.
3. To update the number of data rows that can be reserved by a single user, update the slider for **Reservation count**. Value range is 1 - 5.
4. To update the number of hours that a data row can be reserved to a single user, update the slider for **Reservation duration (hours)**. Value range is 1 - 24.

<Warning>
  Labelers who pause their work on a data row may lose unsubmitted work if they exceed their data row reservation time out.
</Warning>

### How to clear reservations

1. Go to the project in Annotate.
2. Go to **Settings → Queue**. There you will see the reservation queue: which Data Rows are reserved, by which user, and which are only queued.
3. Filter by the **Queue state**.
4. Select **Clear all reservations** to drop all current holds and re-serve rows based on current priorities.

Clearing reservations can impact unsubmitted work, depending on the situation:

* **During active labeling**: Labelers may lose any unsubmitted work when clearing the reservation, unless they later take the same asset, which is uncommon and not guaranteed.
* **For expired reservations**: Clearing expired reservations doesn’t affect the availability of the data row, but the row must be claimed before the queue updates.

## Real-time labeling

When you enable real-time labeling, it allows your team to label continuously. This can be helpful if you have labeling time constraints or if you need to improve your labeling team’s efficiency by enabling a continuous flow in the labeling queue.

### To enable real-time labeling

1. Select a project and go to **Settings** → **Automation**.
2. Toggle on **Real-time labeling**.
3. When all assets in the queue have been labeled, the labeler will see a button that says **Wait for next asset**. When a labeler clicks on **Wait for next asset**, Labelbox will display a real-time screen that will automatically display the next task that enters the queue.

## Best practices

* To avoid lost work, Labelbox recommends that labelers consult their administrator if they anticipate being away from their computer for over 24 hours. Any unsubmitted work done by the original labeler will be lost if you clear the reservations.
* The timer starts counting label time after labelers click **Edit**. It stops counting review and label times if our system detects 5 minutes of inactivity. When we detect user activity again, we restart the timer. This is something Labelbox does intentionally to record only active screen time on the application.
* All labeling work should be limited to one browser tab, and no mobile devices should be used. Duplicate labels may be unintentionally created when a user has multiple tabs open and is actively labeling.


# Limits
Source: https://docs.labelbox.com/horizon/guides/limits

Product limits broken down by subscription type.

## General

| Limit | Free | Starter | Enterprise | Education |
| - | - | - | - | - |
| API rate limit (requests/min) | 1,500 | 5,000 | 20,000 | 5,000 |
| Max users per organization | 30 | No limit | No limit | No limit |
| [Labelbox units (LBUs)](/horizon/guides/billing#labelbox-units-lbus) | 500 LBU free each month | Fixed rate: \$0.10 per LBU | No Limit | No Limit |

## Import

| Limit | Free | Starter | Enterprise | Education |
| - | - | - | - | - |
| Max data rows | 10M | No limit | No limit | 10M |
| Max data rows per single API upload task | 150K | 150K | 150K | 150K |
| Max data rows per bulk classification job | 100K | 100K | 100K | 100K |
| Ground truth annotation import limit | 10,000 | 2M | 10M | 2M |
| Max character length for `global_keys` | 200 | 200 | 200 | 200 |
| Max character length for raw text `row_data` | 64K | 64K | 64K | 64K |

## Export

| Limit | Free | Starter | Enterprise | Education |
| - | - | - | - | - |
| Max number of running exports in a project | 10 | 10 | 10 | 10 |
| Max number of running exports in an organization | 20 | 20 | 20 | 20 |
| Max number of data rows using exports: `export()` | No limit | No limit | No limit | No limit |
| Max entries for `global_keys` filter | 2,000 | 2,000 | 2,000 | 2,000 |
| Max entries for `data_row_ids` filter | 2,000 | 2,000 | 2,000 | 2,000 |

## Schema

| Limit | Free | Starter | Enterprise | Education |
| - | - | - | - | - |
| Max ontologies | 25 | No limit | No limit | No limit |
| Max features per ontology\*\* | 4,000 | 4,000 | 4,000 | 4,000 |
| Max custom metadata fields | 5 | 100 | 500 | 100 |
| Max dimensions per custom embedding | 2048 | 2048 | 2048 | 2048 |
| Max custom embeddings per workspace | 100 | 100 | 100 | 100 |

\*\* The ontology feature limit applies to tools, options, classifications, and subclassifications.

## Annotate

| Limit | Free | Starter | Enterprise | Education |
| - | - | - | - | - |
| Max projects | 50 | No limit | No limit | 100,000 |
| Max project tags | 3 | 5 | No Limit | 3 |
| Max labels | No limit | No limit | No limit | No limit |
| Max data rows per workflow | 10,000 | N/A | N/A | 10,000 |
| Max tasks per workflow | 20 | 20 | 20 | 20 |
| Attachments layer limit | 9 | 9 | 9 | 9 |
| Max frames per video | 20,000 | 20,000 | 20,000 | 20,000 |
| Max messages per data row | 250 | 250 | 250 | 250 |
| Max character limit | 5,000 | 5,000 | 5,000 | 5,000 |
| Max unlabeled LLM data rows | N/A | 20,000 | 20,000 | 20,000 |
| Max tags per project | 3 | 5 | No limit | 3 |
| Max model configuration\*\* | N/A | 10\* | 10 | N/A |

\*\* Only Foundry subscribers can currently set this up.

## Catalog

| Limit | Free | Starter | Enterprise | Education |
| - | - | - | - | - |
| Max slices per workspace | 100 | 100 | 100 | 100 |
| Max data rows per bulk add metadata action | 100,000 | 100,000 | 100,000 | 100,000 |
| Max anchors per similarity search | 20 | 20 | 20 | 20 |
| Min characters for *find text* filter | 3 | 3 | 3 | 3 |
| Max characters for *find text* or *natural language* filter | 100 | 100 | 100 | 100 |
| Max words for *find text* or *natural language* filter | 10 | 10 | 10 | 10 |

## Model

| Limit | Free | Starter | Enterprise | Education |
| - | - | - | - | - |
| Max data rows per model run | 150,000 | 1M | 1M | 1M |
| Model run limit | 20 | 20 | 1,000 | 20 |
| Max slices per workspace | 100 | 100 | 100 | 100 |
| Max schemas per ontology for auto metrics | 50 | 50 | 50 | 50 |

## Performance

Exceeding these limits may cause performance issues.

| Limit | Free | Starter | Enterprise | Education |
| - | - | - | - | - |
| Annotations per label | 5,000 | 5,000 | 5,000 | 5,000 |
| Annotations per label (video) | 1,000 | 1,000 | 1,000 | 1,000 |
| Max data rows per batch | 100,000 | 100,000 | 100,000 | 100,000 |
| Max image size (megapixels) | 81MP | 81MP | 81MP | 81MP |
| Asset size (local import) | 256MB | 256MB | 256MB | 256MB |
| Asset size (cloud storage) | 1GB | 1GB | 1GB | 1GB |


# Live multimodal chat evaluation
Source: https://docs.labelbox.com/horizon/guides/live-multimodal-chat-evaluation

Learn how to create live multimodal chat evaluation projects for ranking and classifying model outputs through live, multi-turn conversations.

The **live multimodal chat evaluation** editor allows you to evaluate generative models by comparing their responses in live, multi-turn conversations. You can select up to 10 models, including popular foundation models and custom models, input prompts to trigger model outputs, and add labels to rate, rank, and refine their performance. The editor supports various data types, including text, images, videos, audio, and PDFs.

## Set up live multimodal chat evaluation projects

The following steps walk you through how to set up a live multimodal chat evaluation project on the Labelbox platform. To learn how to set up a live multimodal chat evaluation project using the SDK, see [Multimodal chat evaluation projects](/horizon/developer-guides/multimodal-chat-evaluation-project).

### Step 1: create a project

1. On the Annotate projects page, click the **+ New project** button.
2. Select **Multimodal chat**, and then select **Live multimodal chat**.
3. Provide a **name** and an optional **description** for your project.
4. Select the type of **project configuration** from:
   1. **Text Chat with Media**: Standard MMC editor with text prompts and media attachments
   2. **Audio Prompt Conversation**: Users record audio prompts
   3. **Video Prompt Conversation**: Users record video prompts
   4. **Realtime Camera Chat**: Users have a realtime camera conversation with the model
   5. **Realtime Audio Chat**: Users have a realtime audio conversation with the model
   6. **Realtime Screen Capture Chat**: Users have a realtime screen capture conversation with the model
5. Configure the data source by selecting from:
   * **Create a new dataset** or **Append data to existing dataset**, and then specify the number of data rows you want to generate.
   * **None of the above** to skip generating data rows during project creation and generate them later.

### Step 2: Add data

If you didn't generate data rows during project creation or want to add more data rows, select **Add data from Catalog** or **Generate new data**.

If you choose to generate new data, select **Create a new dataset** or **Append data to existing dataset**, and then specify the name of the data set and the number of data rows you want to generate.

### Step 3: Select models

Click the **+ Add model** button to select models for your output evaluation project. Depending on your model selection, you can attach images, videos, and documents (PDF) to your prompt.

Currently, you can choose those foundation models integrated by [Foundry](/horizon/guides/foundry) :

| Model | Attachments type |
| - | - |
| AWS Nova Lite | Image, video, and document (PDF) |
| AWS Nova Micro | Image and document (PDF) |
| AWS Nova Pro | Image, video, and document (PDF) |
| AWS Nova Sonic Realtime | Audio |
| Claude 3.5 Haiku | Image and document (PDF) |
| Claude 3.5 Sonnet | Image and document (PDF) |
| Claude 3.7 Sonnet | Image, video, and document (PDF) |
| Claude 3.7 Sonnet Think | Image, video, and document (PDF) |
| Claude 3 Haiku | Image and document (PDF) |
| Claude 3 Opus | Image and document (PDF) |
| DeepSeek R1 | N/A |
| Google Gemini 1.5 Flash | Image, video, and document (PDF) |
| Google Gemini 1.5 Pro | Image, video, and document (PDF) |
| Google Gemini 2.0 Flash Experimental | Image, video, and document (PDF) |
| Google Gemini 2.0 Flash Thinking Mode | Image and document (PDF) |
| Google Gemini 2.5 Pro | Image and document (PDF) |
| Google Gemini Flash Experimental | Image, video, and document (PDF) |
| Google Gemini Pro | N/A |
| Google Gemini Pro Experimental | Image, video, and document (PDF) |
| Grok | N/A |
| Grok 3 | N/A |
| Llama 3.1 405b | N/A |
| Llama 3.2 | N/A |
| Llama 4 Maverick Instruct | N/A |
| OpenAI GPT 4 | N/A |
| OpenAI GPT 4.1 | Image and document (PDF) |
| OpenAI GPT-4o | Image and document (PDF) |
| OpenAI GPT-4o mini Transcribe | Audio |
| OpenAI GPT-4o Transcribe | Audio |
| OpenAI GPT-o1 | Image and document (PDF) |
| OpenAI GPT-o1-mini | Image and document (PDF) |
| OpenAI GPT-o1-preview | Image and document (PDF) |
| OpenAI o3 | Image and document (PDF) |
| OpenAI o4-mini | Image and document (PDF) |
| Whisper | Audio |

### Step 4: Configure models

Once you have chosen a model, you will be prompted to choose a **Model configuration** or create a new one. You can either add a unique name for the configuration or use the auto-generated one. **Model configuration** stores your model configuration attributes and can be reused at a later stage, which allows you to evaluate your configurations by prompting the model directly using the *send a message* text input. Each model configuration name must be unique.

You can render your content in markdown or text format. Use this toggle in the editor to switch the view. This option also allows you to format model output accordingly.

<Warning>
  ### Markdown editor size limit

  When using the Markdown editor, limit the character count to fewer than 6,000 characters.
</Warning>

<Info>
  ### LaTeX support

  To add LaTeX formatting, wrap your math expressions using backticks and dollar signs. The editor supports both inline and block LaTeX formatting. For example, to add LaTeX formatting for `x=2`, put `$$x = 2$$`.
</Info>

Repeat the model configuration process to have a 1-10 model. You can use the same model with different parameters if your use case focuses on a specific model evaluation. See [limits](/horizon/guides/limits#annotate) for your account limits.

#### Add attachment

Depending on the model capability, you can attach image, video, text and PDF files to your prompts. To add one or more attachments, click the paper clip icon at the prompt level and select from **Add from a public link** and **Upload from computer**. Once you have entered a valid URL to insert a supported type of attachment or upload a valid local file, click **Save** and repeat to add more attachments if necessary.

#### Customize system prompt

### Step 5: Set up an ontology

[Create an ontology](/horizon/guides/labelbox-ontology#create-a-new-ontology) for evaluating model responses.

The editor supports the following options:

| Feature | Description | Export format |
| - | - | - |
| **Message selection** | Select single or multiple responses that meet specific criteria. | [Payload](/horizon/developer-guides/export-multimodal-chat-annotations#message-selection) |
| **Message ranking** | Rank multiple model-generated responses to determine their relative quality or relevance. | [Payload](/horizon/developer-guides/export-multimodal-chat-annotations#message-ranking) |
| **Message step reasoning** | (Text conversations only, no multimodal support) Evaluate the accuracy of each step broken down from responses and label it as **correct**, **neutral**, or **incorrect**. Provide a **justification** for incorrect steps and regenerate the conversation from that step. | [Payload](/horizon/developer-guides/export-multimodal-chat-annotations#message-step-reasoning) |
| **Classification - Radio** | Select one option from a predefined set. | [Payload](/horizon/developer-guides/export-multimodal-chat-annotations#classification-radio) |
| **Classification - Checklist** | Choose multiple options from a list. | [Payload](/horizon/developer-guides/export-multimodal-chat-annotations#classification-checklist) |
| **Classification - Free text** | Add free text annotations. | [Payload](/horizon/developer-guides/export-multimodal-chat-annotations#classification-free-form-text) |

Classification tasks can apply globally to the entire conversation or individually to a message. They can also nest subclassification tasks.

<Info>
  ### Message step reasoning best practices

  Message step reasoning is an [experimental](/horizon/guides/product-release-phases#experimental) feature. For projects using ontologies with the step reasoning task, ensure that prompts lead to responses that can be easily broken down into clear steps.
</Info>

### Step 6: Complete model setup

You need to click the **Complete setup** button to lock in your selection of models and generate the labeling queue. Once clicked, you won't be able to alter or remove model selection.

### Step 7: Complete annotation tasks

Click the **Start labeling** button to chat with your configured models and add annotations to evaluate the responses. If multiple models are selected in the setup, their response order is random at each turn to prevent bias. The display order may also change when you refresh the browser.

You can continue to prompt models after the initial input by sending a new prompt. If you have made a mistake in your prompt or encountered a blocker, you can reset your prompt and the model outputs.

For each prompt, you can generate additional responses or write your own using the Markdown editor. Each time you submit a written response, the [AI critic](/horizon/guides/code-grammar-assistance) tool automatically checks for grammar and code errors and proposes suggestions. Repeat step 7 to complete all tasks in your [workflow](/horizon/guides/workflows).

### Customize prompts

You can add custom prompts to control how the model responds, such as rendering outputs in different languages and formats. Labelbox applies system prompts in the following order of precedence:

1. **Data row-level system prompt**: A system prompt is set at the data row level and the text field is not empty.
2. **System prompt**: The **Customize system prompt** option is selected and set as part of model configuration.
3. **Labelbox default prompt**: If you don’t configure either of the above, Labelbox applies a default prompt that ensures proper LaTeX formatting.

<Info>
  ### Latex formatting

  When rendering a LaTeX math expression, wrap the expression in double dollar signs. For example: `$$x^2=4$$`. When rendering LaTeX inside a code block, also use double dollar signs. For example: \`\`\`$E=mc^2$\`\`\`. If you use a data row-level system prompt and expect Markdown or LaTeX rendering, include the correct LaTeX delimiters in your prompt to match your project settings.
</Info>


# LLM human preference
Source: https://docs.labelbox.com/horizon/guides/llm-human-preference-editor

A guide to creating an LLM human preference project for comparing & classifying model outputs on conversation text data.

With the LLM human preference editor, you can create human preference data for model comparison or RLHF (reinforcement learning with human feedback). You can compare model outputs side-by-side and select the most favorable model output on a conversational text thread by assigning the model output a classification.

This editor solves two important problems that are critical to ensuring responsible AI aligned with human preferences:

* Model comparison: Conduct the evaluation and comparison of model configurations for a given use case and decide which one to pick.
* RLHF: Create preference data for training a reward model for RLHF based on multiple outputs from a single model.

## Supported classifications for model response classification

| Classification type | Import format | Export format |
| - | - | - |
| Radio | [See sample](/horizon/developer-guides/import-llm-response-evaluation-annotations#classification-radio-single-choice) | [See sample](/horizon/developer-guides/export-llm-response-evaluation-annotations#classification-radio) |
| Free-form text | [See sample](/horizon/developer-guides/import-llm-response-evaluation-annotations#classification-free-form-text) | [See sample](/horizon/developer-guides/export-llm-response-evaluation-annotations#classification-free-form-text) |
| Checklist | [See sample](/horizon/developer-guides/import-llm-response-evaluation-annotations#classification-checklist-multi-choice) | [See sample](/horizon/developer-guides/export-llm-response-evaluation-annotations#classification-checklist) |

## Set up an LLM human preference project

For this version of the model comparison editor, Labelbox assumes you already have a large language model set up in your own environment that you can use to generate preliminary model responses on your conversational text data.

### Step 1: Import conversation data & model responses

Import using your conversation data & model responses.

To import secured URLs, we recommend setting up your IAM delegated access integration. For instructions on setting this up, see [IAM integration](/horizon/developer-guides/cloud-storage-iam-integration).

1. In your cloud bucket, create a JSON file for each conversational text data row. Each JSON file should contain the model output. You may import content as markdown or text. Use this sample as a guide for creating the JSON files for each conversational text thread and the model output on that thread.

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "type": "application/vnd.labelbox.conversational",
    "version": 1,
    "messages": [
      {
        "messageId": "message-0",
        "timestampUsec": 1530718491,
        "content": "Hi! How can I help?",
        "user": {
          "userId": "Bot 002",
          "name": "Bot"
        },
        "align": "left",
        "canLabel": false
      },
      {
        "messageId": "message-1",
        "timestampUsec": 1530718503,
        "content": "I'm looking to buy a vacuum cleaner this Black Friday. Help me with the best deal?",
        "user": {
          "userId": "User 00686",
          "name": "User"
        },
        "align": "right",
        "canLabel": true
      },
      {
        "messageId": "message-2",
        "timestampUsec": 1530718516,
        "content": "Of course!! I'll be your personal shopping assistant. So tell me, what are you looking for? Is it house or an apartment? What kind of floors? Finally, do you have a budget?",
        "user": {
          "userId": "Bot 002",
          "name": "Bot"
        },
        "align": "left",
        "canLabel": false
      },
      {
        "messageId": "message-3",
        "timestampUsec": 1530718528,
        "content": "Hmm...I have a 2 bedroom apartment with hardwood floors. The square footage is around 1000 sqft. And yes on budget, looking to land something below 600 bucks",
        "user": {
          "userId": "User 00686",
          "name": "User"
        },
        "align": "right",
        "canLabel": true
      }
    ],
    "modelOutputs": [
      {
        "title": "Response A",
        "content": "I have 2 options for you\n- The Dyson V15 [Product page](https://www.dyson.com/vacuum-cleaners/cordless/v15)\n- The Shark Stratos [Product page](https://www.sharkclean.com/products/shark-stratos-cordless-vacuum-with-free-steam-mop-zidIZ862HB)",
        "modelConfigName": "GPT-3.5 with temperature 0"
      },
      {
        "title": "Response B",
        "content": "Although I have a couple of options, I would recommend the Dyson V15 [Product page](https://www.dyson.com/vacuum-cleaners/cordless/v15) based on what you need.\n",
        "modelConfigName": "GPT-4 with temperature 0.2"
      }
    ]
  }
  ```
</CodeGroup>

2. In a separate JSON file, put the URLs to the cloud-hosted JSON files containing the conversation text data and model outputs. This is the file you will import to Labelbox. Use the sample file below as a guide for formatting your import file.

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  [
      {
        "row_data": "https://storage.googleapis.com/labelbox-datasets/conversational-sample-data/pairwise_shopping_1.json",
        "global_key": "global_key_1"
      },
      {
          "row_data": "https://storage.googleapis.com/labelbox-datasets/conversational-sample-data/pairwise_shopping_2.json",
          "global_key": "global_key_2"
      },
      {
          "row_data": "https://storage.googleapis.com/labelbox-datasets/conversational-sample-data/pairwise_shopping_3.json",
          "global_key": "global_key_3"
      }
  ]
  ```
</CodeGroup>

3. Upload your data via Python SDK

To send your data to Labelbox, refer to our guidelines on [conversational text](/horizon/developer-guides/conversational-text) upload

To learn how to upload the URLs to your cloud-hosted JSON files (and import human preference prelabels with Model-assisted labeling) via the Python SDK, see this [developer guide](/horizon/developer-guides/import-llm-response-evaluation-annotations).

#### Alternative data types

You can also make this editor multi-modal (image, video, audio) by adding HTML to your JSON payload.

**Image**

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
      "type": "application/vnd.labelbox.conversational",
      "version": 1,
      "messages": [
          {
              "messageId": "message-0",
              "content": "Ideogram prompt: Generate an image of a red tulip",
              "user": {
                  "userId": "User 000",
                  "name": "User"
              },
              "align": "left",
              "canLabel": false
          }
      ],
      "modelOutputs": [
          {
              "title": "Image 1",
              "content": "<img style='width:50%' src=\"https://storage.googleapis.com/labelbox-datasets/image_sample_data/a-stunning-high-resolution-image-of-a-vibrant-red--FKZ_pkdaRDKZxQC1hJ45yw-9OoHdU5OQzyhpL-cwSz1aw.jpeg\" />",
              "modelConfigName": "ideogram_image_1"
          },
          {
              "title": "Image 2",
              "content": "<img style='width:50%' src=\"https://storage.googleapis.com/labelbox-datasets/image_sample_data/a-stunning-close-up-image-of-a-vibrant-red-tulip-t-mqx4mXo_S5ubGn7kU-oxJA-9OoHdU5OQzyhpL-cwSz1aw.jpeg\" />",
              "modelConfigName": "ideogram_image_2"
          }
      ]
  }
  ```
</CodeGroup>

Using the markdown view, your images will be rendered in the editor.

#### video

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
      "type": "application/vnd.labelbox.conversational",
      "version": 1,
      "messages": [
          {
              "messageId": "message-0",
              "content": "<video style='width:100%' src='https://storage.googleapis.com/gtv-videos-bucket/sample/ForBiggerMeltdowns.mp4' controls></video>",
              "user": {
                  "userId": "User 000",
                  "name": "User"
              },
              "align": "left",
              "canLabel": false
          }
      ],
      "modelOutputs": [
          {
              "title": "Transcription 1",
              "content": "[background music] Argh... I am a monster... arghhh....arghhh",
              "modelConfigName": "Gemini_LLM_Response"
          },
          {
              "title": "Transcription 2",
              "content": "[background music] Ahhhh... I'm a monster ahhhhhhh...ahhhhh",
              "modelConfigName": "GPT_LLM_Response"
          }
      ]
  }
  ```
</CodeGroup>

Using the markdown view, your videos will be rendered in the editor.

**audio**

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
      "type": "application/vnd.labelbox.conversational",
      "version": 1,
      "messages": [
          {
              "messageId": "message-0",
              "content": "LLM sound prompt : firework finals popping",
              "user": {
                  "userId": "User 000",
                  "name": "User"
              },
              "align": "left",
              "canLabel": false
          }
      ],
      "modelOutputs": [
          {
              "title": "Sound 1",
              "content": "<audio controls> <source src=\"<https://storage.googleapis.com/labelbox-datasets/audio-sample-data/Firework_1.m4a>\" type=\"audio/mpeg\"> </audio>",
              "modelConfigName": "LLM_sound_1"
          },
          {
              "title": "Sound 2",
              "content": "<audio controls> <source src=\"<https://storage.googleapis.com/labelbox-datasets/audio-sample-data/Firework_2.m4a>\" type=\"audio/mpeg\"> </audio>",
              "modelConfigName": "LLM_sound_2"
          }
      ]
  }
  ```
</CodeGroup>

Using the markdown view, your audio will be rendered in the editor.

### Step 2: Create an LLM human preference project

After you have imported your data, go to [Annotate](/horizon/guides/annotate-overview) and click **+ New project**. Select **LLM human preference**. Provide a name and optional description for your project and configure your quality mode settings.

### Step 3: Select data rows from Catalog

During the project setup, click **Add data**. This will bring you to Catalog, where you can select the data rows you wish to label in this project. Use the Catalog filters to query your data rows.

### Step 4: Create the project ontology

Create an ontology for classifying the model responses on each data row.

### Step 5: Classify model responses

You have two options for assigning classifications to model responses.

1. Label from scratch: Upload model outputs and have your team assign classifications in the editor from scratch.
2. Import pre-labels via Model-assisted labeling: Upload classification predictions to your model outputs via Model-assisted labeling to give your labeling team a warm start. To learn how to import pre-labels, see [Import LLM human preference annotations](/horizon/developer-guides/import-llm-response-evaluation-annotations).

#### Toggle markdown view

You can render your content in markdown or text format. Use this toggle in the editor to switch the view. This option also allows you to have other types of media, including images and videos to render.


# How to manage your payment & billing information
Source: https://docs.labelbox.com/horizon/guides/manage-billing-details

Learn how to view your billing details and update your payment method.

Managing your Labelbox billing details ensures your subscription remains active and your team has uninterrupted access to Labelbox. This guide will walk you through how to access your billing portal, update payment methods, change your primary billing contact, and review your past invoices.

## View your billing details

Accessing your billing portal is the first step to managing your account's financial information. Here, you can get a high-level overview of your payment methods and subscription status.

To view your billing details:

1. Open your Labelbox account and navigate to the **Billing** tab.
2. Click the **Manage** button displayed to the right of your current account type.
3. Once you click **Manage**, you will be redirected to the **Manage your Labelbox plan** view.

*Note: Depending on your subscription plan (such as Enterprise accounts), the* **Manage** *button may not be available. If you do not see this button and need to adjust your billing information, please reach out to the support team for assistance.*

## Add a new payment method

You may need to add a new credit card or payment method if your current one is expiring or if your team is switching to a different corporate account.

To add a new payment method, navigate to the **Payment Method** section and click the **Add payment method** link. Follow the prompts to securely enter your new payment information.

## Remove a payment method

Keeping your payment options up-to-date helps prevent billing issues and accidental charges on obsolete cards.

To remove an outdated payment method, click the **Remove (x)** button displayed next to the payment details you wish to delete.

*Important:* You must have at least one active payment method on file to maintain your subscription. If you are replacing an expired payment method, make sure to add the new method first before attempting to remove the old one.

## Change your billing contact details

Your billing contact is the individual or department responsible for receiving all billing-related correspondence, including invoices and renewal notices. Keeping this information current ensures invoices always reach the right inbox (e.g., your finance department).

To view or update your billing contact, go to the **Billing Information** section and select **Update information**. From here, you can easily modify the name, address, email address, and phone number associated with your account's billing.

## Review your invoices and payments

You can easily access your past invoices in the platform, which is especially useful for expense reporting, accounting, or tax purposes.

Use the **Invoice History** section to review past invoice details and check on current payment statuses.

* To view or download a specific invoice or receipt, simply click on the invoice date.
* If you have any questions or notice a discrepancy on an invoice, please contact Support for help.


# Manage your cloud integrations
Source: https://docs.labelbox.com/horizon/guides/manage-cloud-integrations



All management tasks for your cloud integrations can be found by navigating to **Settings > Workspace > Integrations**.

This page provides a central dashboard where you can view, update, and manage all your configured cloud storage connections.

## View your integrations

The Integrations page displays a list of all your active connections. This table gives you an at-a-glance overview of each integration's status and details.

| Column name | Description |
| - | - |
| Name | The custom name you assigned to the integration for easy identification (e.g., "Production Data Bucket"). |
| Provider | The cloud provider for this integration (AWS, GCP, or Azure). |
| Created at | The date the integration was first configured in Labelbox. |
| Last updated | The last time any details of the integration were modified. |
| Last used | The most recent time the integration was used by Labelbox to access data for a project. |
| Status | Indicates the health of the connection. A green checkmark means the connection is working correctly. |

## Update an integration

You can modify certain details of an integration after it has been created. This is useful if you need to update credentials or change the name.

To update an integration:

1. Navigate to the **Settings > Integrations** page.
2. Locate the integration you wish to update in the list.
3. Click the three-dot menu (...) on the far right of the integration's row and select **Edit**.
4. A modal window will appear where you can modify the integration's details.

What you can update:

* **Integration Name:** You can change the custom name at any time.
* **Credentials:** You can update the credentials used to connect (e.g., paste a new JSON key for GCP, update a client secret for Azure, or change the Role ARN for AWS).

<Note>
  Note: You cannot change the cloud provider (e.g., from AWS to GCP) or the specific resource being accessed (e.g., change the S3 bucket name) after an integration is created. To connect to a different resource, you must create a new integration.
</Note>

## Delete an integration

Deleting an integration permanently removes the connection from Labelbox. This action will break any datasets that rely on this integration to access their data.

<Warning>
  Warning: This action cannot be undone. Datasets linked to this integration will no longer be able to load their data. Proceed with caution.
</Warning>

To delete an integration:

1. Navigate to the **Settings > Integrations** page.
2. Locate the integration you wish to delete.
3. Click the three-dot menu (...) on the far right and select **Delete**.
4. A confirmation prompt will appear. To proceed, you must type the name of the integration to confirm your choice.
5. Click **Delete**.


# Manage members and groups
Source: https://docs.labelbox.com/horizon/guides/manage-members-and-groups

Instructions for managing members and groups permissions.

A key part of managing any labeling operation is controlling who has access to what. Labelbox gives you powerful, flexible tools to manage your team, whether you have a small group of labelers or a large, multi-team organization. This guide explains the core concepts of members and groups and provides step-by-step instructions for managing them.

## Key concepts

Before diving into the "how-to," it's important to understand the two main components of team management in Labelbox:

* A ***member*** is any individual user account with access to your Labelbox organization. Every person you invite to Labelbox is a "member." You can assign roles and project access to members one by one. This is great for small teams or for users who need unique permissions.
* A ***group*** is a collection of members that you can manage as a single unit. Instead of assigning projects and roles to each person individually, you can assign them to a group. Everyone in that group automatically inherits the group's access and permissions.

## Required permissions

To add members to a project and set their roles and permissions, you must have one of the following two roles:

* **Tenant admins** can manage member roles across multiple workspaces under the same organization and create new workspaces.
* **Admins** can manage member roles within one workspace and can't make changes to other workspaces under the same organization.

| Permission | Tenant admin | Admin |
| - | - | - |
| Add members to the workspace | <Icon icon="check" /> | <Icon icon="check" /> |
| Remove members from the workspace | <Icon icon="check" /> | <Icon icon="check" /> |
| Create groups | <Icon icon="check" /> | <Icon icon="check" /> |
| Add member to existing groups | <Icon icon="check" /> | <Icon icon="check" /> |
| Add groups to projects | <Icon icon="check" /> | <Icon icon="check" /> |
| Manage roles and permissions of workspace members | <Icon icon="check" /> | <Icon icon="check" /> |
| Create custom roles | <Icon icon="check" /> | <Icon icon="check" /> |
| Create new workspaces under the organization | <Icon icon="check" /> | |
| Create custom API keys | <Icon icon="check" /> | <Icon icon="check" /> |

## Manage individual members

You can always grant a specific user access to a project or your entire workspace. This is useful for assigning administrative roles or for one-off project assignments.

### Invite members with a workspace-wide role

A workspace-wide role gives a user the same level of access to *every project* in the workspace. This is best for administrators, team managers, or stakeholders who need broad visibility.

1. Navigate to **Workspace settings** and select the **Members** tab.
2. Click **+ Invite members**.
3. Enter the email address(es) of the user(s) you want to invite.
4. Check the **Workspace-wide role** box.
5. Select a workspace-level role (e.g., Admin, Team Manager) from the **Role** dropdown menu.
6. Click **Invite**.

### Remove a member from the workspace

1. Navigate to **Workspace settings** in the bottom-left corner and select the **Members** tab.
2. Use the search bar to find the member whose role you want to modify.
3. Click the three-dot icon at the far right of the member's row.
4. Select **Remove** from the dropdown menu.
5. Confirm the action. The user will be removed from the workspace and will lose access to all of its projects.

### Invite members to specific projects

You can also give a user a role on one or more specific projects without giving them access to the entire workspace. This is the most common way to grant access to labelers or reviewers who only need to work on specific tasks.

1. Go to **Workspace settings** and select the **Members** tab.
2. Click **+ Invite members**.
3. Enter the email address(es) of the user(s).
4. Use the **Search projects** bar to find and select all the projects you want to assign to the user(s).
5. In the **Role in groups and projects** dropdown, select a project-level role (e.g., Labeler, Reviewer).
6. Click **Invite**.

### Remove a member from a specific project

1. Follow steps 1-3 above to open the member's detailed profile view.
2. Find the project you wish to remove the member from.
3. Click the **"X"** icon to the right of the project role.
4. Confirm the action. The user will lose access to that project but will retain access to any other projects they are assigned to.

## Manage teams at scale with groups

While you can manage everyone individually, using groups becomes essential as your team grows. Groups allow you to:

* **Manage at scale:** Onboard an entire team to a new project in a single step, rather than inviting dozens of people one by one.
* **Improve efficiency:** Update permissions for multiple users at once. If a team's responsibilities change, you only need to edit the group's settings.
* **Maintain clarity:** Keep your projects organized by creating groups that reflect your company's team structure (e.g., "Data Science Team," "Pedestrian Annotation Team," "External QA").

<Tip>
  **Best practice**: We recommend using groups as the primary way to manage project access.

  Managing individual permissions should be reserved for unique exceptions.
</Tip>

### Create a new group

First, you'll create a group and assign it to the relevant projects.

1. Navigate to the **Groups** tab in **Workspace settings**.
2. Click **New Group**.
3. Enter a descriptive name (e.g., "Video Annotation Team") and select a color.
4. Under **Assign Projects**, search for and select the projects this group will work on.
5. *Optional:* You can add initial members now, or add them later.
6. Click **Confirm**.

### Add members to a group

Once a group is created, you can add members. When you add a member to a group, they are automatically granted the appropriate role on all projects assigned to that group.

1. Go to the **Groups** tab and click on the group you want to modify.
2. Click **Add members**.
3. Enter the members' email addresses and click **Submit**. They will now have access to all of the group's projects.

<Info>
  To search for a member:

  * Select either **Workspace** or **Project** before searching, as each member belongs to only one scope, and searches can only be performed within the selected scope.
  * Use quick search options only if you are a labeling services customer. Otherwise, use the standard search options.
</Info>

### Manage project access for a group

If your team's focus shifts, you can easily add or remove projects from a group. This will update the access for all members of the group simultaneously.

1. Navigate to the **Groups** tab.
2. Find the group you wish to edit and click the three-dot icon, then select **Edit**.
3. In the **Assign Projects** section, add or remove projects as needed.
4. Click **Confirm** to save the changes.


# Requesting managed labeling services for your project
Source: https://docs.labelbox.com/horizon/guides/managed-services

Learn how to request fully managed labeling services for your labeling project.

Labelbox provides access to a skilled workforce to help you annotate and review large volumes of data. This allows you to scale your data labeling operations efficiently.

This guide focuses on setting up and managing the **Managed Services** offering.

## How to request managed labeling services

Follow these steps to get a new labeling project started with a dedicated workforce.

1. From the Labelbox app home page, under **Managed services**, click on **Select a project to start**.
2. You can either:
   * **Select an existing project**: Select an existing project from the project list that you want to add a labeling workforce to.
   * **Create a new project**: Alternatively, you can click **+ New project with labeling services** to start from scratch. If you create a new project, you will need to complete the standard project setup steps first.
3. **Provide clear instructions**: Once your project is selected, you will be taken to the project overview page where you can complete the **Labeling services project setup**. Click on **Add instructions**. This is a crucial step where you must provide detailed guidance for the labelers. You can either type directly in the editor or upload a file (like a PDF or Word document) with your guidelines. Click **Save** when you are finished.

After saving the instructions, a tracker will appear on the project overview page, showing the status of your request. The initial status will be **Requested**. You can still modify your instructions at this stage by clicking **Edit instructions**.

## Supported skills and expertise

The labeling workforce is powered by the Alignerr community, which consists of highly-educated experts proficient in over 30 languages and various advanced subjects. This enables them to assist with critical and complex tasks, including:

* Reinforcement Learning with Human Feedback (RLHF)
* Supervised fine-tuning (SFT)
* Multimodal LLM evaluation and preference ranking
* Red teaming and LLM chat arena tasks
* Specialized tasks for text-to-image, video, and audio
* Coding and AI agent-related tasks

## Security and privacy

Labelbox is committed to ensuring the security and privacy of your data, which involves rigorous security measures and strict privacy protocols to protect sensitive information.

### Workforce partner security

Data protection is a top priority for Labelbox. To ensure the most qualified partnerships, our Workforce and Business Process Outsourcing (BPO) vendors undergo a detailed security review regarding the security practices they have implemented. Our partners comply with a wide range of industry compliance standards including ISO 27001, SOC 2, HIPAA, and GDPR.

Notably, all Labelbox workforce partners have thorough role-based access controls (RBAC), ethical business practices policies (including fair wage), and data transfer policies to keep customer data safe and secure. In addition, all partners ensure that their employees strictly follow physical access controls that include, but are not limited to, secure on-premise entry protocols, clean desk protocol, and prohibiting the use of mobile phones in the production environment.Labelbox does not consider partnerships with companies that do not adhere to the robust security, privacy, and ethical practices of Labelbox.

### Data privacy in Labelbox workforce services

In projects utilizing Labelbox workforce services, both BPO service providers and our Workforce members shall not see any personally identifiable information (PII) of customers that can identify an individual or the Organization they are a part of.

* **For Labelbox Workforce**: They cannot see emails, user names, project names, descriptions, tags, and organization names for the projects they work on.
* **For Customers**: They cannot see the emails and user names of the BPO members working on their projects.

These privacy settings are automatically applied to all relevant projects to protect sensitive information. For example, on the Performance Dashboard, emails are obfuscated on both sides to prevent Workforce and Customers from identifying each other. For instance, an email like `jane.doe@example.com` would be turned into something like `usr.name.cptsivzdhkn492z1umb59sakh2@internal.labelbox.com`.


# Connect Azure Blob Storage to Labelbox via IAM Delegated Access
Source: https://docs.labelbox.com/horizon/guides/microsoft-azure-blob-storage

Learn how to import your Azure Blob Storage data to Labelbox via IAM delegated access.

This guide provides a complete set of step-by-step instructions for securely connecting your Microsoft Azure Blob Storage data to Labelbox using IAM delegated access.

## Prerequisites

Before you begin, please ensure you have the following:

* Permissions to register applications and assign roles within your Azure subscription.
* The name of the Azure Storage Account you wish to connect.
* Configure [Cross-Origin Resource Sharing (CORS)](/horizon/guides/create-cors-headers) on your storage account.

<Warning>
  Labelbox is not currently compatible with Azure Data Lake Storage (ADLS) Gen2. You must use Azure Blob Storage for this integration.
</Warning>

## Step 1: Add the Labelbox application to your Azure tenant

First, you need to authorize the official Labelbox enterprise application within your Azure Active Directory tenant.

1. In Labelbox, navigate to **Settings > Integrations** and click **New integration**.
2. Select **Microsoft Azure**.
3. On the integration page, click the **Add Labelbox to Azure tenant** button.
4. You will be redirected to a Microsoft login page to grant permission. This one-time action installs an enterprise application in your Azure tenant that allows Labelbox to handle delegated access securely.

## Step 2: Grant permissions in Azure

Next, you will assign the necessary roles to the Labelbox application to allow it to access your storage account and container.

### Part A: Assign 'Storage blob delegator' role

1. In your Azure portal, navigate to the **Storage account** you want to connect.
2. Go to the **Access control (IAM)** page.
3. Click **Add > Add role assignment**.
4. For the **Role**, select **Storage Blob Delegator**.
5. In the **Members** tab, click **Select members** and search for the `labelbox_azure_delegated_access` application.
6. Select the application and save the role assignment.

### Part B: Assign 'Storage blob data reader' role

1. Navigate to the **container** within your storage account that you want to connect.
2. Go to its **Access control (IAM)** page.
3. Click **Add > Add role assignment**.
4. For the **Role**, select **Storage Blob Data Reader**.
5. Assign this role to the same `labelbox_azure_delegated_access` application.
6. Save the role assignment.

<Note>
  Role-Based Access Control (RBAC) changes in Azure can take up to 30 minutes to take effect.
</Note>

## Step 3: Complete and validate the integration in Labelbox

Finally, return to Labelbox to complete the setup and validate the connection.

1. Go back to the Azure integration page in Labelbox that you opened in Step 1.
2. Enter your Azure **Tenant ID** and the **Container URL**.
3. Click **Save integration**.
4. Labelbox will automatically validate the integration. You can check the status on the **Integrations > Manage integrations** page. If the validation fails, you can review the error messages to troubleshoot your configuration.

## Step 4: Upload your data

Once the connection is successfully established, you can begin uploading your data to Labelbox.

When preparing your import file, make sure you use correctly formatted Azure Blob Storage URLs. **Example**: `https://your-storage-account.blob.core.windows.net/your-container/image.jpg`. To learn how to format your import file, visit these guides:

<Columns>
  <Column>
    <Card title="Import image data" icon="plus" href="/horizon/developer-guides/import-image-data" />

    <Card title="Import text data" icon="plus" href="/horizon/developer-guides/import-text-data" />

    <Card title="Import document data" icon="plus" href="/horizon/developer-guides/import-document-data" />

    <Card title="Import HTML data" icon="plus" href="/horizon/developer-guides/import-html-data" />
  </Column>

  <Column>
    <Card title="Import video data" icon="plus" href="/horizon/developer-guides/import-video-data" />

    <Card title="Import geospatial data" icon="plus" href="/horizon/developer-guides/import-geospatial-data" />

    <Card title="Import audio data" icon="plus" href="/horizon/developer-guides/import-audio-data" />
  </Column>
</Columns>

Your dataset should now be set up with IAM delegated access. Labelbox will use the roles you assigned to securely access data from your Azure Blob Storage container.

This guide provides the exact steps to configure Azure Active Directory and Labelbox for IAM Delegated Access.

## Optional security measures

To enhance the security of your Azure integration, Labelbox recommends implementing the following optional configurations. These steps help ensure that access to your data is strictly controlled.

### Set container access to private

To prevent any possibility of unauthorized public access, you should configure your blob container’s access level to **Private**.

1. In the Azure portal, navigate to the storage container you have connected to Labelbox.
2. From the container's settings menu, select **Change access level**.
3. Set the **Public access level** to **Private (no anonymous access)**.
4. Click **OK** to save the changes.

This setting ensures that all requests to the container must be authorized, and no data can be accessed anonymously. To learn more, visit the [Microsoft docs](https://learn.microsoft.com/en-us/azure/storage/blobs/anonymous-read-access-configure?tabs=portal).

### Restrict access by IP address

For an additional layer of security, you can configure your storage account’s firewall to allow access only from trusted IP addresses. If you enable this feature, you must add the IP addresses for both the Labelbox servers and your organization’s users who require direct access to the data. To learn more, visit the [Microsoft docs](https://learn.microsoft.com/en-us/azure/storage/common/storage-network-security?tabs=azure-portal#grant-access-from-an-internet-ip-range).

1. In the Azure portal, navigate to the **Storage account** that contains your connected container.
2. Under **Security + networking**, select **Networking**.
3. In the **Public network access** tab, select **Enabled from selected virtual networks and IP addresses**.
4. In the **Firewall** section, add the IP address ranges for Labelbox servers and any internal users who need access.
5. Click **Save** to apply the firewall rules.

By implementing these two configurations, you can significantly strengthen the security of your data while maintaining seamless integration with Labelbox.

### Integrate with Microsoft Entra ID

Use the official Labelbox application for Microsoft Entra ID to simplify user management and enable Single Sign-On (SSO).

You can install the pre-verified Labelbox enterprise application directly into your Azure tenant.

Microsoft provides a comprehensive tutorial with detailed steps for the entire configuration process.

**Get Started: [Follow the Labelbox setup tutorial on Microsoft Learn](https://www.google.com/url?q=https%3A%2F%2Flearn.microsoft.com%2Fen-us%2Fentra%2Fidentity%2Fsaas-apps%2Flabelbox-tutorial "https://learn.microsoft.com/en-us/entra/identity/saas-apps/labelbox-tutorial")**


# Monitor labeling performance
Source: https://docs.labelbox.com/horizon/guides/monitor

Monitor the performance of your labeling project and labeling service workforce.

The [Monitor](https://app.labelbox.com/monitor) page allows you to monitor labeling operations and performance across all projects in real-time. It provides filters and visualizations to help you identify outlier labelers and make decisions to improve productivity and efficiency.

The Monitor page has the following sections:

* **Project Performance**: Shows performance metrics for each project, allowing you to monitor and analyze performance at the project level.
* **Member performance**: Shows performance metrics for individual labelers, allowing you to assess and compare the performance of each team member, including your own and requested [labeling services](/horizon/guides/labeling-services) workforce.
* **Performance charts**: Visualizes key performance metrics indicating the throughput, efficiency, and quality of the labeling process, including all charts available on the [performance dashboard](/horizon/guides/performance-dashboard).

To read these numbers from your own tools, see the [Performance API](/horizon/developer-guides/performance-api).

<Info>
  **Monitor is an enterprise only feature**

  Only organizations on the enterprise plan can access the Monitor page. To upgrade to the enterprise plan, please [contact sales](https://labelbox.com/sales/).
</Info>

## Performance charts

The Monitor page displays metrics in the following categories (separated by subtab):

* **Throughput**: Tracks the volume of labeling, annotation, and review activities over time.
* **Efficiency**: Measures the time taken for labeling, reviewing, and rework activities to assess process efficiency.

By default, the page displays all available metrics, but you can adjust the number, order, and density of metrics displayed on the performance tables and charts. Each metric automatically detects and highlights outlier data for your attention. For example, if you see a decline in quality scores, you can quickly investigate and take corrective action, such as redistributing tasks or providing additional guidance to your team. You can also apply these metrics as filters to find certain projects and members.

### Throughput

The Performance charts in the Monitor tab provide visual aids to help you understand how your data rows are progressing through the workflow steps over a selected time period.

The Throughput graphs in the Performance charts section of the Production Monitor display the following:

| **Metric** | **Description** |
| - | - |
| **Done** | The number of data rows in the Done state of the project workflow. Hover over the chip in the upper right corner to see the number of data rows moved to Done in the selected time period. |
| **Labels** | The number of labeled data rows over a specified time period, including deleted labels by default. |
| **Annotations** | The number of annotations (features) created over a specified time period, including deleted annotations by default. |
| **Reviews** | The count of *approve* and *reject* actions performed on labeled data rows within a project. |
| **Total time** | The total time spent on labeling, reviewing, and reworking data rows. |
| **Labeling time** | The total time spent labeling data rows. Time increments when a labeler skips or submits an asset in the labeling queue. |
| **Review time** | The total time spent reviewing labeled data rows. For reviewers, time increments when they view an asset in the review queue or in the data row browser view. For labelers, time increments when they view submitted labels. |
| **Rework time** | The total time spent on reworking labeled data rows. Time increments when a labeler submits an asset in the rework queue, navigates to the review queue to make edits, or approves/rejects the asset in the data row browser view. |

#### Average daily throughput

The **Done**, **Labels**, **Annotations**, and **Reviews** charts in the **Performance charts** > **Throughput** section contain a chip that displays the average daily throughput for the data rows in that chart. Hover over the chips to see a further breakdown of the average daily throughput.

### Efficiency

The Monitor page has the following efficiency metrics:

| Metric | Description |
| - | - |
| **Avg time per label** | The average labeling time spent per label. Avg time per label = Total labeling time / number of labels submitted |
| **Avg review time** | The average review time per data row. Avg review time = Total review time / number of data rows reviewed |
| **Avg rework time** | The average rework time per data row. Avg rework time = Total rework time/ number of data rows reworked |
| **AHT per labeled data row** | Includes creating, reviewing, and reworking time. Calculated as total time / number of labeled data rows. Includes all data rows with at least one label. |
| **AHT per done data row** | Includes creating, reviewing, and reworking time. Calculated as total time / number of done data rows. Only includes data rows marked as Done in the workflow. |
| **AHT per created label** | Includes creating, reviewing, and reworking time. Calculated as total time / number of labels created. Includes all created labels (skipped, abandoned, and submitted). |
| **AHT per submitted label** | Includes creating, reviewing, and reworking time. Calculated as total time / number of submitted labels. Excludes skipped and abandoned labels. |
| **AHT per done label** | Includes creating, reviewing, and reworking time. Calculated as total time / number of labels on done data rows. Excludes skipped and abandoned labels. |


# Multi-factor authentication
Source: https://docs.labelbox.com/horizon/guides/multifactor-authentication

This article describes how to set up multifactor authentication (MFA) for Labelbox accounts.

Labelbox supports **multi-factor authentication** (MFA), also known as **two factor authentication** (2FA), which adds an extra layer of security to prevent unauthorized access. Workspace administrators can activate MFA for accounts managed by Labelbox.

If you use [single sign on](/horizon/guides/single-sign-on) (SSO) to manage access to Labelbox and your SSO provider supports MFA, we strongly recommend using your provider's MFA implementation. To learn more, consult your provider's docs.

## Activate multi-factor authentication

To activate MFA for all users of your Labelbox workspace:

1. Sign in to Labelbox as a workspace administrator.

2. From the account menu, select **Workspace settings** and then select the **Workspace** tab.

3. Locate the **Multi factor authentication** section and find the *Disabled for all workspace users* setting. Select this to activate MFA for all workspace users.
   If a checkmark already exists and the option is titled *Enabled* for all workspace users, MFA is already enabled.

When you change the setting, a background process starts working on the process. Use the **Notifications** page to follow progress.

## Labelbox authentication process

When MFA is enabled, you are prompted to enter a code to complete the sign-on process. This code is generated by an authenticator app associated with your credentials.

You can associate an app (and device) with your credentials the first time you sign in to Labelbox after MFA is enabled. When you do this, a prompt displays a QR code. Scan this code from an authentication app running on a device typically in your possession, such as a mobile phone. You can use any major authentication app, such as [Google Authenticator](https://play.google.com/store/apps/details?id=com.google.android.apps.authenticator2), [Microsoft Authenticator](https://support.microsoft.com/en-us/account-billing/download-and-install-the-microsoft-authenticator-app-351498fc-850a-45da-b7b6-27e523b8702a), [Twilio Authy](https://authy.com/guides/twilio/), and so on.

When you register an authentication app, a code appears in your app. Enter this code into the Labelbox prompt that appears when you sign in to Labelbox.

Once MFA is enabled for your account, you will enter a code from your authenticator app each time you sign in to Labelbox.

## Deactivate multi-factor authentication

To deactivate MFA for all users of your workspace:

1. Sign in to Labelbox using an account as a workspace administrator.

2. From the account menu, select **Workspace Settings** and then select the **Workspace** tab.

3. Locate the **Multi factor authentication** section and then find the *Enabled for all workspace users setting*. Remove the checkmark to deactivate MFA for all workspace users.

## Recommendations

When using (or considering) MFA with Labelbox:

* Single sign on (SSO) providers frequently offer MFA as part of their service. If you're using SSO to manage access to Labelbox, we strongly recommend using your provider's MFA implementation to simplify and consolidate the support process.
* If one of your users needs to reset their MFA configuration, they should not contact Labelbox support directly. Instead, they should contact a workspace administrator. For security reasons, the administrator should confirm the request before contacting Labelbox support.


# Multimodal chat evaluation
Source: https://docs.labelbox.com/horizon/guides/multimodal-chat-evaluation-editor

Learn how to use the multimodal chat evaluation editor to evaluate generative models across multiple data types, compare model outputs, and refine performance with supported annotation tasks.

The multimodal chat evaluation editor allows you to evaluate generative model responses across multiple data types, including text, images, videos, audio, and PDFs. You can select up to 10 different models, including common foundation models and your custom models, to rate and rank their responses for:

* **Model comparison**: Evaluate and compare model configurations for a given use case and decide which one to pick. You can also compare model outputs directly from prompts using predefined criteria to perform a ranking or selection.
* **RLHF**(reinforcement learning with human feedback): Create preference data for training a reward model for RLHF based on multiple outputs from a single model or different model.

The editor has the following modes:

* [Live multimodal chat](/horizon/guides/live-multimodal-chat-evaluation) supports live, multi-turn conversations with models for evaluation. Unlike other editors, your team can type prompts to trigger inferencing on the given prompt and return model outputs without importing data into a dataset.
* [Offline multimodal chat](/horizon/guides/offline-multimodal-chat-evaluation) allows you to import existing conversations for annotating model responses.

## Supported annotation types

| Feature | Description | Export format |
| - | - | - |
| **Message ranking** | Rank multiple model-generated responses to determine their relative quality or relevance. | [Payload](/horizon/developer-guides/export-multimodal-chat-annotations#message-ranking) |
| **Message selection** | Select single or multiple responses that meet specific criteria. | [Payload](/horizon/developer-guides/export-multimodal-chat-annotations#message-selection) |
| **Message step reasoning** | (Text conversations only, no multimodal support) Evaluate the accuracy of each step broken down from responses and label it as **correct**, **neutral**, or **incorrect**. Provide a **justification** for incorrect steps and regenerate the conversation from that step. | [Payload](/horizon/developer-guides/export-multimodal-chat-annotations#message-step-reasoning) |
| **Prompt rating** | Flag prompt with issues and select an issue category. | [Payload](/horizon/developer-guides/export-multimodal-chat-annotations#prompt-rating) |
| **Fact-checking** | Verify the factual information in the response using these rating options: **Accurate**, **Inaccurate**, **Disputed**, **Unsupported**, **Can’t confidently assess**, or **No factual information**. For **Accurate**, **Inaccurate**, and **Disputed**, provide a justification explaining your choice. | [Payload](/horizon/developer-guides/export-multimodal-chat-annotations#fact-checking) |
| **Classification - Radio** | Select one option from a predefined set. | [Payload](/horizon/developer-guides/export-multimodal-chat-annotations#classification-radio) |
| **Classification - Checklist** | Choose multiple options from a list. | [Payload](/horizon/developer-guides/export-multimodal-chat-annotations#classification-checklist) |
| **Classification - Free text** | Add free text annotations. | [Payload](/horizon/developer-guides/export-multimodal-chat-annotations#classification-free-form-text) |

Classification tasks can apply globally to the entire conversation or individually to a message. They can also nest subclassification tasks.

<Info>
  ### Message step reasoning best practices

  Message step reasoning is an [experimental](/horizon/guides/product-release-phases#experimental) feature. For projects using ontologies with the step reasoning task, ensure that prompts lead to responses that can be easily broken down into clear steps.
</Info>

## Code and grammar assistance

The multimodal chat evaluation editor has the following built-in code and grammar assistance tools:

* [AI critic](/horizon/guides/code-grammar-assistance#ai-critic): Detects code and grammar issues and suggests improvements for your input.
* [Code editors](/horizon/guides/code-grammar-assistance#code-editors): Lets you write and edit code using the Monaco code editor or Visual Studio Code for an improved coding experience.

## Analytics view for annotations

The multimodal chat evaluation editor has some special metrics that provide insights into the project overview.

### Ranking

Depending on the number of model outputs you have configured, a horizontal ranking graph will appear to provide a visual analysis of LLM (large language model) win rate.

For each position, how often a model output was chosen. A longer bar means it was chosen more often in that position.

### Selection

You can also have a selection with different defined topics to further your use case.

### Model variance histograms

This chart shows a model's position in terms of how often it wins, with a variance chart showing how consistent it is in that position against other models.


# Natural language search
Source: https://docs.labelbox.com/horizon/guides/natural-language-search

A guide for using natural language search in Catalog.

You can use Labelbox's *natural language* search to surface data rows that match any expression you provide. This natural language search engine gives your team an edge by helping you find high-impact data rows in an ocean of data (e.g., rare data or edge cases).

We recommend using the native natural language search engine within our Catalog product.

## How natural language search works

Natural language search is powered by *vector embeddings*. A vector embedding is a numerical representation of a piece of data (e.g., an image, text, document, or video) that translates the raw data into a lower-dimensional space.

Recent advances in the machine learning field enable some neural networks (e.g., [CLIP vision model by OpenAI](https://openai.com/research/clip) or [all-mpnet-base-v2 text model](https://huggingface.co/sentence-transformers/all-mpnet-base-v2)) to recognize a wide variety of visual concepts in images, texts, or documents and associate them with keywords.

You can now surface images in Catalog by describing them in natural language. For example, type in *"a photo of birds in the sunset"* to surface images of birds in the sunset. You can also surface text, conversational text, or documents by describing them in natural language. For example, type in *"disappointed movie reviews"* to surface data rows that likely contain negative movie reviews.

<Info>
  ### Character/word limit for natural language search

  To view the character or word limit for the natural language filter, visit our [limits](/horizon/guides/limits) page.
</Info>

## Supported media types

Labelbox supports natural language search for several data modalities. For each media type, the same neural network embeds both the user query and the data row. Below is the list of neural networks used for each media type.

| Asset type | Supported | Embedding |
| - | - | - |
| **Image** | <Icon icon="check" /> | [CLIP-ViT-B-32](https://huggingface.co/sentence-transformers/clip-ViT-B-32) (512 dimensions) |
| **Video** | <Icon icon="check" /> | Google [Gemini Pro Vision](https://ai.google.dev/models/gemini). First two (2) minutes of content is embedded. Audio signal is not used currently. This is a paid add-on feature available upon request. |
| **Text** | <Icon icon="check" /> | [all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) (768 dimensions), based on the first 64K characters |
| **HTML** | <Icon icon="check" /> | [all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) (768 dimensions), based on the first 64K characters |
| **Document** | <Icon icon="check" /> | [CLIP-ViT-B-32](https://huggingface.co/sentence-transformers/clip-ViT-B-32) (512 dimensions) and [all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) (768 dimensions), based on the first 64K characters |
| **Tiled imagery** | <Icon icon="check" /> | [CLIP-ViT-B-32](https://huggingface.co/sentence-transformers/clip-ViT-B-32) (512 dimensions) |
| **Audio** | <Icon icon="check" /> | Audio is transcribed to text. [all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) (768 dimensions) |
| **Conversational text** | <Icon icon="check" /> | [all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) (768 dimensions), based on first 64K characters |

## How to search data using natural language

In the gallery view of Catalog, select the **Natural language** filter. Then, decide if you want to do a **Visual** search or **Text** search. Finally, input the description of the data you are looking for. The description must have at least 3 characters and at most 10 words.

### Prompt engineering

Prompt engineering involves trying several prompts until finding one that works well. Labelbox recommends trying several natural language descriptions (or *prompts*) until the natural language search surfaces the data you are looking for. Users have reported that small tweaks to the prompt can help return more relevant data.

<Tip>
  **Pro tip: Refine your prompt with positive biases and negative biases.**
</Tip>

You can refine your prompt by adding positive biases and negative biases, using this prompt structure: `[my prompt] / [more of this positive bias] / [less of this negative bias]`.

For example, I can refine my search for `a photo of birds in the sunset`, by asking for `purple` sunsets and not `red` sunsets, with the prompt: `a photo of birds in the sunset / purple / red`. This returns only images of birds in a purple sunset, and not in a red sunset.

### Set the score range

Natural language search surfaces the data rows whose embeddings are closest to the prompt. This is measured using cosine distance, a number between 0 and 1. The more similar the embeddings, the higher the natural language score.

By default, Labelbox returns embeddings with a natural language score between 0.5 and 1. You can customize this range by setting the minimum and maximum values of the natural language search slider.

### Combine natural language search & other searches

You can combine natural language search with other filters in Catalog. Some filters are best used for targeting *unstructured* data, and others for targeting *structured* data.

Combine natural language search with the following filters to target data rows by structured data:

<CardGroup>
  <Card title="Metadata" icon="square-1" href="/horizon/developer-guides/metadata" />

  <Card title="Annotations" icon="square-2" href="/horizon/guides/annotate-overview" />

  <Card title="Datasets" icon="square-3" href="/horizon/developer-guides/dataset" />

  <Card title="Projects" icon="square-4" href="/horizon/developer-guides/project" />
</CardGroup>

Combine natural language search with the following filters to search unstructured data:

<CardGroup>
  <Card title="Similarity search" icon="square-1" href="/horizon/guides/similarity" />

  <Card title="Text search" icon="square-2" href="/horizon/guides/find-text" />
</CardGroup>

## Automate data curation with slices

After populating filters in Catalog, you can save these filters as a [slice](/horizon/guides/slices) of data. When you save a filter as a slice, you will not need to populate the same filters repeatedly. Also, slices are dynamic, so any new incoming data row in Catalog will appear in the relevant slices.

Read the following resources to learn how to take action on the filtered data.

<CardGroup>
  <Card title="Refine the similarity search" icon="square-1" href="/horizon/guides/similarity#adding-anchors" />

  <Card title="Send filtered data rows to a labeling project as a batch" icon="square-2" href="/horizon/guides/batches" />

  <Card title="Add metadata to the filtered data rows" icon="square-3" href="/horizon/guides/datarow-metadata" />
</CardGroup>


# Notifications
Source: https://docs.labelbox.com/horizon/guides/notifications

View and set notification preferences for your tasks and operations.

When you perform operation tasks using the UI or the SDK, you can check the status of these tasks on the [Notifications](https://app.labelbox.com/notifications) page. Examples of tasks include:

* Importing and exporting data
* Sending model predictions for human review
* Assigning benchmark agreement scores
* Deleting data rows and projects

The notifications page displays tasks performed in the last 30 days and allows you to search for specific tasks by task name, type, status, and performer. Additionally, you can set up email alerts for specific tasks.

If you are an admin, project lead, or team manager, you can view and receive notifications for tasks performed by all users in the workspace or associated with your managed project.

## Set up email notifications

On your [user profile](https://app.labelbox.com/user-profile) page, you can enable email notifications for the following types of tasks:

* **Assignment-related notifications** when your group is assigned or unassigned from a project.
* **Batch-related notifications** when a batch is added to a project or completed.
* **Issue-related notifications** when an issue is created or resolved on a project.

If you have projects with [labeling services](/horizon/guides/labeling-services), you can set up the following additional email notifications:

* **Project-related notifications**:

  * If you are a customer who requested labeling services, you will be notified about project phase changes, labelers being promoted to production, and issue resolutions.
  * If you are a labeler, you will be notified when you are assigned to or removed from a project, when a project is promoted to production, or when an issue is opened on your data row.

* **Digest email summaries** (project manager only): receive daily summaries of the project's activity from the previous day.


# Offline multimodal chat evaluation
Source: https://docs.labelbox.com/horizon/guides/offline-multimodal-chat-evaluation

Learn how to create offline multimodal chat evaluation projects for ranking and classifying model outputs on conversation text data.

The **offline multimodal chat evaluation** editor allows you to evaluate generative models by importing existing conversations and adding annotations to model responses. The editor supports various data types, including text, images, videos, audio, and PDFs.

## Set up offline multimodal chat evaluation projects

The following steps walk you through how to set up an offline multimodal chat evaluation project on the Labelbox platform. To learn how to set up an offline multimodal chat evaluation project using the SDK, see [Multimodal chat evaluation projects](/horizon/developer-guides/multimodal-chat-evaluation-project).

### Step 1: create a project

1. On the [Annotate projects page](https://app.labelbox.com/projects), click the **+ New project** button.

2. Select **Multimodal chat**, and then select **Offline multimodal chat**.

3. Provide a **name** and an optional **description** for your project.

### Step 2: add data

1. Click the **Add data** button to select a [conversation v2 JSON](/horizon/developer-guides/import-multimodal-chat-data#sample-conversation-v2-json) dataset or create a new dataset. Alternatively, you can [import data using the SDK](/horizon/developer-guides/import-multimodal-chat-data).

### Step 3: Set up an ontology

[Create an ontology](/horizon/guides/labelbox-ontology#create-a-new-ontology) for evaluating model responses.

The editor supports the following options:

| Feature | Description | Export format |
| - | - | - |
| **Message ranking** | Rank multiple model-generated responses to determine their relative quality or relevance. | [Payload](/horizon/developer-guides/export-multimodal-chat-annotations#message-ranking) |
| **Message selection** | Select single or multiple responses that meet specific criteria. | [Payload](/horizon/developer-guides/export-multimodal-chat-annotations#message-selection) |
| **Message step reasoning** | Break responses into steps and evaluate the accuracy of each step by selecting from **correct**, **neutral**, and **incorrect**. Add your rewrite with justification for incorrect steps. | [Payload](/horizon/developer-guides/export-multimodal-chat-annotations#message-step-reasoning) |
| **Classification - Radio** | Select one option from a predefined set. | [Payload](/horizon/developer-guides/export-multimodal-chat-annotations#classification-radio) |
| **Classification - Checklist** | Choose multiple options from a list. | [Payload](/horizon/developer-guides/export-multimodal-chat-annotations#classification-checklist) |
| **Classification - Free text** | Add free text annotations. | [Payload](/horizon/developer-guides/export-multimodal-chat-annotations#classification-free-form-text) |

Classification tasks can apply globally to the entire conversation or individually to a message. They can also nest subclassification tasks.

<Info>
  ### Experimental feature

  Message step reasoning is an [experimental](/horizon/guides/product-release-phases#experimental) feature. Currently, you can't import step reasoning labels using the SDK.
</Info>

### Step 4: Complete annotation tasks

Click the **Start labeling** button to add annotations to evaluate the responses. Complete all tasks in your [workflow](/horizon/guides/workflows).


# Get started with Labelbox
Source: https://docs.labelbox.com/horizon/guides/overview

Labelbox accelerates the creation of high-quality, differentiated data by combining on-demand expert labeling services with the industry-leading data labeling platform.

Labelbox is the data factory for generative AI, providing the highest quality training data for frontier and task-specific models. Labelbox’s comprehensive platform combines on-demand labeling services with our industry-leading data labeling platform.

<Card title="Connect an AI assistant" icon="plug" href="/horizon/guides/labelbox-mcp">
  Ask an AI assistant to summarize project progress, inspect quality, and use Labelbox tools with your current permissions.
</Card>

## Data labeling solutions

Labelbox offers a comprehensive labeling service that rapidly delivers labeled data with real-time feedback, built-in automation, and quality control. Powered by the [Alignerr](https://intercom.help/alignerr/en/articles/9540511-what-is-alignerr) community of highly educated experts proficient in major languages and advanced subjects, the labeling service excels in critical post-training and GenAI tasks, such as:

* Reinforcement Learning with Human Feedback (RLHF)
* Supervised fine-tuning (SFT)
* Multimodal LLM evaluation
* Preference ranking
* LLM chat arena
* Red teaming
* Text-to-image, video, and audio tasks
* Coding and AI agent tasks

Labelbox provides expert labeling in 30+ languages and is capable of more regional and domain-specific languages upon request.

To learn more about how to set up labeling services, see [Labeling services](/horizon/guides/labeling-services).

## An all-in-one platform

Labelbox’s unified platform allows you to manage the entire AI model lifecycle, from data labeling to model training and post-training tasks. Begin with annotating datasets using Labelbox's tools, or opt for expert labeling services when necessary. From there, you can use foundational model-assisted features to automate the labeling and training process and collaboration tools to enable team-wide review and alignment.

## Annotate labels

Collaboratively annotate data with the internal team, your own vendor, or Labelbox data labeling services. Labelbox Annotate supports 10+ built-in editors for multimodal chat, LLM evaluation, prompt/response generation, computer vision, natural language processing, and more.

To get started with Annotate:

<CardGroup>
  <Card title="Set up labeling project" icon="wrench" href="/horizon/guides/what-is-a-project" />

  <Card title="Set up ontology (task design)" icon="screwdriver-wrench" href="/horizon/guides/labelbox-ontology" />

  <Card title="Import ground truth annotations" icon="file-import" href="/horizon/guides/import-ground-truth" />
</CardGroup>

## Train and evaluate models for RLHF

Generate preference data to fine-tune large language models for RLHF using outputs from one or more foundational models. Labelbox Foundry integrates foundational models into your Labelbox workflow.

To learn more about how to leverage foundational models on Labelbox, see [Foundry](/horizon/guides/foundry).

## Catalog datasets

Integrate data from 25+ different cloud sources or your own data solutions within a unified, and intuitive interface. Turn data into insight and action with vector & traditional search, data exploration, and curation.

To get started with Catalog:

<CardGroup>
  <Card title="Create a dataset" icon="database" href="/horizon/guides/datasets-datarows" />

  <Card title="Bulk Classification" icon="list-check" href="/horizon/guides/bulk-classification" />

  <Card title="Data curation" icon="server" href="/horizon/guides/catalog-overview#how-catalog-works" />
</CardGroup>

## Monitor and manage teams

Monitor labeling project and workforce performance with metric filters to identify your top performers, including your own team members and the labeling service professionals.

To learn more about available tools and user roles, see [Team management](/horizon/guides/team-management).

## Run and test custom models

Add your custom models to run labeling tasks and training experiments to diagnose and quickly improve their performance.

To get started with Model:

<CardGroup>
  <Card title="Overview" icon="circle-info" href="/horizon/guides/models-overview" />

  <Card title="Import model predictions for model error analysis" icon="file-import" href="/horizon/guides/upload-model-predictions" />

  <Card title="Model error analysis" icon="chart-line" href="/horizon/guides/improve-model-performance" />
</CardGroup>


# Performance dashboard
Source: https://docs.labelbox.com/horizon/guides/performance-dashboard

Information on the project performance dashboard and the included metrics.

The performance dashboard helps you manage labeling operations in your Labelbox projects by reporting on the throughput, efficiency, and quality of your labeling process. You can see analytics for the overall project and for individual labelers. Filters are available to help you focus on specific information and dimensions.

The performance of your data labeling operation is broken down into four components, each with unique views to help you understand the overall performance:

* **Individual**
* **TEQ** (Throughput, Efficiency, and Quality)
* **Participation**
* **Instruction Quiz** (see [Quiz](/horizon/guides/quiz#instruction-quiz-metrics))

You can use the filters at the top of the performance dashboard to analyze relevant subsets of data. When active, these filters apply to all metric views, including Throughput, Efficiency, and Quality. The following filters are available:

| Filter | Description |
| - | - |
| Batch | Filters the graphs and metrics by data rows belonging to a specific batch. |
| Label actions - Labeled by | Filters the graphs andmetrics by data rows that have been labeled by a specific labeler. |
| User group | Filters the graphs and metrics to work by the current members of one or more user groups attached to the project. |
| Label actions - Deleted | Filters the graphs and metrics by data rows based on their deletion status. You can either `Exclude` data rows with deleted labels or `Include` all labels created in the project. |

To read these numbers from your own tools, see the [Performance API](/horizon/developer-guides/performance-api).

## Throughput, efficiency, and quality (TEQ)

This section provides three separate views to help you understand the performance of your labeling operation: Throughput, Efficiency, and Quality.

### Throughput view

The throughput view provides insight into the amount of labeling work being produced and helps you answer questions like:

* *How many assets were labeled in the last 30 days?*
* *What is the review time and rework time for labeled assets?*
* *What is the average amount of labeling work being produced?*

These metrics are available for all members of the project and for individual members. The throughput view displays the following charts:

| Chart | Description |
| - | - |
| Done | Displays the daily count of data rows in the *Done* step of the project workflow. See [Workflows](/horizon/guides/workflows) for more definitions. |
| Labels | Displays the count of labeled data rows, including labels that were later deleted. For Benchmark or Consensus data rows, a single data row can have multiple labels, so the Labels count may exceed the number of data rows in the same period. |
| Annotations | Displays the count of annotations created, including labels that wer later deleted. |
| Reviews | Count of "Approve" and "Reject" actions for labels created in the project. For information on approve and reject actions in the review queue, see [Workflows](/horizon/guides/workflows). |
| Total time | Displays the total time spent (inclusive of Labeling, Review and Rework time) on labels created in the project. |
| Labeling time | Displays the total *labeling time* spent on data rows, while the timer is on. |
| Review time | Displays the total *review time* spent on labeled data rows, while the timer is on. |
| Rework time | Displays the total *rework time* spent on labeled data rows, while the timer is on. |

#### Forecast

The forecast feature shows your average daily throughput and completed work (data rows, labels, reviews, annotations) and uses that to forecast completion dates. To see this, enable the **Forecast** toggle.

#### Average daily throughput

The **average daily throughput** is calculated by dividing the total units completed in the selected date range by the number of calendar days in that range.

Where:

* Total units completed = sum of the relevant throughput metric over the selected period (e.g., total labels created, total reviews, etc)
* Number of calendar days = inclusive of days between start and end date (e.g., Jan 1–Jan 7 → 7 days). This aligns the “average per day” with how the forecast projects completion over calendar time.

For example, if a project has 2,800 labels created over a 14-day window, the average daily throughput for labels would be 200 labels/day (2,800 labels / 14 days).

Note that metric units in this view can vary:

* For **Labels created / Reviews received / Annotations created**, average daily throughput is in units like “labels/day”, “reviews/day”, “annotations/day”.
* For **time-based metrics**, the forecast uses those as inputs, but the primary “average daily throughput” surfaced is for the count-based metrics (data rows/labels/reviews/annotations), per the spec and launch comms.

### Efficiency view

The efficiency view helps you visualize the time spent per unit of work, per labeled asset, or per review. These metrics help answer questions such as:

* *What is the average amount of time spent labeling an asset?*
* *How can I reduce the time spent per labeled asset?*

These metrics are available for individual project members and at the project level. The efficiency view includes these charts:

| Chart | Description |
| - | - |
| Avg time per label | Displays the average *labeling time* spent per label. Avg time per label = Total labeling time / number of labels submitted |
| Avg review time | Displays the average *review time* per data row. Avg review time = Total review time / number of data rows reviewed |
| Avg rework time | Displays the average *rework time* per data row. Avg rework time = Total rework time / number of data rows reworked |
| AHT per labeled data row | Displays the total time across all modes for all data rows divided by the number of data rows. |
| AHT per done data row | Displays the total time across all modes for data rows in the ‘Done’ task queue divided by the number of data rows in ‘Done’. |
| AHT per created label | Displays the total time across all modes for ‘Created’ labels divided by the number of ‘Created’ labels. |
| AHT per submitted label | Displays the total time across all modes for ‘Submitted’ labels divided by the number of ‘Submitted’ labels |
| AHT per done label | Displays the total time across all modes for labels in the ‘Done’ task queue divided by the number of labels in ‘Done’. |

<Note>
  "Skipped" time is excluded from labeling time, and "abandoned" (unsaved) work is counted toward labeling time. You can use filters to include or exclude time spent on "deleted" labels.
</Note>

### Quality view

The quality view helps you understand the accuracy and consistency of the labeling work being produced. These metrics answer questions like:

* *What is the average quality of a labeled asset?*
* *How can I ensure label quality is more consistent across the team?*

These metrics are available for individual project members and for the project as a whole. The quality view includes the following charts, which will only display if the corresponding features (Benchmark and Consensus) are enabled for your project:

| Metric | Description |
| - | - |
| Benchmark | Shows the average benchmark scores on labeled data rows within a specified time frame. |
| Benchmark distribution | Shows a histogram of benchmark scores (grouped by 10) for labeled assets within a specified time frame. |
| Consensus | Shows the average consensus score of labeled assets over a selected period. This is the average agreement score of consensus labels within themselves (for a given data row). |
| Consensus distribution | Shows a histogram of consensus scores (grouped by 10) for labeled assets plotted over the selected period. |

## Individual member performance

Click **Individual** in the Performance dashboard to view individual metrics for each team member that has worked on the project. The performance metrics are separated by intent (**labeling** and **reviewing**) and are shown as distinct views in the table.

Team members are listed individually in separate rows and will only appear if they have actively performed tasks during the selected period.

### Labeling metrics

Here are the metrics displayed in the **Individual** → **Labeling** view.

| Metric (Labeling) | Description |
| - | - |
| Labels created | The number of labels created by team members during the selected period. |
| Labels skipped | The number of labels skipped by team members during the selected period. |
| Labeling time (submitted) | Total *labeling time* team member spent creating and submitting labels during the selected period. |
| Labeling time (skipped) | Total *labeling time* team member spent working on labels that were ultimately skipped. (The **Skip** button was clicked.) |
| Avg time per label | Average *labeling time* of submitted and skipped labels. Calculated by dividing total labeling time by the number of labels submitted or skipped. Displays *N/A* when no labels have been submitted or skipped. |
| Reviews received | The number of Workflow review queue actions (*Approve* and *Reject*) received during the selected period on data rows labeled by the team member. |
| Avg review time (all) | Average *review time* on labels created by the team member. Calculated by dividing the total *review time* for all team members by the number of labels that have been reviewed. |
| Avg rework time (all) | Average *rework time* on labels created by the team member. Calculated by dividing the total time spent by all team members and the number of labels reworked. |
| Rework time (all) | Total *rework time* spent on labels created by the team member in the selected period. |
| Review time (all) | Total *review time* spent on labels created by the team member during the selected period. |
| Rework % | Percentage of labels created by the team member that were reworked during the select time period. Calculated by dividing the number of labels created by the team member that had *any* rework done by the number of labels created by the team member. |
| Approval % | Percentage of data rows labeled by the team member that were approved during the selected period. Calculated by dividing the number of data rows with one or more approved labels by the number of review actions (*Approve* or *Reject*). Labels pending review are not included. |
| Benchmark score | Average agreement score with benchmark labels for labels created by the team member during the selected period. |
| Consensus score | Average agreement score with other consensus labels for labels created by the team member during the selected period. |

### Reviewing metrics

Here are the metrics displayed in the **Individual** → **Reviewing** view.

| Metric (Reviewing) | Description |
| - | - |
| Data rows reviewed | Data rows that have had *review time* spent by the user (per filter selection). |
| Data rows reworked | Data rows that have had *rework time* spent by the user. |
| Avg review time | Average *review time* spent by the user in the period selected. Numerator = Total *review time* spent by the user Denominator = Number of labels reviewed by the user |
| Avg rework time | Average *rework time* spent by the user in the selected period. Numerator = Total rework time spent by the user Denominator = Number of labels reworked (approved or rejected) by the user |
| Total time | Sum of *review time* and *rework time* spent by the user during the period selected. |
| Rework % | Percentage (%) of data rows reviewed by the user that have been reworked. Numerator = Number of labels that have had both *rework time* and *review time* spent by the user. Denominator = Number of data rows that have had [*review time*](#review-time) spent by the user. |
| Approval % | Data rows with an *approve* action by the user as a percentage (%) of data rows with any review action by the user. Numerator = Number of data rows with *approve* action by the user (in workflow). Denominator = Number of data rows with either an *approve* or *reject* action by the user (in workflow). |

## Participation histogram

The **Participation** histogram is a chart in the Performance Dashboard that visualizes the distribution of labeler activity across a project.

Specifically, it shows:

* How many labelers fall into different activity buckets (e.g., by number of tasks completed or labels created)
* It helps admins see at a glance whether participation is concentrated among a few heavy contributors or spread evenly across the workforce

This is distinct from aggregate throughput charts — rather than showing total output over time, it shows the *spread* of engagement across individual contributors. It's useful for identifying:

* Labelers who haven't participated at all
* Labelers who are highly active vs. barely active
* Overall workforce engagement health for a project

## Timer details

To help you measure efficiency, the timer log tracks how much time is spent on each data row. The time is broken down into three categories: labeling, reviewing, and reworking.

### How time is tracked in your project

A data row goes through many different states, which are tracked in the timer log.

| Type of timer | When it is tracked |
| :- | :- |
| Labeling time | This is the time the **original labeler** spends creating the label. It starts when they open an asset and stops when they **skip** or **submit** it. It also includes any time the original labeler spends going back to edit their own work. |
| Review time | This is the time a **different user** (not the original labeler) spends looking at a label in either the data row browser or the review queue. |
| Rework time | This is the time a **different user** spends fixing a label. It's tracked when they **submit** an asset from the rework queue, or when they **edit and save/approve/reject** the asset from the main data row view. |

<Note>
  Important notes

  * **Inactivity Pause:** If a user is inactive for more than five minutes, the timer will automatically pause. It will resume as soon as they become active again.
  * **Consensus and Benchmarks:** If your project uses consensus or benchmarks, the time recorded for a data row will be the total time spent across *all* labels created for that data row.
</Note>

### How importing labels impacts the timer

Importing labels affects labeling time. When you import:

* **Ground truth labels:** No label time is recorded on the **Data rows** tab or the performance dashboard for that label. Label time is displayed as zero (0). When team members modify the label, time is recorded as review time.
* **Model-assisted learning (MAL) pre-labels:** No label time is recorded on the **Data rows** tab or the performance dashboard for that label. Label time is displayed as zero (0). When team members open the data row in the editor and click **Edit**, the time spent before selecting **Skip** or **Submit** is recorded as labeling time.


# Predict with Foundry
Source: https://docs.labelbox.com/horizon/guides/predict-with-foundry



This guide will walk you through the end-to-end process of running your first model in Foundry. By the end, you'll have a set of model-generated predictions ready for review.

## Step 1: Select Your data

Your journey begins in the **Catalog**. The Catalog is where you manage and explore all your data within Labelbox.

1. Navigate to the **Catalog** tab.
2. Use the filters to find the data you want the model to process. You can filter by dataset, metadata, and other attributes.
3. Select the checkboxes next to the data rows you wish to include in your model run.
4. Once you've made your selection, click **Predict with Foundry.**

This will take you to the Foundry model gallery with your data selection pre-loaded.

## Step 2: Choose a model

Now it's time to choose a model to generate your predictions.

1. From the model gallery, you'll see a variety of pre-trained foundation models.
2. Use the filters on the left to narrow down the models based on the task you want to perform (e.g., Object Detection, Text Summarization).
3. Once you've found a suitable model, click on the model card to proceed.

## Step 3: Configure the model run

This is the final configuration step before launching your model run.

1. **Select an ontology:** This is a critical step. You need to map the classes the model can predict to the feature classes in your project's ontology. This ensures the predictions are in a format that your project understands.
2. **Set confidence threshold (optional):** Use the slider to set a minimum confidence score. The model will only generate predictions that meet this threshold. This is useful for filtering out low-quality or uncertain predictions.
3. **Generate previews (optional):** Before committing to a full run, click the **Generate previews** button. This will run the model on a small sample of your selected data, allowing you to quickly verify that the configuration is correct and the model is performing as expected.
4. **Submit the model run:** The **Submit** button appears only when your settings are valid for the selected model, ontology, and underlying data types. For example, you cannot submit a model run for models that require an ontology until you select an ontology. When satisfied with the settings and generated previews, submit your model run.

## Step 4: Track your model run

Your model run is now in progress.

1. You will be redirected to the **Models** page, where you can see a list of all your model runs.
2. Your new run will appear at the top with a "Running" status. The time it takes to complete depends on the model and the amount of data.
3. You can click on the model run to see its progress and details. You'll receive a notification in Labelbox once the run is complete.

## Next steps

<Card title="Send predictions for human review" icon="sparkles" />

<Card title="Create a Foundry app" icon="sparkles" />

<Card title="Export predictions" icon="sparkles" />


# Prompt and response generation
Source: https://docs.labelbox.com/horizon/guides/prompt-and-response-generation-editor

Learn how to create fine-tuning datasets of prompts and responses for LLMs.

You can prepare a dataset of prompts and responses to fine-tune large language models (LLMs). Labelbox supports dataset creation for a variety of fine-tuning tasks including summarization, classification, question-answering, and generation.

## Set up prompt and response generation

When you set up a prompt and response generation project, you will be prompted to specify *how* you will use the editor with the following three options:

* **Humans generate prompts and responses**: In the editor, the prompt and response fields will be required. This will indicate to your team they should create a prompt and a response from scratch.
* **Humans generate prompts**: In the editor, only the prompt field will be required. This will indicate to your team that they should create a prompt from scratch.
* **Humans generate responses to uploaded prompts**: In the editor, a previously uploaded prompt will appear. Your team will need to create responses for that prompt.

<Info>
  ### Benchmark and consensus support

  [Benchmarks](/horizon/guides/benchmark) and [Consensus](/horizon/guides/consensus) are only available for the **Humans generate responses to uploaded prompts** option.
</Info>

### Specify prompts and/or responses

When setting up the project,

* If you select **Humans generate prompts**, you will need to specify a prompt for your labelers to reference so they can generate more prompts. Prompts are restricted to free-form text format. You can optionally set a character minimum and maximum for prompt data.

* If you select **Humans generate prompts and responses**, you will need to specify a prompt *and* a response for your labelers to reference so they can generate more prompts and responses. See [Supported prompt formats](#supported-prompt-formats) and [Supported response format](#supported-response-formats) for supported format types.

<Warning>
  ### Markdown editor size limit

  When using the Markdown editor to specify a prompt or response, limit the character count to fewer than 6,000 characters.
</Warning>

### Import prompts

During the project setup, if you select **Humans generate responses to uploaded prompts** as your LLM data generation workflow, you will need to create an import file containing links to a set of prompts in text format. Then, upload your import file and send the prompts to the project.

Follow these steps to upload prompt data to Labelbox:

1. Create the import file containing links to the prompts. See [Import text data](/horizon/developer-guides/import-text-data) to learn how to structure your import file.
2. [Upload the import file](/horizon/guides/datasets-datarows#create-a-dataset-via-the-app-ui) to Labelbox.
3. Save the prompts in a batch and send the batch to your project. See [Batches](/horizon/guides/batches) for instructions.

<Info>
  ### Data row size limit

  To view the maximum size allowed for a data row, visit our [limits](/horizon/guides/limits) page.
</Info>

## Supported prompt formats

If you selected an LLM data generation workflow that involves generating a prompt in the editor, you will need to specify a prompt to use as the ontology. Each LLM data generation ontology is limited to one prompt.

| Feature | Import format | Export format |
| - | - | - |
| **Prompt - Free-form text** | N/A | [See payload](/horizon/developer-guides/export-text-annotations#free-form-text) |

## Supported response formats

If you selected an LLM data generation workflow that involves generating a response in the editor, you will need to specify a set of responses to use as the ontology. Below are the supported formats you may include when you are specifying responses in your ontology. Responses can be applied at the global level and/or nested within other responses. LLM data generation ontologies support multiple responses.

| Feature | Import format | Export format |
| - | - | - |
| **Response - Text** | N/A | [See payload](/horizon/developer-guides/export-text-annotations#radio) |
| **Response - Radio** | N/A | [See payload](/horizon/developer-guides/export-text-annotations#checklist) |
| **Response - Checklist** | N/A | [See payload](/horizon/developer-guides/export-text-annotations#free-form-text) |

### Response - Text

Create a text response by selecting a response type of **Text** during ontology creation. You can optionally set a character minimum and maximum for text-type responses.

### Response - Radio

Create a radio response by selecting a response type of **Radio** during ontology creation. Radio responses support nested sub-classifications.

### Response - Checklist

Create a checklist response by selecting a response type of **Checklist** during ontology creation. Checklist responses support nested sub-classifications.


# Quality analysis
Source: https://docs.labelbox.com/horizon/guides/quality-analysis

Learn how to utilize benchmarks and consensus scoring to analyze the quality of your labels.

Labelbox provides two quality analysis approaches to ensure the accuracy and performance of your labels:

* [Benchmarking](/horizon/guides/benchmark) allows you to designate a labeled data row as a gold standard for other labels. It’s useful for assessing the accuracy of annotations.
* [Consensus scoring](/horizon/guides/consensus) compares annotations on the same data row by different labelers to determine a consensus winner. It’s useful for facilitating immediate corrective actions to improve both training data and model performance.

You can apply one or both benchmarking and consensus scoring to an **Annotate** project.

## View and assess quality analysis performance

On a project’s **Data Rows** tab, you can view the average benchmark and consensus agreement scores for each benchmark data row, along with the number of labels associated with the consensus agreement. Additionally, you can assess performance using these scores on the **Performance** tab:

* **Labeler performance**: In the **Members Performance** section, click on an individual labeler to see their average benchmark and consensus scores.
* **Overall project performance**: Under **Performance charts** > **Quality**, the left histogram shows the average benchmark score by date range, and the right histogram displays the number of labels within specific benchmark score ranges.

## Export quality analysis scores

You can export labels along with their benchmark and consensus agreement scores using the [SDK](/horizon/developer-guides/export-overview). For every exported label with a benchmark or consensus agreement, you can find their `benchmark_score` and `consensus-score` values in the [`performance_details`](/horizon/developer-guides/export-glossary#performance_details) section of exported labels in the resulting JSON file.

For a benchmark data row labeled multiple times, all non-benchmark labels contain a `benchmark_reference_label` field, which is the ID of the benchmark label they reference. The benchmark label itself doesn’t have a `benchmark_reference_label` field or an associated benchmark score, as it serves as the standard for comparison, not as a label being compared.

## Calculation methodologies

Benchmark and consensus agreement scores for different types of labels are calculated using different methodologies. The scores for each annotation type are averaged to produce an overall benchmark score for a data row, which helps assess the accuracy and consistency of labels.

### Benchmarking methodologies

#### Object-type annotations

The benchmark agreement for bounding box, polygon, and segmentation mask annotations is calculated using Intersection over Union (IoU). The agreement between point annotations and polyline annotations is calculated based on proximity.

1. First, Labelbox compares each annotation to its corresponding benchmark annotation to generate IoU scores for each annotation. The algorithm first finds the pairs of annotations to maximize the total IoU score, then it assigns an IoU value of 0 for the unmatched annotations.

2. Then, Labelbox averages the IoU scores for each annotation belonging to the same annotation class to create an overall score for that annotation class.

*"Tree" annotation class agreement = (0.99 + 0.99 + 0.97 + 0 + 0) / 5 = 0.59*

#### Classifications

The calculation for each classification type varies. One commonality, however, is that if two classifications of the same type are compared and there are no corresponding selections between the two classifications at all, the agreement will be 0%.

* A **radio** classification can only have one selected answer. Therefore, the agreement between the two radio classifications will either be 0% or 100%. 0% means no agreement and 100% means agreement.

* A **checklist** classification can have more than one selected answer, which makes the agreement calculation a little more complex. The agreement between two checklist classifications is generated by dividing the number of overlapping answers by the number of selected answers.

For child classifications, if two annotations containing child classifications have 0 agreement (resulting in a false positive), the child classifications will automatically be assigned a score of 0 as well.

Labelbox then creates a score for each annotation class by averaging all of the per-annotation scores.

For example, when Image X loads in the editor, a labeler has 3 classification questions to choose from (Q1, Q2, Q3), each with two answers. The green boxes indicate the benchmark answers.

Say, for example, a labeler answers Q1 correctly but answers Q2 and Q3 incorrectly.

**For classifications, the benchmark agreement is calculated based on: 1. How many *unique* answer schemas the labeler selects AND 2. Out of those selected, how many are *correct*?**

* Q1-A: 1

* Q1-B: N/A \<-- not included in the final calculation.

* Q2-A: 0

* Q2-B: 0

* Q3-A: 0

* Q3-B: 0

So the final benchmark calculation for the classifications on Image X is:

(1 + 0 + 0 + 0 + 0) / 5 = .20

#### Overall score

Labelbox averages the scores for each annotation class (object-type & classification-type) to create an overall score for the asset. Each annotation class is weighted equally. Below is a simplified example:

Benchmark score = (*tree* annotation class agreement + *radio* class agreement) / total annotation classes

*0.795 = (0.59 + 1.00) / 2*

For text and conversations, such as human-generated responses from [prompt and response generation](/horizon/developer-guides/import-prompt-and-response-annotations), Labelbox also creates a model-based similarity score that accounts for various ways of expressing the same idea, such as using active versus passive voice, synonyms for the same concept, and other variations in writing style.

You can use this metric as an initial indicator of label quality, the clarity of your ontology, and/or the clarity of your labeling instructions.

### Consensus scoring methodologies

#### Labels used in calculation

While you can submit any number of labels for a given asset, Labelbox only includes a subset of those labels when calculating the consensus score. For most data types, up to 20 labels are used in the calculation. For the free-form text classification type, only 5 labels are used in the calculation. These limits help ensure efficient and consistent consensus score evaluations, especially in large-scale labeling projects.

#### Object-type annotations

Consensus agreement for bounding box, polygon, and segmentation mask annotations is calculated using Intersection over Union (IoU). The agreement between point annotations and polyline annotations is calculated based on proximity.

1. First, Labelbox compares each annotation to its corresponding annotation to generate IoU scores for each annotation. The algorithm first finds the pairs of annotations to maximize the total IoU score, then it assigns the IoU value of 0 to any unmatched annotations.

2. Labelbox then averages the IoU scores for each annotation belonging to the same annotation class to create an overall score for that annotation class.

"Tree" annotation class agreement = (0.99 + 0.99 + 0.97 + 0 + 0) / 5 = 0.59

#### Text (NER) annotations

The consensus score for two text entity annotations is calculated at the character level. If two entity annotations do not overlap, the consensus score will be 0. Overlapping text entity annotations will have a non-zero score. When there is overlap, Labelbox computes the weighted sum of the overlap length ratios, discounting for already counted overlaps. Whitespace is included in the calculation.

1. Since the consensus agreement for NER is calculated at the character level, spans of text are partly inclusive. For example, If two labelers make an overlapping text entity annotation on the word "house" and the first labeler submits an annotation with `house` and the second labeler submits an annotation on the same word in the text file with `hous`, the agreement score between these two annotations would be 0.80.
2. Labelbox then averages the agreements for each annotation created using that annotation class to create an overall score for that annotation class.

#### Classifications

The calculation method for each classification type is different. One commonality, however, is that if two classifications of the same type are compared, and there are no corresponding selections between the two classifications at all, the agreement will be 0%.

* A **radio** classification can only have one selected answer. Therefore, the agreement between the two radio classifications will either be 0% or 100%. 0% means no agreement, and 100% means agreement.

* A **checklist** classification can have more than one selected answer, which makes the agreement calculation a little more complex. The agreement between two checklist classifications is generated by dividing the number of overlapping answers by the number of selected answers.

For **child classifications**, if two annotations containing child classifications have 0 agreement (false positive), the child classifications will automatically be assigned a score of 0 as well.

Labelbox then creates a score for each annotation class by averaging all of the annotation scores.

For example, when Image X loads in the editor, the labelers have 3 classification questions to choose from (Q1, Q2, Q3) each with two answers.

Say, for example, these 2 labelers have the same answer for Q1 but different answers for Q2 and Q3.

**For classifications, the consensus agreement is calculated based on how many *unique* answer schemas are selected by all labelers.**

* Q1-A: 1 (both labelers picked this answer)

* Q1-B: N/A (neither labeler picked this answer) \<-- not included in the final calculation.

* Q2-A: 0 (Labeler 1 selected, Labeler 2 did not)

* Q2-B: 0 (Labeler 2 selected, Labeler 1 did not)

* Q3-A: 0 (Labeler 1 selected, Labeler 2 did not)

* Q3-B: 0 (Labeler 2 selected, Labeler 1 did not)

So the final consensus calculation for the classifications on Image X is:

(1 + 0 + 0 + 0 + 0) / 5 = .20

#### Overall score

Labelbox averages the scores for each annotation class (object-type & classification-type) to create an overall score for the asset. Each annotation class is weighted equally. Below is a simplified example:

Consensus score = (annotation class agreement + radio class agreement) / total annotation classes

*0.795 = (0.59 + 1.00) / 2*

For text and conversations, such as human-generated responses from [prompt and response generation](/horizon/developer-guides/import-prompt-and-response-annotations), Labelbox also creates a model-based similarity score that accounts for various ways of expressing the same idea, such as using active versus passive voice, synonyms for the same concept, and other variations in writing style.

You can use the metric as an initial indicator of label quality, the clarity of your ontology, and/or your labeling instructions.


# Quiz
Source: https://docs.labelbox.com/horizon/guides/quiz

Configure project quizzes as a quality gate for labelers before and during project work.

Quizzes are a **project-level** quality check for labelers and Alignerrs. Use them to confirm readiness before work starts, reinforce quality standards, or reassess contributors after guidelines change. Configure and manage quizzes from the project overview, separately from ontology schema and [labeling instructions](/horizon/guides/instructions-and-quizzes).

Validating that labelers understand your instructions is one common use case; quizzes can also test edge cases, quality criteria, or other project-specific knowledge.

For the contributor experience (review instructions → take quiz → results), see [Labeling instructions](/horizon/guides/instructions-and-quizzes#use-quizzes-for-quality).

## Configure a project quiz

Save project instructions before you configure a quiz. Then:

1. Go to your project overview.
2. In the sidebar, click **Quiz** (or **Click to configure quiz** if no quiz exists yet).
3. On the **Project Quiz** page, create questions with AI or add them manually.
4. Configure attempt policy, pass threshold, feedback, and whether the quiz is required.
5. Click **Save quiz**.

Saving the quiz does not change ontology schema or instructions. Saving ontology schema or instructions does not change the quiz.

## Automatically generate quiz questions

To generate questions from your project instructions:

1. Open the project **Quiz** page.
2. Under **Generate with AI**, choose how many questions to create (1–10; default 3).
3. Click **Generate Quiz using AI**.

AI generation uses the project's instruction text and any uploaded PDF as a source. Videos and external links in instructions are not included. Instructions must be at least 50 characters (and at most 8,000 characters) for generation. You can edit generated questions or write your own to cover quality standards beyond the instruction text.

You can also click **Add Question using AI** on an existing quiz to append one AI-generated question at a time.

## Create quiz questions manually

1. Open the project **Quiz** page.
2. Click **Add Questions Manually** (or **Add Question Manually** when a quiz already exists).
3. Enter each question and expected answer, then click **Save quiz**.

Questions and answers can be up to 1,000 characters each.

## Quiz settings

| Setting | Description |
| - | - |
| **Attempt policy** | **Retake until pass** (unlimited retries), **One attempt only**, or **Limited attempts** (2–10). |
| **Pass threshold (%)** | Percentage of questions that must be answered correctly to pass (1–100; default 100). |
| **Show question feedback** | When enabled, contributors see per-question grader feedback after submission. |
| **Required to start work** | When enabled (default), contributors must pass before labeling or starting the Hubstaff timer. When disabled, the quiz remains available but failing it does not block work. |

## Force retakes and lockouts

After contributors have already passed, editing questions and checking **Force users to retake the quiz** on save requires those contributors to pass the updated quiz. You can also click **Force retake now** to require a retake without changing the questions—for example after a quality standard or guideline update.

With **One attempt only** or **Limited attempts**, a required quiz that is failed for the last time locks the contributor out until an admin clears the failure. Use **Manage lockouts** on the project Quiz page to review failed answers and reset a lockout (a reason is required). Force retake does not clear final-failure lockouts.

Optional quizzes never lock contributors out of work, even if attempt limits are exhausted.

<Tip>
  **Best practices**

  * Focus questions on quality standards, edge cases, and decisions that affect label accuracy
  * Cover the concepts contributors need to apply consistently, not only onboarding facts
  * Keep questions and answers concise (up to 1,000 characters each)
  * Use force retake when standards change and you need contributors to re-qualify
  * Note that videos and external links in instructions are not included in AI quiz generation
</Tip>

## Instruction Quiz metrics

The **Instruction Quiz** tab on the project Performance dashboard provides analytics for quiz performance when a project quiz is configured. Use these metrics to gauge contributor readiness, spot weak quality areas, and decide when to revise questions or force a retake.

### How to access your quiz analytics

To view quiz analytics for your project, please follow these steps:

1. Navigate to your project.
2. Go to the **Performance** tab.
3. Select the **Instruction Quiz** tab at the top of the page.
4. Use the date range picker to filter analytics by your desired time period.

### An overview of the metrics

The dashboard displays the following key performance indicators at the top:

| Metric | **Description** |
| - | - |
| Total quiz attempts | The total number of times labelers have taken the quiz during the selected period. |
| Overall pass rate | The percentage of quiz attempts that passed. An attempt passes when the share of correctly answered questions meets the quiz's configured pass threshold. |
| Average score | The mean LLM-generated score across all quiz attempts on a 1–5 scale. |
| Unique users | The number of distinct labelers who have attempted the quiz. |
| Total questions | The number of distinct questions answered in attempts during the selected period. |
| Average time to pass | The average time from the first attempt to the first passing attempt. This excludes the time spent on the first attempt itself. |
| Average attempts to pass | The average number of quiz attempts needed for labelers to pass the quiz for the first time. |

### Understanding the visual analytics

The dashboard includes two key distribution charts to help you visualize the quiz data.

* **Pass attempt distribution:** This chart shows how many attempts labelers need to pass the quiz and helps you identify if the quiz difficulty is appropriate. For example, if most labelers pass on the first attempt, the quiz may be too easy. If they need many attempts, it may be too difficult.
* **Score distribution:** This chart displays the range of scores across all attempts and shows how scores are distributed on the 1–5 scale. This helps you identify if most labelers are performing well or struggling.

### Analyzing the question performance table

This table shows detailed metrics for each quiz question so you can identify any problematic questions.

| Metric | **Description** |
| - | - |
| Question | The full text of the quiz question. Questions marked “Past question” are from previous quiz versions. |
| Average score | The average score (1–5 scale) for this question across all attempts. |
| Average attempts to pass | The average number of attempts needed for labelers to answer this question correctly. |
| Improvement trend | The score difference between the first and last attempts, showing if labelers improve over time. |
| Average time to pass | The average time spent to successfully pass this question (displayed in MM:SS format). |

You can use this table to identify the following:

* Questions with low average scores may need clearer wording or better supporting guidelines.
* Questions requiring many attempts suggest the topic needs a better explanation or training.
* Negative improvement trends indicate labelers aren't learning from retakes.

### Analyzing the user performance table

This table shows individual labeler performance, so you can identify labelers who may need additional training or support.

| Metric | **Description** |
| - | - |
| Email | The labeler's email address. |
| Attempts | The total number of quiz attempts, with the count of passed attempts in parentheses. |
| Average score | The average score across all of this user's attempts (1–5 scale). |
| Improvement trend | The score difference between the user's first and last attempts. |
| Pass rate | The percentage of this user's attempts that achieved a passing score. |
| Time to pass | The time from the first attempt to the first successful pass (displayed in MM:SS format). |
| Quiz status | This shows “Passed” if the user has successfully passed at least once, and “Not Passed” otherwise. |

### Using analytics to improve your quiz

Based on the analytics data, you can take several actions to improve your quiz.

* **If overall pass rates are low:**
  * Clarify the quality standards or guidelines the quiz is testing.
  * Consider breaking down complex concepts into simpler explanations.
  * Add more examples that illustrate correct decisions.
* **If specific questions have low scores:**
  * Revise the question or expected answer.
  * Ensure the question accurately tests the intended knowledge or quality criterion.
  * Adjust the expected answer to be more flexible when appropriate.
* **If labelers need many attempts to pass:**
  * Simplify your quiz questions or make the criteria more explicit.
  * Add practice examples to your guidelines.
  * Consider lowering the **Pass threshold (%)** if appropriate.
* **If improvement trends are negative:**
  * Review the feedback provided by the AI scoring.
  * Ensure questions test applied understanding, not memorization.
  * Consider whether the quiz is testing the right quality concepts.


# Manage roles and permissions
Source: https://docs.labelbox.com/horizon/guides/roles-and-permissions

A guide for understanding and modifying member roles and permissions at the workspace level and the project level.

Control access to your data and projects by assigning roles with specific permissions to your team members.

Member roles define what a user can see and do within Labelbox. Roles can be assigned with a **workspace-wide** scope, giving access to all projects, or a **project-based** scope, limiting access to specific projects. These two scopes are mutually exclusive.

Member roles can be assigned to individual members or *groups*, which consist of members that share the same project assignments, whose permissions can be managed together. Member roles can be set to the following scopes:

| Scope | Description |
| :- | :- |
| Workspace-wide roles | Can access all projects with their default role. This role can't be overridden at the project level. |
| Project-based using Groups | Can access all projects assigned to their group. They inherit their role from the **Default role** in their user profile. |
| Project-based using manual assignment | Can be manually assigned to specific projects. Admins can specify their user role, or it will be inherited from the **Default role** in their user profile. |
| Custom Role | Can be customized by selecting from available permissions. Allows granular control. |

## Workspace-wide roles

A workspace-wide role applies to every project in the workspace and cannot be overridden at the project level. This is ideal for administrators or managers who need broad access.

**To change a member's workspace-wide role:**

1. Navigate to **Workspace settings** in the bottom-left corner and select the **Members** tab.
2. Use the search bar to find the member whose role you want to modify.
3. Locate the member in the list. Their current role will be displayed in the **Workspace Role** column.
4. Click on the current role name (e.g., "Admin", "Team Manager"). A dropdown menu will appear.
5. Select the new workspace-wide role from the list. The change is saved automatically.

| Permission | Admin | Read-only Admin | Data Admin | Reviewer | Project Lead | Team Manager | Labeler |
| - | - | - | - | - | - | - | - |
| View datasets in Catalog | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | | <Icon icon="check" /> | | |
| Import data/create new datasets in Catalog | <Icon icon="check" /> | | <Icon icon="check" /> | | | | |
| Apply filters in Catalog | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | | <Icon icon="check" /> | | |
| Create slices in Catalog | <Icon icon="check" /> | | <Icon icon="check" /> | | <Icon icon="check" /> | | |
| Create new batches | <Icon icon="check" /> | | | | <Icon icon="check" /> | | |
| Rename a batch | <Icon icon="check" /> | | | | <Icon icon="check" /> | | |
| Delete batches | <Icon icon="check" /> | | | | <Icon icon="check" /> | | |
| Remove queued labels from a batch | <Icon icon="check" /> | | | | <Icon icon="check" /> | | |
| Delete non-queued labels from a batch | <Icon icon="check" /> | | | | <Icon icon="check" /> | | |
| Create/modify ontologies | <Icon icon="check" /> | | | | | | |
| Create/modify/delete projects | <Icon icon="check" /> | | | | | | |
| Invite new members to an organization | <Icon icon="check" /> | | | | | | |
| View projects and labels | <Icon icon="check" /> | <Icon icon="check" /> | | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> |
| Add/modify members on a project | <Icon icon="check" /> | | | | <Icon icon="check" /> | <Icon icon="check" /> | |
| Create/modify labels | <Icon icon="check" /> | | | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | own only(Except in Rework task) |
| Review labels | <Icon icon="check" /> | | | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | own only |
| Add a data row as a benchmark | <Icon icon="check" /> | | | | <Icon icon="check" /> | <Icon icon="check" /> | |
| Receive benchmark data rows for labeling | <Icon icon="check" /> | | | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> |
| Delete labels | <Icon icon="check" /> | | | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | |
| View own & team's performance metrics | <Icon icon="check" /> | <Icon icon="check" /> | | own only | <Icon icon="check" /> | <Icon icon="check" /> | own only |
| Create/update/delete/ resolve/reopen issues | Issues & comments set at project-level | | | | <Icon icon="check" /> | | |
| Create/update/delete comments | Issues & comments set at project-level | | | | <Icon icon="check" /> | | |
| Create an export | <Icon icon="check" /> | | | | <Icon icon="check" /> | | |
| Models - full access | <Icon icon="check" /> | | | | | | |
| View Models | <Icon icon="check" /> | <Icon icon="check" /> | | | | | |
| Access the Workflow tab in projects | <Icon icon="check" /> | <Icon icon="check" /> | | | <Icon icon="check" /> | <Icon icon="check" /> | |
| Move data rows between steps | <Icon icon="check" /> | <Icon icon="check" /> | | | <Icon icon="check" /> | <Icon icon="check" /> | |
| Request labeling services | <Icon icon="check" /> | | | | <Icon icon="check" /> | | |

## Project-based roles

Project-based roles are assigned on a per-project basis, either by manually adding a member to a project or by adding them to a group. This allows for more granular access control.

<Note>
  If a user's project access is granted through a group, you cannot modify their role on a project-by-project basis. You must either remove them from the group or change the group's project assignments.
</Note>

**To change a member's role on a specific project:**

1. Navigate to **Workspace settings** and select the **Members** tab.
2. Use the search bar to find the member.
3. Click on the member's name to open their detailed profile view. This view lists all the projects and groups they are assigned to.
4. Under the **Projects** section, find the project you want to modify.
5. Click on the current role name next to the project (e.g., "Labeler", "Reviewer"). A dropdown menu will appear.
6. Select the new project-based role from the list. The change for that specific project is saved automatically.

| Permission | Project-based  project lead | Project-based  team manager | Project-based reviewer | Project-based  labeler |
| - | - | - | - | - |
| Import data/create new datasets | | | | |
| Create new batches | <Icon icon="check" /> | | | |
| Rename a batch | <Icon icon="check" /> | | | |
| Delete batches | <Icon icon="check" /> | | | |
| Remove queued labels from a batch | <Icon icon="check" /> | | | |
| Delete non-queued labels from a batch | <Icon icon="check" /> | | | |
| Edit projects | <Icon icon="check" /> | | | |
| Invite new members to an organization | <Icon icon="check" /> | <Icon icon="check" /> | | |
| Add/modify existing org members on a project | <Icon icon="check" /> | <Icon icon="check" /> | | |
| Create/modify labels | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | own only(Except in Rework task) |
| Review labels | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | own only |
| Add a data row as a benchmark | <Icon icon="check" /> | <Icon icon="check" /> | | |
| Receive benchmark data rows for labeling | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> |
| Delete labels | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | |
| View own & team's performance metrics | <Icon icon="check" /> | <Icon icon="check" /> | own only | own only |
| Create an export | <Icon icon="check" /> | | | |
| Create issues | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | on own assets only |
| Update/delete issues | own only | own only | own only | on own assets only |
| Resolve/Reopen issues | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | own only |
| Create/update/delete comments | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | on own assets only |
| Access the Workflow tab | <Icon icon="check" /> | <Icon icon="check" /> | | |
| Move data rows between steps | <Icon icon="check" /> | <Icon icon="check" /> | | |
| Request labeling services | <Icon icon="check" /> | | | |

## Custom roles

For more granular control, Enterprise customers can create up to five custom project-level roles.

**To create a custom role:**

1. On the **Workspace settings** page, select **User Roles**.
2. Click **+ Create User Role**.
3. Select the specific permissions for this role.
4. Add a name and description, then click **Create**.

Admins can define custom roles at project level by selecting from the following permissions:

| Category | Permission |
| - | - |
| Admin | Add new members to the organization |
| | Access and interact with the API |
| | Cancel ongoing tasks |
| | Create new API keys for accessing services |
| | Delete members from an org |
| | Manage webhooks for the system |
| | Modify the membership of projects or organizations |
| | Modify organizational membership details |
| | Modify the organization’s profile |
| | Refresh organization entitlements |
| | Remove members from projects or organizations |
| | View performance metrics |
| | View project or organization members |
| | View project settings |
| | View user groups |
| | Manage workspaces within the system |
| Annotate | Clear all reservations made within the system |
| | Create new batches for processing data |
| | Create new benchmarks for performance evaluation |
| | Create export tasks for data |
| | Create comments on issues |
| | Create new labels for data |
| | Create prediction tasks or models |
| | Create new projects |
| | Create updates for projects |
| | Create reviews for tasks or projects |
| | Delete benchmarks |
| | Delete labels |
| | Delete a project |
| | Detach datasets from projects or tasks |
| | Manage data collections |
| | Manage categories for issues |
| | Change the status of issues |
| | Modify ontologies |
| | Modify the consensus settings of a project |
| | Modify project settings |
| | Change the status of projects |
| | Modify the workflow of projects |
| | View all labels within the system |
| | View benchmarks |
| | View custom metadata |
| | View ontologies |
| | View comments on issues made by others |
| | View labels created by other users |
| | View your own comments on issues |
| | View labels created by you |
| | View your own performance metrics |
| | View project details |
| | View project consensus settings |
| | View datasets associated with a project |
| | View external organizations associated with a project |
| | View the project’s labeling interface |
| | View the workflow of projects |
| | View the reporting dashboard |
| | View the schema tab |
| | View webhooks |
| | Clear your own reservations within the system |
| Catalog | Create new datasets within the system |
| | Create IAM (Identity and Access Management) integrations |
| | Create slices of data for analysis |
| | Delete individual data rows |
| | Delete datasets from the system |
| | Delete IAM (Identity and Access Management) integrations |
| | Modify batches of data rows |
| | Modify datasets within the system |
| | Modify IAM integrations |
| | Upload files to the system |
| | View batches of data |
| | View the data catalog |
| | View datasets within the system |
| Model | Delete model diagnostics |
| | Delete prediction tasks or models |
| | Manage model configurations |
| | Modify model diagnostics |
| | View model diagnostics |

## Workforce provider roles

When using an external labeling service, their team members are granted special roles with limited permissions. They can only see the labels created by their own team, ensuring your internal work remains private.

| Permission | Workforce admin | Workforce team manager | Workforce reviewer | Workforce labeler |
| - | - | - | - | - |
| Import data/create new datasets | | | | |
| Attach a dataset to a project | <Icon icon="check" /> | | | |
| Detach a dataset from a project | <Icon icon="check" /> | | | |
| Add/edit/delete projects | <Icon icon="check" /> | | | |
| Invite new members to an organization | <Icon icon="check" /> | | | |
| Add/modify existing org members on a project | <Icon icon="check" /> | <Icon icon="check" /> | | |
| Create/modify labels | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> (own only) |
| Review/modify others' labels | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> | |
| Delete labels | <Icon icon="check" /> | <Icon icon="check" /> | | |
| View own & team's performance metrics | <Icon icon="check" /> | <Icon icon="check" /> | | |
| Create an export | <Icon icon="check" /> | <Icon icon="check" /> | <Icon icon="check" /> (own only) | <Icon icon="check" /> (own only) |


# Run Foundry apps
Source: https://docs.labelbox.com/horizon/guides/run-foundry-apps

Shows how to run Foundry apps from the app, from REST endpoints, and using the SDK

Foundry apps are designed to streamline your model runs by automating the process. You can run your apps in several ways:

* From the **Catalog** in the Labelbox app.
* Using the **REST API** for integration with external workflows.
* With the **Python SDK** for more complex, scripted workflows.

This guide will cover each of these methods in detail.

## Run Foundry app from Catalog

1. Go to the **Catalog** and select the data rows you want your Foundry app to process.
2. When prompted to select a model, click on the **Apps** tab.
3. Select your Foundry app from the list.
4. From here, you can generate previews or submit the model run, just like any other Foundry model run.

## Run Foundry app using REST API

You can use the REST API to run a Foundry app, which is useful for triggering predictions from external systems. You can use REST endpoints to get predictions for:

* **Data rows** that are already in your Labelbox Catalog datasets.
* **Raw data assets** that are not stored in Labelbox.

The Labelbox app provides assistance in defining REST queries for common tasks. To get started:

1. Go to the **Model** section and open your Foundry app.
2. Select the **Developers** tab.
3. **Authentication**: If you have not already [generated an API](https://docs.labelbox.com/horizon/developer-guides/create-api-key) key, select **Generate API key** and then follow the prompts. (Remember to save your API key in a safe place.)
4. **Choose your endpoint**: Select the endpoint that corresponds to the task you want to perform. The app provides sample request bodies to help you define your REST query.

### Predict an existing data row

To create a prediction job for a data row that is already in Labelbox, you'll need to specify the data row's ID in the request body. Note that `type` is set to `dataRow` and that `<data_row_id>` specifies the appropriate value.

Example cURL request:

```shellscript theme={"theme":"css-variables"}
curl --location 'https://app.labelbox.com/api/v1/foundry-app/:appId/predict' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer <API_KEY>' \
--data '{
    "type": "dataRow",
    "id": "<data_row_id>"
}'
```

### Predict a raw asset

You can also use Foundry apps to get predictions for raw data assets that are not stored in your Labelbox Catalog.

Example cURL request:

<CodeGroup>
  ```shellscript Creating a prediction job on raw asset with url theme={"theme":"css-variables"}
  curl --location 'https://app.labelbox.com/api/v1/foundry-app/:appId/predict' \
  --header 'Content-Type: application/json' \
  --header 'Authorization: Bearer <API_KEY>' \
  --data '{
      "type": "text | image",
      "url": "<asset_url>",
      "createDataRow": "true",
      "addToModelRun": "true"
  }'
  ```

  ```curl Raw text content theme={"theme":"css-variables"}
  curl --location 'https://app.labelbox.com/api/v1/foundry-app/:appId/predict' \
  --header 'Content-Type: application/json' \
  --header 'Authorization: Bearer <API_KEY>' \
  --data '{
      "type": "text",
      "text": "<text content>",
      "createDataRow": "true",
      "addToModelRun": "true"
  }'
  ```

  ```curl Raw image content theme={"theme":"css-variables"}
  curl --location 'https://app.labelbox.com/api/v1/foundry-app/:appId/predict' \
  --header 'Content-Type: application/json' \
  --header 'Authorization: Bearer <API_KEY>' \
  --data '{
      "type": "image",
      "base64": "<base64 encoding of the image>",
      "createDataRow": "true",
      "addToModelRun": "true"
  }'
  ```
</CodeGroup>

Request body parameters:

| Parameter | Description |
| - | - |
| `type` | Set to either `text` or `image`. |
| `url` | A public URL pointing to the asset. |
| `text` | The text content of the asset. |
| `base64` | A Base64-encoded string that defines an image. |
| `createDataRow` | Set to `true` (default) to create a new data row. |
| `addToModelRun` | Set to `true` (default) to add the data row and prediction to a model run. |

When `createDataRow` and `addToModelRun` are true, the data row is added to a dataset named `App {Name} - inference data`, and the prediction is added to a model run named `App {Name} - inferences`, where `{Name}` is the name of your Foundry app.

Example response:

```json theme={"theme":"css-variables"}
{
    "id": "449f0b35-38c6-4fb1-bb47-144e958d08a4",   <--- Job ID
    "organizationId": "clah7z9b610d707wpfpf35066",
    "userId": "clah7z9br10d807wpboxt920r",
    "modelAppId": "bfe3927c-1f60-44ec-9114-c6a5f6ff0de2",
    "dataRowId": "clqbkkn656s6s07975dhqnotz",
    "status": "in_progress",
    "createdAt": "2024-03-25T19:04:44.850617+00:00"
}
```

The `id` field includes the Job ID, which can be used to retrieve results when the job completes.

### Get prediction results

Prediction jobs run asynchronously, so you'll need to make a second query to get the results. This query requires the `Job ID` that was provided in the response to your initial request.

Example cURL request:

```shellscript theme={"theme":"css-variables"}
curl --location 'https://app.labelbox.com/api/v1/foundry-app/prediction/:id' \
--header 'Authorization: Bearer <API_KEY>'
```

The prediction results will be included in the response. You can also view the results in the **Predictions** tab of your Foundry app in the Labelbox UI.

Example response:

```json theme={"theme":"css-variables"}
{
    "id": "449f0b35-38c6-4fb1-bb47-144e958d08a4",
    "organizationId": "clah7z9b610d707wpfpf35066",
    "userId": "clah7z9br10d807wpboxt920r",
    "modelAppId": "bfe3927c-1f60-44ec-9114-c6a5f6ff0de2",
    "dataRowId": "clqbkkn656s6s07975dhqnotz",
    "status": "success",
    "annotations": [
        {
            "objects": [],
            "relationships": [],
            "classifications": [
                {
                    "name": "describe_photo",
                    "feature_id": "af4e3cd30b904392a370559bf0a92064",
                    "text_answer": {
                        "content": "N/A"
                    }
                },
                {
                    "name": "has_people",
                    "feature_id": "69bda23bc9df4dcab86ace5c95cb6503",
                    "radio_answer": {
                        "name": "no",
                        "feature_id": "c0c57d73d3ba4884af88357bf6c3a025"
                    }
                },
                {
                    "name": "has_chairs",
                    "feature_id": "88f84d30ea4745be98a66baa23d685ed",
                    "radio_answer": {
                        "name": "no",
                        "feature_id": "289e63b18ff746c899a1eb6c0c358fe3"
                    }
                }
            ]
        }
    ],
    "price": 0,
    "createdAt": "2024-03-25T19:04:44.850617+00:00",
    "finishedAt": "2024-03-25T19:05:01.704949+00:00"
}
```

### Find a prediction job

You can search for previous prediction jobs using the following filters:

* **Model App ID**: The ID of your Foundry app.
* **Job status**: `in_progress`, `success`, `error`, `canceled`, or `retried`.
* **Creation date**: A range of dates in ISO 8601 format.

<CodeGroup>
  ```shellscript Search for a prediction job by model id theme={"theme":"css-variables"}
  curl --location 'https://app.labelbox.com/api/v1/foundry-app/:appId/predictions' \
  --header 'Content-Type: application/json' \
  --header 'Authorization: Bearer <API_KEY>' \
  --data '{
    "filters": {
      "modelRunId": "string",
    },
    "pagination": {
      "limit": "number",
      "cursor": "string"
    }
  }'
  ```

  ```curl Search for a prediction job by status theme={"theme":"css-variables"}
  curl --location 'https://app.labelbox.com/api/v1/foundry-app/:appId/predictions' \
  --header 'Content-Type: application/json' \
  --header 'Authorization: Bearer <API_KEY>' \
  --data '{
    "filters": {
      "status": "in_progress|success|error|canceled|retried",
    },
    "pagination": {
      "limit": "number",
      "cursor": "string"
    }
  }'
  ```

  ```curl Search for a prediction job by creation date theme={"theme":"css-variables"}
  curl --location 'https://app.labelbox.com/api/v1/foundry-app/:appId/predictions' \
  --header 'Content-Type: application/json' \
  --header 'Authorization: Bearer <API_KEY>' \
  --data '{
    "filters": {
      "createdFrom": "string (ISO8601 date format)",
      "createdTo": "string (ISO8601 date format)"
    },
    "pagination": {
      "limit": "number",
      "cursor": "string"
    }
  }'
  ```
</CodeGroup>

## Run a Foundry app using the Python SDK

For more advanced workflows, you can use the Python SDK to run your Foundry apps and process the results. For example, you could write a script that runs an app and then sends the predictions to be annotated by a human reviewer.

The **Developers** tab of your Foundry app in the Labelbox UI provides a sample Python script that shows how to:

1. Select a set of data rows.
2. Run a Foundry app on those data rows.
3. Retrieve the prediction results.
4. Send the prediction results to be annotated.

This sample script is a template and will need to be customized with the specific IDs for your project, which you can find in the Labelbox app.


# Search and filter your data in Catalog
Source: https://docs.labelbox.com/horizon/guides/search

Learn how to use the filters in Catalog to help you gather insights about your data, surface high-impact subsets of data rows, and remove unwanted data.

Catalog's true power lies in its advanced search and filtering engine. You can build highly specific queries to isolate the exact data you need for any task, from data cleaning to active learning workflows.

## How to use filters

The filter bar at the top of the Catalog page is your primary interface for building queries.

### Step-by-step instructions:

1. Click the **Add Filter** dropdown to see the available filtering options.
2. Select a filter (e.g., `Metadata`).
3. Configure the filter's operator and value (e.g., `where metadata.weather is "rainy"`).
4. To add another condition, click **Add Filter** again. By default, new filters are joined by an **AND** operator, meaning data must match all conditions. You can change this to **OR** if needed.
5. Once you have a query you want to reuse, click the **Save as Slice** button to name and save your filter set. This creates a dynamic slice you can return to at any time.
   <Tip>
     Tip: To send a saved slice to other members in your workspace, simply copy the URL.
   </Tip>

## Basic search filters

These are the foundational filters based on structured information associated with your data rows.

You can filter by the following basic search filters.

| Attribute | Description |
| - | - |
| Annotation | Find data rows with labels that contain or do not have certain counts of annotations |
| Batch | Find data rows that belong to a particular batch |
| Data row | Filter based on [global key](/horizon/developer-guides/data-row-global-keys), data row ID, created at, and last activity at |
| Dataset | Find data rows that belong to a particular dataset |
| [Issue](/horizon/guides/issues-comments) | Find labelled data rows that have Issue |
| Label Actions | Find data rows that have a label(s) or haven't been labeled yet |
| [Media attributes](/horizon/developer-guides/media-attributes) | Find data rows based on their media type (e.g., image, video, text) or other attributes computed upon upload (e.g., video duration, height, width) |
| [Metadata](/horizon/guides/datarow-metadata) | Find data rows that contain a certain metadata field and/or value |
| Model | Find data rows used in [Foundry](/horizon/guides/foundry) |
| Model prediction | Find data rows that have [Foundry](/horizon/guides/foundry) prediction(s) |
| Project | Find data rows that are associated with a specific labeling project |
| Benchmark agreement | Find data rows with certain [benchmark](/horizon/guides/benchmark) agreement metrics by label or feature |
| Consensus agreement | Find data rows with certain [consensus](/horizon/guides/consensus) agreement metrics by label or feature |

## Advanced search techniques (AI powered)

Go beyond metadata with searches that understand the *content* of your data, powered by embeddings.

You can also search by the following.

### Similarity search

1. Select one or more data rows in the gallery view to act as your "anchors."
2. In the selection menu that appears, choose "Find similar data."
3. Catalog will return a new set of results, ordered by similarity to your selected anchors. This is perfect for finding more examples of a rare edge case you've discovered.

### Natural language

Follow these steps to filter using natural language:

1. Go to the main search bar at the top of the page.
2. Type a plain English description of what you're looking for (e.g., "a person walking a dog on a sunny day" or "customer reviews mentioning 'poor service'").
3. Press Enter. Catalog will translate your text into a semantic search and return the most relevant results from your dataset. You can combine this with other filters for even more specific queries.

### Find text

Follow these steps to find data rows that contain a particular keyword

1. Click "Add Filter" and select `Find Text`.
2. Enter the exact word or phrase you want to find within your text-based data assets (like .txt files or documents).
3. This will return all data rows where the specified text is present in the content of the document itself.

## Next steps

<CardGroup>
  <Card title="Refine the similarity search" icon="square-1" href="/horizon/guides/similarity#adding-anchors" />

  <Card title="Send filtered data rows to a labeling project as a batch" icon="square-2" href="/horizon/guides/batches" />

  <Card title="Add metadata to the filtered data rows" icon="square-3" href="/horizon/guides/datarow-metadata" />
</CardGroup>


# Enhance security with IP address allow-list
Source: https://docs.labelbox.com/horizon/guides/server-addresses

List of Labelbox server IP addresses and step-by-step guides for setting IP allow-listing for AWS S3, GCS, and Azure.

This guide explains how to add an extra layer of security to your cloud storage bucket by creating a policy that only allows connections from Labelbox's servers. This ensures that only Labelbox can access your data, even if your access keys were inadvertently exposed.

This is an optional but highly recommended security measure for any organization that wants to enforce the strictest possible access controls on their data.

## How IP allow-listing works

Think of an IP allow-list as a digital gatekeeper for your data. By applying an IP-based policy to your storage bucket, you are instructing your cloud provider to check the source of every incoming request.

If the request originates from an IP address on your approved list (in this case, Labelbox's servers), it is allowed to proceed. If the request comes from *any* other IP address, it is immediately rejected, adding a network-level layer of security.

This policy works in conjunction with IAM Delegated Access, meaning a request is only successful if it both comes from a Labelbox IP address and correctly assumes the IAM role you configured.

## Labelbox server IP addresses

To configure your allow-list, you will need the following list of Labelbox's egress IP addresses.

Last updated: March 27, 2026

```bash theme={"theme":"css-variables"}
35.223.142.181
34.135.127.45
35.224.108.6
34.10.80.254
```

<Note>
  Disclaimer: This list is subject to change as we grow and improve our infrastructure. To prevent any disruption of service, we strongly recommend that you subscribe to our [product release notes](https://docs.labelbox.com/horizon/release-notes) to be notified of any changes to this list.
</Note>

## Step-by-step implementation guides

### AWS S3

For AWS S3 buckets, use an IP address [bucket policy](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html) (Amazon AWS docs) to permit access to specific addresses.

1. Navigate to the Amazon S3 console and select the bucket you have connected to Labelbox.
2. Go to the **Permissions** tab.
3. Scroll down to the **Bucket policy** section and click **Edit**.
4. Add the following JSON policy to the editor. This policy explicitly denies all `GetObject` actions unless the request comes from one of Labelbox's IP addresses.
   Remember to replace `YOUR-BUCKET-NAME` with the actual name of your S3 bucket.

In the following example, access is permitted only from the address shown. You will likely need to update the example to suit your needs.

<CodeGroup>
  ```json JSON theme={"theme":"css-variables"}
  {
    "Version": "2012-10-17",
    "Id": "S3PolicyId1",
    "Statement": [
      {
        "Sid": "IPAllow",
        "Effect": "Allow",
        "Principal": "*",
        "Action": [ "s3:GetObject" ],
        "Resource": "arn:aws:s3:::examplebucket/*",
        "Condition": {
           "IpAddress": {"aws:SourceIp": "54.240.143.0/24"},
        }
      }
    ]
  }
  ```
</CodeGroup>

To learn more, see [Blocking public access to your Amazon S3 storage](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-control-block-public-access.html) (Amazon AWS docs).

### Google Cloud Platform (GCP)

In GCP, IP-based restrictions are typically managed at the network level using **VPC Service Controls**. This allows you to create a secure perimeter around your projects and data.

1. **Identify or Create a Service Perimeter:** Navigate to the VPC Service Controls page in the Google Cloud Console. You can either add rules to an existing perimeter or create a new one that includes the project containing your GCS bucket.
2. **Configure an Ingress Rule:** Within your perimeter configuration, create an **ingress rule**. This rule defines what is allowed to access the services inside the perimeter from the outside.
3. **Set the Source:** In the "From" attributes of the ingress rule, select **IP subnets** as the source.
4. **Add Labelbox IPs:** Add the list of Labelbox Server IP Addresses to the specified IP subnets.
5. **Set the Service:** In the "To" attributes, specify the services the source is allowed to access. At a minimum, this should include the **"Storage API" (**`storage.googleapis.com`**)**.
6. Save your perimeter configuration.

For more detailed instructions, please refer to Google's official documentation on [configuring ingress and egress policies](https://cloud.google.com/vpc-service-controls/docs/ingress-egress-rules "https://cloud.google.com/vpc-service-controls/docs/ingress-egress-rules").

### Microsoft Azure

In Azure, you can restrict access to your Blob Storage account by configuring its built-in firewall.

1. Navigate to your **Storage Account** in the Azure Portal.
2. In the left-hand navigation pane, under **Security + networking**, select **Networking**.
3. Under the **Firewalls and virtual networks** tab, select the option for **"Enabled from selected virtual networks and IP addresses"**.
4. **Add Labelbox IPs:** In the **Firewall** section, there is an **Address range** text box. Add the Labelbox Server IP Addresses to this list in CIDR notation (e.g., `35.232.254.112/32`). You will need to add each IP address individually.
5. Click **Save**. The firewall will now block all traffic that does not originate from the specified IP addresses.

For more detailed instructions, please refer to Azure's official documentation on [configuring Azure Storage firewalls and virtual networks](https://www.google.com/url?q=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fstorage%2Fcommon%2Fstorage-network-security "https://docs.microsoft.com/en-us/azure/storage/common/storage-network-security").


# Generate signed URLs
Source: https://docs.labelbox.com/horizon/guides/signed-urls

Learn how signed URLs work, what the advantages are, and explore docs for creating signed URLs.

A signed URL is a web link that provides temporary, secure access to a single, specific file in your cloud storage bucket. Think of it as a time-sensitive guest pass to your data.

Each URL is "signed" with your own secret cryptographic keys. This signature validates the request and is only valid for a limited time, which you define (e.g., 7 days). After the URL expires, the link no longer works, and access to the file is revoked.

This method allows you to grant Labelbox access to your data without configuring a permanent IAM trust relationship. However, it shifts the responsibility for managing access credentials entirely to you.

## How signed URLs work

1. **You: Generate URLs** -- You write and run a script on your own infrastructure that generates a unique signed URL for *each data asset* you want to label. You must set an expiration date for these URLs.
2. **You: Format and upload** -- You create a JSON file that maps each data asset to its corresponding signed URL. You then import this JSON file when creating a new dataset in Labelbox.
3. **Labelbox: Renders data** -- When a user opens a labeling task, Labelbox retrieves the corresponding signed URL from your imported JSON file. The Labelbox application uses that URL to fetch the data directly from your bucket and render it in the editor.
4. **Your cloud provider: Validates** -- Your cloud provider inspects the signature on the URL. If the signature is valid and the URL has not expired, it serves the file to the Labelbox application. If not, it returns an error.

Signed URLs permit access to resources without requiring updates to policies or other configuration settings. These URLs are usually temporary and only valid for a limited period. Cloud providers generate a hash value for each signed URL to authenticate access requests. If the hash value is missing or expired, access to the resource is denied.

Here's an example signed URL with an access token generated by the server hosting the file as the hash value:

```text theme={"theme":"css-variables"}
http://example.com/filename?hash=DMF1ucDxtqgxwYQ==
```

## Key advantages

* **Fast to start:** This method is the quickest way to begin a project, as it bypasses the need for initial IAM configuration in your cloud environment. It's excellent for short-term projects, pilots, or proof-of-concepts.
* **Granular, explicit control:** You have explicit, file-level control over exactly which assets are accessible and for precisely how long.

## Step-by-step instructions

Here is a step-by-step guide to using signed URLs with Labelbox.

### **Prerequisites**

Before you start, make sure you have the following:

* Your data is stored in a cloud storage bucket (AWS S3, Google Cloud Storage, or Azure Storage).
* You have the necessary permissions in your cloud environment to generate signed URLs.
* You have a local environment set up to run scripts (e.g., Python, Node.js) for generating the signed URLs.

### Step 1: Generate your signed URLs

For each file you want to import into Labelbox, you need to generate a signed URL. This is typically done by writing a script that uses your cloud provider's SDK.

Refer to your cloud provider's official documentation for instructions and code samples:

* AWS S3: [Upload objects with presigned URLs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/PresignedUrlUploadObject.html)
* Google Cloud Storage: [Create a GET-signed URL for an object using Cloud Storage libraries](https://docs.cloud.google.com/storage/docs/samples/storage-generate-signed-url-v4)
* Microsoft Azure Storage: [Create a user delegation SAS](https://learn.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas)

### Step 2: Create your JSON file

Once you have generated your signed URLs, you will need to create a JSON file that you will upload to Labelbox. This file maps each of your data assets to its corresponding signed URL.

For details on the JSON format for different data types, please refer to the following documentation:

<Columns>
  <Column>
    <Card title="Images" icon="sparkles" href="/horizon/developer-guides/import-image-data" />

    <Card title="Text" icon="sparkles" href="/horizon/developer-guides/import-text-data" />

    <Card title="Geospatial" icon="sparkles" href="/horizon/developer-guides/import-geospatial-data" />

    <Card title="Conversational text" icon="sparkles" href="/horizon/developer-guides/import-conversational-text-data" />
  </Column>

  <Column>
    <Card title="Videos" icon="sparkles" href="/horizon/developer-guides/import-video-data" />

    <Card title="Audio" icon="sparkles" href="/horizon/developer-guides/import-audio-data" />

    <Card title="Documents" icon="sparkles" href="/horizon/developer-guides/import-document-data" />

    <Card title="HTML" icon="sparkles" href="/horizon/developer-guides/import-html-data" />
  </Column>
</Columns>

### Step 3: Import your JSON file into Labelbox

With your JSON file created, you can now import it into Labelbox to create your dataset.

1. Go to **Catalog**.
2. Navigate to the **Import data** section.
3. Select the option to upload a JSON file.
4. Upload the JSON file you created in the previous step.
5. Labelbox will process the file and create a new dataset with your data.

## Security best practices

* **Use Short-Lived Expiration Times:** Set the expiration time for your signed URLs to be as short as possible. This minimizes the risk of unauthorized access if a URL is accidentally exposed. For most labeling projects, an expiration of 7 days is sufficient.
* **Principle of Least Privilege:** Only generate signed URLs for the specific files that need to be labeled. Avoid generating URLs for entire buckets or directories.
* **Audit Regularly:** Keep logs of the signed URLs you generate and review them periodically to ensure there is no unusual activity.


# Enabling single sign-on (SSO) for your organization
Source: https://docs.labelbox.com/horizon/guides/single-sign-on

Use this guide to learn how Labelbox supports SSO and what is needed to enable SSO for your organization.

Single sign-on (SSO) allows your team to log in to Labelbox using the same identity provider you use for other services—no separate Labelbox passwords required. Once SSO is enabled for your organization, Labelbox automatically detects your domain at login and redirects users to your identity provider for authentication.

Single sign-on is an add-on available to [Enterprise customers](/horizon/guides/update-account-plan). To request this add-on, contact [Sales](https://labelbox.com/sales/).

## Before you begin

To set up SSO, you'll need to gather the following information from your identity provider. Most of this can be found in your identity provider's admin console or extracted from a metadata XML file.<br /><br />**For SAML providers (recommended):**

* **Sign-in URL**: The login URL provided by your identity provider (sometimes called the SSO URL or SAML endpoint). This can be extracted from a metadata XML file if you have one.
* **X.509 signing certificate**: The public certificate your identity provider uses to sign SAML assertions. This should be in PEM or CER format and can also be extracted from a metadata XML file.
* **Email domain(s)**: The domain(s) associated with users who will log in via SSO (e.g., `yourcompany.com`).

**For OIDC providers:**

* **Client ID**: A unique identifier assigned to Labelbox by your identity provider when you register it as an application. Think of it as the "username" that tells your identity provider which application is requesting authentication.
* **Client Secret**: A confidential key paired with the Client ID, used to verify that the authentication request is genuinely coming from Labelbox. Treat this like a password, never share it publicly.
* **Email domain(s)**: The domain(s) associated with users who will log in via SSO (e.g., `yourcompany.com`).

<Tip>
  If your identity provider provides a metadata XML file, you can share that directly with Labelbox support. We can extract the required values from it.
</Tip>

## How to enable SSO

SSO is configured by the Labelbox team on your behalf. To get started:

1. **Gather the required information** listed above from your identity provider.
2. **Contact Labelbox support** at [support@labelbox.com](mailto:support@labelbox.com) with your SSO details. If you have a metadata XML file, attach it to your message.
3. **Labelbox configures the connection**: Our team will set up the SSO connection for your organization using the information you provide.
4. **Test the integration**: Once configured, you'll be notified to test the login flow. Open the Labelbox sign-in page, enter an email address on your domain, and confirm you're redirected to your identity provider correctly.
5. After a successful test, SSO will be fully enabled for your organization.

## Logging in with SSO

Once SSO is enabled for your organization:

1. Navigate to [app.labelbox.com](http://app.labelbox.com).
2. Enter your work email address.
3. Labelbox automatically detects that SSO is configured for your domain and redirects you to your identity provider.
4. Authenticate with your identity provider credentials as you normally would.
5. You're redirected back to Labelbox and logged in.

<Note>
  Note: Users do not need to enter a Labelbox password when SSO is enabled. Authentication is handled entirely by your identity provider.
</Note>

## Supported identity providers

Labelbox supports single sign-on using **SAML 2.0** and **OpenID Connect (OIDC)** with standard-compliant identity providers. Commonly used providers include:

* SAML 2.0
* OpenID Connect
* Microsoft Entra ID
* Active Directory Federation Services (ADFS)
* Okta
* and other enterprise identity providers

You can install a verified version of the Labelbox Enterprise Application in your Azure tenant to manage user access and configure SSO directly. For detailed steps, see the [Labelbox tutorial](https://learn.microsoft.com/en-us/entra/identity/saas-apps/labelbox-tutorial) on Microsoft Entra ID.

## Troubleshooting

**I entered my email but wasn't redirected to my identity provider.**<br />Make sure you're using an email address that matches the domain registered for SSO. If you believe SSO should be configured for your domain, contact [support@labelbox.com](mailto:support@labelbox.com).<br /><br />**I'm getting an authentication error after being redirected.**<br />This is often caused by a misconfigured attribute mapping or an incorrect certificate. Check with your identity provider admin to confirm the sign-in URL and certificate are correct, then reach out to Labelbox support with any error details.<br /><br />**Some users on my domain can't log in via SSO.**<br />Confirm that those users have been assigned the Labelbox application in your identity provider. Users must be provisioned in your identity provider to authenticate via SSO.

*Need help? Reach out to [support@labelbox.com](mailto:support@labelbox.com) and our team will assist you.*


# Save a filter as a slice
Source: https://docs.labelbox.com/horizon/guides/slices

Create, explore, and updates slices in Catalog.

Think of a "slice" as a saved search. It's a dynamic, saved collection of your data that you've grouped together using one or more filters. For example, you could create a slice of all images that have a "dog" label and were uploaded in the last week.

Because slices are dynamic, they automatically update. Any new data you add to Labelbox that matches your saved filters will automatically appear in that slice.

## How to create a slice

1. **Start with a search**: In the **Catalog**, use the [filter](/horizon/guides/filtering-and-sorting) options to narrow down your data. You can filter by metadata, annotations, dataset, and more.
2. **Save your search**: Once you're happy with your filtered results, click the **Save slice** button.
3. **Name your slice**: Give your slice a descriptive name (between 3 and 30 characters) and add an optional description.
4. **Find your slice**: Your newly created slice will now appear in the **Slices** list on the left-hand panel of the Catalog.

After you create a slice, the slice will show up in your list of **Slices** in the left side panel of Catalog.

<Info>
  **Limits**

  See [this page](/horizon/guides/limits) to learn the limits for creating slices.
</Info>

## Explore your slices

To explore a slice, go to the left side panel of the Catalog, click on **Slices**, and click on the slice name you want to explore.

The filters associated with the slice will appear instantly. All data rows matching the filters will also appear, including data rows uploaded after the slice was created.

Once you are done exploring a slice, you can:

* Click on a different slice name to explore a different slice.
* Click on **Datasets** and a dataset name to explore a dataset.
* Click on **All datasets** to explore your complete Catalog.

## Update a slice

Over time, you might need to adjust the filters associated with a slice. Upon navigating to a slice, you can modify, add, or delete attributes of the filters.

**To update a slice:**

1. Navigate to the slice you want to change.
2. Modify, add, or delete filters as needed.
3. Click **Update slice**. You'll be prompted to select one of the following options.
   * **Save** the changes to the current slice to (update it with the newly configured filters).
   * **Save as new** a distinct slice with the newly configured filters.

## Other actions

When you select a slice, you can access its dedicated menu to:

* **Edit** the slice (aka, change its name or description).
* **Copy** the slice ID that you can use with the Python SDK (see [link](/horizon/developer-guides/slice)).
* **Export** the slice (see [Export data rows from Catalog](/horizon/developer-guides/export-overview#export-data-rows-from-catalog)).
* **Delete** the slice. Be careful. This can't be undone.


# Team management overview
Source: https://docs.labelbox.com/horizon/guides/team-management

An overview of the tools Labelbox offers to help you manage your workspace and team members.

Labelbox provides a comprehensive suite of tools to help you manage your projects and teams efficiently. You can organize your work into distinct workspaces, monitor real-time progress through detailed performance metrics, assign roles and permissions for granular access control, and set up notifications to stay on top of all project activities.

This guide provides a complete overview of team management in Labelbox, broken down into the following sections:

<Card title="Understanding your workspace" icon="sparkles" href="/horizon/guides/workspaces">
  Learn how to use workspaces to organize your projects, data, and users.
</Card>

<Card title="Managing members & groups" icon="sparkles" href="/horizon/guides/manage-members-and-groups">
  Get step-by-step instructions for inviting users and managing team access at scale.
</Card>

<Card title="Understanding roles & permissions" icon="sparkles" href="/horizon/guides/roles-and-permissions">
  A detailed breakdown of every role and permission available in Labelbox.
</Card>

<Card title="Monitoring performance & staying updated" icon="sparkles" href="/horizon/guides/monitor">
  Learn how to track labeling performance and configure notifications.
</Card>


# Technical requirements & best practices
Source: https://docs.labelbox.com/horizon/guides/technical-prerequisites

A guide for optimizing your Labelbox experience.

To ensure a smooth and efficient experience on the Labelbox platform, it's important to meet our technical requirements and follow our recommended best practices. This guide consolidates all the necessary information to help you optimize performance, from system setup to project execution.

## System and hardware requirements

Your computer's hardware plays a significant role in the performance of the Labelbox editor. Below are the minimum requirements to use the platform and the recommended specifications for an optimal experience.

| Detail | Minimum requirement | Recommended for optimal performance |
| - | - | - |
| CPU | 64-bit | - |
| Memory (RAM) | 8 GB | 16 GB or greater |
| Display resolution | 720p (1280x720) | - |
| Disk space | 4 GB available | - |

<Note>
  Effective resolutions below 1280x720 may not display all user interface elements correctly.
</Note>

## Browser and network configuration

For the best performance and compatibility, please adhere to the following browser and network settings.

### Browser compatibility

Labelbox is certified and exclusively optimized for **Google Chrome**. While the platform might function on other browsers, we cannot guarantee support for any issues you might encounter.

To render graphics with optimal performance, especially for 3D labeling, you must **enable hardware acceleration** in Chrome.

1. Go to Chrome `Settings` > `System`.
2. Toggle on **Use graphics acceleration when available**.

If you see a "There's a glitch" error even with hardware acceleration enabled, you may need to update your video drivers. Please refer to your device vendor's documentation for instructions.

### Network speed

| Requirement | Speed |
| - | - |
| Minimum | 40 Mbps |
| Recommended | 200 Mbps or faster |

## Best practices for data and projects

How you manage your data and set up your projects can significantly impact editor performance and labeling efficiency.

| Best practice | Recommendation |
| - | - |
| **Optimize asset size (256 MB or smaller)** | The most effective way to improve editor performance is to limit your asset size. We recommend images no larger than 4000x4000px and videos with frame rates no greater than 30fps. |
| **Use IAM delegated access** | For optimal security and performance, use **IAM delegated access** to connect your cloud storage to Labelbox. This keeps your assets in your own storage and grants Labelbox only the limited access it needs. |
| **Use a Content Delivery Network (CDN)** | If your labelers are in different geographic locations or time zones from where your data is stored, a CDN can significantly speed up the delivery of assets to them. |
| **Conduct a trial run** | Before starting a large project, set up a small test project that simulates your actual labeling environment (time of day, location, network, and machines). This will help you identify and resolve any latency issues upfront. |

### How Labelbox maximizes labeling speed

The Labelbox editor is designed to maximize labeling speed and data integrity using a few key background strategies:

* **Asset caching**: When a labeler begins a session, the editor pre-loads and caches the assets in their queue. This ensures that the next asset loads instantly upon completion of the previous one.
* **Disabled controls during load**: To prevent errors and ensure data integrity, all labeling tools and editor controls remain inactive until an asset is fully loaded.
* **EXIF data stripping**: To guarantee that labels are always drawn and stored at the correct orientation, we strip any EXIF data from your assets.

## Account and interface enhancements

* **Multi-factor authentication (MFA)**: We strongly recommend enabling MFA to add an extra layer of security to your account and reduce the risk of unauthorized access.
* **Dark mode**: To reduce eye strain, especially during long labeling sessions, you can switch to Dark Mode at any time. Click the **Dark mode** icon in the Labelbox menu to toggle between light and dark themes.


# Text
Source: https://docs.labelbox.com/horizon/guides/text-editor

Guide for labeling text data.

When creating a project, select **Text**.

## Import text data

To learn how to text data to Labelbox, visit our docs on [Importing text data](/horizon/developer-guides/import-text-data).

<Info>
  ### Data row size limit

  To view the maximum size allowed for a data row, visit our [limits](/horizon/guides/limits) page.
</Info>

## Supported annotation types

Below are all of the annotation types you may include in your ontology when you are labeling text data. Classification-type annotations can be applied at the global level and/or nested within an object-type annotation.

| Feature | Import annotations | Export annotations |
| - | - | - |
| **Entity** | [See payload](/horizon/developer-guides/import-text-annotations#entity) | [See payload](/horizon/developer-guides/export-text-annotations#text-entity-named-entity) |
| **Relationships** | [See payload](/horizon/developer-guides/import-text-annotations#relationship) | [See payload](/horizon/developer-guides/export-text-annotations#relationship) |
| **Radio classification** | [See payload](/horizon/developer-guides/import-text-annotations#classification---radio-single-choice) | [See payload](/horizon/developer-guides/export-text-annotations#radio) |
| **Checklist classification** | [See payload](/horizon/developer-guides/import-text-annotations#classification---checklist-multi-choice) | [See payload](/horizon/developer-guides/export-text-annotations#checklist) |
| **Free-form text classification** | [See payload](/horizon/developer-guides/import-text-annotations#classification---free-form-text) | [See payload](/horizon/developer-guides/export-text-annotations#free-form-text) |

### Text entity

Create an entity by clicking the desired starting character and dragging to select a sequence of characters in the unstructured text.

The characters in a text file are not restricted to a single annotation. Therefore, entity annotations can overlap completely or partially.

#### Full word selection

When labeling text, you can hover over and annotate full words with one click rather than dragging your cursor and labeling character by character. This feature allows for vastly more efficient labeling and decreases the likelihood of labeling errors.

To toggle between **Word** and **Character** mode, click on the **Settings** icon in the text editor and select an option. You can also toggle between approaches using the hotkeys `Shift` + `W` for **Word** mode and `Shift` + `C` for **Character** mode. The default setting for all users is **Word** mode.

### Relationship

With annotation relationships, you can create and define relationships between entity annotations in the text editor. You can then use these annotation relationships to consolidate labeling workflows and potentially reduce the number of language models needed.

Follow these steps to create a relationship between two entity annotations:

1. In the editor, create two entity annotations.

2. Select the relationship annotation from the Tools menu, then click on one of the entity annotations. To create the relationship, move your cursor to another entity and click to create the relationship.

3. Add an optional subclassification to the relationship.

## Text-specific hotkeys

| Function | Hotkey | Description |
| - | - | - |
| Create a relationship | `Option` + Click (source) + Click (target) | Select an entity to be the source of a relationship, then select another entity to be the target. |


# Geospatial / tiled imagery
Source: https://docs.labelbox.com/horizon/guides/tiled-editor

Guide for labeling geospatial / simple tile data.

When creating a project, select either **Geospatial** (GeoTIFF or COG/NTR asset) or **Simple tile** (slippy map).

## Import geospatial data

For more information on the import format, see the documentation on [Geospatial / Tiled Imagery import formats](/horizon/developer-guides/import-geospatial-data).

<Info>
  ### Data row size limit

  To view the maximum size allowed for a data row, visit our [limits](/horizon/guides/limits) page.
</Info>

## Supported annotation types

These are the annotation types you may include in your ontology for labeling tiled imagery data. Classification-type annotations can be applied globally and/or nested within an object-type annotation.

| Feature | Import annotation | Export annotation |
| - | - | - |
| **Bounding box** | [See payload](/horizon/developer-guides/import-geospatial-annotations#bounding-box) | [See payload](/horizon/developer-guides/export-geospatial-annotations#bounding-box) |
| **Polygon** | [See payload](/horizon/developer-guides/import-geospatial-annotations#polygon) | [See payload](/horizon/developer-guides/export-geospatial-annotations#polygon) |
| **Polyline** | [See payload](/horizon/developer-guides/import-geospatial-annotations#polyline) | [See payload](/horizon/developer-guides/export-geospatial-annotations#polyline) |
| **Point** | [See payload](/horizon/developer-guides/import-geospatial-annotations#point) | [See payload](/horizon/developer-guides/export-geospatial-annotations#point) |
| **Radio classification** | [See payload](/horizon/developer-guides/import-geospatial-annotations#classification-radio-single-choice) | [See payload](/horizon/developer-guides/export-geospatial-annotations#classification---radio) |
| **Checklist classification** | [See payload](/horizon/developer-guides/import-geospatial-annotations#classification-checklist-multi-choice) | [See payload](/horizon/developer-guides/export-geospatial-annotations#classification---checklist) |
| **Free-form text classification** | [See payload](/horizon/developer-guides/import-geospatial-annotations#classification-free-form-text) | [See payload](/horizon/developer-guides/export-geospatial-annotations#classification---free-form-text) |

<Info>
  ### Use polygon for segmentation mask use cases

  Use the polygon tool if your use case requires segmentation masks for the following reasons:

  1. The polygon tool with free-hand annotations will allow you to draw the necessary shapes and the export will return the vertices in the coordinate system of your geo file.
  2. You will not have to deal with large PNGs that are returned when using segmentation masks.
</Info>

### Bounding box

Create a bounding box by starting at one corner and dragging your cursor to create the shape around an object in the tiled image.

You can also click and drag to reposition the bounding box on the tiled image.

### Polygon

Create a polygon annotation by clicking to create each point in the shape. Click the first point to close the polygon.

#### Pen tool

In order to create a more free-hand shape, click and hold down your mouse, and the polygon tool automatically adds vertices in the cursor's path. The slower you move your cursor, the closer the points will be together. Alternatively, let the cursor go to draw straight lines between points.

The pen tool is only available when creating polygons. Click the pen icon with the plus sign to use the pen tool.

You can also incrementally add to a completed polygon using the pen tool.

#### Erase tool

You can use the erase tool to clean up the edges of your polygon. Click the pen icon with the minus sign to use the erase tool.

### Polyline

Use the polyline tool to label lines in the tiled imagery editor. Click on the last point to complete the line.

### Point

Use the point tool to label precise locations on the tiled image.

## How Slippy maps work

Tiled imagery uses a slippy map tool for labeling map tiles of the earth at various zoom levels. *Slippy map* is a term that refers to modern web maps that allow for zooming and panning.

Map tiles are structured like a pyramid of zoom levels. The topmost layer is the largest in size and only includes one tile. The tiles in each subsequent layer decrease in size but increase in number and resolution. You can use the `minZoom` and `maxZoom` parameters to specify the zoom level limits. The zoom parameter is an integer between 0 (zoomed out) and 18 (zoomed in). 18 is normally the maximum, but some tile servers may exceed that.

In order to use our tiled imagery editor, you must use a tile server to create your map tiles. When you import your map tiles, you can use bounds to specify the area of interest and to break up your map tiles into appropriate labeling tasks.

Tiled imagery supports the following coordinate reference systems (CRS):

| CRS | Description |
| - | - |
| EPSG:3857 | A projected coordinate system measured in meters on a flattened surface. It is the projection of the EPSG:4326 coordinate system onto a square for viewing on a web app. Uses the Spherical Mercator projection (also known as the Web Mercator projection) to render within a web page. |
| EPSG:4326 | A geographic coordinate system used by GPS to express locations on the earth using GeoJSON longitude and latitude. Also known as the World Geodetic System (WGS84). |
| Simple | A basic geographic coordinate reference system that maps `x` to longitude and `y` to latitude. Also uses the Equirectangular or Plate Carrée projection to render within a web page. |


# How to update your account plan
Source: https://docs.labelbox.com/horizon/guides/update-account-plan

Shows how to upgrade, downgrade, or cancel your Labelbox account.

As your data management and labeling needs evolve, you may need to adjust your Labelbox subscription. Whether you are scaling up to unlock advanced tools and higher usage limits, or scaling down between large projects, managing your plan is straightforward.

This guide explains how to upgrade, downgrade, or cancel your Labelbox account.

## Upgrade your account plan

Upgrading your account gives your team access to more features, higher usage capacities, and potentially dedicated support. The steps to upgrade depend on the tier you are moving to:

* **To upgrade to a Starter plan:** Navigate to your **Billing** tab, locate the **Starter** plan and click **Upgrade**. Add your payment details and select **Click to upgrade** to finalize the setup.
* **To upgrade to an Enterprise plan:** Please [contact sales](https://labelbox.com/sales/) to create a tailored plan for your organization's needs.

**Payment and activation context:**\
To upgrade any plan (including from a Free tier), you must have a valid payment method on file. You can manage your payment methods directly in the **Billing** tab.

* **Credit cards:** Your upgrade goes into effect immediately once the payment is accepted.
* **ACH transfers (US only):** Your upgrade becomes active as soon as your bank confirms the transfer.

## Downgrade your account plan

If you anticipate lower usage in the future, you can downgrade your plan. Downgrades do not happen immediately; instead, they take effect at the beginning of your next billing cycle (typically the 1st of the next month). This means you will keep your current plan's benefits for the remainder of the month.

* **To downgrade from Starter to a free plan:**
  1. Go to **Workplace Settings** to open your organization settings.
  2. Navigate to the **Billing** tab.
  3. Click the **Manage** button next to your plan's name.
  4. Choose **Cancel subscription** in the management window, and confirm the cancellation by clicking **Cancel subscription** again.
* **To downgrade from an Enterprise plan to a lower plan (or change billing frequency):** Please [contact Sales](https://labelbox.com/sales/).

**Note:** When a downgrade is pending, you will see a reminder banner on your Billing tab. If you change your mind before the new billing cycle starts, you can easily cancel the downgrade by clicking renew on your plan.

## Cancel your account plan

If you no longer need to use Labelbox, you can cancel your subscription entirely.

**Important context:** Before canceling your plan, it is highly recommended that you clean up your workspace by deleting your data rows, datasets, and projects, as you will lose access to this information.

* **To cancel a Free, Starter, or Standard plan:** Open your **Workplace Settings** and navigate to the **Billing** tab. Scroll to the bottom and click the **Cancel plan** button.
* **To cancel an Enterprise plan:** Please [contact Sales](https://labelbox.com/sales/).


# Send predictions for human review
Source: https://docs.labelbox.com/horizon/guides/use-foundry-predictions



Once your model run is complete, you'll have a set of predictions. This page explains how to view, analyze, and use them.

## View and analyze predictions

You can view the predictions generated by your model run in two ways:

**From the Models tab**

1. Go to the **Models** tab and select your completed model run.
2. In the **Predictions** tab of the model run, you'll see a list of all data rows with generated predictions.
3. Click on any data row to open it in the viewer and see the model's predictions overlaid on the asset.

**From Catalog**

1. Navigate to the **Catalog**.
2. Use the filters to find your model run. This will show you all the data rows that were part of that run.
3. Click on a data row to view its predictions.

## Step-by-step instructions

One of the most powerful features of Foundry is its ability to create a tight feedback loop between your model and your human labeling teams. This process, often called *model-assisted labeling* or *active learning*, allows you to use your model's predictions as a starting point for your labelers, dramatically accelerating the creation of high-quality training data.

Instead of labeling from scratch, your team can simply review, correct, or approve the labels generated by the model. By focusing human effort on the most uncertain or incorrect predictions, you can improve your model's performance more efficiently.

### Workflow and strategy

Before sending predictions to a project, consider your goal. You don't need to send every prediction for review. A targeted approach is more effective. Common strategies include:

* **Focusing on Low-Confidence Predictions:** This is the core of most active learning workflows. By filtering for predictions where the model had a low confidence score, you are prioritizing the data that the model found most difficult to understand. Correcting these examples provides the most valuable signal for fine-tuning your next model.
* **Targeting Specific Classes:** If you know your model struggles with a particular object or class, you can filter for all predictions of that class and send them for review to improve the model's performance on that specific area.
* **Random Sampling for Quality Control:** To get a general sense of your model's performance across all classes, you can send a random sample of predictions for review. This helps you spot systemic issues and calculate overall accuracy metrics.

### Step 1: Navigate to your model run

1. From the main menu, select **Model**.
2. Find the completed model run you want to work with and click on it to open its details page.

### Step 2: Filter and select predictions

1. Click on the **Predictions** tab within your model run. You will see a list of all data rows that have predictions.
2. Use the filter panel on the left to narrow down the predictions you want to send for review. The most common filter is **Confidence**, which you can use to isolate predictions below a certain threshold (e.g., `confidence < 0.75`).
3. Once you have filtered the list, select the specific predictions you want to send. You can use the checkbox at the top of the list to select all filtered predictions.

### Step 3: Initiate "Send to Project"

With your desired predictions selected, click the Send to project button located at the bottom of the screen. A configuration dialog will appear.

### Step 4: Configure the project and ontology mapping

This is a critical step to ensure the predictions are correctly imported into your labeling project.

1. **Select Project:** Choose the destination labeling project from the dropdown menu. This project's queue will receive the predictions as new labeling tasks.
2. **Map Ontology:** You must map the model's output features to your project's ontology.
   * On the left, you'll see the feature classes from the model (e.g., `car`, `person`).
   * On the right, use the dropdowns to select the corresponding feature class from your project's ontology.
   * This "translation" step ensures that a prediction for `car` from the model is correctly understood as a `car` annotation in your project.

### Step 5: Submit for review

1. After confirming the project and ontology mapping, click **Submit**.
2. The selected predictions will be added to the chosen project's labeling queue.

When your labelers open a task that was sent from Foundry, they will not see a blank asset. Instead, they will see the data asset pre-labeled with the model's predictions.


# Connect Google Cloud Storage to Labelbox via IAM Delegated Access
Source: https://docs.labelbox.com/horizon/guides/using-google-cloud-storage

Learn how to import your GCS bucket data to Labelbox via IAM delegated access.

This guide provides a complete set of step-by-step instructions for securely connecting your Google Cloud Storage (GCS) bucket to Labelbox. The process uses a dedicated service account to grant Labelbox delegated access to your data.

## Prerequisites

Before you begin, please ensure you have the following:

* Permissions to create service accounts and assign IAM roles in your Google Cloud Platform (GCP) project.
* The name of the GCS bucket you want to connect.
* Configure [Cross-Origin Resource Sharing (CORS)](/horizon/guides/create-cors-headers) on your GCS account.

## Step 1: Create a custom service account in GCP

First, you will create a new service account in your GCP project that Labelbox will use to access your data.

1. In the Google Cloud Console, navigate to **IAM & Admin > Service Accounts**.
2. Click **Create Service Account**.
3. Enter a **Service account name** (e.g., `labelbox-integration-svc`) and a brief description.
4. Click **Create and Continue**. You do not need to grant this service account access to the project or grant users access to this service account in this step.
5. Click **Done**.
6. From the list of service accounts, copy the **email address** of the service account you just created. You will need it in the next steps.

## Step 2: Assign IAM roles to the service account

Next, you need to grant the new service account the necessary permissions to read data from your GCS bucket and to create security tokens.

### Part A: Grant access to the GCS bucket

1. Navigate to your GCS bucket by going to **Cloud Storage > Buckets**.
2. Select the bucket you want to connect and go to the **Permissions** tab.
3. Click **Grant Access**.
4. In the **New principals** field, paste the email address of the service account you created in Step 1.
5. In the **Assign roles** dropdown, select the **Storage Object Viewer** role.
6. Click **Save**.

### Part B: Grant service account token creator role

1. Navigate back to **IAM & Admin > Service Accounts** and select the service account you created.
2. Go to the **Permissions** tab for that service account.
3. Click **Grant Access**.
4. In the **New principals** field, paste the same service account email address.
5. In the **Assign roles** dropdown, select the **Service Account Token Creator** role.
6. Click **Save**.

## Step 3: Grant Labelbox access to your service account

Now, you will authorize the official Labelbox service account to act on behalf of the service account you created.

1. Navigate back to **IAM & Admin > Service Accounts** and select the service account you created.
2. Go to the **Permissions** tab for that service account.
3. Click **Grant Access**.
4. In the **New principals** field, paste the official Labelbox service account email you created in Step 1 part 6 above.  For example:`gdsa-CUID@labelbox-prod.iam.gserviceaccount.com`.
5. In the **Assign roles** dropdown, select the **Service Account Token Creator** role.
6. Click **Save**.

## Step 4: Complete the integration in Labelbox

Finally, return to Labelbox to complete the integration setup.

1. In Labelbox, navigate to **Settings > Integrations** and click **New integration**.
2. Select **Google Cloud Platform**.
3. On the integration page, provide the following details:
   * **Integration Name:** A descriptive name for your integration.
   * **GCS Bucket Name:** The name of your GCS bucket (e.g., `my-bucket`).
   * **Service Account Email:** The email address of the custom service account you created in Step 1.
4. Click **Save integration**.

Labelbox will automatically validate the connection. Once validated, your integration is complete and you can begin creating datasets with your GCS data.

## Step 5: Upload your data

Once the connection is successfully established, you can begin uploading your data to Labelbox.

<Columns>
  <Column>
    <Card title="Import image data" icon="plus" href="/horizon/developer-guides/import-image-data" />

    <Card title="Import text data" icon="plus" href="/horizon/developer-guides/import-text-data" />

    <Card title="Import document data" icon="plus" href="/horizon/developer-guides/import-document-data" />

    <Card title="Import HTML data" icon="plus" href="/horizon/developer-guides/import-html-data" />
  </Column>

  <Column>
    <Card title="Import video data" icon="plus" href="/horizon/developer-guides/import-video-data" />

    <Card title="Import geospatial data" icon="plus" href="/horizon/developer-guides/import-geospatial-data" />

    <Card title="Import audio data" icon="plus" href="/horizon/developer-guides/import-audio-data" />
  </Column>
</Columns>

If a dataset is signed by a GCP IAM integration, Labelbox will attempt to sign all data rows with this integration.

The value of `rowData` for each Data Row will be updated as follows (the `queryParams` contain signing information):

`<https://storage.googleapis.com/${bucket}/${key}?{queryParams}`>

<Info>
  Please ensure that you are using `gsutil` URIs during data import (JSON file or Python SDK).

  Example: `gs://gcs-lb-demo-bucket/test.png`
</Info>

You should now be set up with IAM delegated access. Labelbox will use the roles you assigned to securely access data from your GCS bucket.


# Videos
Source: https://docs.labelbox.com/horizon/guides/video-editor

Learn how to add labels to video data

With the video editor, you can apply scalable annotations to video data for training AI models.

## Import video data

To learn how to import video data to Labelbox, visit the documentation on [importing video data](/horizon/developer-guides/import-video-data).

<Info>
  ### Data row size limit

  To view the maximum size allowed for a data row, see [limits](/horizon/guides/limits).
</Info>

## Supported annotation types

Below are the annotation types you may include in your ontology when labeling video data. Classifications can be applied globally, at the frame level, or nested within another annotation.

| Feature | Import annotations | Export annotations |
| - | - | - |
| **Bounding box** | [See payload](/horizon/developer-guides/import-video-annotations#bounding-box-frame-based) | [See payload](/horizon/developer-guides/export-video-annotations#bounding-box) |
| **Polyline** | [See payload](/horizon/developer-guides/import-video-annotations#polyline-frame-based) | [See payload](/horizon/developer-guides/export-video-annotations#polyline) |
| **Point** | [See payload](/horizon/developer-guides/import-video-annotations#point-frame-based) | [See payload](/horizon/developer-guides/export-video-annotations#point) |
| **Segmentation mask** | [See payload](/horizon/developer-guides/import-video-annotations#masks-frame-based) | [See payload](/horizon/developer-guides/export-video-annotations#mask) |
| **Radio classification** | [See payload](/horizon/developer-guides/import-video-annotations#classification-radio-frame-based) | [See payload](/horizon/developer-guides/export-video-annotations#classification---radio) |
| **Checklist classification** | [See payload](/horizon/developer-guides/import-video-annotations#classification-checklist-frame-based) | [See payload](/horizon/developer-guides/export-video-annotations#classification---checklist) |
| **Global free-form text classification** | [See payload](/horizon/developer-guides/import-video-annotations#classification-free-form-text) | [See payload](/horizon/developer-guides/export-video-annotations#classification---free-form-text) |

### Bounding box

To create a bounding box, use your cursor to place the top left and bottom right corners of the shape. By default, the shape will be created in the current frame and all frames following. You can use the annotation toggle (or the hotkey `T`) to adjust the length of the object instance.

To reposition the bounding box, simply click and hold, then use your mouse or trackpad to reposition the annotation on the frame. You can also click and drag the corners to resize the bounding box. Whenever an object is created or edited on a video frame, a keyframe is created, and all frames between keyframes will interpolate.

#### AI-based bounding box tracking

To track an object automatically, simply draw a bounding box around the object and then press the **Track Object** button that appears right above the timeline.

By default, it tracks for 10 frames at a time, but you can change it to a higher number or track until the end of the video by pressing the settings button next to **Track Object**.

Watch this video to better understand how to use bounding box tracking:

<Frame>
  <iframe name="wistia_embed" />
</Frame>

### Polyline

To create a polyline, use your cursor to place each vertex of the line and click the last point again (or press `F` to end the shape. By default, the shape will be created in the current frame and all frames following. You can use the annotation toggle (or the hotkey `T`) to adjust the length of the object instance.

To reposition the polyline, simply click and drag each vertex to reposition it on the frame.

You can also click and drag the polyline to change its position. Whenever an object is created or edited on a video frame, a keyframe is created, and all frames between keyframes will interpolate.

### Point

To create a point, put your cursor where you want to place the point and click and release to create it. By default, the shape will be created in the current frame and all frames following. You can use the toggle Annotations menu (or the hotkey `T`) to adjust the length of the object instance.

To reposition the point, simply click and drag it to reposition it on the frame. Whenever an object is created or edited on a video frame, a keyframe is created, and all frames between keyframes will interpolate.

### Segmentation mask

Unlike other tools, segmentation masks can only be created on individual frames. Multiple masks may be created on the same frame.

To edit a segmentation mask, you can use the add/remove tools in the top navigation bar to add to and remove from a segmentation mask. Segmentation masks cannot be repositioned.

#### Pen tool

To create a segmentation mask, click and drag to free-hand draw a shape. You can also click and release to place vertices with straight edges in between.

#### Brush tool

With the brush tool, you can use brush strokes to quickly draw a mask. You can change the brush size using the slider at the top. You can select a circle or square shape for the brush tool.

<Frame>
  <iframe name="wistia_embed" />
</Frame>

<Warning>
  ### Limited view during playback

  Please note that segmentation masks currently will not display during video playback due to performance constraints. Please use the arrow keys or navigate to a frame directly to review a mask.
</Warning>

### Frame classification

To create a frame classification, activate the classification question, input the answer value, and select from supported types:

* Radio
* Checklist
* Free-form text

<Info>
  ### Free-form text

  Free-form text classification at the frame level is currently only available in the UI and is not yet available for SDK Annotation importing.
</Info>

### Global classification

Global classifications apply to the entire video rather than to individual frames, so they do not appear in the timeline. To create a global classification activate the classification question, input the answer value, and select from supported types:

* Radio
* Checklist
* Free-form text

## Video editor components

| Component | Description |
| - | - |
| *Tools menu* | Contains the list of annotation classes (AKA ontology) for your project. |
| *Annotations menu* | Lists all of the annotations that have been created so far. |
| *Progress bar* | A visual indicator of the temporal progression of the frame sequence. You can hover over the progress bar to quickly preview any frame in the video. |
| *Classification panel* | Appears only for frame classification-type annotations (e.g., radio). Here you can apply select answer(s) on a classification question and toggle the classification on/off. |
| *Timeline* | Displays a temporal visualization of all created annotations on the video clip. Indicates the frames on which the annotation was created. You can hover over the timeline to preview any frame in the video. |
| *Keyframe* | A keyframe is a frame in the sequence where an annotation starts, ends, or is modified in some way. A keyframe is indicated on an annotation track as a small circle. The frames between two keyframes are called interpolated (tweened) frames. This is because Labelbox uses interpolation to incrementally adjust the size and position of an annotation on frames in between keyframes. Note: interpolation is not applied to classifications. |
| *Volume* | Sometimes, to make a certain classification in a video, you need to be able to hear and see the content in the frame sequence. When a video has audio, your labeling team will be able to hear the audio during video playback (but not when hovering or stepping through with arrow keys). Labelers can adjust the volume level and click the volume button to mute/unmute audio playback. You can use the video editor to classify videos with both audio and visual content or audio content alone. To use the video editor to label audio only, please reach out to support for details. |

## Navigating the video editor

Use the progress bar to navigate the video in its entirety, or use the timeline to navigate the current window of the video. The width of the current window can be adjusted from the player's settings to the right of the timestamp.

You can scrub a preview of any part of the video by hovering over a certain part of the progress bar or timeline. Clicking within the progress bar or timeline will bring the playhead to the clicked position.

Press play/pause or use `Shift` + `Space` to control the playback of the video. The left and right arrow keys can be used to move backward one frame and forward one frame, respectively.

To navigate to a specific frame, click on the current frame number, replace it with the frame number you want to navigate to, and press `Enter`.

Hover over the top edge of the timeline or annotations menu, and click and drag to change the height of the timeline.

## Audio transcription

You can now transcribe audio directly within the video and audio editors for both global and temporal text classifications. This feature allows you to record audio and have it automatically converted into text, streamlining your labeling workflow.

### Enable the audio transcription feature

Before you can use the transcription feature, you must enable it in your project's ontology for the desired text classifications.

1. Navigate to your ontology settings.
2. Select the text classification you want to enable transcription for (this can be a global or a temporal/per-frame classification).
3. Locate the toggle switch labeled **Record audio for transcription**.
4. Enable the toggle for each text classification where you want to use this feature.
5. Save your changes.

### Use audio transcription in the video editor

Once enabled, you can use the feature for both global and frame-based classifications.

**Global classifications**

1. Select the global classification text field.
2. Click the **Record Audio** button that appears below the text area.
3. Speak to record your classification. The transcribed text will automatically populate the field.

**Temporal (per-frame) classifications**

1. Select a frame or a range of frames on the video timeline.
2. Select the temporal classification text field.
3. Click the **Record Audio** button.
4. Speak to record your classification. The transcription will be applied to the selected frame range.

## Video-specific hotkeys

| Function | Hotkey | Description |
| - | - | - |
| Play/Pause | `Shift` + `Space` | Play or pause the video playback. |
| Move backward one frame | `Left` | Move back one frame in the sequence. |
| Move forward one frame | `Right` | Move forward one frame in the sequence. |
| Add/remove a keyframe | `K` | Add/remove a keyframe for the selected annotation on the selected frame. |
| Add/remove an object | `T` | Toggle the presence of an object on or off in the video timeline. |
| Select multiple contiguous frames | `Shift` + `Select` | Select multiple, contiguous frames in the video. |
| Select multiple discontiguous frames | `Shift` + `Cmd` + `Select` | Select multiple, discontiguous frames in the video. |
| Advance to the previous keyframe | `Opt` + `Left` | Jump to the previous keyframe on an annotation instance track. |
| Advance to the next keyframe | `Opt` + `Right` | Jump to the next keyframe on an annotation instance track. |
| Undo | `Cmd`/`Ctrl` + `Z` | Undo an action. |
| Redo | `Cmd` + `Shift` + `Z` (Mac) / `Ctrl` +`Shift` + `Z` (Windows) | Redo an action. |


# View account usage
Source: https://docs.labelbox.com/horizon/guides/view-billing-details

Shows how to sign in to the Labelbox app to view your account, verify your current plan, and to manage your billing details.

<Note>
  **Usage tab is visible to organization administrators only**

  If you don't see the Usage tab in your Workspace settings, verify with your account administrator that you have the appropriate permissions.
</Note>

## Org usage overview

To view account details and verify your current plan:

1. Go to **Workplace settings → Usage**.
2. Your current plan is displayed in the **Org usage overview** section.

If your current plan name includes "(legacy)", your current plan has been retired. You'll need to select a new plan to renew or otherwise change your plan benefits. Note that legacy plan upgrades can't be rolled back; you cannot downgrade to a retired plan.

## LBU usage

The LBU usage section displays the following information:

* **Line chart**: shows LBU usage over time. Hover over the last day of a given month to see monthly totals.
* **Pie chart**: shows a breakdown of total LBU usage, by Annotate, Model, and Catalog.
* **Date filters**: let you toggle between a monthly and multi-month view.

LBU consumption is updated each day at midnight (00:00) UTC. Activity performed today will appear after the next update. The Usage tab reflects consumption and activity as of the last update.

## Data row usage

The **Data row usage** section shows the LBU usage per product and data modality.

Clicking through each line item will give you a clear breakdown of your LBU consumption per data modality.

## Labeling service usage

The **Labeling service usage** section displays the usage and project billing rates for projects using [labeling services](/horizon/guides/labeling-services).


# Webhooks
Source: https://docs.labelbox.com/horizon/guides/webhooks

A guide for managing webhooks in the UI.

A webhook is simply a way for Labelbox to automatically send information to your server when a specific event occurs. Think of it as a real-time notification system. Instead of your application constantly asking Labelbox if there are any updates (a process called "polling"), Labelbox actively pushes the updates to your application the moment they happen. This saves time, bandwidth, and resources.

## Event delivery and volume

When building your webhook integration, it is important to understand how Labelbox delivers events to your application:

* **Delivery order:** Labelbox does not guarantee the sequential delivery of webhook events (e.g., the exact chronological order in which a label was created, updated, or deleted). Your system should be designed to handle events that may arrive out of order.
* **Outgoing rate:** The rate of incoming webhooks depends entirely on which topics you subscribe to and your usage volume (for instance, how rapidly your team is creating labels or moving data rows).

## Security and payload formatting

To keep your data secure, all Labelbox webhook events are sent exclusively via **HTTPS**. Additionally, Labelbox attaches a unique signature to every event. By configuring your endpoint to verify this signature, you can confidently confirm that the incoming message genuinely originated from Labelbox.

When parsing the data sent by Labelbox, keep the following in mind:

* **Format:** The `Content-Type` header for all webhook events is always strictly `application/json`.
* **Payload size:** There is no maximum upper limit for the webhook payload. The payload size grows based on the number of annotations on a label, and Labelbox does not restrict the number of annotations you can have per data row.

## Labelbox server addresses

All webhook traffic from Labelbox originates from specific IP addresses. To ensure you receive these events without interruption, please review the [Labelbox server addresses](https://docs.labelbox.com/horizon/guides/server-addresses#labelbox-server-ip-addresses "/horizon/guides/server-addresses") and add them to your firewall's allowlist or relevant security policies.

## Webhook events

Labelbox supports a variety of webhook events, which can be configured for a specific project or across your entire organization.

| Entity | Event Types | Description | Method & Payload |
| - | - | - | - |
| Label | LABEL\_CREATED, LABEL\_UPDATED, LABEL\_DELETED | Triggers when labels have been created, updated, or deleted. | HTTP POST |
| Model run training | Created (HTTP POST) | Triggers when training of a model run has been initiated Note this webhook does not yet conform to standard webhook subscriptions. It must be registered in the Model User Interface. | HTTP POST |
| Models list | Queries (HTTP GET) | Callback to retrieve a list of model types for a given model training backend | HTTP GET |
| Workflow | MOVE | Triggers when a workflow action happens, e.g. after a data row was approved or rejected during a review step, or after it was manually moved to a different queue. | HTTP POST |

**Understanding label event payloads**<br />When you receive a label-related event (such as `LABEL_CREATED`), the first `id` in the JSON payload always corresponds to that specific label's ID.

```json theme={"theme":"css-variables"}
Event: LABEL_CREATED
Payload: {
    "id": "cl0sjr9nb9tfi0zatagsq5zw1",
}
```

## How to create webhooks

You can easily set up and manage your webhooks without writing any code. Simply navigate to the [Webhooks tab](https://www.google.com/url?q=https%3A%2F%2Fapp.labelbox.com%2Fworkspace-settings%2Fwebhooks "https://app.labelbox.com/workspace-settings/webhooks") located in your **Workspace settings**.

You can choose to set up a webhook at the **workspace-level** or **project-level:**

<Warning>
  When creating a webhook, you must decide upfront whether it will be **project-specific** or **organization-wide (global)**. Once a webhook is created, its scope is permanent and cannot be modified. If you need a different scope, you will need to delete the existing webhook and create a new one.
</Warning>

**Global webhook**

* Applies to **all projects** in your organization
* Triggers on events across the entire workspace
* Useful when you want a single endpoint to receive notifications from everything

**Project-specific webhook**

* Scoped to a **single project** only
* Only triggers on events within that project
* Useful when different projects need to send data to different endpoints, or when you only care about events from one particular project

## Developer guide

To learn how to manage your webhooks programmatically, visit this developer guide

<Card title="Webhooks developer guide" icon="sparkles" href="/horizon/developer-guides/webhook" />


# Project overview
Source: https://docs.labelbox.com/horizon/guides/what-is-a-project

Understand what a project is and how to navigate the project overview dashboard.

A project in Labelbox is the central workspace where you manage your data, configure your labeling interface, and collaborate with your team to create high-quality training data. Think of it as the primary hub for any specific dataset you intend to label.

From the moment you create a project, it becomes the single source of truth for that labeling initiative. Within a project, you will:

* **Connect your data:** Link the images, text, or other documents you need to label.
* **Define your ontology:** Create the specific classes and categories that your team will use for annotation.
* **Assign your team:** Invite and manage the labelers and reviewers who will work on the data.
* **Monitor progress:** Track the status of labeling, review, and rework tasks.
* **Measure quality:** Use tools like Consensus and Benchmark to ensure your labels are accurate and consistent.

When you open a project, you'll land on the main **Project Dashboard**. This provides a high-level overview of your project's health and progress.

## Project overview

The **Overview** tab is your command center for a specific labeling project. It provides a comprehensive, at-a-glance summary of your project's health, progress, and quality. Think of it as the first place you should look to understand what's happening, identify potential issues, and ensure your project is on track to meet its goals.

### Project metrics

The **Participation** view is an embedded version of the Participation histogram from the Performance tab. Click on **View details** to open the chart in the Performance tab.

The **Pipeline** view displays of all statuses in a project and a count of data rows in each status. It also provides a count of the issues.

Status descriptions:

| Status | Description |
| - | - |
| **Initial labeling task** | This is the very first step in the workflow where a data asset is assigned to a labeler to be annotated for the first time. The labeler's task is to create the initial set of labels according to the project guidelines. |
| **Rework (all rejected)** | This step occurs after a label has been rejected during a review. The data asset is sent back to a labeler—often the original one—to correct the specific errors that the reviewer identified. Once the corrections are made, the asset is submitted again for another review. |
| **Initial review task** | This is a quality assurance step that follows the initial labeling task. A different team member (a reviewer) inspects the labels to ensure they are accurate and meet the project's quality standards. The reviewer can either approve the asset, or reject it and send it to the rework step. |
| **Skipped** | This status is applied when a labeler chooses to skip a data asset in their queue without annotating it. This action returns the asset to the general labeling queue so it can be assigned to another labeler. A labeler might skip an asset if the data is unclear, corrupted, or if they are unsure how to apply the correct labels. |
| **Done** | This is the final state for a data asset in the workflow. It signifies that the asset has successfully passed all required labeling and review steps and is considered complete and ready to be used for model training. |
| **Issues** | This is not a linear step but a collaborative feature used to flag problems. An "issue" can be created on a data asset at any stage to highlight a problem, ask a question, or start a discussion. This action pauses the asset in the workflow until the issue is formally resolved by the team. |

### Workflow tasks

From this section you can enter a specific step in the labeling workflow or quickly add an entirely new step.

### Analytics view

This section lists all the features in a project along with the count of annotations for each.

For each feature, you can see the following:

* **Name** of the feature.
* **Count** of data rows that contain the feature.
* **% Share** indicating the percentage of data rows that contain the feature.
* A horizontal bar chart that depicts the percent share. You can also click the bar to navigate to a filtered view of the data rows tab including only the data rows that contain the relevant feature.


# Workflows
Source: https://docs.labelbox.com/horizon/guides/workflows

Create customizable, multi-step review and rework pipelines using the node-based editor

A workflow in Labelbox is a customizable, multi-step pipeline for reviewing and reworking your labeled data. It allows you to create a structured quality assurance process to ensure the accuracy and consistency of your annotations.

Think of a workflow as a conveyor belt. A data row moves from one task to the next in a sequential order. When a data row completes a task, its status is updated, and it moves to the next appropriate task in the workflow.

Workflows are built using an interactive, node-based editor where you can define the path that your data rows will follow.

## Building your workflow

You can customize the default workflow by adding your own tasks and logic to create a process that fits your specific needs.

By default, every project comes with a simple, four-step workflow:

| Task | Description |
| - | - |
| Initial labeling task | This is the starting point for all data rows that need to be labeled. You can customize the "Initial review task" by renaming it and adding specific filtering criteria to it. |
| Initial review task | After a data row is labeled, it moves to this first review step. |
| Rework | If a label is rejected during any review step, the data row is sent here to be corrected. |
| Done | Once a data row has successfully passed all the required steps in the workflow, it is marked as "Done". |

Data rows automatically move to the next task only when the previous step is completed. As a data row moves, its status updates automatically and cannot be changed manually.

For example, a data row with the To Label status must first be labeled before it can proceed to a review or rework task.

### Create a new task

You can add custom review and rework tasks to your workflow. To do so:

1. Go to the **Workflow** tab and click **Edit workflow**.
2. Click the **+** button in the workflow editor.
3. Select **Step → Review task** or **Custom rework task**.
4. In the task configuration settings (Review task and Custom rework task only), you will need to provide:
   * **Task name:** A descriptive name for your task.
   * **Instructions:** Clear instructions for the users who will be working on this task.
   * **Group assignment:** The user group that will be responsible for this task.
   * **Maximum number of labels per task queue:** This setting puts a **per-contributor cap** on how many labels a single user can create in that specific task queue. Once an individual contributor hits that cap, they stop receiving more tasks from that queue even if tasks remain, unless the field is left empty for **no limit**.
   * **Per-user/group task limits:** This setting lets you define *who* the task caps apply to, at a more granular level:
     * For ***specific users***: you can set a **maximum number of tasks** that each listed user can complete in that queue. Once a user hits their per-user limit, they stop receiving additional tasks from that queue, even if more tasks are available.
     * For ***specific groups***: you can set a **cumulative task limit per group** for that queue (e.g., a reviewer group can do at most N tasks total across that queue). Once the group collectively reaches that limit, no one in that group will receive more tasks from that queue, even if other groups or users can.
   * **Individual assignment** (custom rework tasks only): For custom rework tasks, you can configure this setting to ensure that any data rows sent to rework get re-assigned to the original **label creator**.

<Info>
  To view the maximum number of tasks per workflow, see [limits](/horizon/guides/limits).
</Info>

### Add filtering logic to tasks

You can use filtering logic to control which data rows are sent to which tasks. This allows you to create specialized review steps for different types of data. To add logic:

1. Go to the **Workflow** tab and click **Edit workflow**.
2. Click the **+** button in the workflow editor and select **Step → Logic**.
3. Click **+ Add** to choose one or more filters.
4. Select **All** or **Any** to specify whether all or any of the filter conditions must be met.

The following filters are available:

| Filter | Description |
| - | - |
| Labeled by | The user who labeled the data row. |
| Annotation | The label added to the data row. |
| Labeled at | The date range for when the label was added. |
| Sample probability | The likelihood of a data row being included in the task. For example, a sample probability of 40% means each data row that meets the other filters has a 40% chance of being included in the task. |
| Consensus average | A range of average label-level consensus scores. |
| Feature consensus | A range of average feature-level consensus scores. You can select more than one feature schema in the ontology. |
| Dataset | The dataset where the data row comes from. |
| Issue category | The issue category of issues on the data row. |
| Batch | The batch where the data row is sent. |
| Metadata | The non-annotation metadata information on the data row. |
| Model prediction | The Foundry model predicted label of the data row. |
| Labeling time | The labeling time spent on the data row. |
| Review time | The review time spent on the data row. |
| Natural language | A natural language expression and confidence score range matching data rows with similar vector embeddings. |
| Label feedback | The label feedback score range. |

<Info>
  **Sample probability** is processed after all other filters. It's a probability, not a guarantee, so it's possible that none or only a few data rows are included for the task, even if many data rows meet the other filters.
</Info>

### Connect logic and tasks

Once you have created your tasks and logic, you can connect them in the workflow editor by dragging and dropping the arrows between the nodes. This allows you to visually map out the flow of your review process.

For complex workflows, you can add multiple **Rework** and **Done** terminal nodes to the editor to make your workflow easier to read. This is a visual aid and does not create additional tasks.

<Frame>
  <iframe title="Connect logics to tasks" />
</Frame>

## Managing your workflow

Workflows are designed to be flexible, allowing you to optimize your quality assurance process as your project evolves.

To modify a task or logic node, simply click on it in the editor to open its settings. You can delete a node by clicking the trash can icon.

To change the connections between nodes, select the arrow connecting them and either drag it to a different node or click the trash can icon to delete it.

To see a breakdown of how many data rows are in each workflow status, click the **info** icon next to the editor.

## Advanced workflow actions

Here are some more advanced actions you can perform within your workflow to manage your data quality.

### Add terminal nodes

**Terminal** nodes represent the default **Rework** and **Done** tasks.

You can create *multiple* terminal nodes on the editor instead of connecting everything to a single **Rework** or **Done** node, making your workflow easier to read and visually cleaner. Adding multiple terminals does not create additional tasks and is purely for improving the visual layout.

<Frame>
  <iframe title="Add multiple terminals" />
</Frame>

### Reject and rework data rows

When a label is rejected, the data row is automatically sent to the "Rework" task.

* **To send a single data row to rework:**
  1. Select the task in the **Workflow** tab.
  2. Click **Start Reviewing**.
  3. Click **Reject** on the data rows you want to send to rework.
* **To send multiple data rows to rework in bulk:**
  1. Go to the **Data rows** tab.
  2. Use the filters to find the data rows you want to send to rework.
  3. Select the data rows (up to 10,000 at a time).
  4. From the dropdown menu, select **Move to task** and choose **Rework**.

### Use benchmarks in your workflow

At any stage of the project, if a label on a data row is set as a benchmark, the data row (and thus any current and future labels made on the data row) will move to the **Done** task until the benchmark status is removed from the label.

If you are trying to review a data row with a benchmark label, remove the benchmark from the label and then proceed to move the data row to the appropriate task.

### View audit logs

Labelbox automatically creates an audit log for every data row that enters a workflow. This log provides a complete history of all the events that have occurred on a data row, which is useful for investigating its review history.

To view the audit log:

1. From the **Workflow** tab, click on a task to view the data rows within it.
2. Click on a data row to open it.
3. Click on the **clock** icon to view the audit log.


# Features
Source: https://docs.labelbox.com/horizon/guides/working-with-features

Instructions for creating and modifying features.

A *feature* within the context of Labelbox is an individual entry in an ontology. It contains all of the information required for rendering a specific annotation.

In other words, a feature is the master definition of:

1. How you want your model to do a given task (i.e., create predictions).

2. How your labeling team should label an asset (i.e., create ground truth).

You can think of a collection of features as a blueprint for providing structure to your data in the form of annotations. In the editor, you will see the list of features (AKA the ontology) in the **Tools** panel.

There are two kinds of features: *objects* (e.g., bounding box, polygon, etc) and *classifications* (e.g., radio, checklist, etc). A feature can have multiple deeply nested sub-classifications.

An annotation is simply a single instance of a feature.

## Key definitions

| Component | Definition |
| - | - |
| Object | Objects are features that generally are visible on the stage (canvas). This includes bounding boxes, segmentation masks, named entities, and relationships. |
| Classification | Classification is essentially a tag that can exist in any context (global or inside an object). Labelers assign classifications by completing a form in the Editor. |
| Nested classification | Classifications can be global (i.e., it applies to the entire asset) or they can be nested within an object-type annotation (i.e., a child of an object-type annotation). |
| Relationship | A relationship is a specialized annotation type that connects two annotations and allows you to name that connection. |

## Objects versus classifications

Labelbox separates its annotations into two general categories: objects and classifications. An easy way to remember the difference between the two is that object-type annotations are visible on the editor stage and classifications are not.

Structurally, objects cannot be nested within other objects. However, classifications can be nested within objects. Classifications can also be at the global level (can apply to the entire asset).

In this sample ontology, there is one object feature (named "Tree") and 3 classification features.

The classification question, "Is the image blurry?", is at the global level. This means this classification applies to the entire data row.

The classification questions, "Does the tree have leaves?" and "Is the tree alive?", are sub-classifications that apply only within the scope of the "Tree" object.

## Create features from the Schema tab

To create a new feature in the UI, navigate to **Schema** > **Features** and click the **Create** button on the right.

Features will fall into one of three categories: objects, classifications or relationships.

### Create objects

To create an object feature, you need to provide the following information:

* **Tool name**: Name of the feature that will appear in the **Tools** menu in the editor. If you want to add an existing feature to your ontology, you can look it up by name.

* **Tool color**: Color of the feature in the editor. Helps labelers identify the objects on the editor stage and in the **Tools** menu.

* **Tool type**: Indicates the annotation type (ex. bounding box, segmentation mask, etc)

* **Sub-classifications**: If you need a sub-classification (AKA nested classification) on the object, you can specify that here. This is an optional setting you can use to provide more information on the object being labeled.

### Create classifications

To create a classification feature, you need to provide the following information:

* **Classification name**: Name of the classification that will appear in the **Tools** menu in the Editor.

* **Input type**: Indicates the answer choice type (ex. radio, free text, etc)

* **Required**: Should this classification be answered in order to submit a label? If yes, then enable the Required option.

* **Searchable**: If you have a large list of answer options, the Searchable flag allows you to search the options during labeling.

## Create features from a project

You can also dynamically create new features from within a project. To do this, you must:

1. Select a project.

2. Go to **Settings** > **Label editor** and click **Edit**.

3. In the edit mode, click the **+** button under the object or classification section

4. Fill out the necessary information to create the feature.

5. Hit the checkmark or **Done** at the bottom of the screen.

<Info>
  ### Parent feature ID

  While working with deeply nested features, you may see feature ID and schema node ID for every child entity. They will soon be removed entirely. For all useful workflows, you now only need to care about the feature ID that you can see and copy from the UI.
</Info>

## Edit a feature

You can make changes to a feature definition anytime.

* If the feature has one or more annotations, you can only change everything except the tool type.

* If the feature has no annotations, you can make changes to every setting.

In general, proceed with caution when making changes to a feature that is already in use in existing labeling projects.


# Understanding your workspace
Source: https://docs.labelbox.com/horizon/guides/workspaces

A guide for navigating and managing your workspace in Labelbox.

This section introduces you to the concept of workspaces, the foundation for organizing projects, data, and users in Labelbox.

## What is a workspace?

Workspaces allow large organizations to manage multiple Labelbox instances with a single account. Think of each workspace as a separate, self-contained instance of Labelbox with its own projects, datasets, and members.

This is useful for separating work between different business units or maintaining distinct development and production environments. Annotations from all workspaces are aggregated for billing purposes.

A **Tenant Admin** is a special role that has administrative access across all workspaces within an organization. They can view, add, and manage users across multiple workspaces. The first user of a new enterprise account is automatically a Tenant Admin. To grant this role to additional users, please contact our support team.

## Switch between workspaces

If your organization has multiple workspaces, you can easily switch between them.

**To switch workspaces:**

1. Click your initials in the bottom-left corner of the navigation panel.
2. Select **Switch workspace** from the menu.
3. Click on the name of the workspace you wish to access.

## Request additional workspaces

The ability to have multiple workspaces is available to organizations on the Enterprise plan.

To request an additional workspace for your organization, please contact sales to upgrade or reach out to our support team if you are already on the Enterprise plan.


# Release Notes
Source: https://docs.labelbox.com/horizon/release-notes



<Update label="Subscribe to the Labelbox release notes">
  To subscribe to the Labelbox release notes, please fill out the [subscription form](https://learn.labelbox.com/releasenotessubscription/).
</Update>

<Update label="September 3, 2026">
  ## App

  *Added*

  * **SSO for multi-business-unit companies:** We added support for per-business-unit SSO routing, so companies with multiple IDPs can each configure their own under one email domain (rather than the current one-IDP-per-domain assumption). Available for Enterprise customers only. Contact Labelbox support for access.
  * **SAML/OIDC group-claim role mapping:** The platform now supports mapping identity-provider group claims to Labelbox roles automatically at sign-in, so upstream access-review processes could drive role assignment. Available for Enterprise customers only. Contact Labelbox support for access.
  * **Two-stage video labeling mode:** Video projects can now enable a two-stage labeling workflow (Project Settings → Advanced) that splits boundary-marking from annotation. Stage 1 lets labelers quickly mark when each event starts and ends without describing it yet, with optional dense-annotation auto-placement and support for multiple passes (e.g., labeling a left arm and right arm separately). Stage 2 then loads each marked segment for annotation, with modifier-key/mouse seeking and an optional ontology view, so labelers can move through segments quickly with keyboard-driven typing.
  * **Gemini 3.7 Flash** is now available in the Model tab.
  * **"Created by" on project details:** Users with permissions to edit project details can now see who created a project directly in the UI.

  *Changed*

  * **Data row tab layout persistence:** Column widths, order, and hidden/visible columns on the data rows tab are now preserved per project.
  * **Fixed consensus scoring for conversational tasks with reused schema IDs:** Consensus calculation previously compared only the overall set of labels used across a conversation, so it could report 100% agreement even when labelers disagreed on which specific messages received which label. It now aligns answers per message before scoring agreement.
  * **Fixed "Maximum number of labels per task queue" accepting 0:** In the Workflows tab, setting the maximum number of labels for a task queue to 0 is no longer allowed, the field now enforces a minimum of 1 with inline validation. Previously, a value of 0 was silently treated as "no limit," which could be confusing.

  *Removed*

  * Google Gemini 3 Pro Preview is no longer available in the Model tab.
  * Google Imagen 4 is no longer available in the Model tab.
  * GPT-4o mini TTS is no longer available in the Model tab.

  ## Python SDK

  To view the latest updates to our Python SDK, see the [<u>What's new</u>](/horizon/developer-guides/whats-new) section in the developer guides.
</Update>

<Update label="August 5, 2026">
  ## App

  *Added*

  * **Project-level quizzes with multiple question types:** Quizzes now live at the project level (previously one shared quiz per ontology), managed from a new Quizzes tab. Beyond free-text, you can add auto-graded single-select, multi-select, and true/false questions. Quizzes also support draft/publish states, attempt limits, pass thresholds, required/optional gating, and contributor preview.
  * **New models are now supported in Foundry:** Gemini 3.5 Flash, Gemini 3.5 Flash-Lite, and Gemini 3.6 Flash.

  *Changed*

  * **Fixed the priority filter in the Data Rows tab:** The priority filter in the Data Rows tab was not working correctly and has been fixed and rolled out to all Enterprise-tier organizations.
  * **Consensus for text entity annotations in the document editor:** The document labeling editor now supports consensus agreement scoring for text entity annotations, matching consensus support already available for other annotation types.
  * **Bulk class changes for object annotations:** You can now change the class of a feature object in bulk instead of updating annotations one at a time. To do this, go to the editor and select the annotations that you want to update. If the object annotations are the same type, you may select “Change class” and update their class. If the objects are the same type or different types, you may bulk delete them by selecting them in the editor and clicking “Delete”.
  * **Issue and comment exports now show emails instead of internal IDs:** Exporting issues or comments (via the SDK or the UI) now returns the person's email address for the “created by,” “resolved by,” and “labeler” fields instead of internal user IDs.

  *Removed*

  * **The Model experiments feature is no longer available** as of July 15, 2026. You can no longer create new experiments or import model runs into experiments.

  ## Python SDK

  To view the latest updates to our Python SDK, see the [<u>What's new</u>](/horizon/developer-guides/whats-new) section in the developer guides.
</Update>

<Update label="July 7, 2026">
  ## App

  *Added*

  * **Markdown preview support for text classification fields in the image editor**: Enable "Markdown preview for text classes" in advanced project settings to reveal an "Edit with markdown" button beneath the text box. Click it, enter your markdown, and save. Afterward, the field will display the rendered markdown. Subclassifications are also supported.
  * **Claude Sonnet 5** is now available in the Model tab.

  *Changed*

  * **The Project list in Annotate** got the following UI improvements:
    * A new card-based layout displaying key information for each project.
    * Each project card displays progress broken down by workflow stage (In Generation, In Review, In Rework, and Done) shown as percentage bars so you can see at a glance where work stands across the pipeline.
    * Updated filters include workflow stage filters, plus improved behavior for tagging and sorting.
    * The UI adapts based on user role to show only relevant project information based on who is viewing.
  * **Instructions quiz generation now enforces daily usage limits** of 100 requests per user and 1,000 requests per organization, and input instructions are capped at 8,000 characters.

  *Removed*

  * The **Prompt and response generation project type** is no longer available in Annotate.
  * **Code runner** is no longer available in the Multi-model chat editor.
  * We have removed the capability for **individual data row inferences in Foundry apps**. This means you can no longer use the API to create an inference on a single data row or raw data via Foundry apps. Foundry model jobs for model Apps are not impacted and continue to work as expected.

  *Future*

  * Effective July 15th, the **Model experiments** feature will no longer be available. After July 15th, you will no longer be able to create new experiments or model runs. Existing experiment data, including model runs, predictions, and metrics, will be accessible for export until July 15th.
    ## Python SDK
    To view the latest updates to our Python SDK, see the [<u>What's new</u>](/horizon/developer-guides/whats-new) section in the developer guides.
</Update>

<Update label="June 4, 2026">
  ## App

  *Added*

  * **Starter tier re-enabled**: The Starter tier is back and available for new and existing users. You can select or switch to it from your organization's billing settings.
  * The following model has been added to the **Model** tab:
    * Opus 4.8

  *Changed*

  * **Refreshed color & styling**: We made some updates to our app UI. Here's what you may notice
    * Warmer light mode: The light theme now features warmer tones that are easier on your eyes.
    * Refreshed styling: You will notice updated buttons, cleaner spacing, and polished visual details.
    * Same functionality: All features, tools, and workflows remain exactly where they were before.
  * **Updated max users per group to 100K**: Organizations can now create and manage up to 100,000 user groups, making it easier to scale access control across large teams. This limit has been automatically updated for all orgs.

  *Fixed*

  * **Admins can now edit workflows without clearing active reservations**: Previously, editing a workflow would disrupt labelers actively working on tasks by clearing their reservations. This fix ensures that in-progress work is preserved when admins make workflow changes (in the initial labeling queue only).

  ## Python SDK

  To view the latest updates to our Python SDK, see the [<u>What's new</u>](/horizon/developer-guides/whats-new) section in the developer guides.
</Update>

<Update label="May 4, 2026">
  ## App

  *Added*

  * **AI critics in audio/video editors:** AI critics have been fully integrated into the audio and video editors and works across all classification types (text, radio, checklist; global, temporal, nested). It can be used for tasks like spotting grammar issues in transcriptions, validating the quality and completeness of text summaries, checking responses for contradictions or inconsistent selections, and more. See our docs on [AI critic](/horizon/guides/ai-critic) for more details.
  * **Search functionality in the document editor:** You can now use Ctrl+F (or Cmd+F on Mac) to search within the document editor. This keyboard shortcut opens an inline search bar, allowing you to quickly locate and highlight text within a document.
  * **“Multi-label” without consensus scoring**: Allows you to set up a project with *N* labels per data row with no scoring. To enable scoreless multi-labeling for a project, go to project advanced settings.
  * **Read only predictions**: Allows you to import labels that have read-only annotations for normal workflows. To import read-only predictions, you add `readonly=True` to your prediction/MAL annotation objects. This setting allows labelers to review the pre-labels but they cannot modify them.
  * **Foundry updates**:
    * The following models are now available in Foundry
      * OpenAI GPT-5.5
      * OpenAI GPT-5.5 Pro
      * Claude Opus 4.7
    * The following models in Foundry are no longer supported:
      * OpenAI GPT-5.1
      * OpenAI GPT-5.2
      * OpenAI GPT-5.2 Pro

  ## Python SDK

  To view the latest updates to our Python SDK, see the [<u>What's new</u>](/horizon/developer-guides/whats-new) section in the developer guides.
</Update>

<Update label="April 6, 2026">
  ## App

  *Added*

  * **Custom roles compatible filters:** Previously, there were no guardrails to prevent admins from adding incompatible permissions when creating custom roles. Now, when you create a new custom role, you can select Project, Workspace, or All to quickly filter for the compatible permissions at each of these scope levels
  * **Audio transcription in audio & video editors:** You can now transcribe audio directly within the video and audio editors for both global and temporal text classifications. This feature allows you to record audio and have it automatically converted into text, streamlining your labeling workflow. To learn more see our [<u>Audio transcription docs</u>](/horizon/guides/video-editor#audio-transcription).

  *Removed*

  * The Public Demo workspace has been sunsetted and is no longer available for use.

  ## Python SDK

  To view the latest updates to our Python SDK, see the [<u>What's new</u>](/horizon/developer-guides/whats-new) section in the developer guides.
</Update>

<Update label="March 4, 2026">
  ## App

  *Added*

  * **AI critic in the Multi-modal chat editor**: You can now use natural language to automatically critique any element in the MMC editor before label submission. This makes it easier to catch issues in real time and standardize QA across projects while reducing manual review overhead.
  * **AI generated rubrics in the Multi-modal chat editor**: You can now automatically generate rubrics from your task instructions and examples inside the Multi-modal editor, enabling you to spin up consistent, detailed rubrics for new projects and iterate on grading criteria without starting from scratch.
  * **Improved auto-segmentation in the Video editor (Meta’s SAM3)**: The Video editor now uses Meta’s SAM3 (Segment Anything Model) for auto-segmentation, producing higher-quality masks and faster object selection on video frames.
  * **Assign data rows to individuals in Consensus mode**: You can now assign specific data rows to individual contributors even when using Consensus projects. Enables more control over who labels what (e.g., for training, audits, or specialization) without giving up the quality benefits of consensus.
  * **New “Media Duration” Throughput chart**: In the Throughput section in the Performance Dashboard, there is a new chart called “Media Duration” that shows the total time of media duration recorded over a time series across all labelers.

  *Changed*

  * **Default roles when creating groups**: When you create a Group, you’ll now be prompted to select a default role for members. This change is designed to reduce the changes of misconfiguring access for group members and makes it easier to configure baseline permissions consistently.
  * **Project overview tiles**: In the Project overview under project metrics, the number of hours is now recorded in the “Done” tile.

  *Removed*

  * The following models in Foundry are no longer supported: Claude Sonnet 3.7, Claude Haiku 3.5
  * The "Annotation usage" chart under Workspace settings > Usage has been removed. This legacy chart was originally built to support older annotation-based billing and is no longer used for billing calculations.

  ## Python SDK

  To view the latest updates to our Python SDK, see the [<u>What's new</u>](/horizon/developer-guides/whats-new) section in the developer guides.
</Update>

<Update label="February 6, 2026">
  ## App

  *Added*

  * In the Monitor tab and project’s Performance > TEQ > Throughput charts, the “Done”, “Labels”, “Annotations”, and “Reviews” tiles now have a chip in the top right corner that displays Average daily throughput when you hover over them.
  * In the project’s Performance > TEQ > Throughput section, you can toggle ON the “Forecast” setting to display the forecast for the next time period. See the [<u>Performance Dashboard docs</u>](/horizon/guides/performance-dashboard) for more details.
  * For Video projects, the following controls now exist in Project settings > Advanced:
    * “Video Timeline V2”: When this is toggled ON, the video editor timeline will be updated to the new UI.
    * “Timeline minimum visible duration”: Use this setting to define the max zoom level control for your timeline (i.e., how many seconds from left border to right border). For example, if this setting is set to 1, the entire span of the timeline view will be set to 1 second.
    * “Disable automatic split at playhead for classifications/objects”: When this is toggled ON, the segment under the playhead in the timeline view will be selected and any change will be applied to the entire segment. To make a partial change to a segment at a specific point in the timeline, click “Split” to create a new segment and then apply the change.
  * For Audio projects, the following controls now exist in Project settings > Advanced:
    * “Audio Timeline V2”: When this is toggled ON, the video editor timeline will be updated to the new UI.
    * “Timeline minimum visible duration”: Use this setting to define the max zoom level control for your timeline (i.e., how many seconds from left border to right border). For example, if this setting is set to 1, the entire span of the timeline view will be set to 1 second.
    * “Disable automatic split at playhead for classifications”: When this is toggled ON, the segment under the playhead in the timeline view will be selected and any change will be applied to the entire segment. To make a partial change to a segment at a specific point in the timeline, click “Split” to create a new segment and then apply the change.
  * The Audio & Video editors received the following improvements:
    * Segments shorter than 300ms now show a warning icon to help users identify and optionally remove invalid micro-segments, reducing annotation noise.
    * You can now jump to a specific time in the audio timeline by typing the desired timestamp.
    * Start and end handles now appear on the timeline segments, making it easier to resize segments by dragging.
    * You can now use the option+arrow shortcut to navigate keyframes.
  * With the new AI critic in the Multi-modal chat editor, you can use natural language to target any element in the editor and auto-generate a critique for each element. You can enable this by going to project settings > Advanced > toggle “Enable AI Critic” ON. In order to use this feature, Foundry billing must be enabled for your organization.

  *Changed*

  * In the Audio editor, you can now scroll beyond the end of the audio to access annotation groups that were previously cut off from view.
  * In the Audio and Video editors, when you select an issue the timeline automatically scrolls to the nearest relevant keyframe.

  *Future*

  * Starting February 9th, the new Audio/Video timeline V2 will be enabled for new projects by default. Projects created before February 9th will continue to default to the timeline V1, and you can enable or disable it anytime in the Advanced project settings.
  * Per Claude, claude-3-7-sonnet will be removed as it was deprecated in December and will be retired on February 19th. We recommend moving to claude-4-5-sonnet.

  ## Python SDK

  To view the latest updates to our Python SDK, see the [<u>What's new</u>](/horizon/developer-guides/whats-new) section in the developer guides.
</Update>

<Update label="January 5, 2026">
  ## App

  *Added*

  * The new timer details in the editor provide comprehensive time tracking for labeling, review, and rework modes. The timer will show progressive warnings with visual alerts as the labeler approaches the target maximum time, exceeds the target maximum time, and reaches the cutoff threshold.
  * If network connection issues occur while labeling, the editor displays a warning modal to notify the user of the issue and another modal when the network connection is restored. The timer will stop when a network connection issue is detected and resume when the issue is resolved.
  * In the project’s Workflow tab, you can configure the maximum task limit for specific users or the cumulative task limit for specific groups. To configure this setting, go to the Workflow tab, select the Initial labeling task, select Edit workflow, and configure the values in the Per-user/group task limits section.
  * You can now route rework tasks back to the original labeler. To enable this, go to the Workflow tab, select the Rework step, and update the Individual assignment field to Label creator.
  * You can now configure a maximum number of rework attempts per contributor. This allows you to set a threshold, and once that threshold is reached, the task will be routed to a different contributor. To configure this setting, go to your project’s Workflow tab, open the Rework step, make sure the Individual assignment setting is set to Label creator, and set the value for Max rework attempts per contributor.
  * Reviewers can now assign a score on a per-label basis, and the Labelbox platform will automatically calculate an average score across all attempts for each contributor. When a labeler’s average score falls below the configured threshold, the system can automatically remove them from the project or queue. To configure this setting, go to your project’s Advanced settings and set a value for Max rework attempts. Make sure the Individual assignment in your Rework step in Workflows is set to Label creator.

  *Changed*

  * In the Audio editor, when you change the timeline resolution/zoom, the timeline now scrolls to keep the playhead in view instead of jumping back to the start.

  ## Python SDK

  To view the latest updates to our Python SDK, see the [<u>What's new</u>](/horizon/developer-guides/whats-new) section in the developer guides.
</Update>

<Update label="December 3, 2025">
  ## App

  *Added*

  * The Multi-modal chat editor has a new “Data capture” project configuration. When projects are set to this mode, you can upload videos, images, PDFs, or other files as attachments for general-purpose data collection without a model. You may enable this configuration at project setup.
  * In the Advanced project settings, you’ll find a new option called “Authentic input protection”. When enabled, this feature blocks paste actions, detects suspiciously rapid entry, and automatically creates an issue on the data row when either of these suspicious behaviors is detected. This is available for all project types.
  * In the Multi-modal chat editor, you can now define minimum and maximum character counts in the prompt. Admins can enable this feature in the Advanced project settings (see “Min prompt message character length” and “Max prompt message character length”).
  * In the Multi-modal chat editor, there is an option to display model responses in a vertical message layout. Admins can enable this feature by toggling ON the “Vertical response messages” toggle in the Advanced project settings.
  * The following are now available in the Model tab:
    * Gemini 3 Pro Preview
    * GPT-5.1
    * Claude Opus 4.5

  *Changed*

  * The Multi-modal chat editor has received some improvements that significantly boost speed and performance when working with data rows containing many classifications.

  *Removed*

  * The following have been removed from the Model tab:
    * Llama 3.1

  ## Python SDK

  To view the latest updates to our Python SDK, see the [<u>What's new</u>](/horizon/developer-guides/whats-new) section in the developer guides.
</Update>

<Update label="November 4, 2025">
  ## App

  *Added*

  * The Labelbox app now displays charts for average handling time (AHT). To learn more about these new AHT charts, visit our [docs](/horizon/guides/performance-dashboard). You will find new charts in the following places:
    * Annotate > Performance > TEQ > Efficiency
    * Monitor
  * All editors now display a timer, giving labelers and reviewers visibility into how long they've spent on each task. This is now enabled for all customers.
  * In the Audio editor, you can now select a range of frames on the audio timeline and modify all classification values that belong to that portion of the timeline. This multi-selection support encompasses all operations available in the audio editor.
  * The Multi-modal chat editor now supports a multiple conversation interface that allows labelers to work on multiple conversations at once. To enable this, click the "Conversation list" icon in the top navbar in the editor.
  * In the Multi-modal chat editor, project admins can use the new "Response count" field to set multiple model response generations for every turn simultaneously. This "Response count" is supported for both single and multiple models connected.
  * Project admins can now enable the "Thresholding" toggle for Multi-modal chat projects. When enabled, labelers can evaluate each response as passing or failing, and then the editor will display pass\@k scores automatically after all evaluations are completed.
  * In the Multi-modal chat editor, you can now clip your recorded audio segments.
  * In the Multi-modal chat editor, admins can configure the editor to display specific classifications for each turn of a conversation. To enable this setting, go to the project setup page, create a new classification, set "Scope" to "Narrow", and under "Narrow scopes" select "Turn".
  * The following are now hosted in the Model tab:
    * Gemini 2.5 Flash Image Preview
    * Claude Sonnet 4.5
    * Claude Sonnet 4.5 Vertex
    * GPT-4o Diarize
    * Veo 3.1
    * Veo 3.1 Fast

  ## Python SDK

  To view the latest updates to our Python SDK, see the [**What’s new**](https://docs.labelbox.com/horizon/developer-guides/whats-new) section in the developer guides.
</Update>

<Update label="October 3, 2025">
  ## App

  *Added*

  * The z-order visualization capability is now available in the Catalog view.
  * In the audio editor, it is now possible to select a range of the timeline and modify all classification values in bulk.
  * In the Model tab, Claude Sonnet 4.5 is now supported.

  *Changed*

  * In the video editor, you can now move a sub-classification at the global level to the frame level.
  * You can now conditionally allow sub-classifications based on more than one parent classification.

  ## Python SDK

  To view the latest updates to our Python SDK, see the [<u>What's new</u>](/horizon/developer-guides/whats-new) section in the developer guides.
</Update>

<Update label="September 2, 2025">
  ## App

  *Added*

  * The Performance Dashboard now contains a bar chart that shows the number of labeling tasks per labeler.
  * The SAM2 auto-segmentation model is now integrated into the video editor. Now, a labeler can click a few positive and negative points for objects on a single frame they want to track and select a frame range. SAM2 will automatically add segmentation masks to the object over the selected frame range, reducing manual work.
  * The Models tab now supports the following models:
    * Open AI GPT-5
    * Claude Opus 4.1
    * Veo 3
    * Google Imagen 4
  * The Multi-modal chat has been updated with the following:
    * You can now add turn-level instructions as metadata (see `turnInstructions` in the [<u>docs</u>](/horizon/guides/datarow-metadata)).
    * The Multi-modal chat editor now supports video generation, starting with Veo 3.
  * You can now edit the behavior of the rework task in workflows to send rejected labels back to the original label creator only.
  * If "Skip Data Row" in the advanced project settings is toggled on, when a data row is skipped five times, it will move to the next task queue.

  *Changed*

  * When a data row gets skipped in the editor, the data row will stay in the “To label” queue instead of moving to the “Review” queue.
  * The default real-time resolution setting in the video editor is now set to 1080p.
  * We made the following improvements to the Multi-modal chat editor
    * When you upload an image during a turn, the image won’t be saved as an attachment until you submit the turn.
    * When you reset a turn, you have the option to leave the image attachment on the turn or manually remove it.
  * The hotkey for "Submit / Save" in the editor has been updated to "E E".

  ## Python SDK

  To view the latest updates to our Python SDK, see the [What’s new](/horizon/developer-guides/whats-new) section in the developer guides.
</Update>

<Update label="August 1, 2025">
  ## App

  *Added*

  * The following models are now hosted in the Model tab:
    * Deepseek R1
    * OpenAI o3 deep research
    * OpenAI o4-mini deep research
    * Grok-4
    * Kimi K2 Instruct
  * Timer details (elapsed time) are now available in the editor. Click the timer icon to view elapsed labeling time on the current data row.
  * In the editor, polygons connected by a relationship will be highlighted for better visibility.
  * Bounding box objects can now be toggled to force square shapes.

  *Changed*

  * Our docs have been migrated to a new platform. You should see an improvement in readability and searchability.

  *Removed*

  * The following models were removed from the Model tab:
    * o1-preview
    * o1-mini
  * The Census integration has been sunsetted and is no longer supported.

  *Future* By the end of 2025, we’ll be expanding our list of whitelisted IPs. We will send out more details about this upcoming change soon.

  ## Python SDK

  To view the latest updates to our Python SDK, see the [What’s new](/horizon/developer-guides/whats-new) section in the developer guides.
</Update>

<Update label="July 1, 2025">
  ## App

  *Added*

  * Previously, the Multi-modal chat editor only supported XLS, PDF, and images.
  * Now, all file types can be uploaded. If a model cannot read a particular file type, a warning will be displayed to the user.
  * The Multi-modal chat editor now supports rubrics that can be created and customized at the per-prompt level. Currently, the scoring mechanisms supported for rubrics are binary responses and numerical scores.
  * Instead of having to select “Start reviewing” to approve or reject a data row, you can now approve/reject a data row by selecting a data row from the Data rows tab in a project. This will open the Task browser, where you will be able to approve/reject the data row. From this view, you can also choose the winning label for Consensus projects.
  * In the Workflow tab, when you edit the initial labeling task, you can specify the maximum number of labels for that task. You can use this mechanism to ensure that data rows are evenly distributed among the labelers assigned to the project.
  * The following models are now available in the Model tab:
    * Qwen3 235B
    * Perplexity Sonar Pro

  *Changed*

  * You can now edit model config parameters in a Multi-modal chat project without having to create an entirely new project.
  * Previously, if a model failed in the Multi-modal chat editor, the user would need to regenerate all responses. Now, if one model fails and others succeed, the successful model responses will show, and the user can trigger a regeneration of just the model response that failed.
  * Now, the Multi-modal chat editor displays more descriptive error codes when a model fails to generate a response.
  * The Multi-modal chat editor now supports models responding with multiple messages and a variety of asset types, including images and videos.
  * The Multi-modal chat editor now supports human rewrites for 2+ models.
  * The Catalog filter, “Label Actions”, has a new option to filter for skipped data rows (“Is skipped”).
  * Now, when you create relationships in the editor with the relationship tool, the relationship will be validated upon creation.

  *Removed*

  * The following models have been removed from the Model tab:
    * Gemini 1.5 Flash ​

  ## Python SDK

  To view the latest updates to our Python SDK, see the [What’s new](/horizon/developer-guides/whats-new) section in the developer guides.
</Update>

<Update label="June 3, 2025">
  **Release notes**

  ## App

  *Added*

  * When you create a project and select the Multi-modal chat editor, you’ll be prompted to select from a dropdown of preconfigured settings for your project. If needed, you can update these settings in the advanced project settings section.
  * New models added to Foundry
    * Amazon Nova Sonic
    * Amazon Nova Premier
    * Open AI GPT-4o Image Generation
    * Meta Llama 3.3
    * Anthropic Claude 4

  *Changed*

  * The “start transcribing” button in the audio editor will be disabled unless your organization has access to Foundry.
  * The issues creation flow in the Multi-modal chat editor has been slightly updated to display a more discreet issues icon (caution icon). Clicking this icon allows you to create, resolve, and reopen issues in the task.

  *Removed*

  * Removed these models from Foundry
    * Google Gemini 2.0 Flash Experimental
    * xAI Grok Beta

  *Future*

  * On June 30, 2025, we will sunset the [Census integration](/horizon/guides/census-integration) due to a lack of usage. If you have any questions or concerns, please contact [support@labelbox.com](mailto:support@labelbox.com).

  ## Python SDK

  To view the latest updates to our Python SDK, see the [What's new](/horizon/developer-guides/whats-new) section in the developer guides.

  Read More
</Update>

<Update label="May 5, 2025">
  **Release notes**

  ## App

  *Added*

  * The new form-based UI in the Multi-modal chat editor makes the labeling experience more intuitive, displays instructions at each labeling task, and makes the tools menu more accessible.
  * In the Multi-modal chat editor, you’ll see a minimap on the right that indicates the location of the errors to address before the task can be submitted. Selecting the red markers in the minimap will bring you to each error in the conversation. To learn more, read this [blog post](https://labelbox.com/blog/ai-evaluation-labelboxs-redesigned-multimodal-chat-editor-04-2025/).
  * We added turn-based audio and video support to the Multi-modal chat editor to enable labelers to generate video training data on real-world examples. You can enable this in the project settings.
  * We added real-time audio and video support to the Multi-modal chat editor to enable labelers to submit video as a response to the model in real time. You can enable this in the project settings.
  * The new “Model selection” task option in the Multi-modal chat editor makes it more intuitive for labelers to select which model response is better, indicate that both are bad, or are “good”. You can configure this task to function as a likert scale. To configure this task at the global level, go to Schema → Ontologies → Create → Model chat evaluation → Model selection tasks → Global → Add. To configure this task at the turn-level, go to Schema → Ontologies → Create → Model chat evaluation → Model selection tasks → Per turn → Add.
  * Foundry now supports the following models:
    * Gemini 2.5 Pro
    * Meta Llama 4.0 Maverick
    * Grok 3
    * OpenAI GPT 4.1
    * OpenAI GPT-4o Transcribe
    * OpenAI GPT-4o mini Transcribe
    * OpenAI GPT-4o mini TTS
    * OpenAI o3
    * OpenAI o4-mini

  *Changed*

  * In the Data Rows tab, you can now expand column widths and reorder columns in the table.
  * Within a project, the "Automation" tab has now been renamed to the "Import labels" tab. Here you can find your import jobs used for MAL predictions, ground truth, or benchmarks. There is also helpful code recommended that includes information from the ontology assigned to the project.

  *Removed*

  * April 30, 2025, we started introducing changes to the Python SDK that will break compatibility with Export v2 non-streamable methods in version 3.66 and earlier. To see which methods will be impacted by this change, visit our [Deprecations](/horizon/guides/deprecations#export-v2-nonstreamable-methods) page.

  *Future*

  * On June 30, 2025, we will sunset the [Census integration](/horizon/guides/census-integration) due to lack of usage. If you have any questions or concerns, please contact [support@labelbox.com](mailto:support@labelbox.com).

  ## Python SDK

  To view the latest updates to our Python SDK, see the [What's new](/horizon/developer-guides/whats-new) section in the developer guides.

  Read More
</Update>

<Update label="April 2, 2025">
  **Release notes**

  ## App

  *Added*

  * We have rolled out an improved experience for managing workspace members and groups to all tiers. To learn more about these changes, read [Manage members and permissions](/horizon/guides/manage-members-and-groups).

  *Changed*

  * When you select the Workflows tab in a project, you’ll notice the UI has been updated. This new Workflow editor allows you more flexibility when creating and arranging the steps in your project workflow. To learn more, see our docs on [Workflows](/horizon/guides/workflows).
  * If you have MFA enabled for your organization, you will see a checkbox upon login that offers to “Remember this device for 30 days”.
  * Integrating AWS S3 via IAM now requires you to create a custom trust policy instead of selecting the Labelbox AWS account. This change gives you more granular control, making the integration more flexible and customizable. See our docs on [AWS S3 integration](/horizon/guides/import-aws-s3-data) to learn more.

  *Future*

  * On April 30, 2025, we will introduce changes to the Python SDK that will break compatibility with Export v2 non-streamable methods in version 3.66 and earlier. To see which methods will be impacted by this change, visit our [Deprecations page](/horizon/guides/deprecations#export-v2-nonstreamable-methods).
  * On June 30, 2025, we will sunset the [Census integration](/horizon/guides/census-integration) due to lack of usage. If you have any questions or concerns, please contact [support@labelbox.com](mailto:support@labelbox.com).

  ## Python SDK

  To view the latest updates to our Python SDK, see the [What's new](/horizon/developer-guides/whats-new) section in the developer guides.

  Read More
</Update>

<Update label="March 3, 2025">
  **Release notes**

  ## App

  *Added*

  * In the image editor, you can now record an audio clip as a response, and the editor will automatically transcribe and save the audio recording as a classification. To learn more, read our Image editor guide.
  * In the Services tab, we introduced an NLP search option to make it easier to find candidates that have certain skills or specializations. To use the NLP search, go to the Services tab, select the “Natural language” filter, and enter one or more keywords to narrow down the search results.
  * When you are labeling code in the Multi-modal chat editor, you now have access to a full VSCode IDE web instance and a remote host environment, allowing you to work on entire code repositories, run CLI tools, execute code, use debuggers, write tests, install additional VSCode extensions, and use Clari copilot to write/edit code blocks. To use this feature, create a project with the Multi-modal chat editor, edit the code block, and select the “Edit in coder” button to open the IDE web instance.
  * The following models are now supported in the Model tab (see “Hosted by Labelbox” section):
    * Anthropic Claude 3.7 Sonnet
    * Anthropic Claude 3.7 Sonnet Think
    * Google Gemini 2.0 Flash
    * Google Gemini 2.0 Flash-Lite
    * Google Gemini 2.0 Pro

  *Changed*

  * The home page was improved recently to highlight our labeling services and to make our core features easier to find.
  * We are gradually rolling out an improved experience for managing workspace members and groups. To ensure a smooth transition, these changes will be rolled out tier by tier. To learn more about these changes, read [New member and group management experience](/horizon/guides/new-member-and-group-management-experience).

  *Removed*

  * On February 13th, we disabled the “Reporting” tab (AKA Enterprise Dashboard) for all customers. The Reporting tab has been replaced by the new Monitor tab.
  * On February 28th, we disabled the Catalog cluster view + Smart select feature for all customers.

  *Future*

  * On April 24, 2025, we will sunset exports v2 non-streamable for all customers using Python SDK versions 3.66 or earlier.

  ## Python SDK

  * To view the latest updates to our Python SDK, see the [What's new](/horizon/developer-guides/whats-new) section in the developer guides. Read More
</Update>

<Update label="February 4, 2025">
  **Release notes**

  ## App

  *Added*

  * The following models are now hosted and available in the Model tab:
    * OpenAI o1 & o3-mini
    * OpenAI Whisper
    * Google Gemini 2.0 Flash Thinking
    * Google Gemini 2.0 Flash Experimental
    * Amazon Nova Micro, Lite, and Pro
  * To increase both the speed and accuracy of extracting text from a PDF, labelers can now draw bounding boxes around text in a document, and the PDF editor will automatically extract the text from the bounding box.
  * The Audio editor now automatically transcribes speech to text to make labeling audio files easier.

  *Changed*

  * The Multi-modal chat editor now includes an improved experience for editing and running code. To learn more, see our [Code runner docs](/horizon/guides/code-grammar-assistance#code-editor).
  * The AI critic in the Multi-modal chat editor now has an improved experience for getting suggestions on coding tasks.
  * The Services tab is now easier to navigate and has an improved experience for filtering results.

  *Future*

  * On February 13, 2025, we will sunset the “Reporting” tab (AKA Enterprise Dashboard) for all customers. The Reporting tab has been replaced by the new Monitor tab. To learn more about this change, refer to [this guide](/horizon/guides/migration-guide-reporting-page-to-monitor).

  ## Python SDK

  To view the latest updates to our Python SDK, see the [What's new](/horizon/developer-guides/whats-new) section in the developer guides.

  Read More
</Update>

<Update label="January 6, 2025">
  **Release notes**

  ## App

  *Added*

  * The Multi-modal chat editor now has a built-in code runner that runs and detects code blocks. This feature allows labelers and reviewers to execute code directly in the editor and see the results. Code execution results are persisted and included in exports. To learn more, see our [Code runner docs](/horizon/guides/code-grammar-assistance#code-runner). This feature is currently in beta.
  * Polygons, polylines, bounding boxes, and points now have a z-order and are rendered so that those in front occlude or block out those behind. Their z-order can be changed using a new view in the Objects panel. To learn more, see our docs on [Editor settings](/horizon/guides/image-editor#setting-object-positions).
  * The new “Polygon snapping” feature allows points on lines, polygons, and bounding boxes to align or attach to polygon edges. You can toggle on polygon snapping in the [Editor settings](/horizon/guides/image-editor#editor-settings).
  * To encourage labeler understanding of project instructions, labelers will be enforced to read and acknowledge project instructions before working on a project. If project instructions are modified, labelers will be required to re-read instructions before they can resume labeling in the project.Enforced labeling instructions **only** impacts customers who opt into using Alignerr labeling services for their projects.
  * In the PDF editor, you can now create bounding boxes on the page and use the relationship tool to connect the bounding boxes to classifications. This feature is available upon request (email [support@labelbox.com](mailto:support@labelbox.com) to request access).
  * In the new labeling services marketplace, you can view Alignerr labeler profiles and select labelers for your next project based on their expertise and availability. To browse Alignerr labeler profiles, select “Services” from the left nav bar. To learn more, see our docs on [Alignerr Connect](/horizon/guides/alignerr-connect).

  *Changed*

  * After a series of bug fixes, the custom roles feature now has an improved user experience. Fixes mostly include changes to make permission settings more reliable.
  * You can now download the history of changes made to a user group as a CSV file. To do this, go to Workspace settings > Groups > Export > Export history.
  * The latest timer mechanism has now been rolled out to all remaining organizations. This new timer mechanism offers more accurate measurements for labeling, reviewing, and reworking time.

  *Future*

  * On February 13, 2025, we will sunset the “Reporting” tab (AKA Enterprise Dashboard) for all customers. The Reporting tab has been replaced by the new Monitor tab. To learn more about this change, refer to [this guide](/horizon/guides/migration-guide-reporting-page-to-monitor).

  ## Python SDK

  To view the latest updates to our Python SDK, see the [What's new](/horizon/developer-guides/whats-new) section in the developer guides.

  Read More
</Update>

<Update label="December 3, 2024">
  **Release notes**

  ## App

  *Added*

  * In-editor suggestions are now available in the Multi-modal chat and Prompt & response editors. You can use this AI-powered tool to double-check your grammar and wording to ensure your prompts are written correctly. This feature is enabled on a project-by-project basis.
  * The new “step-by-step reasoning” task in the Multi-modal chat editor allows labelers to classify each step in the model response as “correct” or “incorrect”. Then, the model regenerates its response starting from the incorrect segment while preserving the previous responses. To learn more about how this works, read [this guide](https://labelbox.com/blog/multi-step-reasoning-teach-llms-to-think-critically/).
  * The new “fact-checking” task in the Multi-modal chat editor allows labelers to easily rate any step in the response as “accurate”, “inaccurate”, “disputed”, “unsupported”, etc.
  * The new “prompt-rating” task in the Multi-modal chat editor allows labelers to select from a pre-set list of toggles to signal that the content should not be labeled for a specific reason (e.g., offensive content, contains PII, not understandable, etc).

  *Removed*

  * On November 7th 2024, we removed the native support for YOLO models in Foundry. If you would like to set up your own YOLO model for inferencing, refer to our [custom model integration docs](/horizon/guides/custom-model-integration#create-model-integrations-for-bounding-box-and-mask-tasks).
  * On November 7th 2024, we disabled the model fine-tuning feature, meaning the image fine-tuning is no longer available. We may re-enable this feature in the future.
  * On November 25th 2024, we sunsetted the DICOM editor in our platform for all customers. Labeling DICOM data is no longer supported in Labelbox.

  *Future*

  * On February 13th 2025, we will sunset the “Reporting” tab (AKA Enterprise Dashboard) for all customers. The Reporting tab has been replaced by the new Monitor tab. To learn more about this change, refer to [this guide](/horizon/guides/migration-guide-reporting-page-to-monitor).

  ## Python SDK

  To view the latest updates to our Python SDK, see the [What's new](/horizon/developer-guides/whats-new) section in the developer guides. Read More
</Update>

<Update label="November 4, 2024">
  **Release notes**

  ## App

  *Added*

  * In the video editor, classifications can now be arbitrarily nested. Nested classifications can also be displayed in the timeline.
  * When you are configuring classifications in an ontology, you will see an option to toggle on/off the Likert scale option. Enabling the Likert scale option will automatically populate labeler classification responses with the first response option listed.
  * Now, when you enter a response in the multi-modal chat editor, you’ll see a button for “Get suggestions”. When you enable this AI-powered tool, you will get suggestions (code or grammar) to catch any mistakes you may have overlooked in your response. This feature is currently in beta.
  * When you are in review mode, you will now see Consensus scores next to the annotation names in the Tools menu.
  * You can now enable email notifications to stay up-to-date on changes related to project assignments, batch updates, and issue activity. To learn more, see our docs on [Notifications](/horizon/guides/notifications).

  *Fixed*

  * The Usage page got some UI improvements and additional filters for better usability. Changes include fixes to LBU calculation for custom dates, graphing capabilities, and LBU display on modal dialogs.
  * The toggle-off and backspace issues in the video editor have also been fixed.

  *Future*

  * On November 7th 2024, Labelbox will sunset native support for YOLO models in Foundry. If you would like to set up your own YOLO model for inferencing, refer to our [custom model integration](/horizon/guides/custom-model-integration#create-model-integrations-for-bounding-box-and-mask-tasks) docs.
  * On November 7th 2024, Labelbox will disable YOLOv8 for the model fine-tuning feature. This means that image fine-tuning will not be usable. We may re-enable image fine-tuning in the future.

  ## Python SDK

  * To view the latest updates to our Python SDK, see the [What's new](/horizon/developer-guides/whats-new) section in the developer guides. Read More
</Update>

<Update label="October 1, 2024">
  **Release notes**

  *Added*

  * The new Labelbox Microsoft Entra application allows you to install a verified version of the Labelbox Enterprise application into your Azure tenant. To learn how this works, see [these docs](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/aad.labelbox?tab=Overview).
  * The Performance tab in Annotate now contains an option to enable outlier detection metrics. These metrics highlight and bucket labelers who are outliers in key areas for quick remediation.
  * Customers who want to connect their own models to Labelbox now have a self-serve, product-integrated UI to bring their own models for LLM, classification, text, and NER use cases. Bounding boxes and masks will still require manual onboarding for the time being. To learn more, see our docs on [Custom model integration](/horizon/guides/custom-model-integration#create-model-integrations).
  * The new Monitor page is now available for Enterprise customers. The Monitor contains charts, tables, and filters to display project and member performance across workspaces. It also contains new bulk actions and outlier detection metrics. To learn more, see our [Monitor](/horizon/guides/monitor) docs.

  *Changed*

  * Asset proxy (the feature that hides raw data row URLs from being exposed in the platform) is now supported for all project types. To request access to this feature, contact our support team.

  ## Python SDK

  To view the latest updates to our Python SDK, see the [What's new](/horizon/developer-guides/whats-new) section in the developer guides. Read More
</Update>

<Update label="September 5th, 2024">
  **Release notes**

  ## App

  *Added*

  * Now, you can subscribe to email notifications for the following: 1) when labelers are assigned to (or unassigned from) a project, 2) when an issue is created, resolved, or commented on and 3) when a batch of data rows is added to a project or has been labeled. To manage email notifications, go to your user profile in the app UI.

  *Changed*

  * The Multimodal chat editor has a new and improved UI. Key changes include: 1) When ranking, selecting, and classifying, the turn remains horizontal and 2) Instead of a left side panel, each turn is self-contained with its own tasks and classifications.
  * When you create new workflow tasks, you have more task filters to choose from: Issue category, Dataset, Batch, Metadata, Model prediction, Labeling time, Review time, Natural language, and Label feedback.
  * When you add task filters to workflow tasks, you can select "Match: Any" to apply OR logic or select "Match: All" to apply AND logic to the filters.
  * You will see an updated flow when requesting labeling services via the UI.

  *Removed*

  * Export v1 has been sunsetted for all customers.

  ## Python SDK

  To view the latest updates to our Python SDK, see the [What's new](/horizon/developer-guides/whats-new#version-3770-2024-08-09) section in the developer guides. Read More
</Update>

<Update label="August 5, 2024">
  **Release notes**

  ## App

  *Added*

  * The multimodal chat editor now supports offline multimodal chat evaluation projects. To learn how this works, see our [Multi-modal chat evaluation docs](/horizon/guides/multimodal-chat-evaluation-editor#set-up-offline-multimodal-chat-evaluation-projects).
  * The Multimodal chat evaluation editor now supports LaTeX math expressions. To learn more, see our [Multi-modal chat evaluation docs](/horizon/guides/multimodal-chat-evaluation-editor#step-2-configure-models).
  * Enterprise customers will see an updated flow for requesting labeling services in the app UI.
  * You can now use the Benchmarks & Consensus tools for prompt & response generation projects.
  * You can now use the Benchmarks & Consensus tools for video projects.
  * Benchmarks & Consensus are now enabled for the following workflows: Send data rows from Model to Annotate, send data rows from Annotate to a model experiment, and Bulk classification in Catalog.
  * In the Data rows tab in Annotate, you will see an improved search and filtering experience for data rows with Benchmarks.
  * The audio editor now supports temporal classifications and a timeline, so you can assign classifications to specific points in the audio timeline.
  * When you compare model runs in the metrics and cluster view in Model, it will only display the metrics calculations for the data rows that appear in both model runs.

  *Changed*

  * You can now configure your projects to use Benchmarks & Consensus simultaneously. If you have a project that has already been set up with Benchmarks, you can enable Consensus later in the project settings. However, it is still not possible to disable once set.
  * In the Data row browser, you can now set a Consensus label as a Benchmark label. You can do so by selecting a data row in the Data rows tab that has a Consensus label and select Add as Benchmark. This will calculate both the Benchmark and Consensus score for the data row.
  * The left side panel is now resizable across all editors.
  * The default reservation count for the labeling queue has been updated from 10 to 4. Admins can adjust the queue parameters in the project settings.
  * The Auto-segment tool in the image editor has been upgraded to use SAM 2.

  *Removed*

  * The custom editor has been sunsetted for all customers.
  * The following data connector libraries have been archived: labelbase, labelspark, labelpandas, labelsnow, labelbox-bigquery. To learn more, see our [Deprecations page](/horizon/guides/deprecations#data-connector-libraries).
  * The External workforce tab has been removed from the project settings.

  ## Python SDK

  The latest version of our Python SDK is v3.76.0. See our [full changelog on GitHub](https://github.com/Labelbox/labelbox-python/blob/develop/libs/labelbox/CHANGELOG.md#version-3760-2024-07-29) for more details on what was added recently.

  ### Version 3.76.0 (2024-07-29)

  *Added*

  * Added Project get\_labeling\_service(), request\_labeling\_service() and get\_labeling\_service\_status()
  * Added project and ontology creation for prompt response projects: Client create\_prompt\_response\_generation\_project(), create\_response\_creation\_project()
  * Added is\_benchmark\_enabled, is\_consensus\_enabled to Project

  *Updated*

  * Made Project quality modes a list to allow combining more than 1 quality mode per project

  *Notebooks*

  * Added back in export migration guide
  * Added correct data param to video notebook

  *Other*

  * Use connection pool for all http and graphql requests

  ### Version 3.75.1 (2024-07-16)

  *Removed*

  * Project MEDIA\_TYPE JSON

  ### Version 3.75.0 (2024-07-10)

  *Added*

  * Added Project set\_project\_model\_setup\_complete() method
  * Added user group management
  * Refactor Dataset create\_data\_rows\_sync to use upsert
  * Added upload\_type to Project
  * Added prompt classification for python object support
  * Alias `wait_xxx` functions

  *Fixed*

  * Predictions missing during Catalog slice Export
  * Prevented adding batches to live chat evaluation projects
  * Added missing media types
  * Deprecate Project setup\_editor and add Project connect\_ontology
  * Bumped dateutil max version
  * Bumped version rye
  * Updated create ontology for project setup

  Read More
</Update>

<Update label="July 5, 2024">
  **Release notes**

  ## App

  *Added*

  * For video projects, you can now assign free-form text classifications to individual frames or a range of frames.
  * The Usage tab in Workspace settings now contains LBU usage per product and data modality. It also displays usage and project billing costs for Boost Workforce and Boost Workforce Express projects. To learn more, see our docs on [Account details](/horizon/guides/view-billing-details#data-row-usage).
  * Enterprise and pay-as-you-go customers will see an improved setup flow when configuring a data warehouse integration. When you initiate a new integration in the Integrations tab, you'll see step-by-step instructions to guide you through the setup process. To learn more, see our [Census integration](/horizon/guides/census-integration) docs.
  * Admins can define their own user roles with any set of permissions that Labelbox provides and allow those roles to be assigned to any user. To learn more, see our docs on [Member roles](/horizon/guides/roles-and-permissions#member-roles).
  * Benchmarks and Consensus are now supported for projects created using the “Prompt and response generation” editor. This is only supported when you select the “Humans generate responses to uploaded prompts” option.
  * Claude 3.5 Sonnet is now available in Foundry.
  * Images are now supported for Claude 3 Opus, Claude 3.5 Sonnet, and Claude 3 Haiku in Foundry.

  *Changed*

  * You can now configure your project settings to use the Benchmarks and Consensus tools at the same time.
  * Contract-based customers will see an updated flow when requesting Boost workforce services.
  * We made some improvements to our annotation import formats to make them more intuitive. To see the updated import formats, click through the links at the top of the [Import ground truth doc](/horizon/guides/import-ground-truth).

  *Fixed*

  * A fix that blocks concurrent edits to the same label has addressed the platform issues causing duplicate classifications, multiple segmentation groups, and options with deleted questions.
  * The root cause of certain data integrity issues, such as missing pixels in raster segmentation masks and multiple/duplicate classifications, has been addressed.
  * A fix in our queueing system has boosted Export v2's overall performance, significantly improving export performance on larger scales.

  *Removed*

  * Claude Instant, Claude 2, and Claude 3 Sonnet have been removed from Foundry due to lack of usage.
  * The custom editor has been sunset for all customers. Custom editor projects are no longer accessible (exporting is also disabled) via the app and the SDK. Custom editor projects will no longer appear in your project list.

  ## Python SDK

  The latest version of our Python SDK is v3.74.0. See our [full changelog on GitHub](https://github.com/Labelbox/labelbox-python/blob/develop/libs/labelbox/CHANGELOG.md#version-3740-2024-06-24) for more details on what was added recently.

  ### Version 3.74.0 (2024-06-24)

  *Added*

  * Include predictions in export
  * Upsert label feedback method Client `upsert_label_feedback()`

  ### Version 3.74.0 (2024-06-24)

  *Added*

  * Include predictions in export
  * Upsert label feedback method Client `upsert_label_feedback()`

  *Removed*

  * Removed deprecated class `LabelList`

  ### Version 3.73.0 (2024-06-20)

  *Added*

  * Conversational data row checks
  * UI ontology mode support
  * Empty data row validation

  *Fixed*

  * Numpy semver locked to \< 2.0.0

  ### Version 3.72.2 (2024-06-10)

  *Added*

  * SLSA provenance generation

  ### Version 3.72.1 (2024-06-06)

  *Fixed*

  * Fix `client.get_project()` for LLM projects
  * Throw user-readable errors when creating a custom embedding Read More
</Update>

<Update label="June 5, 2024">
  **Release notes**

  ## App

  *Added*

  * Now you can browse video assets (as well as their annotations and predictions) in the detailed view within Catalog, Model, and Annotate.
  * Our Foundry product is now available as GA. Foundry enables you to use powerful foundation models to prelabel your data and integrate directly with Annotate workflows. To learn more, read our docs on [Foundry](/horizon/guides/foundry).
  * Now, when you create an API key, you can specify its scope to a particular permission level. See our docs on [API keys](/horizon/developer-guides/create-api-key) to learn more.
  * You can now [clone projects](/horizon/guides/create-a-project#duplicate-a-project) and their settings (ontology, members, name, tags, issue categories). To do this, go to Annotate, select a project, click on the project name at the top, and select Duplicate project.
  * We introduced a new role called tenant admin. This tenant admin role has the permission to create new workspaces and invite team members to any workspace. See our docs on [Tenant Admin](/horizon/guides/workspaces#tenant-admin) to learn more.
  * OpenAI's GPT-4o, as well as Anthropic's Claude 3 Sonnet and Claude 3 Haiku models, are now available in Foundry.
  * When you open an experiment from Model, you'll see a new view called "List view". This new view displays each data row as a row in a list. Each row displays a preview of the asset and any annotation and/or prediction information for the data row. See our docs on [Model runs](/horizon/guides/model-runs) to learn more.
  * Now, in the data rows tab, you can filter data rows by agreement score at the feature level. To do this, go to the Data rows tab, filter by "Consensus average," and select "Feature". You can further narrow down your results by feature schema. To learn more, see our docs on the [Concensus](/horizon/guides/consensus).
  * We released a new [live multimodal chat evaluation editor](/horizon/guides/live-multimodal-chat-evaluation) that enables you to enter a prompt, get a response from multiple models, and use an ontology to rank and select the best model responses.

  *Removed*

  * On March 16, 2024, we sunset Export v1 for pro, standard, and select enterprise customers.

  *Future*

  * On June 30, 2024, we will sunset the custom editor for all customers.

  ## Python SDK

  The latest version of our Python SDK is v3.72.0. See our [full changelog on GitHub](https://github.com/Labelbox/labelbox-python/blob/develop/libs/labelbox/CHANGELOG.md#version-3720-2024-06-04) for more details on what was added recently.

  ### Version 3.72.0 (2024-06-04)

  *Added*

  * Update Dataset create\_data\_rows to allow upload of an unlimited number of data rows
  * New Dataset methods for iam\_integraton: add\_iam\_integration, remove\_iam\_integration

  *Notebooks*

  * Added model evaluation SDK method notebook
  * Added quick start notebook geared towards new users

  ### Version 3.71.0 (2024-05-28)

  *Added*

  * project.get\_overview() to be able to retrieve project details
  * project.clone() to be able to clone projects
  * ExportTask.get\_buffered\_stream to replace ExportTask.get\_stream

  *Fixed*

  * ExportTask.result / ExportTask.errors parsing content incorrectly
  * Lack of exceptions related to updating model config

  ### Version 3.70.0 (2024-05-20)

  *Added*

  * Added chat model evaluation support
    * client.create\_model\_config()
    * ModelConfig project\_model\_configs()
    * ModelConfig add\_model\_config()
    * ModelConfig delete\_project\_model\_config()
    * ProjectModelConfig delete()
    * client.create\_model\_evaluation\_project()
  * Update existing methods to support chat model evaluation project
    * client.create\_ontology()
    * client.create\_ontology\_from\_feature\_schemas()
  * Coco deprecation message

  *Fixed*

  * Fixed error reporting for client.create\_project()
  * Do not retry http 422 errors

  *Notebooks*

  * Send\_to\_annotate\_from\_catalog functionalities outside Foundry

  *Fixed in Notebooks*

  * Fixed meta notebook
  * Modified queue\_management.ipynb to remove some parameters
  * Update\_huggingface.ipynb

  ## Python SDK

  *Added*

  * Dark mode (sun/moon icon on the top right-hand side)

  *Fixed*

  * All example notebooks from our SDK repository are now in the [Python tutorials](/horizon/tutorials) tab

  Read More
</Update>

<Update label="May 7, 2024">
  **Release notes**

  ## App

  *Added*

  * Our new warehouse integration p rovides an easy, no-code solution to keep your datasets in Labelbox in sync with the tables in your data warehouse. You can use this integration to connect over 25 different data sources, including Big Query, Databricks, Snowflake, and Google Sheets. To learn more, see our [Census integration](/horizon/guides/census-integration) docs.
  * You can use the new fine-tuning capability (beta) to fine-tune a YoloV8 object detection model with custom features. To run fine-tuning, you'll need to create a model experiment, import data rows and ground truth, and add an ontology to use for fine-tuning. Then, select the "fine-tune model" button to configure the training job. Docs on this feature are coming soon.
  * You can now include custom and auto-generated embeddings when you export your data rows via Export v2. You can find an example of exported embeddings on this page: [Export image annotations](/horizon/developer-guides/export-image-annotations#sample-project-export).
  * When you create an issue, you now have the option to assign the issue to a category. Issue categories can be viewed and managed in the Issues tab in a project. To learn more, see our docs on [Issues](/horizon/guides/issues-comments#create-an-issue).
  * The following model + annotation types are now supported in Foundry:
    * Video bounding box detection via GroundingDINO model
    * Video segmentation mask detection via GroundingDINO + SAM model
    * Video frame-based classification via Google Gemini 1.5 Pro (Beta) model
    * Video global classification via X-Clip, Gemini Pro Vision, Gemini 1.5 Pro (Beta) models
  * We introduced new ways to create model experiments in the UI: 1) From the Manage selection dropdown in Catalog, you can send selected data rows to a new experiment or an existing experiment. 2) From Model, you can now click Create -> Experiment to create a new experiment. 3) From an existing experiment, you can now click + New Model Run to append data rows to the experiment.
  * LlaMa v3 is now supported in Foundry.

  *Changed*

  * Our updated home page provides you with key actions, featured reads, and entry points to make it easier to get started on projects.
  * For projects that are configured with Boost Workforce Express, labeler instructions are now added as part of the ontology. We also enabled hyperlinking in the ontology Instructions to allow you to link a Google doc as labeler instructions.
  * With the new Catalog search experience, you can quickly search your data in Catalog by typing into the data search filter bar. You no longer need to specify a filter to find the data you are looking for. See [Filters](/horizon/guides/search#data-search) for more details.
  * The rate limit for Gemini 1.5 Pro increased from 5 to 600 requests per minute for Labelbox Foundry.

  *Removed*

  * On April 19, we sunsetted Export v1 for free, edu, and starter customers. To learn more about this deprecation, see our [migration guide](/horizon/guides/migration-guide-export-v1-to-export-v2).

  *Future*

  * On May 15, we will begin sunsetting Export v1 for pro, standard, and enterprise customers. To learn more, see our [migration guide](/horizon/guides/migration-guide-export-v1-to-export-v2).
  * On June 30, we will be sunsetting the custom editor for all customers.

  ## Python SDK

  The latest version of our Python SDK is v3.69.1. See our [full changelog on GitHub](https://github.com/Labelbox/labelbox-python/blob/develop/libs/labelbox/CHANGELOG.md) for more details on what was added recently.

  ### Version 3.69.1 (2024-05-01)

  Fixed- Fixed a bug with certain types of content not being returned as a result of `ExportTask.result` or `ExportTask.errors`

  ### Version 3.69.0 (2024-04-25)

  *Added*

  * Support to export embeddings from the SDK

  *Fixed*

  * Used OpenCV's headless library in replacement of OpenCV's default library

  ### Version 3.68.0 (2024-04-16)

  *Added*

  * Added support for embeddings.
  * Introduced the use of 'rye' as a package manager for SDK contributors.
  * Implemented a unified `create` method for `AnnotationImport`, `MEAPredictionImport`, and `MALPredictionImport`.
  * Enhanced annotation upload functionality to accept data row IDs, global keys, or external IDs directly for `labelbox.data.annotation_types.label`

  *Fixed*

  * Ensure items in `dataset.upsert_data_rows` are not empty
  * Streamable export fix to report export\_v2 errors as list of dictionaries, compatible with older releases

  ### Version 3.67.0 (2024-04-05)

  *Added*

  * Added `SECURITY.md` file
  * Made export\_v2 methods use streamable backend
  * Added support for custom embeddings to dataset *create data row(s)* methods
  * Added ability to upsert data rows via `dataset.upsert_data_rows()` method
  * Added `AssetAttachment` with an ability to `update()` and `delete()`

  *Updated*

  * Added check for 5000 labels per annotation per data row

  *Fixed*

  * Errors and Failed data rows are included in the `task.result` for `dataset.create_data_rows()`
  * Fixed 500 error handling and reporting

  *Notebooks*

  * Updated import notebook for image data
  * Added attachment PDF example, removed requirements around text\_layer\_url
  * Included the `get_catalog()` method to the export notebook
  * Added workflow status filter to export\_data notebook for projects
  * Send predictions to a project demo
  * Removed model diagnostic notebooks Read More
</Update>

<Update label="April 2, 2024">
  **Release notes**

  ## App

  *Added*

  * You can now filter and sort by custom metrics at the prediction level in the Model runs view. This new filter capability supports autogenerated metrics as well as custom metrics at the prediction level. To learn more, see [Filters](/horizon/guides/filtering-and-sorting#filtering-on-custom-metrics-per-prediction-basis).
  * When you enter the Schema tab, you'll see a new tab for Embeddings. In this part of the app, you can view all of your autogenerated and custom embeddings for your organization. Here, you can also create custom embeddings via the UI. See [Embeddings](/horizon/guides/embeddings) to learn more about the Embeddings subtab.
  * You can now add PDFs as attachments to data rows via the `PDF_URL` attachment type. To view an example of importing a PDF as an attachment, see [Attachments](/horizon/developer-guides/attachments).
  * If you are a new user and do not have any data in the Catalog yet, Labelbox will show a zero-state screen with guidance on how to add data rows.
  * If you are a new user and do not have any projects in Annotate yet, Labelbox will show a zero-state screen with guidance on how to create your first project.
  * Foundry now includes support for Claude 3 Opus and Google Gemini 1.5 Pro.
  * The new inferencing endpoints (beta) enable you to run any Foundry app to generate predictions on your data via a REST API. You can use this REST API to 1) pass raw data without creating data rows or a dataset, 2) pass raw data and specify which dataset to create the data row, or 3) pass a specific data row that has already been created in Labelbox. To learn more, see our docs on [Foundry apps](/horizon/guides/run-foundry-apps#run-app-using-rest-api).

  *Changed*

  * When you import PDFs as data rows, you no longer need to specify the `text_layer_url` as Labelbox now automatically generates the text layers when you import PDF assets. This means you no longer need to create the text layers and include them when importing PDF assets.
  * We now render the image overlay attachments in the order they were created. This means they are no longer randomly ordered when you view them in the editor.
  * Now when you click on a relationship edge in the editor, the relationship will be selected in the objects panel. If the relationship has a subclassification, the subclassification will open in the left panel. This improvement impacts the text and conversational text editors. To learn more, see our docs on Text relationships.

  *Fixed*

  * The issue where data rows were stuck in the "To label" queue has been resolved. Now data rows that have been labeled or skipped should be moved to the next workflow step, reliably.
  * The issue causing some assets to not load properly in the data row browser has been resolved.
  * We implemented a fix to ensure feature names with underscores are automatically formatted with surrounding spaces for accurate model processing.

  *Future*

  * On April 20, 2024, we will sunset Export v1 for Free, EDU, and Starter tiers. On April 27, 2024, we will sunset Export v1 for Standard, Pro, and Enterprise tiers. To learn more, see the [Export v1 to v2 migration guide](/horizon/guides/migration-guide-export-v1-to-export-v2).
  * On Jun 30, 2024, we will be sunsetting our custom editor for all customers.

  ## Python SDK

  The latest version of our Python SDK is v3.66.0. See our [full changelog on GitHub](https://github.com/Labelbox/labelbox-python/blob/master/CHANGELOG.md#version-366-2024-03-20) for more details on what was added recently.

  ### Version 3.66.0 (2024-03-20)

  *Added*

  * Added support for Python 3.11, 3.12
  * Added update method to attachments

  *Notebooks*

  * Improved notebooks for integration and model diagnostics
  * Removed databricks integrations notebooks

  *Updated*

  * Updated README for clarity and contribution guidelines

  *Removed*

  * Removed support Python 3.7 as it has been end of life since June 2023 Read More
</Update>

<Update label="March 5, 2024">
  **Release notes**

  ## App

  *Added*

  * Smart select (beta) in Catalog offers three different ways to curate your data within a dataset: random, ordered, and cluster-based. To access this feature, go to Catalog, select a dataset that is larger than 100 data rows, and select "Smart select".
  * Cluster view (beta) in Catalog is a new interface that helps you discover and find similar data rows as well as labeling or model mistakes. See [Cluster view](/horizon/guides/cluster-view) for more details.
  * Natural language search and similarity search in Catalog now support video and audio assets.
  * You can now 1) import custom metrics per prediction and 2) filter/sort by prediction-level custom metrics. These added functionalities make it easier to use our model error analysis tools in the Model product. To learn more about prediction-level metrics on model runs, see [Filters](/horizon/guides/filtering-and-sorting).
  * You can now invoke Foundry models from within a project to generate pre-labels. In the project overview, you'll see "suggested models" to help you select the best model for generating predictions. This feature is in beta.
  * The new Audit trail feature provides a way for you to view all operations (create, update, and delete) on an annotation. This feature makes monitoring changes to annotations over time easier for auditing purposes.
  * The new enhanced video player (beta) in Catalog offers new capabilities to help you curate, evaluate, and visualize video data. These new capabilities include basic play/pause controls, support for predictions, pre-labels, ground truth (bounding boxes, polylines, and points), and support for comparing model runs.
  * If you export from an image or video labeling project that contains data rows labeled with segmentation masks, you will see a new property in the export file called composite\_mask. Composite masks are a grouping of all segmentation masks on a label. To learn more, see [Export image annotations](/horizon/developer-guides/export-image-annotations#mask).
  * If you export from a model run that contains segmentation mask predictions, you will see a new property in the export file called composite\_mask. To learn more, see [Export image annotations](/horizon/developer-guides/export-image-annotations#mask).
  * You can now have a workflow filter based on consensus average. For example, you can use this filter to send data rows with a consensus score over 80% straight to "Done". To do this, create a new workflow step and select consensus average from the workflow filters.
  * Boost express makes it easier for you to request help labeling data. Once you've set up a project, you can request up to fifteen labelers to work on your data. See our docs on [Boost express](/horizon/guides/labeling-services) to learn more.
  * The billing tab in your Workspace settings now has a button that enables you to easily unsubscribe from Foundry.

  *Changed*

  * When you select a project in Annotate, you will see a new project overview UI that makes navigating your project easier.
  * We have disabled the ability to download videos from the preview mode for all users.
  * When you are in a project, you will now see a "Start" button with a dropdown menu containing options for labeling, reviewing, and any other tasks you have configured for your project.

  *Fixed*

  * Google SSO users that have a profile picture on their account can now log in as expected.
  * The issue causing large numbers of data rows to be stuck in the “To Label” task has been fixed.
  * The Enterprise dashboard now displays the correct information for your organization.
  * In the Data row details panel, the row data can no longer be displayed with the security token.

  *Future*

  * Starting in mid-April, we will sunset export v1 on a rolling basis. To view the sunset date for your tier, see the [Export v1 to v2 migration guide](/horizon/guides/migration-guide-export-v1-to-export-v2).

  ## Python SDK

  The latest version of our Python SDK is v3.65.0. See our [full changelog on GitHub](https://github.com/Labelbox/labelbox-python/blob/master/CHANGELOG.md#version-3650-2024-03-05) for more details on what was added recently.

  ### Version 3.65.0 (2024-03-05)

  *Notes*

  * Rerelease of 3.64.0

  ### Version 3.64.0 (2024-02-29)

  *Added*

  * `Client.get_catalog` Add catalog schema class. Catalog exports can now be made without creating a slice first
  * `last_activity_at` filter added to export\_v2, allowing users to specify a datetime window without a slice

  *Removed*

  * Review related WebhookDataSource topics

  *Notebooks*

  * Added get\_catalog notebook
  * Update custom metrics notebook
  * Update notebooks for video and image annotation import

  ### Version 3.63.0 (2024-02-19)

  *Added*

  * Ability for users to install and use sdk with pydantic v.2. *while still maintaining support for pydantic v1.*
  * `ModelRun` `export()` and `export_v2()` add model\_run\_details to support splits

  *Notebooks*

  * Add composite mask notebook

  ### Version 3.62.0 (2024-02-12)

  *Added*

  * Support custom metrics for predictions (all applicable annotation classes)
  * `FoundryClient.run_app` Add data\_row identifier validation for running foundry app
  * `Client.get_error_status_code` Default to 500 error if a server error is unparseable instead of throwing an exception

  *Updated*

  * `DataRowMetadata, DataRowMetadataBatchResponse, _UpsertBatchDataRowMetadata` Make data\_row\_id and global\_key optional in all schema types

  *Fixed*

  * `ExportTask.__str__` Fix returned type in ExportTask instance representation

  *Removed*

  * `Project.upsert_review_queue`

  *Notebooks*

  * Update notebooks to new export methods
  * Add model slice notebook
  * Added support for annotation import with img bytes Read More
</Update>

<Update label="February 5, 2024">
  **Release notes**

  ## App

  *Added*

  * When you select a project in Annotate, you will now see a count for “Skipped” data rows in the project overview.
  * The text search filter in Catalog now supports text attachments.
  * The new Cluster view (beta) in Catalog is a visual tool that allows you to explore relationships between data rows, identify edge cases and outliers, select for pre-labeling or human review, and quickly classify large datasets in bulk. Read our docs on [Cluster view (beta)](/horizon/guides/cluster-view) to learn more.
  * The new Cloud bucket sync (beta) provides a simple way for you to sync the data in your cloud buckets to Labelbox (supported for AWS S3 buckets, Google Cloud Storage, and Microsoft Azure Blob Storage). You must have IAM delegated access configured in order to use the cloud bucket sync (beta) in Catalog.
  * Foundry now has the following improvements:
    * GPT4v will now support PDF files
    * Gemini Pro Vision supports video classification use cases
    * Improved speed for preview mode for GPT-like models and llava\llama
    * Increased the speed on the submit job mode

  *Changed*

  * We removed the technical limitation that prevented customers from exporting over 700K data rows at a time. Read our docs on [streamable exports](/horizon/developer-guides/export-overview#streamable-exports) and [Limits](/horizon/guides/limits#export) to learn more about the new export limits.
  * Our new flat-rate LBU pricing system makes understanding your usage in Labelbox easier. To learn more about the recent LBU pricing changes, see our [Manage account and billing](/horizon/guides/billing) docs.
  * When you select a project in Annotate, you will see a new project overview UI that shows labeling progress, workflow tasks, and other key configurations for your project.
  * We made some improvements to increase the reliability of routing data rows within Annotate workflows. This improvement mostly helps large-scale organizations that rely heavily on review workflows and workflow history for their operations.

  *Fixed*

  * When you update the metadata field on a data row, the change should be reflected in the “Last activity” filter in Catalog.
  * Creating multiple relationships between points and other vector annotations in the editor now works as expected.
  * The issue where tracked bounding boxes appear outside the editor canvas has been fixed.
  * The issue of the editor not allowing you to unhide a single annotation when the whole annotation group is hidden has been fixed. You can now hide/unhide individual annotations as expected.
  * The hotkeys for zooming in and out of an asset in the editor have been fixed.
  * Now, when a user navigates back to a skipped data row in the editor, the user can’t navigate to the next asset without filling in all required fields or discarding the changes.
  * The sequence of undoing/redoing annotations and then creating new annotations now works as expected.

  *Future*

  * Starting in April, we will sunset export v1 on a rolling basis. To view the sunset date for your tier, see the [Export v1 to v2 migration guide](/horizon/guides/migration-guide-export-v1-to-export-v2).

  # Python SDK

  The latest version of our Python SDK is v3.61.2. See our [full changelog on GitHub](https://github.com/Labelbox/labelbox-python/blob/master/CHANGELOG.md#version-3612-2024-01-29) for more details on what was added recently.

  ### Version 3.61.2 (2024-01-29)

  *Added*

  * `ModelSlice.get_data_row_identifiers` for Foundry data rows

  *Fixed*

  * `ModelSlice.get_data_row_identifiers` scoping by model run id

  ### Version 3.61.1 (2024-01-25)

  *Fixed*

  * Removed export API limit (5000)

  ### Version 3.61.0 (2024-01-22)

  *Added*

  * `ModelSlice.get_data_row_identifiers`
    * Fetches all data row ids and global keys for the model slice
    * NOTE Foundry model slices are note supported yet

  *Updated*

  * Updated exports v1 deprecation date to April 30th, 2024
  * Remove `streamable` param from export\_v2 methods

  ### Version 3.60.0 (2024-01-17)

  *Added*

  * Get resource tags from a project
  * Method to CatalogSlice to get data row identifiers (both uids and global keys)
  * Added deprecation notice for the `upsert_review_queue` method in project

  *Notebooks*

  * Update notebook for Project `move_data_rows_to_task_queue`
  * Added notebook for model foundry
  * Added notebook for migrating from Exports V1 to V2

  ### Version 3.59.0 (2024-01-05)

  *Added*

  * Support `set_labeling_parameter_overrides` for global keys
  * Support `bulk_delete` of data row metadata for global keys
  * Support `bulk_export` of data row metadata for global keys

  *Fixed*

  * Stop overwriting class annotations on prediction upload
  * Prevent users from uploading video annotations over the API limit (5000)
  * Make description optional for foundry app

  *Notebooks*

  * Update notebooks for Project `set_labeling_parameter_overrides` add support for global keys

  Read More
</Update>

<Update label="January 4, 2024">
  **Release notes**

  ## App

  *Added*

  * Foundry is now available to all users. Foundry enables you to use foundational models to predict annotations for your data and to create model runs to compare and diagnose the behavior of different models against your data and requirements. To learn more, read our [docs](/horizon/guides/foundry).
  * In the Catalog, there is now an easy way to send predictions straight to Annotate. Once you’ve generated predictions with Foundry, you can send them to Annotate either as prelabels or as labels for human review, either as prelabels or as annotations associated with a specific labeling step. To learn more, read these [docs](/horizon/guides/foundry-annotate-predictions).
  * Catalog similarity search now works on up to 1 billion data rows at a time.
  * We've updated our Create dataset workflow to highlight the following import options: Python SDK script, Cloud buckets integration, Data warehouse sync, or local file upload. Go to Catalog and select +New to see the new import flow.

  *Fixed*

  * When you navigate to Model --> Create --> Experiment, the Experiment page now opens as expected.
  * For ontologies containing required radio and/or checklist per message classifications, if a user attempts to remove the classification answer, they will be shown an error message and the classification value will be restored.
  * In the Data Rows tab, if a user inserts a value for Media attribute --> Attribute value --> Duration, the filter will now be applied correctly and the correct results will be shown.
  * Undo/redo behavior in the editor is limited to the current asset only.
  * When an Issue has resolved status, the Resolve button now has “Reopen” text instead, as it reopens a resolved Issue.

  ## Python SDK

  The latest version of our Python SDK is v3.58.1. See our [full changelog](https://github.com/Labelbox/labelbox-python/blob/master/CHANGELOG.md#version-3581-2023-12-15) in GitHub for more details on what was added recently.

  ### Version 3.58.1 (2023-12-15)

  *Added*

  * Support to export all projects and all model runs to `export_v2` for a `dataset` and a `slice`

  *Notebooks*

  * Update exports v2 notebook to include methods that return `ExportTask`

  ### Version 3.58.0 (2023-12-11)

  *Added*

  * `ontology_id` to the model app instantiation
  * LLM data generation label types
  * `run_foundry_app` to support running model foundry apps
  * Two methods for sending data rows to any workflow task in a project, that can also include predictions from a model run, or annotations from a different project

  *Fixed*

  * Documentation index for identifiables

  *Removed*

  * Project.datasets and Datasets.projects methods as they have been deprecated

  *Notebooks*

  * Added note books for Human labeling(GT/MAL/MEA) + data generation (GT/MAL)
  * Remove relationship annotations from text and conversational imports Read More
</Update>


# Subscribe to the Labelbox release notes
Source: https://docs.labelbox.com/horizon/release-notes/subscribe



To subscribe to the Labelbox release notes, please fill out the [subscription form](https://learn.labelbox.com/releasenotessubscription/).


# Python Tutorials
Source: https://docs.labelbox.com/horizon/tutorials



## Basics

| Notebook | Google Colab | GitHub |
| - | - | - |
| Quick start | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/basics/quick_start.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/basics/quick_start.ipynb) |
| Basics | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/basics/basics.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/basics/basics.ipynb) |
| Data rows | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/basics/data_rows.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/basics/data_rows.ipynb) |
| Data row metadata | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/basics/data_row_metadata.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/basics/data_row_metadata.ipynb) |
| Batches | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/basics/batches.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/basics/batches.ipynb) |
| Projects | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/basics/projects.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/basics/projects.ipynb) |
| Ontologies | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/basics/ontologies.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/basics/ontologies.ipynb) |
| User management | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/basics/user_management.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/basics/user_management.ipynb) |
| Custom embeddings | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/basics/custom_embeddings.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/basics/custom_embeddings.ipynb) |

## Exports

| Notebook | Google Colab | GitHub |
| - | - | - |
| Export data | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/exports/export_data.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/exports/export_data.ipynb) |
| Exporting to CSV | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/exports/exporting_to_csv.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/exports/exporting_to_csv.ipynb) |
| Export v1 to v2 migration support | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/exports/export_v1_to_v2_migration_support.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/exports/export_v1_to_v2_migration_support.ipynb) |
| Composite mask export | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/exports/composite_mask_export.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/exports/composite_mask_export.ipynb) |

## Project configuration

| Notebook | Google Colab | GitHub |
| - | - | - |
| Multimodal chat project | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/project_configuration/multimodal_chat_project.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/project_configuration/multimodal_chat_project.ipynb) |
| Prompt response projects | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/project_configuration/prompt_response_projects.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/project_configuration/prompt_response_projects.ipynb) |
| Webhooks | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/project_configuration/webhooks.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/project_configuration/webhooks.ipynb) |
| Project setup | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/project_configuration/project_setup.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/project_configuration/project_setup.ipynb) |
| Queue management | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/project_configuration/queue_management.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/project_configuration/queue_management.ipynb) |

## Annotation import

| Notebook | Google Colab | GitHub |
| - | - | - |
| LLM data generation | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/annotation_import/LLM_data_generation.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/annotation_import/LLM_data_generation.ipynb) |
| Tiled | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/annotation_import/tiled.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/annotation_import/tiled.ipynb) |
| PDF | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/annotation_import/pdf.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/annotation_import/pdf.ipynb) |
| Audio | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/annotation_import/audio.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/annotation_import/audio.ipynb) |
| Offline multimodal chat evaluation | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/annotation_import/offline_multimodal_chat_evaluation.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/annotation_import/offline_multimodal_chat_evaluation.ipynb) |
| Text | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/annotation_import/text.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/annotation_import/text.ipynb) |
| Conversational LLM | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/annotation_import/conversational_LLM.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/annotation_import/conversational_LLM.ipynb) |
| HTML | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/annotation_import/html.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/annotation_import/html.ipynb) |
| Prompt response | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/annotation_import/prompt_response.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/annotation_import/prompt_response.ipynb) |
| Image | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/annotation_import/image.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/annotation_import/image.ipynb) |
| Video | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/annotation_import/video.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/annotation_import/video.ipynb) |
| Conversational | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/annotation_import/conversational.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/annotation_import/conversational.ipynb) |

## Integrations

| Notebook | Google Colab | GitHub |
| - | - | - |
| Langchain | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/integrations/langchain/langchain.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/integrations/langchain/langchain.ipynb) |
| Huggingface custom embeddings | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/integrations/huggingface/huggingface_custom_embeddings.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/integrations/huggingface/huggingface_custom_embeddings.ipynb) |
| Import YOLOv8 annotations | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/integrations/yolo/import_yolov8_annotations.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/integrations/yolo/import_yolov8_annotations.ipynb) |
| Meta SAM | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/integrations/sam/meta_sam.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/integrations/sam/meta_sam.ipynb) |
| Meta SAM video | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/integrations/sam/meta_sam_video.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/integrations/sam/meta_sam_video.ipynb) |

## Model experiments

| Notebook | Google Colab | GitHub |
| - | - | - |
| Custom metrics demo | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/model_experiments/custom_metrics_demo.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/model_experiments/custom_metrics_demo.ipynb) |
| Model slices | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/model_experiments/model_slices.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/model_experiments/model_slices.ipynb) |
| Model predictions to project | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/model_experiments/model_predictions_to_project.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/model_experiments/model_predictions_to_project.ipynb) |
| Custom metrics basics | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/model_experiments/custom_metrics_basics.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/model_experiments/custom_metrics_basics.ipynb) |

## Prediction upload

| Notebook | Google Colab | GitHub |
| - | - | - |
| PDF predictions | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/prediction_upload/pdf_predictions.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/prediction_upload/pdf_predictions.ipynb) |
| Text predictions | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/prediction_upload/text_predictions.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/prediction_upload/text_predictions.ipynb) |
| Geospatial predictions | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/prediction_upload/geospatial_predictions.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/prediction_upload/geospatial_predictions.ipynb) |
| Conversational predictions | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/prediction_upload/conversational_predictions.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/prediction_upload/conversational_predictions.ipynb) |
| Conversational LLM predictions | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/prediction_upload/conversational_LLM_predictions.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/prediction_upload/conversational_LLM_predictions.ipynb) |
| Video predictions | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/prediction_upload/video_predictions.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/prediction_upload/video_predictions.ipynb) |
| Image predictions | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/prediction_upload/image_predictions.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/prediction_upload/image_predictions.ipynb) |
| HTML predictions | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/prediction_upload/html_predictions.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/prediction_upload/html_predictions.ipynb) |

## Foundry

| Notebook | Google Colab | GitHub |
| - | - | - |
| Object detection | [Open in Colab](https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/foundry/object_detection.ipynb) | [Open in GitHub](https://github.com/Labelbox/labelbox-notebooks/tree/main/foundry/object_detection.ipynb) |


# Agent file history
Source: https://docs.labelbox.com/recursion/agent-file-history

Let an agent's sessions search and read the deliverables its earlier sessions saved, by path, metadata, date, and text.

An agent's file history is every deliverable its sessions saved: files written with `write_artifact` or placed in the deliverables folder, including those saved by the sessions' subagents and teammates. Turn on **File history** for an agent, and each new session can search that history and read what it finds, so work from earlier runs is not lost or redone.

<Note>
  You need the Developer or Admin role to change an agent. The User role can view an agent's Files tab. See [Organizations and roles](/recursion/organizations-and-roles). Deliverables are what a history holds. See [Deliverables and artifacts](/recursion/artifacts).
</Note>

## Turn on file history

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Agents** and open the agent.
    2. On **Configuration**, under **Tools**, select **File history**.
    3. Click **Save**.
  </Tab>

  <Tab title="cURL">
    `fileHistoryEnabled` is one more field of the definition. Send it in `createAgent`, or in `createAgentVersion` together with every other field and the current `base_agent_version_id`, as in [Update an agent](/recursion/agents#update-an-agent). A field left out of a new version is cleared.

    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/agents/5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55/versions' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      -d '{
        "base_agent_version_id": "e82b5d19-7c40-4a6f-91d3-3f5a2c8b0e64",
        "name": "Quarterly analyst",
        "model": "<model-id>",
        "system": "You write the quarterly finance report.",
        "fileHistoryEnabled": true
      }'
    ```
  </Tab>
</Tabs>

Sessions started after the change get the tools. Sessions already running keep the tools they started with until you [move them to the new version](/recursion/sessions#move-a-session-to-another-version), but a search fails once the setting is off.

## Make deliverables easy to find

Metadata makes a search exact. Ask the agent to label what it saves:

| Where | Pattern |
| - | - |
| Agent system prompt | "Save each report with `write_artifact` and metadata `kind`, `quarter`, and `status`." |
| Task message | "Save the review as `reviews/acme.md` with metadata `customer_id: acme` and `status: draft`." |

A search can then ask for `kind` equal to `report` and `quarter` equal to `Q3` and get exactly those files. See [Label deliverables with metadata](/recursion/artifacts#label-deliverables-with-metadata).

## Browse an agent's files

Search and preview the deliverables an agent's sessions have saved.

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Agents** and open the agent.
    2. Click **Files**.
    3. Search file contents, filter by path or metadata, and click a file to preview or download it.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/agents/5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55/files/search' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      -d '{
        "filter": {
          "pathPrefix": "reports/",
          "metadata": [
            {"key": "kind", "op": "eq", "value": "report"},
            {"key": "quarter", "op": "in", "value": ["Q2", "Q3"]}
          ]
        },
        "text": "revenue",
        "sort": "relevance",
        "limit": 20
      }'
    ```
  </Tab>
</Tabs>

The response lists the matching files with the session that saved each one:

```json theme={"theme":"css-variables"}
{
  "items": [
    {
      "fileId": "8f966221-d636-4cd2-b905-ade8341e8844",
      "path": "reports/q3.md",
      "filename": "q3.md",
      "mediaType": "text/markdown",
      "byteSize": 4210,
      "sha256": "…",
      "metadata": {"kind": "report", "quarter": "Q3"},
      "ownerAgentId": "5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55",
      "rootSessionId": "b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38",
      "createdAt": "2026-09-25T17:21:40Z",
      "contentState": "indexed",
      "snippet": "…Revenue grew twelve percent over the quarter…"
    }
  ],
  "nextCursor": null,
  "total": 1,
  "asOf": "2026-09-28T02:40:00Z",
  "coverage": {"metadata": "complete", "content": "complete"}
}
```

`GET /managed-agents/v1/agents/{agentId}/files/{fileId}` returns one file of the history.

## What a session can do

The session's agent uses two tools. You can also name them in prompts, for example "Search your file history for last quarter's report before writing this one."

| Tool | What it does |
| - | - |
| `search_agent_files` | Finds files by `pathPrefix`, `filenamePrefix`, `mediaTypes`, `rootSessionIds`, a `createdAt` range, metadata conditions, and `text` found in the file. Without conditions it lists the newest files. |
| `read_agent_file` | Reads one file a search returned, by `file_id`. Text comes back in windows; images are shown. |

<Accordion title="Metadata conditions">
  Each condition names a top-level metadata `key`, an `op`, and usually a `value`. All conditions must hold.

  | `op` | Matches |
  | - | - |
  | `eq` | The stored value equals `value`, which is required. `null` matches a stored `null`, not a missing key. |
  | `in` | The stored value equals any value in an array. |
  | `exists` | The key is present, whatever its value, `null` included. |
  | `gt`, `gte`, `lt`, `lte` | Numbers compared with numbers, and strings with strings. |

  Values are never converted between types: `"3"` does not match `3`, and `true` does not match `"true"`. Nested objects and arrays are stored and returned but are not compared.
</Accordion>

<Accordion title="Text search">
  `text` finds words in the contents of text files: Markdown, plain text, JSON, CSV, YAML, code, and similar. All words must appear; put a phrase in quotes, write `OR` between words to match either, and prefix a word with `-` to exclude it. Results can be ranked by `relevance` instead of newest first, and each result carries a short `snippet`.

  Some files can't be text-searched in full: images, PDFs, and other binary formats, and text beyond the first 1 MiB of a file. The result's `coverage` says how many candidates were not fully searched, so an empty result is never mistaken for a complete one. Set `strictContent` to fail instead when coverage is partial.
</Accordion>

<Accordion title="Result pages">
  Results come in pages of up to 100 (20 by default, 50 for a session's tool), newest first unless ranked by relevance. Every page reads the same snapshot, named by `asOf`: files saved after the first page are not added part-way through. Pass `nextCursor` back with an otherwise unchanged request for the next page. `total` counts the snapshot's matches under current access, so a file deleted between pages lowers it. A cursor expires after an hour.
</Accordion>

## How it works

* **The history belongs to the agent.** A deliverable is kept in the history of the agent the session was started from, including deliverables saved by its subagents, teammates, and other agents it delegated to. Those helpers search the same history while they work in that session. A session started from another agent never sees it.
* **Sessions search on demand.** A session is told the history exists, but no files are loaded into it up front. The agent decides when a search is worth making.
* **Earlier sessions only.** A session sees deliverables saved before it started, not its own work in progress.
* **Current access is checked every time.** A deleted file, a deleted session's deliverables, and deliverables of sessions with restricted access are left out. Turning **File history** off stops the next search of every running session.

## What can go wrong

The most common problems:

* **A session has no `search_agent_files` tool.** **File history** was off when the session started. Turn it on, then move the session to the new version or start a new session.
* **A file saved in this session is not found.** A session sees only files saved before it started. Use `read_artifact` for the session's own deliverables.
* **`404` for a file.** The file is not in this agent's history: another agent's file, a deleted file, or a deleted or restricted session. Search the history for the file you want.

<Accordion title="Every file history error">
  | Symptom or message | Cause | Fix |
  | - | - | - |
  | A session has no `search_agent_files` tool | **File history** was off when the session started. | Turn it on, then move the session to the new version or start a new session. |
  | "file history was turned off for this agent" | The setting was turned off while the session ran. | Turn it back on; the session's next search works. |
  | A file saved in this session is not found | A session sees only files saved before it started. | Use `read_artifact` for the session's own deliverables. |
  | `400` on a `filter` field, `text`, `sort`, or `limit` | A condition is malformed or repeated, `eq` has no `value`, an array filter is empty or repeats a value, or `path` is used as a metadata key. | Fix the named field; filter paths with `pathPrefix`. |
  | `400` on `cursor` | The cursor belongs to a different request or has expired. | Repeat the search without a cursor. |
  | `422` `content_index_incomplete` | `strictContent` was set and some candidates could not be text-searched. | Search without `strictContent`, or narrow the search to text files. |
  | `404` for a file | The file is not in this agent's history: another agent's file, a deleted file, or a deleted or restricted session. | Search the history for the file you want. |
</Accordion>

## Next steps

<CardGroup>
  <Card title="Deliverables and artifacts" href="/recursion/artifacts">
    Save deliverables and label them with metadata.
  </Card>

  <Card title="Referenced sessions" href="/recursion/referenced-sessions">
    Read specific earlier sessions' transcripts and deliverables.
  </Card>
</CardGroup>


# Agents
Source: https://docs.labelbox.com/recursion/agents

Create an agent, publish immutable versions of it, tag it, and delete it.

An agent is a reusable definition of how a model works: the model, the system prompt, the tools, the skills, and the credentials it can use. You create an agent once and start any number of [sessions](/recursion/sessions) from it.

Every change to an agent's definition creates a new agent version. A version is immutable, and a session keeps its version until you [move it to another](/recursion/sessions#move-a-session-to-another-version), so an edit alone never changes a session that is already running.

<Note>
  Creating, changing, or deleting agents needs the organization developer or admin role. The organization user role can view agents. See [Organizations and roles](/recursion/organizations-and-roles). For the API, create a key on [API keys](/recursion/api-keys) and export it as `RECURSION_API_KEY`. Pick a model id from `listModels`: your organization can use only the models that list returns. In the API examples, replace `<model-id>` with a `modelId` returned by `listModels`.
</Note>

## Create an agent

One request saves the agent and its first version. Start with a short prompt and only the tools the task needs. You can add more in a later version.

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Agents**.
    2. Click **Create agent**.
    3. Under **Choose a starting point**, click **Blank**, or pick a template to start from its definition.
    4. Under **General**, fill in **Name**, **Model**, and **System prompt**.
    5. Click **Create agent**.

    Some fields, such as `metadata`, have no control in the console form. Set them with the API.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/agents' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      -d '{
        "name": "Release notes writer",
        "description": "Drafts release notes from merged pull requests.",
        "model": "<model-id>",
        "system": "You write release notes. Read the merged pull requests, group the changes by product area, and write one plain sentence for each change."
      }'
    ```
  </Tab>
</Tabs>

A `200` response is the agent, resolved to its first version. Some fields are left out here.

```json theme={"theme":"css-variables"}
{
  "agent_id": "5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55",
  "latest_agent_version_id": "c1a94e07-2f6b-4d18-b3a5-0e7d8c6f4a21",
  "organization_id": "org_01a08a705220724f9a2fe1bcd8638c9d",
  "name": "Release notes writer",
  "description": "Drafts release notes from merged pull requests.",
  "system": "You write release notes. Read the merged pull requests, group the changes by product area, and write one plain sentence for each change.",
  "web_search_enabled": true,
  "mcp_servers": [],
  "skills": [],
  "default_vault_ids": [],
  "default_credential_refs": [],
  "disabled_integration_mcp_providers": [],
  "tags": [],
  "created_at": "2026-09-17T09:12:44Z",
  "updated_at": "2026-09-17T09:12:44Z"
}
```

Keep `agent_id` to [start sessions](/recursion/sessions) and `latest_agent_version_id` to update the agent.

<Accordion title="Agent fields">
  You send these fields when you create an agent and when you create a new version.

  | Field | Description |
  | - | - |
  | `name` | Required. The label shown in lists, at most 256 characters. |
  | `model` | Required. A model id from `listModels`. |
  | `system` | Required. The system prompt added to every conversation. |
  | `description` | A note for people. The model never sees it. |
  | `web_search_enabled` | Whether sessions get the `web_search` tool. Defaults to `true`. See [Tools](/recursion/tools). |
  | `mcp_servers` | Remote [MCP servers](/recursion/mcp-servers) the agent can call, each as `{ "name", "url" }`. |
  | `skills` | [Skills](/recursion/skills) and skill groups, at most 100 entries. The list is ordered; see [Attach skills to an agent](/recursion/skills#attach-skills-to-an-agent). |
  | `built_in_integrations` | Apps the agent may use, each as `{ "connection_id" }` naming an active connection from [Integrations](/recursion/integrations). At most 25. Sessions get each app's tool allow-list. See [Grant apps to an agent](/recursion/integrations#grant-apps-to-an-agent). |
  | `default_vault_ids` | [Vaults](/recursion/vaults) a session receives when its start request names none. Each vault must exist. |
  | `multiagent` | The agents this agent can delegate to, or the team it leads. See [Multi-agent](/recursion/multi-agent) and [Teams](/recursion/teams). |
  | `metadata` | Your own JSON object. The API returns it unchanged. |
  | `model_config` | Inference settings for every turn: `max_tokens`, `temperature`, `top_p`, `reasoning_effort`, and `provider_params`. Send `{}` or omit the field for the model's defaults. `null` is rejected. `reasoning_effort` must be one of the model's `supportedReasoningEfforts` from `listModels`. |
  | `default_credential_refs` | The credentials from the default vaults that the agent receives. Name every credential the agent should get, across all its default vaults. If you omit it, the API selects every credential those vaults hold when you save the agent and stores that selection with the version. A credential added to the vault later does not reach the agent until you save the agent again with it selected. An empty list with a non-empty `default_vault_ids` is rejected. |
  | `disabled_integration_mcp_providers` | Native integration providers whose MCP tools sessions do not receive, such as `github` or `jira`. It doesn't affect built-in integrations. On a new agent it defaults to an empty list, so sessions get the MCP tools of every connection the agent is granted. See [Tools](/recursion/tools#withhold-integration-tools). |
  | `default_rubric` | A Markdown rubric; each list item is one criterion. Every session of the agent that starts with a message is graded against it by default. See [Outcome rubrics](/recursion/outcomes). |
  | `max_concurrent_sessions` | From 1 to 10000. Sessions over the cap wait with `execution_state` `queued`. Subagent sessions do not count. The current value on the agent applies, not the value in a session's version. |

  The API sets `agent_id`, `latest_agent_version_id`, `organization_id`, `tags`, `created_at`, and `updated_at`. [Tags](#tag-agents) live outside versions, so a tag change never creates a version.
</Accordion>

<Accordion title="Retry a create safely">
  If a create request times out, you cannot tell whether the agent exists. To make the retry safe, send an `Idempotency-Key` header with 1 to 256 visible ASCII characters. The console adds a key to every **Create agent** click, so a double click or a retried save creates one agent.

  ```bash theme={"theme":"css-variables"}
  curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/agents' \
    -H "Authorization: Bearer $RECURSION_API_KEY" \
    -H 'Idempotency-Key: release-notes-writer-2026-09-17' \
    -H 'Content-Type: application/json' \
    -d '{"name":"Release notes writer","model":"<model-id>","system":"You write release notes."}'
  ```

  | You send | Result |
  | - | - |
  | The same key and the same request | The original response. No second agent is created. |
  | The same key with any different request identity bytes | `409 idempotency_conflict` |
  | The same key while the first request still runs | `409 idempotency_in_progress`. Wait for the time in `Retry-After`, then retry. |

  The API compares the exact method, escaped path, raw query, and raw body bytes, so keep the request target and body bytes unchanged. A completed response is kept for about 24 hours. Your organization shares one key namespace across all keyed requests, so put the resource name in the key. See [Idempotent mutations](/recursion/api#idempotent-mutations) for the complete retry contract.
</Accordion>

## Update an agent

You cannot edit an agent's definition in place. To change it, create a new version with `createAgentVersion`. New sessions use the new version. Running sessions keep theirs until you [move them](/recursion/sessions#move-a-session-to-another-version): select them on the agent's **Sessions** tab in the console, or send one request for each session.

<Tabs>
  <Tab title="Console">
    1. Open the agent and stay on the **Configuration** tab.
    2. Change the fields.
    3. Click **Save new version**.
    4. If someone else saved first, the console shows a conflict notice. Click **Load the newer version**, then make your change again.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/agents/5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55/versions' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      -d '{
        "base_agent_version_id": "c1a94e07-2f6b-4d18-b3a5-0e7d8c6f4a21",
        "name": "Release notes writer",
        "model": "<model-id>",
        "system": "You write release notes. Group the changes by product area. Put breaking changes first.",
        "disabled_integration_mcp_providers": ["jira"]
      }'
    ```
  </Tab>
</Tabs>

A `201` response is the agent with a new `latest_agent_version_id`. Use that id as `base_agent_version_id` in your next update. Only `latest_agent_version_id` is guaranteed to point at the version you just wrote, so read that version back, as in [Review earlier versions](#review-earlier-versions), when you need its exact content.

### Update rules

* **Send the full definition.** A version is complete in itself. A field you leave out is cleared, not copied from the earlier version. This covers `skills`, `mcp_servers`, `built_in_integrations`, `default_vault_ids`, and every other field. The console sends the full definition for you.
* **Send `disabled_integration_mcp_providers` again.** Leaving it out means an empty list, which turns every service's MCP tools on.
* **Set `base_agent_version_id` to the version you read.** It must equal the agent's current `latest_agent_version_id`. If another writer created a version first, the API returns `409 revision_conflict` and saves nothing, so you cannot overwrite a change you have not seen.

After a `revision_conflict`, get the agent again, apply your change to the new definition, and send the request with the new `latest_agent_version_id`. Branch on `code`, because the `message` text can change.

<Accordion title="The revision conflict response">
  ```json theme={"theme":"css-variables"}
  {
    "code": "revision_conflict",
    "message": "source revision conflict: base_agent_version_id is not the agent's current version",
    "details": {
      "field": "base_agent_version_id",
      "requestId": "0b6f2c1d9a4e4b7f8c3d5e6a7b8c9d0e",
      "retryable": false
    }
  }
  ```
</Accordion>

## Find an agent

List the agents in your organization, or get one by id. The list returns every agent in one response, newest first, without pages. To filter, pass `tag_ids`.

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Agents**.
    2. To narrow the list, type a name, id, model, or tag in the search box, or pick tags in the **Tags** filter.
    3. Click an agent. The **Configuration** tab shows the current version.
  </Tab>

  <Tab title="cURL — list">
    ```bash theme={"theme":"css-variables"}
    curl 'https://api.recursion.labelbox.com/managed-agents/v1/agents?tag_ids=a41d7c02-5e9b-4f36-8a10-2b7c9d3e6f08' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>

  <Tab title="cURL — get">
    ```bash theme={"theme":"css-variables"}
    curl 'https://api.recursion.labelbox.com/managed-agents/v1/agents/5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

```json theme={"theme":"css-variables"}
{
  "agents": [
    {
      "agent_id": "5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55",
      "latest_agent_version_id": "c1a94e07-2f6b-4d18-b3a5-0e7d8c6f4a21",
      "name": "Release notes writer",
      "tags": [
        {
          "tag_id": "a41d7c02-5e9b-4f36-8a10-2b7c9d3e6f08",
          "label": "release",
          "color": "#2F80ED"
        }
      ]
    }
  ]
}
```

Each entry is a full agent. The sample shows a few fields. A get returns one agent, in the same shape as the create response, and always shows the latest version.

<Accordion title="List and get details">
  * Deleted agents are left out of the list.
  * `tag_ids` is a comma-separated string of at most 32 tag ids. An agent must carry every tag you name. A `tag_ids` entry that does not exist in your organization returns `404 not_found`.
  * When your organization has no agents, `agents` can be `null`; treat it as an empty list.
  * Getting a deleted agent, or one from another organization, returns `404 not_found`.
</Accordion>

## Review earlier versions

Versions are numbered from 1. Every session reports the version it runs as `agent_version_id`. A session that was [moved to another version](/recursion/sessions#move-a-session-to-another-version) did its earlier work on a different one, and its events record each move. To see the exact prompt and tools of a version, get that version.

<Tabs>
  <Tab title="Console">
    1. Open the agent and stay on the **Configuration** tab.
    2. Click the **Version** picker. The latest version is marked **current**.
    3. Pick a version to view it. Earlier versions are read-only.
    4. To make an earlier version the latest, click **Restore this version**. The console saves its definition as a new version.
  </Tab>

  <Tab title="cURL — list">
    ```bash theme={"theme":"css-variables"}
    curl 'https://api.recursion.labelbox.com/managed-agents/v1/agents/5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55/versions' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>

  <Tab title="cURL — get">
    ```bash theme={"theme":"css-variables"}
    curl 'https://api.recursion.labelbox.com/managed-agents/v1/agents/5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55/versions/c1a94e07-2f6b-4d18-b3a5-0e7d8c6f4a21' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

The list returns every version, newest first, without pages:

```json theme={"theme":"css-variables"}
{
  "agent_versions": [
    {
      "agent_version_id": "e82b5d19-7c40-4a6f-91d3-3f5a2c8b0e64",
      "agent_id": "5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55",
      "version_number": 2,
      "name": "Release notes writer",
      "system": "You write release notes. Group the changes by product area. Put breaking changes first.",
      "created_by": "usr_01a0867d1b0875be9f9a9938ffb90971",
      "created_at": "2026-09-17T10:03:12Z"
    },
    {
      "agent_version_id": "c1a94e07-2f6b-4d18-b3a5-0e7d8c6f4a21",
      "agent_id": "5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55",
      "version_number": 1,
      "name": "Release notes writer",
      "system": "You write release notes. Read the merged pull requests, group the changes by product area, and write one plain sentence for each change.",
      "created_at": "2026-09-17T09:12:44Z"
    }
  ]
}
```

Each entry holds the full definition at that version. The sample shows part of each entry. A get returns one entry from this list.

<Accordion title="Version details">
  * `created_by` is the user who saved the version and is empty when an automated caller saved it.
  * The version you get must belong to the agent in the path. A version id from another agent returns `404 not_found`.
</Accordion>

## Tag agents

Tags are colored labels that you define once for your organization and apply to agents. Use them to filter the agent list. Tags are mutable and live outside versions: creating, applying, or removing a tag never creates an agent version, never changes the agent's `updated_at`, and is not recorded on sessions.

### Create a tag

A tag has a `label`, a `color` in `#RRGGBB` form, and an optional `description`.

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Agents**.
    2. Click **Manage tags**, then **Create tag**.
    3. Fill in **Label**, **Color**, and an optional **Description**.
    4. Click **Create tag**.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/tags' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      -H 'Idempotency-Key: release-tag-2026-09-28' \
      -d '{"label":"release","color":"#2F80ED","description":"Agents that ship release work."}'
    ```
  </Tab>
</Tabs>

```json theme={"theme":"css-variables"}
{
  "organization_id": "org_01a08a705220724f9a2fe1bcd8638c9d",
  "tag_id": "a41d7c02-5e9b-4f36-8a10-2b7c9d3e6f08",
  "label": "release",
  "color": "#2F80ED",
  "description": "Agents that ship release work.",
  "created_at": "2026-09-17T09:20:05Z",
  "updated_at": "2026-09-17T09:20:05Z"
}
```

Keep `tag_id` to apply the tag. A successful create returns `201` with the tag and a `Location` header for its get URL.

The optional `Idempotency-Key` lets you retry the exact same request if the response is lost; a replay returns the same tag and `Idempotency-Replayed: true`. Reusing the key with a different request returns `409 idempotency_conflict`. Without a key, check the tag list after an uncertain result before creating again.

<Accordion title="Tag rules">
  * `label` is 1 to 32 characters, and labels are unique within your organization, ignoring case. A label that another live tag already uses returns `409 conflict` on `label`.
  * `description` is at most 280 characters.
  * Your organization can have at most 200 live tags.
</Accordion>

### List, get, update, and delete tags

`listTags` returns `{ "items": [...] }` with every live tag in one response. `getTag` returns one tag, or `404 not_found` for a deleted tag or one from another organization.

`updateTag` is a partial update. Fields you omit keep their value, and an empty `description` clears it. Sending `null` for a field returns `400 invalid_request`; omit the field instead. An empty update object `{}` also returns `400 invalid_request`. Renaming or recoloring a tag changes it on every agent that carries it.

An update returns `200` with the current tag. If its response is lost, get the tag before deciding what to do next. If the requested fields already match, the update took effect. If they differ, review the current values before sending a new update; another editor may have changed the tag.

<Tabs>
  <Tab title="Console">
    1. In **Agents**, click **Manage tags**.
    2. Open the tag's row menu and click **Edit**, change the fields, and click **Save tag**.
    3. To delete, open the row menu, click **Delete**, and confirm in the **Delete tag?** dialog.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X PATCH 'https://api.recursion.labelbox.com/managed-agents/v1/tags/a41d7c02-5e9b-4f36-8a10-2b7c9d3e6f08' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      -d '{"color":"#27AE60"}'

    curl -X DELETE 'https://api.recursion.labelbox.com/managed-agents/v1/tags/a41d7c02-5e9b-4f36-8a10-2b7c9d3e6f08' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

A delete returns `204` with no body, including when you repeat it for a tag already deleted in your organization. An unknown tag or one from another organization returns `404 not_found`. The tag disappears from every agent at once, but the agents and their versions are not changed, and the label becomes free to reuse. A deleted tag cannot be restored.

### Apply tags to an agent

`applyAgentTag` adds one tag and `removeAgentTag` removes one. `replaceAgentTags` sets the whole list in one call.

<Tabs>
  <Tab title="Console">
    1. Open the agent and stay on the **Configuration** tab.
    2. In **Tags**, use **Add tag**, or remove a tag from the list.
    3. Click **Save tags**. If you also changed the definition, the button reads **Save changes** and saves both.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X PUT 'https://api.recursion.labelbox.com/managed-agents/v1/agents/5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55/tags/a41d7c02-5e9b-4f36-8a10-2b7c9d3e6f08' \
      -H "Authorization: Bearer $RECURSION_API_KEY"

    curl -X PUT 'https://api.recursion.labelbox.com/managed-agents/v1/agents/5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55/tags' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      -d '{"tag_ids":["a41d7c02-5e9b-4f36-8a10-2b7c9d3e6f08","d93b6e20-4c17-4a85-b2f9-7e1a0c5d8f36"]}'

    curl -X DELETE 'https://api.recursion.labelbox.com/managed-agents/v1/agents/5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55/tags/d93b6e20-4c17-4a85-b2f9-7e1a0c5d8f36' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

`applyAgentTag` and `replaceAgentTags` return the agent's full tag list. `removeAgentTag` returns `204` with no body, including when the tag was already removed.

```json theme={"theme":"css-variables"}
{
  "tags": [
    {
      "organization_id": "org_01a08a705220724f9a2fe1bcd8638c9d",
      "tag_id": "a41d7c02-5e9b-4f36-8a10-2b7c9d3e6f08",
      "label": "release",
      "color": "#2F80ED",
      "description": "Agents that ship release work.",
      "created_at": "2026-09-17T09:20:05Z",
      "updated_at": "2026-09-17T09:20:05Z"
    }
  ]
}
```

<Accordion title="Tag list rules">
  An agent can carry at most 32 tags. `applyAgentTag` and `removeAgentTag` are both safe to repeat. In `replaceAgentTags`, duplicates collapse, and `[]` removes every tag.
</Accordion>

## Delete an agent

A delete is a soft delete. The agent leaves lists, returns `404` on reads, and cannot start new sessions. Its versions stay stored, and the sessions that used it stay readable. A deleted agent cannot be restored.

Deleting an agent also deletes every [automation](/recursion/automations) that runs it, from the console or the API. The agent and its automations go together: if the delete is refused, none of them is deleted. Those automations could not run without the agent, and their triggers would go on holding their event sources. Sessions those automations already started stay readable. To keep an automation, point it at another agent before you delete this one.

<Tabs>
  <Tab title="Console">
    1. Open the agent, or open its row menu in **Agents**.
    2. Click **Delete**.
    3. The **Delete agent** dialog names the automations that will be deleted with the agent. Click **Delete**.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X DELETE 'https://api.recursion.labelbox.com/managed-agents/v1/agents/5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

```json theme={"theme":"css-variables"}
{ "deleted": true }
```

## What can go wrong

The most common problems:

* **`409 revision_conflict` on update.** Someone saved a version after you read the agent. Get the agent, apply your change again, and retry. See [Update rules](#update-rules).
* **`400 invalid_request` on `model`.** The model is not available to your organization. Copy an id from `listModels`; the `message` suggests valid ids.
* **`403 forbidden`.** Your role can view agents but not change them. Use an account or key with the developer or admin role.
* **`413 payload_too_large`.** The request body is over the size limit. Shorten the prompt, or move procedures into a [skill](/recursion/skills).

<Accordion title="Every agent error">
  Every error has required `code` and `message` fields and optional `details`. When present, `details.field` names the request field at fault. Use the generated **Endpoints** reference for exact alternatives and [Errors](/recursion/errors) for recovery guidance.

  | Symptom or code | Cause | Fix |
  | - | - | - |
  | `400 invalid_request` on `name`, `system`, or `max_concurrent_sessions` | A required field is missing, `name` is over 256 characters, or `max_concurrent_sessions` is outside 1 to 10000. | Read `details.field` and `message`, then correct the value. |
  | `400 invalid_request` on `model` | The model is not available to your organization. | Copy an id from `listModels`. The `message` suggests valid ids. |
  | `400 invalid_request` on `model_config` | `model_config` is `null`, or `reasoning_effort` is not one the model supports. | Send `{}` for defaults, and pick a value from the model's `supportedReasoningEfforts`. |
  | `400 invalid_request` on `disabled_integration_mcp_providers` | The list names an unsupported provider. | Use only the values listed in [Tools](/recursion/tools#withhold-integration-tools). |
  | `400 invalid_request` on `built_in_integrations` | More than 25 entries, a non-empty `tools`, or a connection that isn't an active built-in integration with at least one tool. | Read `message`. See [Grant apps to an agent](/recursion/integrations#grant-apps-to-an-agent). |
  | `400 invalid_request` on `skills[n].skill_id`, `skills[n].version`, or `skills[n].skill_group_id` | An attached skill, skill version, or skill group does not exist, is deleted, or belongs to another organization. | Check the id in that entry. See [Attach skills to an agent](/recursion/skills#attach-skills-to-an-agent). |
  | `400 invalid_request` on `tags`, `tag_id`, or `tag_ids` | Your organization already has 200 tags, an agent would carry more than 32, or `tag_ids` holds an empty id. | Delete unused tags, or remove tags from the agent. |
  | `400 invalid_json` | The body is not valid JSON, or a field has the wrong type. | Correct the body. `details.field` names the field when the type is wrong. |
  | `401 unauthorized` | The API key is missing, revoked, or expired. | Send a valid key. See [API keys](/recursion/api-keys). |
  | `403 forbidden` | Your role can view agents but not change them. | See [Organizations and roles](/recursion/organizations-and-roles). Use an account or key with the developer or admin role. |
  | `404 not_found` | The agent, version, tag, default vault, or granted integration connection does not exist, is deleted, or belongs to another organization. | Check the id. The API does not say which case applies. |
  | `409 conflict` on `label` | Another live tag uses the same label, ignoring case. | Pick another label, or reuse the existing tag. |
  | `409 revision_conflict` | `base_agent_version_id` is not the latest version. | Get the agent, apply your change again, and retry. |
  | `409 idempotency_conflict` | You reused an `Idempotency-Key` with a different request. | Use a new key for a new request. |
  | `409 idempotency_in_progress` | The first request with this key is still running. | Wait for the time in `Retry-After`, then retry. |
  | `413 payload_too_large` | The request body is over the size limit. | Shorten the prompt, or move procedures into a [skill](/recursion/skills). |
  | `429 rate_limit_exceeded` | Your organization is over its request rate. | Retry after `Retry-After` only when [the operation's retry contract permits it](/recursion/api#rate-limits). |
  | `503 idempotency_unavailable` | The API could not read or reserve the key, so the request did not run. | Retry with the same key and request bytes. |
  | `503 managed_agents_unavailable` | The service could not check the model or complete the request. | Retry only when `details.retryable` is not `false` and the operation's retry contract permits it. |
</Accordion>

Limits on agent names, skills, apps, concurrency, tags, idempotency keys, request sizes, and rates are in [Limits](/recursion/limits).

## Next steps

<CardGroup>
  <Card title="Tools" href="/recursion/tools">
    See which tools an agent gets and turn some of them off.
  </Card>

  <Card title="Skills" href="/recursion/skills">
    Attach procedures that the agent loads when they apply.
  </Card>

  <Card title="Sessions" href="/recursion/sessions">
    Run the agent in an environment and follow its work.
  </Card>

  <Card title="Multi-agent" href="/recursion/multi-agent">
    Let one agent delegate to others, or lead a team.
  </Card>
</CardGroup>


# Connect to the Recursion MCP
Source: https://docs.labelbox.com/recursion/ai-coding-agents

Connect Claude Code, Codex, Cursor, or another MCP client to the public docs MCP server and to the authenticated Recursion MCP endpoint with an API key.

Recursion offers two MCP servers for coding agents and other MCP clients:

* **The docs MCP server** at `https://docs.labelbox.com/mcp` searches and reads every page here. It's public and needs no key.
* **The Recursion MCP endpoint** at `https://api.recursion.labelbox.com/mcp` finds and runs Managed Agents operations through four tools. It authenticates with an API key and acts as you.

The same API key also lets code the agent writes call the REST API.

```mermaid theme={"theme":"css-variables"}
flowchart TD
  coder["Your coding agent"] -->|"reads"| docs["Docs MCP"]
  coder -->|"calls"| mcp["Recursion MCP"]
  coder -->|"writes"| code["Your script"]
  mcp -->|"API key"| api["Managed Agents API"]
  code -->|"API key"| api
```

<Note>
  You need a coding agent: Claude Code, Codex, or Cursor. The Recursion MCP endpoint and the REST API need an API key: create one under **Settings › API keys** in the console (see [API keys](/recursion/api-keys)). Scripts need `curl` and `jq`, or an HTTP client in your preferred language (see [REST client setup](/recursion/api#rest-client-setup)).
</Note>

<Tip>
  This page connects a local coding agent to Recursion. To give an agent that runs inside Recursion access to another MCP server, configure that server on the agent instead. See [MCP servers](/recursion/mcp-servers).
</Tip>

## Connect the docs MCP server

The docs server at `https://docs.labelbox.com/mcp` is public, so it needs no key. Apart from sending feedback, it's read-only.

<Tabs>
  <Tab title="Claude Code">
    ```bash theme={"theme":"css-variables"}
    claude mcp add --transport http --scope user recursion-docs https://docs.labelbox.com/mcp
    claude mcp list
    ```
  </Tab>

  <Tab title="Codex">
    ```bash theme={"theme":"css-variables"}
    codex mcp add recursion-docs --url https://docs.labelbox.com/mcp
    codex mcp list
    ```
  </Tab>

  <Tab title="Cursor">
    Add the server to `~/.cursor/mcp.json`, then restart Cursor:

    ```json theme={"theme":"css-variables"}
    {
      "mcpServers": {
        "recursion-docs": {
          "url": "https://docs.labelbox.com/mcp"
        }
      }
    }
    ```
  </Tab>
</Tabs>

Start a new session in your coding agent after changing its MCP configuration. The server covers the whole docs site, so tell your agent to stay within the `/recursion` pages.

<Accordion title="Docs server tools and other formats">
  The server offers three tools:

  | Tool | What it does |
  | - | - |
  | Search | Finds relevant pages and returns excerpts with their titles and links. |
  | Docs filesystem | Reads full pages and the API reference with read-only commands such as `cat`, `head`, and `rg`. |
  | Feedback | Reports a page that's wrong, outdated, or unclear. |

  Every page is also available as Markdown: append `.md` to its URL. For agents without MCP, the docs site root serves an index of every page at `/llms.txt` and the full text at `/llms-full.txt`.

  Every code sample on these pages is checked against the API contract whenever the docs change, so operation names, request fields, and paths match what the API accepts.
</Accordion>

## Connect the Recursion MCP endpoint

The authenticated endpoint is `https://api.recursion.labelbox.com/mcp`. Configure it as a Streamable HTTP MCP server that sends `Authorization: Bearer <key>`. Export the key as `RECURSION_API_KEY` in the shell that starts your client, and let the client read it from the environment, so the key never lands in a configuration file. Prefer an organization-scoped key; a tenant-scoped key must also send `x-organization-id` with the organization it should act in.

<Tabs>
  <Tab title="Claude Code">
    Add the server to `.mcp.json` in your project. Claude Code fills in `${RECURSION_API_KEY}` from the environment:

    ```json theme={"theme":"css-variables"}
    {
      "mcpServers": {
        "recursion": {
          "type": "http",
          "url": "https://api.recursion.labelbox.com/mcp",
          "headers": {
            "Authorization": "Bearer ${RECURSION_API_KEY}"
          }
        }
      }
    }
    ```
  </Tab>

  <Tab title="Codex">
    ```bash theme={"theme":"css-variables"}
    codex mcp add recursion --url https://api.recursion.labelbox.com/mcp --bearer-token-env-var RECURSION_API_KEY
    codex mcp list
    ```
  </Tab>

  <Tab title="Cursor">
    Add the server to `~/.cursor/mcp.json`, then restart Cursor. Cursor fills in `${env:RECURSION_API_KEY}` from the environment:

    ```json theme={"theme":"css-variables"}
    {
      "mcpServers": {
        "recursion": {
          "url": "https://api.recursion.labelbox.com/mcp",
          "headers": {
            "Authorization": "Bearer ${env:RECURSION_API_KEY}"
          }
        }
      }
    }
    ```
  </Tab>
</Tabs>

It exposes exactly four tools. Start with `search_docs`, pass its slug to `read_doc`, then use the operation's declared tool and input schema. Every operation runs with the connecting key's tenant, organization, role, and permissions.

<Accordion title="The four tools">
  | Tool | What it does |
  | - | - |
  | `search_docs` | Finds Managed Agents guides, API operations, and recipes: worked examples of a whole task, such as giving a session a credential. |
  | `read_doc` | Reads the exact guide, generated operation schema, or recipe (its steps, CLI and cURL code, and related links) for a discovered slug. |
  | `read_operation` | Executes a discovered read-only operation. |
  | `write_operation` | Executes a discovered create, update, or delete operation. |

  Path parameters in an operation's input use camelCase, such as `vaultId`, while request bodies keep the operation's `snake_case` fields. Follow the input schema `read_doc` returns.

  When an operation declares an `ETag`, its successful result includes `responseHeaders.etag`. If the mutation's discovered input schema declares `If-Match`, copy the complete opaque value from the resource read into that argument. Refresh it after each successful write; do not construct it from the body's `revision`.

  The endpoint rejects browser-origin requests and does not accept session cookies.
</Accordion>

<Accordion title="Operations to call through REST instead">
  The operation tools run only operations whose request and response bodies are JSON. Other operations stay discoverable and their schemas stay readable, but MCP marks them documentation-only. Call these through the REST API instead:

  * Skill bundle uploads and skill version bundle downloads. A skill made of a single `SKILL.md` can be created through MCP; create a skill with scripts or other files through the REST API or the console.
  * The multipart `uploadFile` and the binary `getFileContent`. `listFiles` and `getFile` are callable through `read_operation`.
  * Agent memory settings, learned memories, and memory runs are available through the REST API. The generated SDK does not yet include them. Memory operations are excluded from MCP discovery and operation tools.
  * Moving a session to another agent version. `replaceSessionAgentVersion` is excluded from MCP discovery and operation tools, so an agent can't move a session through MCP. `getSessionAgentVersion` is callable through `read_operation`. See [Move a session to another version](/recursion/sessions#move-a-session-to-another-version).
</Accordion>

## Give the agent API access

Export the key in the shell that starts your coding agent, or save it once on your own computer with the command the console offers when you [create a key](/recursion/api-keys#create-a-key). The agent and the scripts it runs read it from the environment, and the key never appears in a prompt or in your project's files.

```bash theme={"theme":"css-variables"}
export RECURSION_API_KEY='rma_...'
```

Check the key works without printing it:

```bash theme={"theme":"css-variables"}
curl -sS -o /dev/null -w '%{http_code}\n' 'https://api.recursion.labelbox.com/managed-agents/v1/models' \
  -H "Authorization: Bearer $RECURSION_API_KEY"
```

`200` means the key works. `401` means it's missing, expired, or revoked.

<Warning>
  Never paste a key into a chat. Transcripts are stored and can be shared. If you're an AI agent and `RECURSION_API_KEY` isn't set, ask the person to export it and restart you. Don't ask for the value.
</Warning>

<Accordion title="Handle the key safely">
  * **Use a dedicated, short-lived key.** A key acts as you, with your full role, so give each coding agent its own key with a 7- or 30-day expiry and revoke it when you're done.
  * **Prefer an organization-scoped key.** It can reach only one organization and needs no extra header.
  * **Keep keys out of code and version control.** Read `process.env.RECURSION_API_KEY` or `os.environ["RECURSION_API_KEY"]`. Never write the key to a `.env` file that gets committed.
  * **Don't print it.** Tell the agent never to echo, log, or include the key in output.
</Accordion>

## Run a first task

Paste this prompt into your coding agent. It reads the docs first, then writes and runs code.

```text theme={"theme":"css-variables"}
Use the authenticated recursion MCP server. Search for the quickstart and the
operation that lists models, read both results, and call that read-only
operation. Then write a shell script that uses curl, jq, and the
RECURSION_API_KEY environment variable to create an environment and an agent,
start a session that asks the agent for a short summary, poll getSession until
execution_state is idle, and print the session id, stop_reason, and the agent's
last message. Never print the API key.
Run the script and report the results.
```

When it finishes, open **Sessions** in the console to see the same session and its transcript.

Good follow-ups to ask for:

* "Add an `Idempotency-Key` so a retried start can't create a second session."
* "Stream the session's events instead of polling, and resume from the last event id if the connection drops."
* "Read /recursion/errors and handle every retryable error with backoff."

## What can go wrong

The most common problems:

* **The tools don't appear.** The client hasn't reloaded its MCP configuration, or the address is wrong. For the Recursion MCP, `RECURSION_API_KEY` may not be set where the client started. Check both, then restart the client.
* **`401 unauthorized`.** The key isn't exported in the shell that started the client or agent, or it expired or was revoked. Export a valid key, restart, and run the `curl` check above.
* **`400 invalid_request` mentioning `x-organization-id`.** The key has tenant scope. Add the header, or use an organization-scoped key.

<Accordion title="Every connection problem">
  | Symptom | Cause | Fix |
  | - | - | - |
  | The docs tools don't appear | The client hasn't reloaded its MCP configuration, or the address is wrong. | Check the address, then start a new client session or restart the app. |
  | The Recursion MCP tools don't appear, or calls return `401 unauthorized` | `RECURSION_API_KEY` wasn't set in the environment that started the client, or the key expired or was revoked. | Export a valid key, restart the client, and run the `curl` check above. |
  | Search returns pages from other products | The docs server covers the whole site. | Ask the agent to read only paths under `/recursion`. |
  | Scripts get `401 unauthorized` | `RECURSION_API_KEY` isn't exported in the shell that started the agent, or the key expired. | Export it, restart the coding agent, and run the `curl` check above. |
  | Scripts get `400 invalid_request` mentioning `x-organization-id` | The key has tenant scope. | Add the header, or use an organization-scoped key. See [Tenant-scoped keys](/recursion/api#tenant-scoped-keys). |
  | The agent asks you to paste the key | Its instructions don't say how keys are provided. | Export the key, restart the agent, and tell it to read `RECURSION_API_KEY` from the environment. |
</Accordion>

## Next steps

<CardGroup>
  <Card title="API conventions" href="/recursion/api">
    Authentication, retries, paging, and errors in one place.
  </Card>

  <Card title="Quickstart" href="/recursion/quickstart">
    Run the same flow yourself, step by step.
  </Card>

  <Card title="API keys" href="/recursion/api-keys">
    Scopes, expiry, revocation, and safe storage.
  </Card>

  <Card title="MCP servers" href="/recursion/mcp-servers">
    Give a hosted agent tools from another MCP server.
  </Card>

  <Card title="Labelbox MCP server" href="/recursion/labeling-mcp">
    Read your Labelbox workspace from the same client.
  </Card>
</CardGroup>


# Analytics
Source: https://docs.labelbox.com/recursion/analytics

See how many agent session hours your organization's agents used, how their sessions completed, and how their sandboxes started, in the console.

**Analytics** shows what your organization's agents did over a time window: how many agent session hours they used, how their sessions completed, and how their sandboxes started. Use it to find the agent behind a change in usage. Each agent's **Observability** tab tells the story for that one agent, with its errors and speed.

<Note>
  Any organization role that can read sessions can open **Analytics**: user, developer, or admin. The figures cover the organization you are working in. See [Organizations and roles](/recursion/organizations-and-roles). Analytics is a console screen; it has no API-key operation. What your agents cost is on the **Billing** screen; see [Billing](/recursion/billing).
</Note>

## Open Analytics

Pick a view and a time window. The page opens on **Usage** for the **Last 7 days**.

<Tabs>
  <Tab title="Console">
    1. In the [Recursion console](https://recursion.labelbox.com), click **Analytics** in the sidebar.
    2. Choose a view: **Usage**, **Completions**, or **Environments**.
    3. Choose a window: **Last 24 hours**, **Last 7 days**, **Last 30 days**, **Last 90 days**, or **Last year**.
  </Tab>
</Tabs>

Next to the window, a pill says how current the figures are: **Up to date**, or **Behind by** some minutes or hours. Figures are updated as work is recorded and normally trail it by a minute or two, so a session that just finished can take that long to appear.

## Read usage

**Usage** shows one figure: **Agent session hours**, the time your agents' sandboxes ran in the window. It is the time metered for billing: each sandbox run counts from start to stop, idle time included, once the run is charged. A run is charged when its sandbox stops, so a session that is still running isn't counted yet. Environment setup isn't included.

Below it, **Agent session hours by agent** charts the window by day or by hour, stacked by agent. Hover a column to see that period's hours for each agent. Periods are in UTC.

The **Agent** table lists each agent's session hours.

## Narrow to an agent

Click an agent in the **Agent** table, or in the chart's legend, to filter the whole page to it. The filter appears above the figures. Click it to remove it, or click **Clear all**.

## Read completions and environment startup

* **Completions** shows how many root sessions completed, how many complete per hour or per day, and how long they took from creation to completion, including time spent waiting. **Active agents** lists the same for each agent.
* **Environments** shows how many sandboxes started, how many failed to start, and how long startup took, by provider, compute, machine type, and environment.

## Follow one agent

Open an agent from **Agents** and choose **Observability**. It shows the agent's sessions, tool error rate, and agent session hours; activity stacked by agent version, as **Sessions**, **Errors**, or **Session hours**, so you can see whether a new version changed anything, with a session that was moved to another version counted under each version for the activity it had on that version; per-session means and durations; and its tools by use. The agent's **Sessions** view shows its completion figures above the list.

## What can go wrong

* **A session you just ran isn't counted yet.** Check the freshness pill; figures trail work by a minute or two. Session hours count once the session's sandbox stops and its run is charged.
* **Agent session hours say they're still being calculated.** Hours for earlier work are being worked out. This takes a few minutes; the other figures are already complete.
* **You're looking for cost.** Analytics shows usage, not spend. Read what your tenant spent on the **Billing** screen, or one session's cost in [Usage and cost](/recursion/usage-and-cost).
* **Percentiles look rough.** Durations such as an agent's **Tool call latency** are approximate. Each says how many measurements it's based on.

## Next steps

<CardGroup>
  <Card title="Sessions" icon="play" href="/recursion/sessions">
    Launch work and read the sessions behind these figures.
  </Card>

  <Card title="Billing" icon="credit-card" href="/recursion/billing">
    Track what your tenant spends and buy credits.
  </Card>
</CardGroup>


# API conventions
Source: https://docs.labelbox.com/recursion/api

Create an API key in the console, then call the Managed Agents REST API or the authenticated MCP endpoint: base URL, authentication, asynchronous starts, idempotency, pagination, errors, rate limits, and request ids.

You reach Managed Agents from code over REST or MCP, and both use the same API key:

1. **Create an API key** in the [Recursion console](https://recursion.labelbox.com) under **Settings › API keys**. See [API keys](/recursion/api-keys).
2. **Call the REST API** at `https://api.recursion.labelbox.com/managed-agents/v1` with the key as a bearer token. This page covers its rules.
3. **Or connect an MCP client** to the authenticated MCP endpoint at `https://api.recursion.labelbox.com/mcp` with the same key. See [Connect to the Recursion MCP](/recursion/ai-coding-agents).

Every operation follows the same rules for authentication, retries, paging, and errors. Read this page once, then use the generated **Endpoints** reference as the authority for each operation's exact schema, error alternatives, headers, and retry strategy.

## Base URL

Every operation lives under one base path:

```text theme={"theme":"css-variables"}
https://api.recursion.labelbox.com/managed-agents/v1
```

Requests and responses are JSON. Most fields use `snake_case`. A few responses, such as the model list, use `camelCase`, so follow each operation's schema in **Endpoints**.

## Authentication

Send an API key as a bearer token on every request:

```http theme={"theme":"css-variables"}
Authorization: Bearer rma_...
```

Create the key in the console under **Settings › API keys**. See [API keys](/recursion/api-keys) for scopes, expiry, and safe storage.

| Key scope | Extra header | Where requests act |
| - | - | - |
| Organization | None. You may send `x-organization-id`, but it must name the key's organization. | The key's organization. |
| Tenant | `x-organization-id: <organization id>` or `x-organization-id: default`. Required on every request. | The organization the header names, if you belong to it. |

A key acts as you, with the role you hold in that organization at the moment of the request. If an admin changes your role, your keys follow on the next request. Once the organization is resolved, the response carries `recursion-organization-id` and `recursion-tenant-id` headers that name the scope the request ran in.

| Response | Cause |
| - | - |
| `401 unauthorized` | The key is missing, malformed, revoked, or expired, it belongs to a person who left the tenant, or its organization was archived. |
| `400 invalid_request` | A tenant-scoped key sent no `x-organization-id`. |
| `404 not_found` | `x-organization-id` names an organization you can't reach, an archived one, or a different organization than an organization-scoped key's own. |
| `403 forbidden` | Your role in the organization doesn't allow the operation. See [Organizations and roles](/recursion/organizations-and-roles). |

## REST client setup

Export your key, then send it as a bearer token. Every page shows each request as cURL, and **Endpoints** has each operation's exact schema. See [Python and other languages](#python-and-other-languages) to translate the same requests to an HTTP client.

```bash theme={"theme":"css-variables"}
export RECURSION_API_KEY='rma_...'
```

<Tabs>
  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl 'https://api.recursion.labelbox.com/managed-agents/v1/models' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

A successful call returns the models you can use in agents:

The values below illustrate the response shape; use a returned `modelId` when creating an agent.

```json theme={"theme":"css-variables"}
{
  "items": [
    {
      "modelId": "<model-id>",
      "displayName": "<display-name>",
      "family": "<provider-route>",
      "contextWindow": 200000,
      "supportsImageInput": true
    }
  ]
}
```

### Tenant-scoped keys

A tenant-scoped key must name the organization on every request. Use an organization id, or `default` for your tenant's default organization.

<Tabs>
  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl 'https://api.recursion.labelbox.com/managed-agents/v1/models' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'x-organization-id: default'
    ```
  </Tab>
</Tabs>

### Call shape

Call `https://api.recursion.labelbox.com/managed-agents/v1/<path>`. Send JSON bodies with `Content-Type: application/json`.

Send ordinary request bodies as JSON. A body in a media type the API doesn't accept returns `415 unsupported_media_type`. Two kinds of upload use `multipart/form-data` instead: skill bundle uploads and file uploads with `uploadFile`. See [Skills](/recursion/skills) and [Files](/recursion/files).

### Python and other languages

Send the same REST requests the cURL samples show from any HTTP client, with the bearer header on each one. For example, with `httpx`:

```python theme={"theme":"css-variables"}
import os

import httpx

api = httpx.Client(
    base_url="https://api.recursion.labelbox.com/managed-agents/v1",
    headers={"Authorization": f"Bearer {os.environ['RECURSION_API_KEY']}"},
)
models = api.get("/models").json()
```

Translate any cURL sample the same way: the path follows `/managed-agents/v1`, `-d` becomes a JSON body, and each `-H` becomes a header, such as `Idempotency-Key` on `startSession`.

Use a client such as `httpx` or `requests`. Requests sent with the defaults of Python's built-in `urllib` may be refused.

## Asynchronous session starts

`startSession` returns `202 Accepted` as soon as the session is recorded. It doesn't wait for the sandbox or the agent.

```json theme={"theme":"css-variables"}
{
  "session_id": "b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38",
  "status_path": "/managed-agents/v1/sessions/b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38"
}
```

Follow the session with `getSession` or by [streaming its events](/recursion/events). The session record is authoritative. The event stream is written separately and can lag it, so an empty event list doesn't mean the session is still starting.

What success means depends on the work:

| Work | Done when |
| - | - |
| Finished turn | `execution_state` is `idle` and `stop_reason` says why. `end_turn` means the agent finished its turn, not that its work is correct. |
| Failed session | `status` is `failed`. Read `failure.code`, `failure.category`, and `failure.retryable`. See [Errors](/recursion/errors#session-failure-codes). |

## Idempotent mutations

Mutations with declared retry semantics publish their strategy in the API contract. This inventory is generated from that same contract and includes only operations published by the standalone API:

The operations currently in each strategy are:

* `key-required`: `createAutomation`, `createEnvironmentSetupRun`, `createEventSource`, `runAutomation`, `startSession`
* `key-supported`: `createAgent`, `createEnvironment`, `createTag`, `createVault`, `createVaultCredential`, `sendSessionEvents`, `updateAgentMemorySettings`, `updateEnvironment`
* `naturally-idempotent`: `deleteTag`, `replaceSessionAgentVersion`
* `retry-unsafe`: `completeIntegrationInstall`, `createAgentVersion`, `createSkill`, `createSkillVersion`, `deleteAutomation`, `deleteEventSource`, `openAgentAnalyst`, `openAssistant`, `pauseAutomation`, `pauseEventSource`, `publishSkillBundle`, `publishSkillVersionBundle`, `replaceAutomation`, `replaceEventSource`, `resumeAutomation`, `resumeEventSource`, `startIntegrationInstall`, `submitAssistantInteractionDecision`, `updateTag`, `updateVault`, `uploadFile`

A `key-required` operation rejects a missing `Idempotency-Key`. A `key-supported` operation accepts the header optionally. Use a stable key for each logical mutation so an ambiguous network retry cannot repeat its effect.

| Retry | Result |
| - | - |
| Same key, exact method, escaped path, raw query, and raw body bytes | The original successful response is replayed with `Idempotency-Replayed: true`. |
| Same key with any different request identity bytes | `409 idempotency_conflict`. |
| Same key while the first request is still running | `409 idempotency_in_progress` with `Retry-After`. |
| The key cannot be read or reserved | `503 idempotency_unavailable`. The request did not run and may be retried with the same key. |

Keys hold 1 to 256 visible ASCII characters, sent as exactly one header value. A malformed or repeated header returns `400 invalid_request`. Keys share one namespace per organization and across keyed operations, so use a different key for each logical mutation. A completed key is remembered for about 24 hours, and an abandoned in-progress key can be reused after about 1 hour.

Send `Idempotency-Key` as a request header. Retry with the same request target and body bytes: JSON whitespace and key order, query ordering and escaping, and path escaping are significant.

For `createVault`, pass the optional `Idempotency-Key` header. The retired body field `idempotency_key` is rejected. The key also determines the vault id, so an unchanged vault remains recoverable after the response receipt expires. If that vault was changed or deleted, reuse conflicts and a replacement requires a new key.

A `naturally-idempotent` operation is safe to repeat by its own resource semantics. Do not automatically retry a `retry-unsafe` operation after an ambiguous transport failure: inspect the affected resource or operation state first. An undeclared operation has no replay guarantee. Sending `Idempotency-Key` to any of these operations does not add one.

## Pagination

Paged list operations return a continuation token. Each operation documents its own default and maximum page size.

| Field | Use |
| - | - |
| `limit` | Maximum items to return in one page. |
| `page_token` | The previous response's `next_page_token`. Keep the same filters and `limit`. |
| `next_page_token` | Present when more items exist. Stop when it's absent or empty. |
| `after_event_id` | Continue `listSessionEvents`. Pass that response's `next_page_token` here. |

A page can hold fewer items than `limit` and still have a next token, because event pages also have a size budget. A session-list token is valid for one hour and only with the filters and `limit` it was issued for. A token that's no longer valid returns `400 invalid_request` with `details.field` set to `page_token`, so restart the list without it.

## Errors

Every `4xx` and `5xx` response is a flat object with required `code` and `message` fields and optional `details`:

```json theme={"theme":"css-variables"}
{
  "code": "invalid_request",
  "message": "agent_id is required",
  "details": {
    "field": "agent_id",
    "requestId": "0b6f2c1d-9a4e-4b7f-8c3d-5e6a7b8c9d0e"
  }
}
```

| Field | Use |
| - | - |
| `code` | Required. Stable and machine-readable. Branch on this. |
| `message` | Required. Human-readable and changeable. Don't match on it. |
| `details` | Optional code-specific context, measured values, limits, or next actions. |
| `details.field` | The request field, query parameter, or header at fault, when there is one. |
| `details.issues` | Boundary-validation failures as path-prefixed messages, when present. |
| `details.requestId` | The request id. Include it when you report a problem. |
| `details.retryable` | Server advice about transience: `true` means transient, `false` means non-transient, and absence gives no advice. Automatic replay is allowed only when this value is not `false` and the operation's retry strategy permits it. |

Check the HTTP status and `code` on every non-`2xx` response. Use the generated **Endpoints** reference for an operation's exact status and code alternatives. See [Errors](/recursion/errors) for common recovery guidance.

## Rate limits

Requests are limited to 300 per 60 seconds for each of your allowances: all your API keys share one, in every tenant and organization, and your console use has its own, so a busy script doesn't lock you out of the console. Event streams are also limited in how many can be open at once.

| Response | Meaning | What to do |
| - | - | - |
| `429 rate_limit_exceeded` | You sent too many requests in a short window. | If replay is safe, wait for `Retry-After` and retry. |
| `503 service_unavailable` on a stream | The service is at stream capacity. | Close streams you no longer need, then retry with backoff. |

`Retry-After` supplies timing, not replay permission. Honor it only when the operation's generated retry strategy permits replay and `details.retryable` is not `false`. When a permitted retry has no `Retry-After`, back off exponentially with jitter, starting around one second. See [Limits](/recursion/limits) for every other limit.

## Request ids

Every response carries an `x-request-id` header, and error bodies repeat it as `details.requestId`. You can send your own `x-request-id` of up to 128 letters, digits, `_`, `.`, `:`, or `-`, and it's echoed back so you can match it to your own logs.

## Next steps

<CardGroup>
  <Card title="Quickstart" href="/recursion/quickstart">
    Create an environment and an agent, start a session, and read the reply.
  </Card>

  <Card title="Errors" href="/recursion/errors">
    Look up every error code with its cause, fix, and retry guidance.
  </Card>

  <Card title="Limits" href="/recursion/limits">
    Find every size, count, and time limit in one place.
  </Card>

  <Card title="Troubleshooting" href="/recursion/troubleshooting">
    Match a symptom to its cause and fix.
  </Card>
</CardGroup>


# API keys
Source: https://docs.labelbox.com/recursion/api-keys

Create, scope, use, rotate, and revoke the personal rma_ keys that authenticate every Managed Agents API request.

An API key lets your code, scripts, and coding agents call the Managed Agents API as you. Each key is personal, expires, and is scoped to your whole tenant or to one organization.

<Note>
  You need a role with Managed Agents access to see **Settings › API keys** in the console. See [Organizations and roles](/recursion/organizations-and-roles). Keys are created and revoked only in the console, while you're signed in. A key only calls Managed Agents operations: it can't create, list, or revoke API keys.
</Note>

## Choose a scope

| | Organization scope | Tenant scope |
| - | - | - |
| In the console | **Scope** set to an organization, such as **Default**, which starts on the organization you're in. | **Scope** set to **Tenant**. |
| Where it works | One organization. | Every organization in the tenant that you belong to. |
| `x-organization-id` header | Not needed. If you send it, it must name the key's organization. | Required on every request: an organization id, or `default`. |

Prefer organization scope: it reaches only one organization and needs no extra header. Use tenant scope when one key must work in several organizations. See [Organizations and roles](/recursion/organizations-and-roles).

## Create a key

<Tabs>
  <Tab title="Console">
    1. Open the [Recursion console](https://recursion.labelbox.com). In the sidebar, click **Settings**, choose **API keys**, then click **Create key**.
    2. Enter a **Name** that says where the key is used, such as `nightly-report-prod`.
    3. Choose a lifetime in **Expires**: 3 hours, 1 day, 7 days, 30 days, 90 days, or **Custom date** up to one year out. The default is 30 days.
    4. **Linked account** shows you. Requests with this key act as you, and the key stops working if you lose access to its organizations.
    5. Choose the **Scope**: one organization, such as **Default**, or **Tenant**. It starts on the organization you're in.
    6. Click **Create key**.
    7. On **Save your API key**, click **Copy key** and store the key in a secret manager. This is the only time it's shown.
    8. To use the key from your own computer, choose your operating system under **Save it as `RECURSION_API_KEY`**, copy the command, and run it in a terminal. It saves the key for every new terminal: in `~/.zshrc` on macOS, in `~/.bashrc` on Linux, or in your user environment on Windows.
    9. Close the drawer. If you haven't copied the key or a command, the console asks before it closes.
  </Tab>
</Tabs>

The key starts with `rma_`. After you close the drawer, only a hint such as `rma_…a1b2` is shown, and the full key can't be retrieved. A custom date expires at the end of that day in your local time, or one year from now if that comes first.

## Use a key

Send the key as a bearer token. An organization-scoped key needs nothing else. A tenant-scoped key also sends `x-organization-id`.

<Tabs>
  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    # Omit the x-organization-id line for an organization-scoped key.
    curl 'https://api.recursion.labelbox.com/managed-agents/v1/models' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'x-organization-id: default'
    ```
  </Tab>
</Tabs>

Once a request reaches an organization, its response includes `recursion-organization-id` and `recursion-tenant-id` headers, so you can confirm which organization it ran in.

## Rotate a key

Rotate before a key expires, or right away if it might have leaked.

1. Create a new key with the same scope.
2. Deploy it to the service that uses the old key.
3. Confirm the service works with the new key.
4. Revoke the old key.

## Revoke a key

<Tabs>
  <Tab title="Console">
    1. In **API keys**, open the actions menu on the key's row.
    2. Click **Revoke**, then **Revoke key**.
  </Tab>
</Tabs>

The key stops working immediately. Revoking can't be undone. You can revoke an **Active** or **Inactive** key.

<Accordion title="Key statuses">
  | Status | Meaning | Can you revoke it? |
  | - | - | - |
  | **Active** | The key works. | Yes. |
  | **Inactive** | The key is neither expired nor revoked, but your account no longer reaches its tenant or organization, so it doesn't work. | Yes. |
  | **Expired** | The key reached its expiry date. | No. |
  | **Revoked** | You revoked the key. | No. |

  Filter the list by **Status** and **Scope** to find a key.
</Accordion>

## How keys work

* **A key acts as you.** Requests run with the role you hold in the target organization, checked on every request.
* **The secret is shown once.** Store it when you create the key; it can't be retrieved later.
* **Every key expires.** Choose a lifetime of up to 365 days. There's no "never expires".
* **Keys are yours alone.** The list shows only the keys you created, and you're the only one who can revoke them.

## Keep keys safe

* Store keys in a secret manager or your platform's secret store. Load them into an environment variable such as `RECURSION_API_KEY` at run time.
* Run the console's `RECURSION_API_KEY` command only on your own computer. It writes the key to your shell profile or your Windows user environment, and the command stays in your shell history, so don't run it on a shared machine or a server.
* Never commit a key, put it in a system prompt or session message, or paste it into a chat with an AI assistant.
* Use one key per service or coding agent, so you can revoke one without breaking the others.
* Choose the shortest expiry that works. Short-lived keys limit the damage of a leak.
* Never give a key to an agent running inside Recursion. To give an agent credentials, use a [vault](/recursion/vaults).

## What can go wrong

The most common problems:

* **`401 unauthorized`.** The key is missing, mistyped, expired, revoked, or inactive. Check the `Authorization` header and the key's status, and create a new key if needed.
* **`400 invalid_request` mentioning `x-organization-id`.** A tenant-scoped key sent no `x-organization-id`, or the header isn't an organization id or `default`. Send `x-organization-id: default`, or use an organization-scoped key.
* **`404 not_found` on every request.** `x-organization-id` names an organization you can't reach, or a different organization than the key's. Send `default`, or omit the header for an organization-scoped key.

<Accordion title="Every API key error">
  | Code or symptom | Cause | Fix |
  | - | - | - |
  | `401 unauthorized` | The key is missing, mistyped, expired, revoked, or inactive. | Check the `Authorization` header and the key's status. Create a new key if needed. |
  | `400 invalid_request` mentioning `x-organization-id` | A tenant-scoped key sent no `x-organization-id`, or the header isn't an organization id or `default`. | Send `x-organization-id: default`, or use an organization-scoped key. |
  | `404 not_found` on every request | `x-organization-id` names an organization you can't reach, or a different organization than the key's. | Send `default`, or omit the header for an organization-scoped key. |
  | The key worked yesterday and doesn't today | It expired or was revoked. | Check its status under **API keys**, then create a new key. |
  | **Create key** stays disabled | **Name** is empty, or **Custom date** is empty or in the past. | Enter a name, and choose a date between tomorrow and one year from today. |
</Accordion>

Key name and lifetime limits are in [Limits](/recursion/limits).

## Next steps

<CardGroup>
  <Card title="API conventions" href="/recursion/api">
    Set up a client and learn the request rules.
  </Card>

  <Card title="Organizations and roles" href="/recursion/organizations-and-roles">
    See what each role can do with a key.
  </Card>

  <Card title="Connect to the Recursion MCP" href="/recursion/ai-coding-agents">
    Use your key with the authenticated MCP endpoint and give a coding agent a key safely.
  </Card>

  <Card title="Security" href="/recursion/security">
    See how credentials and data are protected.
  </Card>
</CardGroup>


# Deliverables and artifacts
Source: https://docs.labelbox.com/recursion/artifacts

Ask agents for named deliverables, understand which files are kept and when, show images inline, and get results out of a session.

A deliverable is a file an agent produces for you, such as a report, a CSV, or a patch. Agents already know where to save deliverables, so you only need to ask for a named file. At the end of each turn, the files in the deliverables folder are kept with the session, so you and later sessions can use them.

<Note>
  To start sessions, you need the Developer or Admin role in the organization. See [Organizations and roles](/recursion/organizations-and-roles). Every session runs in an environment's sandbox. See [Environments](/recursion/environments).
</Note>

## Ask for a deliverable

Name the file and its format in the task, and say what it must contain. This session asks for a CSV and a short summary.

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Sessions**, then click **Launch session**.
    2. Choose the **Agent** and **Environment**.
    3. In **Opening message**, describe the task and name the deliverables.
    4. Click **Launch session**.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/sessions' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Idempotency-Key: q3-price-audit-2026-09-25' \
      -H 'Content-Type: application/json' \
      -d '{
        "agent_id": "5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55",
        "environment_id": "9d3e7b52-1a4c-4f80-b6e9-2c8a5d0f7e13",
        "message": "Audit the Q3 price exports. Save price-changes.csv and summary.md as deliverables. The price exports are in /workspace/exports. Keep scratch work in /workspace/scratch."
      }'
    ```
  </Tab>
</Tabs>

The API answers `202 Accepted`:

```json theme={"theme":"css-variables"}
{
  "session_id": "b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38",
  "status_path": "/managed-agents/v1/sessions/b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38"
}
```

In the transcript, the agent's `write_artifact` call answers with the path and "it is captured as a session output when this turn ends". A call that includes `metadata` answers that the file was stored as a session output instead. When the session finishes, [download the deliverables](#download-deliverables).

<Tip>
  Keep scratch work out of the deliverables folder. Everything in it is kept, so intermediate files there clutter the result and count toward the per-turn limits.
</Tip>

<Accordion title="Prompt patterns that work">
  | Where | Pattern |
  | - | - |
  | Agent system prompt | "When a task asks for a report, save it as a Markdown deliverable named after the task, and put your working notes in `/workspace/scratch`." |
  | Task message | "Save the comparison as `vendor-comparison.md` with one table row per vendor." |
  | Done criteria | "You're done when `vendor-comparison.md` is saved as a deliverable and has a row for each of the three vendors." |
  | Many outputs | "Save each chart as a PNG under `charts/` in the deliverables and share the most important one." |
</Accordion>

### Label deliverables with metadata

An agent can attach key/value `metadata` when it calls `write_artifact`, for example `{"kind": "report", "ticket": "OPS-12"}`. The metadata is stored on the deliverable's file, returned as `metadata` beside `path` when you list or get the file, and shown under **Metadata** in the file preview. Ask for it in the task: "Save `summary.md` with metadata `kind` set to `summary` and `quarter` set to `Q3`."

Later sessions of the same agent can find deliverables by their metadata when the agent has **File history** on. See [Agent file history](/recursion/agent-file-history).

<Accordion title="Metadata rules">
  * A write that includes `metadata` is kept at once rather than at the end of the turn, so it doesn't count toward the per-turn limits.
  * A write that includes `metadata` replaces the deliverable's earlier metadata, and `{}` clears it. A write that leaves it out keeps it.
  * The metadata stays when the file is changed later by another tool or command.
  * The rules are the same as for uploaded files: at most 32 keys, each up to 64 characters of letters, digits, `_`, `.`, and `-`, and at most 64 KiB of JSON in total. The `path` key is reserved.
</Accordion>

## Show images inline

Ask the agent to share a chart or screenshot: "Plot weekly signups and share the chart." The agent saves the image under `/workspace` and calls `share_file`.

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Sessions**, then open the session.
    2. Find the `share_file` row in the transcript and click the thumbnail.

    The image opens at full size, with its caption and path, and you can page through every image in the session.
  </Tab>

  <Tab title="cURL">
    Through the API, a shared image is an `artifact` event. List events with `image_urls=signed` to add a short-lived `url` to each image block.

    ```bash theme={"theme":"css-variables"}
    curl 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38/events?image_urls=signed' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

In the console, the transcript row for a shared image shows a thumbnail. Through the API, the `artifact` event carries the caption and the image.

<Accordion title="What an image event looks like">
  ```json theme={"theme":"css-variables"}
  {
    "event_id": "0192d4f1-8c3e-7a5b-b1d0-4e6f2a9c7d58",
    "event_type": "artifact",
    "session_id": "b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38",
    "tool_name": "share_file",
    "created_at": "2026-09-25T17:21:40Z",
    "content": {
      "blocks": [
        {"type": "text", "text": "Weekly signups, last 12 weeks"},
        {
          "type": "image",
          "media_type": "image/png",
          "width": 1280,
          "height": 720,
          "byte_size": 84211,
          "uri": "<private image URI>",
          "url": "<short-lived image URL>",
          "url_expires_at": "2026-09-25T17:36:40Z"
        }
      ],
      "metadata": {
        "artifact_kind": "sandbox_file",
        "path": "/workspace/charts/signups.png",
        "caption": "Weekly signups, last 12 weeks",
        "media_type": "image/png",
        "byte_size": 84211,
        "width": 1280,
        "height": 720
      }
    }
  }
  ```

  A signed `url` is omitted when none can be issued for that read. The image is always available through the authenticated session image endpoint, `GET /managed-agents/v1/sessions/{sessionId}/images`, with the image block's `uri` as the `uri` query parameter.
</Accordion>

## Download deliverables

Every kept deliverable is a file in your organization's library with `source: "session_output"`, listed under the root session, including deliverables that subagents and teammates saved.

Open the session and choose the **Files** tab in the side panel. **Outputs** shows each deliverable with its path under the deliverables folder. Click it to preview it, and download it from the preview. Through the API, list a session's deliverables with `listFiles` and `scope_session_id`, then download each with `getFileContent`. See [Download session deliverables](/recursion/files#download-session-deliverables).

Each output keeps its path under the deliverables folder, such as `reports/q3.md`. Saving to the same path again replaces the earlier version at the end of the turn, or at once when the `write_artifact` call includes `metadata`.

<Accordion title="Other ways to use results">
  | Method | How |
  | - | - |
  | The final message | Ask the agent to end with the key result, for example "End with the five largest increases as a table." Read it in the transcript or from the session's events. |
  | Inline images | Ask the agent to `share_file` charts and screenshots. |
  | A follow-up session | Start a new session that references this one. It can list and read the earlier session's deliverables. See [Referenced sessions](/recursion/referenced-sessions). |
  | Your own storage or repository | Grant a credential for your storage, repository host, or API in a vault, allow its host in the environment's network policy, and ask the agent to upload or push the file. See [Vaults](/recursion/vaults). |
</Accordion>

## Where work happens

| Path | What it's for |
| - | - |
| `/workspace` | The agent's workspace. Repositories, scratch files, and build output go here. The session keeps these files after its sandbox stops or is released; they can be used when its workspace reopens. |
| `/workspace/.managed-agents/outputs` | Deliverables. Every file in this folder is kept with the session. |

Files outside the deliverables folder stay in the session workspace for later turns when it reopens, but aren't saved separately as deliverables or shown in the **Files** tab. Deleting the session removes access to its workspace.

<Accordion title="Tools you'll see in transcripts">
  Agents use these tools on their own. You can also name them in prompts.

  | Tool | What it does |
  | - | - |
  | `write_artifact` | Writes a deliverable. Takes a `path` relative to the deliverables folder, such as `report.md` or `tables/prices.csv`, non-empty text `content`, and optional `metadata`. |
  | `read_artifact` | Reads a deliverable back by the same relative path. |
  | `write_file` | Writes any file in the sandbox by absolute path, such as `/workspace/notes.md`. Use it for scratch work. |
  | `read_file` | Reads a file in the sandbox. PNG, JPEG, and WebP files come back as images the agent can see, when its model accepts images. |
  | `share_file` | Shows a PNG, JPEG, or WebP file from the sandbox to you inline in the session, with an optional one-line caption. The agent doesn't see the image again. |

  `write_artifact` takes text. For a binary deliverable, such as a PDF or a spreadsheet, the agent creates the file with a command and saves it anywhere under `/workspace/.managed-agents/outputs`. Files placed there by any tool are kept the same way.
</Accordion>

## What gets kept

At the end of each turn, once the agent stops and nothing is queued, the deliverables folder is checked and every new or changed file is kept. A file that can't be kept is reported as a warning; the session keeps running.

<Accordion title="How files are kept">
  * **Changes.** A file saved again under the same path replaces the earlier version. A file that hasn't changed isn't kept twice.
  * **Children.** Subagents and teammates share the root's sandbox, so their deliverables count as the root session's deliverables.
  * **Problems are reported, not fatal.** A file that can't be kept adds a `session_status` event with `event_status` `warning`, the code `session_output_degraded`, and a message naming the file. The session keeps running. A file refused once is not reported again until it changes.
  * **Earlier versions survive a bad rewrite.** If a kept file is later emptied, grows past the size limit, or becomes unreadable, the earlier kept version stays.
</Accordion>

### Limits

One deliverable can be at most 1 GiB on managed computes or 64 MiB on self-hosted workers. Each turn keeps at most 200 files and normally 512 MiB. A single file over the turn byte budget can still be kept as the first file of that turn. Files that wait for a later turn are kept only if the session has one.

<Accordion title="Every deliverable limit">
  | Limit | Value | What happens past it |
  | - | - | - |
  | Size of one file | 1 GiB on managed computes; 64 MiB on self-hosted workers | The file isn't kept, and a warning names it. |
  | Files kept per turn | 200 | The rest wait for the end of the next turn. |
  | Bytes kept per turn | 512 MiB | The rest wait for the end of the next turn. |
  | Files in the folder | 10,000 | Only the first 10,000 are considered each turn. |
  | File name | 255 characters, no control characters | The file isn't kept, and a warning names it. |
  | Empty file | Not allowed | `write_artifact` refuses it; a file made another way isn't kept, and a warning names it. |
  | Images shown with `share_file` | PNG, JPEG, or WebP; up to 5 MiB, 8,192 pixels on a side, and 40 megapixels | The tool call fails and the agent is told why. |

  A session also has an overall image budget, shared by every tool that returns images, of 500 images and 100 MiB. See [Limits](/recursion/limits).
</Accordion>

## What can go wrong

The most common problems:

* **A deliverable is missing from the Files tab.** It was saved outside the deliverables folder. Ask for it by name as a deliverable, or save it under `/workspace/.managed-agents/outputs`.
* **`session_output_degraded` warning about the per-file limit.** A deliverable exceeds the limit for its compute provider. Split or compress the file.
* **`share_file` fails with "supports PNG, JPEG, and WebP images".** The file is another format, such as SVG or PDF. Ask the agent to render a PNG first.

<Accordion title="Every deliverable error">
  | Symptom or message | Cause | Fix |
  | - | - | - |
  | "content is empty; an artifact must hold something" | `write_artifact` was called with empty content. | Ask the agent to write the file's content. |
  | "is an absolute path, which is a sandbox file" | `write_artifact` got an absolute path outside the deliverables folder. | Use a relative path such as `report.md`, or save with `write_file` if it's scratch. |
  | "must be a plain relative path with no . or .. segments" | The path contains `..`, `./`, a doubled slash, or a trailing slash. | Use a clean relative path such as `reports/q3.md`. |
  | "cannot be a file name" | The last part of the path is empty, too long, or has control characters. | Use a short, printable file name. |
  | "metadata key "path" is reserved" | `write_artifact` got `metadata` with a `path` key, which the platform uses for the deliverable's path. | Use another key name. |
  | `session_output_degraded` warning: "over the 1024 MiB limit" or "over the 64 MiB limit" | A deliverable exceeds the 1 GiB managed-compute limit or the 64 MiB self-hosted limit. | Split or compress the file. |
  | `session_output_degraded` warning: "is empty" | A file in the deliverables folder has no content. | Remove it or write its content. |
  | "The container changed during the artifact write" | The sandbox changed during the write. The file may or may not have been written. | Check whether the file was written before writing it again. For more sandbox work, start a new session. |
  | A deliverable isn't in the **Files** tab | It was saved outside the deliverables folder. | Ask for it by name as a deliverable, or save it under `/workspace/.managed-agents/outputs`. |
  | `share_file` fails with "supports PNG, JPEG, and WebP images" | The file is another format, such as SVG or PDF. | Ask the agent to render a PNG first. |
</Accordion>

## Next steps

<CardGroup>
  <Card title="Files" href="/recursion/files">
    Attach input files and download session outputs.
  </Card>

  <Card title="Referenced sessions" href="/recursion/referenced-sessions">
    Read an earlier session's deliverables from a new one.
  </Card>

  <Card title="Teams" href="/recursion/teams">
    Have several agents produce parts of one deliverable.
  </Card>

  <Card title="Research team tutorial" href="/recursion/use-cases/research-team">
    A team session that ends in a report.
  </Card>
</CardGroup>


# Automations
Source: https://docs.labelbox.com/recursion/automations

Start agent sessions on a schedule, when a Slack, GitHub, or webhook event arrives, or by hand, with the same agent version, environment, prompt, and credentials every time.

An automation saves everything a session needs: a pinned agent version, an environment, an opening prompt, and run defaults such as credentials and files. Its **triggers** decide when it starts a session, and every start is recorded as a **run**, so you can see why each session began and what it was given.

You can manage automations in the console, under **Automations**, or through the REST API with an API key. Both work on the same automations.

<Note>
  You need an agent version and an environment. Saving checks that both exist. Runs need the environment's setup verified: a run on an unverified environment fails with `environment_not_verified`. See [Agents](/recursion/agents) and [Verify the setup script](/recursion/environments#verify-the-setup-script). Slack and GitHub triggers also need an **event source**: connecting Slack or GitHub on [Integrations](/recursion/integrations) creates one. See [Receive events](#receive-events). Creating automations and starting runs by hand needs create permission, changing, pausing, and resuming them needs update, and deleting needs delete. Organization developers and admins have all of them. See [Organizations and roles](/recursion/organizations-and-roles).
</Note>

## Create an automation

Choose the agent version, environment, and instructions, then add the triggers that should start runs.

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Automations**, then click **New automation**. To start from a ready-made automation, click **Start from a template** at the top of the form and choose one; it fills in the name, triggers, and instructions, and you still choose the agent and environment.
    2. Name it, choose the agent, its version, and an environment.
    3. Under **Instructions**, write what the agent should do each time.
    4. Add triggers: **Scheduled**, **Slack**, or **GitHub**. With none, the automation runs only with **Run now**.
    5. Optionally set **Credential access**, **Files**, and, under **Advanced**, **Run metadata**.
    6. Click **Create automation**. It starts paused.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/automations' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Idempotency-Key: weekday-triage-2026-09-30' \
      -H 'Content-Type: application/json' \
      -d '{
        "displayName": "Weekday triage",
        "agentId": "5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55",
        "agentVersionId": "c1a94e07-2f6b-4d18-b3a5-0e7d8c6f4a21",
        "environmentId": "9d3e7b52-1a4c-4f80-b6e9-2c8a5d0f7e13",
        "initialPrompt": {
          "type": "literal",
          "text": "Read the issues opened in acme/api since the last weekday and post a triage summary."
        },
        "runDefaults": { "metadata": { "team": "platform" } },
        "triggers": [
          {
            "type": "schedule",
            "triggerId": "3f6a9c21-7d4e-4b85-a0c3-5e8d2f1b7a64",
            "enabled": true,
            "schedule": { "cron": "0 9 * * 1-5", "timezone": "Europe/London" }
          }
        ]
      }'
    ```
  </Tab>
</Tabs>

A `201` response is the full automation, with `status` `paused` and a `revision`. Some fields are left out here.

```json theme={"theme":"css-variables"}
{
  "automationId": "a2d7e4c9-1b35-4f86-9e02-7c4b8d1f6a53",
  "displayName": "Weekday triage",
  "agentVersionId": "c1a94e07-2f6b-4d18-b3a5-0e7d8c6f4a21",
  "status": "paused",
  "revision": 1,
  "createdAt": "2026-09-30T09:12:04Z",
  "updatedAt": "2026-09-30T09:12:04Z"
}
```

The `ETag` response header is an opaque validator; send it unchanged as `If-Match` on the next change. Do not construct it from the body's `revision`. New automations start paused: test one with **Run now** from its row's **Actions** menu, then resume it so its triggers can start runs. See [Run, pause, and resume](#run-pause-and-resume).

<Accordion title="Required fields">
  Every request field is required: `displayName`, `agentId`, `agentVersionId`, `environmentId`, `initialPrompt`, `runDefaults` (at least `metadata`, which can be `null`), and `triggers` (an empty list for manual runs only). You choose each `triggerId` yourself, as a UUID, and keep it stable across replacements. A trigger id stays with its automation: a new trigger needs a new UUID, even after the trigger or automation that had the old one is deleted.
</Accordion>

<Accordion title="Retries, Idempotency-Key, and If-Match">
  | Operation | Needs | Retrying |
  | - | - | - |
  | `createAutomation`, `runAutomation` | `Idempotency-Key` | Resend the same request with the same key; it replays the first response instead of creating a second automation or run. |
  | `replaceAutomation`, `deleteAutomation`, `pauseAutomation`, `resumeAutomation` | `If-Match` with the current returned `ETag` | Not safe to replay blindly. Read the automation again, then retry with its new `ETag`. |
  | `pauseEventSource`, `resumeEventSource` | `If-Match` with the source's current `ETag` | Read the source again, then retry. |
  | Lists, reads, and status | Nothing extra | Safe to retry. |

  Copy the complete returned `ETag`, quotes included. A missing `If-Match` returns `428 precondition_required`; a stale one returns `412 precondition_failed`. A public contract change can invalidate an ETag even when the body's `revision` is unchanged: read the resource again before retrying. See [Idempotent mutations](/recursion/api#idempotent-mutations) and [Errors](/recursion/errors#automations-and-event-sources).
</Accordion>

## Choose triggers

An automation has up to 20 triggers, and a run starts when any enabled one matches. The trigger types are `schedule`, `slack`, `github`, and `webhook`. Turn a trigger off with `enabled: false` to keep it without it starting runs. You can also start a run by hand at any time, even while the automation is paused.

### Schedule

Starts runs on a cron expression read in a time zone, such as the create example's `0 9 * * 1-5` in `Europe/London`: 9:00 on weekdays. By default, an occurrence is skipped while the previous run's session is still going.

<Accordion title="Schedule fields">
  These fields go inside the trigger's `schedule` object. Every one you save is applied to the schedule.

  | Field | Meaning |
  | - | - |
  | `cron` | A five-field cron expression, such as `0 9 * * 1-5` for 9:00 on weekdays. The console also offers **Hourly**, **Daily**, **Weekdays**, and **Weekly**. |
  | `timezone` | The IANA time zone the expression is read in, such as `Europe/London`. |
  | `overlapPolicy` | What happens when the previous run's session is still going. See **Overlap policies**. The default is `skip`. |
  | `catchupWindowSeconds` | How late a missed occurrence can still run, from 10 seconds to a year. The default is 900, 15 minutes. |
  | `jitterSeconds` | Up to this many seconds of delay, from 0 to 540, so many schedules don't all start on the same second. The default is 10; 0 turns it off. |
  | `startAt`, `endAt` | Optional times the schedule starts and stops, inclusive. Before `startAt` and after `endAt`, the schedule starts no runs. |
</Accordion>

<Accordion title="Overlap policies">
  | `overlapPolicy` | Console label | Behavior |
  | - | - | - |
  | `skip` | **Skip the next run** | The occurrence is skipped. |
  | `buffer_one` | **Queue one run** | One occurrence waits and runs when the previous finishes. |
  | `buffer_all` | **Queue every run** | Every occurrence waits its turn. |
  | `allow_all` | **Run in parallel** | Occurrences run at the same time. |
  | `cancel_other` | **Cancel the previous session** | The running session is cancelled, and the new occurrence runs. |
</Accordion>

<Accordion title="Check a schedule's status">
  The console shows each schedule's next and last run, upcoming runs, and occurrences skipped for overlap or missed outside the catch-up window. It also shows whether each saved schedule is in effect. `getAutomationStatus` returns the same values for each schedule trigger:

  ```bash theme={"theme":"css-variables"}
  curl 'https://api.recursion.labelbox.com/managed-agents/v1/automations/a2d7e4c9-1b35-4f86-9e02-7c4b8d1f6a53/status' \
    -H "Authorization: Bearer $RECURSION_API_KEY"
  ```

  ```json theme={"theme":"css-variables"}
  {
    "automationId": "a2d7e4c9-1b35-4f86-9e02-7c4b8d1f6a53",
    "triggers": [
      {
        "triggerId": "3f6a9c21-7d4e-4b85-a0c3-5e8d2f1b7a64",
        "synchronization": "synced",
        "available": true,
        "paused": false,
        "nextRunAt": "2026-10-01T08:00:07Z",
        "lastRunAt": "2026-09-30T08:00:04Z",
        "upcomingRuns": ["2026-10-01T08:00:07Z", "2026-10-02T08:00:03Z"],
        "skippedOverlap": 0,
        "missedCatchupWindow": 0
      }
    ]
  }
  ```
</Accordion>

### Slack

Starts a run for Slack messages. It needs a Slack event source: connect Slack on the [Integrations](/recursion/integrations) page. Choose the `message` or `app_mention` event and the channels to accept. This trigger starts a run for each new top-level post in an incident channel that mentions a severity, and ignores other Slack apps:

```json theme={"theme":"css-variables"}
{
  "type": "slack",
  "triggerId": "9c4f2e81-5a7d-4b36-8e19-2f6c3d0a7b58",
  "enabled": true,
  "eventSourceId": "d2b8f6a4-3e17-4c95-a0d6-8b1e5f9c2a73",
  "events": ["message"],
  "routing": { "workspaceIds": [], "channelIds": ["C0APQ7Z1K2M"] },
  "botEvents": { "mode": "ignore" },
  "filters": { "textContains": ["sev1", "sev2"], "ignoreThreadReplies": true }
}
```

* **Filter `message` triggers by text.** Without a text filter, every new message in the trigger's channels starts its own run and a paid session, with no cap. The editor warns about this but still saves. Add `textContains` or `textMatches` so only the messages you mean start work.
* **One post starts one run.** Slack sends a message that mentions the app as both a `message` and an `app_mention` event. One post starts one run per automation, even when a trigger lists both events or two of its triggers each match.
* **Every Slack event starts a new session, unless the trigger continues its thread.** By default a reply in the same thread starts another run with its own session. Set `continuation` to `resume_thread` and a later matching post in a thread is delivered to that thread's session instead, with its transcript and workspace. See [Continue a Slack thread](#continue-a-slack-thread).

<Accordion title="Slack trigger fields">
  | Field | Meaning |
  | - | - |
  | `eventSourceId` | The Slack source. |
  | `events` | `message`, `app_mention`, or both. A post that matches both starts one run per automation. |
  | `routing.channelIds` | Channels to accept. An empty list accepts every channel. The console picks channels the Slack app is a member of. |
  | `routing.workspaceIds` | Workspaces to accept. An empty list accepts every workspace the source receives. |
  | `botEvents.mode` | `ignore` to skip messages from other Slack apps, or `allow_external` to accept them. Messages from the source's own app never start runs. |
  | `allowSharedChannels` | `true` to accept channels shared with other organizations. The default is `false`. |
  | `filters` | Optional `textContains`, `textExcludes`, `textMatches` (a regular expression), `ignoreThreadReplies`, and `fromUserIds`. Text matching ignores case. |
  | `filters.fromUserIds` | The people and apps whose messages may start runs: a person's Slack user id or an app's bot id, as the member catalog lists them. An empty or absent list accepts anyone. An app listed here starts runs even when `botEvents.mode` is `ignore`. An app's message matches only by its bot id, even when the app posts on a listed person's behalf. |
  | `continuation` | `new_session` (the default) starts a session for every matching post. `resume_thread` keeps one session per Slack thread. |
  | `firstResponse` | `none` (the default) posts nothing until the agent replies. `platform` posts an acknowledgement, a reaction, and a live status line while the agent works. |

  The console lists up to 1,000 channels the connected Slack app is a member of. Invite the app to a channel to make it selectable. The same list is available from the API for a Slack connection:

  ```bash theme={"theme":"css-variables"}
  curl 'https://api.recursion.labelbox.com/managed-agents/v1/integrations/connections/b4e81f26-9c3d-4a57-8e10-2d6f5a9c7b43/channel-catalog' \
    -H "Authorization: Bearer $RECURSION_API_KEY"
  ```

  The people and apps for `fromUserIds` come from the connection's member catalog, read with the same connection:

  ```bash theme={"theme":"css-variables"}
  curl 'https://api.recursion.labelbox.com/managed-agents/v1/integrations/connections/b4e81f26-9c3d-4a57-8e10-2d6f5a9c7b43/member-catalog' \
    -H "Authorization: Bearer $RECURSION_API_KEY"
  ```
</Accordion>

#### Continue a Slack thread

With `"continuation": "resume_thread"`, an automation keeps one conversation per Slack thread. Every matching post still records a run, and the run's `sessionAction` says what it did:

| `sessionAction` | When | What the agent has |
| - | - | - |
| `started` | The first matching post in a thread. | A new session. |
| `appended` | The thread's session is still working, or waiting on a person. | The message arrives at its next turn. |
| `resumed` | The thread's session had finished, failed, or been cancelled. | The same transcript. The sandbox is reused after a finished or failed run and replaced after a cancel; `/workspace` survives a stopped sandbox, running processes don't. |
| `successor` | The thread's session was deleted, can never resume, or the automation now names another agent. | A new session. Its prompt is the only context, so have it read the thread. |

* **A conversation belongs to one automation and one thread.** Two automations that both continue the same thread keep separate sessions, and a thread in another channel, workspace, or organization never reaches this one.
* **A resumed session keeps its agent version.** Publishing a version and moving the automation's pin changes new threads only. To change the version a thread's session runs, [move that session](/recursion/sessions#move-a-session-to-another-version).
* **Posts that arrive together join one session.** The first one starts it and the rest are appended, each exactly once, in the order they were admitted.
* **A refusal that can pass is retried, not replaced.** If the session is still stopping, its input queue is full, or the tenant is out of credit, the run stays `pending` and is delivered when it can be.
* **Give every Slack trigger of an automation the same `continuation` and `firstResponse`.** One post that matches two triggers starts one run, so the API rejects triggers that disagree.

A common shape is two triggers on one automation: `message` with `ignoreThreadReplies` for new top-level posts in a channel, and `app_mention` for replies that mention the app, including replies in that same channel.

#### Acknowledge a Slack post

With `"firstResponse": "platform"`, the platform reacts to the post, posts an acknowledgement in the thread, and edits that acknowledgement into a one-line status while the agent works. When the turn ends the status line is removed, or replaced by a line saying the run stopped if the session failed or was cancelled. It posts as the Slack app that received the event, with the bot token of the workspace's Slack connection on the [Integrations](/recursion/integrations) page. The reaction needs the `reactions:write` scope, which a Slack connection made before that scope was requested does not hold: reconnect Slack to add it. Without it the acknowledgement and status line still appear. When the workspace has no active Slack connection for that app, the trigger behaves as `none`: the run is unaffected, and a failed run still posts its failure note.

<Accordion title="Messages that never start runs, and stopping a Slack session">
  * **Only newly posted messages start runs.** Editing or deleting a message never starts one.
  * **Direct messages to the app don't start runs.** Use a channel.
  * **Stopping a Slack-started session posts nothing to its thread.** The console reminds you when you force-stop or interrupt one. Reply in the thread yourself, or message the agent to post an update.
</Accordion>

### GitHub

Starts a run for GitHub webhook events. It needs a GitHub event source: connect GitHub on the [Integrations](/recursion/integrations) page. This trigger matches `pull_request` deliveries in `acme/api` with the action `opened` or `synchronize` on `fix/*` branches, and ignores drafts and bots:

```json theme={"theme":"css-variables"}
{
  "type": "github",
  "triggerId": "8b2e4d71-6c39-4a05-9f18-3d7a5c1e9b42",
  "enabled": true,
  "eventSourceId": "e1c7a3f9-4b26-4d80-a5e3-9f2b6d8c1a07",
  "events": ["pull_request"],
  "filters": {
    "repositories": ["acme/api"],
    "actions": ["opened", "synchronize"],
    "branches": ["fix/*"],
    "excludeDrafts": true,
    "ignoreBots": true
  }
}
```

<Note>
  The event only starts the run. To read or change the repository, the agent needs a GitHub grant, and it works through GitHub's MCP tools. No GitHub credential is placed in the sandbox, so `git` and `gh` there aren't authenticated. See [GitHub access](/recursion/github).
</Note>

Three more common triggers. This one hands a pull request to the agent when someone adds the `agent-ready` label:

```json theme={"theme":"css-variables"}
{
  "type": "github",
  "triggerId": "4a7c9e23-8d15-4f60-b2e8-6c1d3a5f9b07",
  "enabled": true,
  "eventSourceId": "e1c7a3f9-4b26-4d80-a5e3-9f2b6d8c1a07",
  "events": ["pull_request"],
  "filters": {
    "repositories": ["acme/api"],
    "actions": ["labeled"],
    "addedLabels": ["agent-ready"],
    "excludeDrafts": true,
    "ignoreBots": true
  }
}
```

This one starts a run when a pull request comment contains the command `/agent fix`:

```json theme={"theme":"css-variables"}
{
  "type": "github",
  "triggerId": "b3e5d8f1-2c74-4a96-8d0b-5f7e1c9a3d62",
  "enabled": true,
  "eventSourceId": "e1c7a3f9-4b26-4d80-a5e3-9f2b6d8c1a07",
  "events": ["issue_comment"],
  "filters": {
    "repositories": ["acme/api"],
    "actions": ["created"],
    "commentOn": "pull_request",
    "textContains": ["/agent fix"],
    "ignoreBots": true
  }
}
```

This one starts a run when a CI workflow run on `main` finishes with a failure:

```json theme={"theme":"css-variables"}
{
  "type": "github",
  "triggerId": "7f2a6c94-1e58-4d3b-a9c7-0b4e8d2f6a15",
  "enabled": true,
  "eventSourceId": "e1c7a3f9-4b26-4d80-a5e3-9f2b6d8c1a07",
  "events": ["workflow_run"],
  "filters": {
    "repositories": ["acme/api"],
    "actions": ["completed"],
    "conclusions": ["failure"],
    "branches": ["main"]
  }
}
```

<Accordion title="GitHub trigger fields and filters">
  | Field | Meaning |
  | - | - |
  | `eventSourceId` | The GitHub source. |
  | `events` | GitHub event names. The console offers only the events that can start runs. A save that names any other event is refused, and the error lists the accepted names. |
  | `filters` | Optional conditions, all of which must match. |

  | Filter | Matches |
  | - | - |
  | `repositories` | Repository full names (`owner/name`), ignoring case. The console's repository picker lists the repositories of the connection's GitHub installation. |
  | `actions` | Delivery actions, such as `opened`, `labeled`, `submitted`, or `completed`. |
  | `branches` | Globs for the pushed branch, the workflow run's head branch, or the pull request's head branch. |
  | `baseBranches` | Globs for the pull request's base branch. |
  | `labelsAny`, `labelsNone`, `addedLabels` | Labels on the pull request or issue, or the label just added. |
  | `senders`, `ignoreBots` | The GitHub account that caused the delivery. |
  | `excludeDrafts`, `merged` | Draft pull requests, and whether a closed pull request was merged. |
  | `commentOn` | For `issue_comment`: `pull_request` or `issue`. |
  | `reviewStates`, `conclusions` | Submitted review states, and completed workflow run conclusions. |
  | `textContains`, `textExcludes`, `textMatches` | The comment or review body, or else the pull request or issue title and body, ignoring case. `textMatches` is a regular expression. |

  In globs, `*` matches any characters, including `/`, and `?` matches one character.
</Accordion>

### Webhook

Starts a run for deliveries to a custom webhook source. The console offers this trigger when your organization has a custom source. Add `filters.fieldEquals` to start runs only for some deliveries. Each condition names a dot-separated `path` into the JSON body, such as `data.level`, and the `value` it must equal, written as JSON text without quotes.

```json theme={"theme":"css-variables"}
{
  "type": "webhook",
  "triggerId": "5d9c1e37-2a84-4f60-b7c3-1e8a4d6f9b25",
  "enabled": true,
  "eventSourceId": "7a4e2c18-9d53-4b61-8f07-2c6e9a3d5b14",
  "filters": { "fieldEquals": [{ "path": "data.level", "value": "critical" }] }
}
```

To send deliveries to the source, see [Send a custom webhook delivery](#send-a-custom-webhook-delivery).

## Set run defaults

`runDefaults` applies to every run, whether a trigger or a person starts it.

| Field | Meaning |
| - | - |
| `metadata` | String key/value pairs copied onto every session, for example to find them later. `null` for none. Their count and size limits are in [Limits](/recursion/limits#automations). |
| `vaultIds` | Vaults each session can use. Leave it out to use the agent's default vaults. An empty list grants no vault access. |
| `credentialRefs` | Specific credentials within `vaultIds`. Leave it out to grant everything in those vaults. |
| `resources` | [Files](/recursion/files) from your organization, each with a `fileId` and a `mountPath` under the session's files directory, such as `/data/input.csv`. |

## Keep untrusted input in check

Event runs put outside text into the session: Slack messages, pull request and issue titles and bodies, comments, and webhook bodies. Treat that text as untrusted input. Someone who can post in the channel or open an issue can try to steer the agent.

* **Restricted environments drop `web_fetch`.** A session an automation starts, or a session started from a Slack thread, gets no `web_fetch` tool when its environment limits internet access (limited or no access). The environment's network policy is then its only way out.
* **Unrestricted environments keep `web_fetch` and full internet access.** An injected instruction could ask the agent to send data anywhere. Use a limited environment for automations that read outside text, and grant only the credentials and repositories the task needs.
* **Webhook runs record only the sender's headers.** The event in a webhook run keeps the headers the sender sent. Headers added on the way to the event source are never recorded.

See [Environments](/recursion/environments#control-network-access) and [Security](/recursion/security#automations-and-event-sources).

## Run, pause, and resume

Start one run by hand to test the automation, then resume it so its triggers can start runs.

<Tabs>
  <Tab title="Console">
    1. In **Automations**, open the automation row's **Actions** menu.
    2. Click **Run now** to start one run, or **Run with input** to add a **JSON payload**. Both need create permission.
    3. Click **Resume** to let its triggers start runs, or **Pause** to stop them.
  </Tab>

  <Tab title="cURL">
    Set `AUTOMATION_ETAG` to the complete `ETag` header returned by the current automation read. Refresh it from each write response before the next change.

    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/automations/a2d7e4c9-1b35-4f86-9e02-7c4b8d1f6a53/runs' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Idempotency-Key: triage-rerun-2026-09-30' \
      -H 'Content-Type: application/json' \
      -d '{"payload": {"repository": "acme/api", "since": "2026-09-29"}}'

    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/automations/a2d7e4c9-1b35-4f86-9e02-7c4b8d1f6a53/resume' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H "If-Match: $AUTOMATION_ETAG"

    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/automations/a2d7e4c9-1b35-4f86-9e02-7c4b8d1f6a53/pause' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H "If-Match: $AUTOMATION_ETAG"
    ```
  </Tab>
</Tabs>

`runAutomation` answers with the run. **What success means:** the run moves from `pending` to `created`, and `sessionId` names the session it started. Follow that session like any other; see [Sessions](/recursion/sessions).

Resume returns the automation with `status` `active`. Pausing stops triggers from starting runs; **Run now** still works. Each change raises the `revision`, so use the new `ETag` for the next one.

<Accordion title="Run input, failed runs, and automatic pauses">
  `payload` is optional and can be any JSON value up to 64 KiB (65,536 bytes) as JSON; a larger one is refused with `400` on `payload`.

  A run that couldn't start a session ends `failed` with an `error` that has a `code`, a `message`, the `field` or `resourceId` responsible when there is one, and `retryable`.

  Resume is refused with `422 automation_reference_invalid` while the automation's environment is deleted, or while it names an agent or agent version that no longer exists; the automation stays paused. If a run fails because something the automation uses no longer exists, such as its agent, environment, a vault, a credential, or a file, the automation is paused, and the run's error says what is gone. Fix the automation, then resume it.
</Accordion>

## See run history

In **Automations**, choose **View runs** from the automation row's **Actions** menu. Each run shows its **Trigger**, when it started, and **Session creation**. Filter by status, trigger, or date, and click a run for its **Run detail** and a link to its session.

In the API, list an automation's runs, then read one. `getAutomationRun` returns one run with its frozen snapshots and `effectiveInitialMessage`.

```bash theme={"theme":"css-variables"}
curl 'https://api.recursion.labelbox.com/managed-agents/v1/automation-runs?automationId=a2d7e4c9-1b35-4f86-9e02-7c4b8d1f6a53&status=failed' \
  -H "Authorization: Bearer $RECURSION_API_KEY"
curl 'https://api.recursion.labelbox.com/managed-agents/v1/automation-runs/6e3b9d14-8a27-4c50-b1f9-0d5c7e2a4b86' \
  -H "Authorization: Bearer $RECURSION_API_KEY"
```

```json theme={"theme":"css-variables"}
{
  "automationRunId": "6e3b9d14-8a27-4c50-b1f9-0d5c7e2a4b86",
  "automationId": "a2d7e4c9-1b35-4f86-9e02-7c4b8d1f6a53",
  "triggerType": "manual",
  "status": "created",
  "sessionId": "b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38",
  "statusUrl": "/v1/automation-runs/6e3b9d14-8a27-4c50-b1f9-0d5c7e2a4b86",
  "createdAt": "2026-09-30T09:20:11Z",
  "updatedAt": "2026-09-30T09:20:13Z"
}
```

<Accordion title="Run filters and paging">
  `listAutomationRuns` returns runs newest first. Filter with `automationId`, `triggerId`, `eventSourceId`, `triggerType`, `status`, `createdAtGte`, and `createdAtLt`, and page with `limit` and `cursor`.
</Accordion>

## Change or delete an automation

Saving a change keeps the automation's current status. Deleting it removes its triggers while preserving records for runs that already started.

<Tabs>
  <Tab title="Console">
    In **Automations**, use the automation row's **Actions** menu to open it for editing or to delete it. The automations list is newest first, and the editor shows the automation's prompt and triggers. Click **Save changes** to save an edit.
  </Tab>

  <Tab title="cURL">
    `listAutomations` returns summaries, and `getAutomation` returns the full automation with its `ETag`. `replaceAutomation` takes the same complete body as create, so send every field you want to keep. Replace and delete need `If-Match`. Set `AUTOMATION_ETAG` to the current returned header and refresh it after a successful replace before deleting.

    ```bash theme={"theme":"css-variables"}
    curl 'https://api.recursion.labelbox.com/managed-agents/v1/automations' \
      -H "Authorization: Bearer $RECURSION_API_KEY"

    curl -X PUT 'https://api.recursion.labelbox.com/managed-agents/v1/automations/a2d7e4c9-1b35-4f86-9e02-7c4b8d1f6a53' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H "If-Match: $AUTOMATION_ETAG" \
      -H 'Content-Type: application/json' \
      -d '{
        "displayName": "Weekday triage",
        "agentId": "5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55",
        "agentVersionId": "d8f25b1c-6e4a-4a93-b7c0-3f1e9d5a2c68",
        "environmentId": "9d3e7b52-1a4c-4f80-b6e9-2c8a5d0f7e13",
        "initialPrompt": { "type": "literal", "text": "Read the issues opened in acme/api since the last weekday and post a triage summary." },
        "runDefaults": { "metadata": { "team": "platform" } },
        "triggers": [
          {
            "type": "schedule",
            "triggerId": "3f6a9c21-7d4e-4b85-a0c3-5e8d2f1b7a64",
            "enabled": true,
            "schedule": { "cron": "0 9 * * 1-5", "timezone": "Europe/London" }
          }
        ]
      }'

    curl -X DELETE 'https://api.recursion.labelbox.com/managed-agents/v1/automations/a2d7e4c9-1b35-4f86-9e02-7c4b8d1f6a53' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H "If-Match: $AUTOMATION_ETAG"
    ```

    A delete answers `204`.
  </Tab>
</Tabs>

Deleting an agent deletes the automations that run it. See [Delete an agent](/recursion/agents#delete-an-agent).

<Accordion title="If someone else saved first">
  If someone else saved the automation while you were editing it, **Save changes** saves nothing and shows **This automation changed elsewhere. Nothing was saved.** Choose **Keep my edits** to replace their version with yours on the next save, or **Load the newer version** to see what they saved.
</Accordion>

### Keep the agent version current

An automation keeps the agent version it was saved with. Later changes to the agent's instructions, tools, and integrations don't apply to its runs until you move it to the new version. In the **Automations** list, an **Older version** label marks automations that run an older version than their agent's latest. When you open one, the drawer names both versions, for example "This automation runs Version 1 of backend-fixer. The agent is now on Version 2", and offers **Use Version 2**. Click it, then **Save changes**. In the API, replace the automation with the new `agentVersionId`. Moving an automation doesn't move sessions it already started; [move those sessions](/recursion/sessions#move-a-session-to-another-version) themselves.

## Receive events

An **event source** is an endpoint that receives events and checks they're genuine. Automations listen to it through Slack, GitHub, and webhook triggers. One source can serve many automations, up to the enabled-trigger limit in [Limits](/recursion/limits#automations).

| Source type (`type`) | Receives | Comes from |
| - | - | - |
| `slack_events_api` | Slack Events API callbacks. | Connecting Slack on **Integrations**. |
| `github_webhook` | GitHub webhook deliveries. | Connecting GitHub on **Integrations**. |
| `custom_webhook` | JSON from your own services. | Custom sources already in your organization. |

Sources created by connecting an account are **connection-managed**: they start active, and you pause and resume them from the list.

<Accordion title="Signing secrets and disconnecting">
  Slack and GitHub sources check every delivery against a signing secret that the connection creates and manages. Signing secrets never reach an agent sandbox. Disconnecting the account removes its source and stops its triggers. Existing automations keep those trigger settings so you can select a new source after reconnecting.
</Accordion>

### Manage sources

In **Automations**, open the **Event sources** tab. Each source's health and last delivery are columns in the list, and its row's **Actions** menu pauses or resumes deliveries. A healthy source has received a delivery, which does not prove the provider can reach it right now.

A connection-managed source is changed and removed through its connection on **Integrations**. A custom source can be deleted when no automation has a trigger on it. A switched-off trigger still holds its source, although the list's **Triggers** column counts only the ones that are on. A refused delete names the automations: the console lists them, and the `409 conflict` response carries them in `details.dependents`.

<Accordion title="Source health">
  | `health` | Meaning |
  | - | - |
  | `never_received` | No delivery has arrived yet. |
  | `healthy` | A delivery has arrived. |
  | `paused` | The source is paused. |
  | `connection_unavailable` | The connection behind the source isn't usable. Check or reconnect it on **Integrations**. |
  | `setup_required` | The connected GitHub app needs its event permissions approved. |
</Accordion>

<Accordion title="Manage sources with the API">
  `listEventSources` and `getEventSource` return sources, and `getEventSourceStatus` returns `health`, `triggerCount`, and `lastDelivery`. Pause and resume need `If-Match` with the source's `ETag`. Set `EVENT_SOURCE_ETAG` to the complete header returned by `getEventSource`.

  ```bash theme={"theme":"css-variables"}
  curl 'https://api.recursion.labelbox.com/managed-agents/v1/event-sources' \
    -H "Authorization: Bearer $RECURSION_API_KEY"

  curl 'https://api.recursion.labelbox.com/managed-agents/v1/event-sources/e1c7a3f9-4b26-4d80-a5e3-9f2b6d8c1a07/status' \
    -H "Authorization: Bearer $RECURSION_API_KEY"

  curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/event-sources/e1c7a3f9-4b26-4d80-a5e3-9f2b6d8c1a07/pause' \
    -H "Authorization: Bearer $RECURSION_API_KEY" \
    -H "If-Match: $EVENT_SOURCE_ETAG"
  ```
</Accordion>

### Send a custom webhook delivery

A custom source receives deliveries at its **Callback URL**, shown in **Event sources**. The full URL is the console's address followed by the callback path, `https://recursion.labelbox.com/agents/v1/event-sources/<eventSourceId>/events`. Give it to the sender, and resume the source when you're ready to receive events.

Send a JSON body with `POST` to the callback URL. For an `hmac_sha256` source, compute the HMAC-SHA256 of the exact body bytes, keyed with the source's signing secret, and send it as lowercase hexadecimal in the signature header, after the prefix. Sign the bytes you send; re-serializing the JSON after signing breaks the signature.

```http theme={"theme":"css-variables"}
POST /agents/v1/event-sources/7a4e2c18-9d53-4b61-8f07-2c6e9a3d5b14/events HTTP/1.1
Content-Type: application/json
X-Signature-256: sha256=5b1c9e0f3a7d2b8e4c6f1a9d3e7b2c5f8a0d4e6b9c1f3a5d7e2b8c4f6a0d9e1b

{"data":{"level":"critical","service":"billing"}}
```

A delivery that's accepted returns `200` with `{"ok": true}`, and matching triggers on active automations start runs. The same signed body is treated as the same delivery, so a retry doesn't start a second run. For an unsigned source, send an `X-Webhook-Id` or `Idempotency-Key` header to name the delivery.

<Accordion title="Every delivery response">
  | Response | Meaning |
  | - | - |
  | `200` | Accepted. Matching triggers on active automations start runs. |
  | `401 unauthorized` | The signature didn't match. |
  | `400 invalid_request` | The body isn't JSON, or the request is malformed. |
  | `404 not_found` | The source doesn't exist or was deleted. |
  | `413 payload_too_large` | The body is over 1 MiB. |
  | `429 rate_limited` | Too many deliveries. Retry after `Retry-After`. |
  | `503 event_admission_unavailable` | The delivery couldn't be recorded. Retry. |
</Accordion>

## How automations run

```mermaid theme={"theme":"css-variables"}
flowchart TD
  schedule["Schedule"] --> run["Run: pending"]
  event["Slack, GitHub,<br>or webhook event"] --> run
  manual["Run now"] --> run
  run -->|"session created"| created["Run: created"]
  run -->|"could not start"| failed["Run: failed"]
```

* **Every run uses the pinned agent version.** Publishing a new agent version doesn't change the automation. See [Keep the agent version current](#keep-the-agent-version-current).
* **Everything is frozen per run.** A run records the automation, trigger, and event exactly as they were, and the session's opening message. Editing the automation later doesn't change runs already started.
* **The opening message** is your saved prompt. For an event run, it's followed by `Event for this run:` and the event as JSON. For a manual run with input, it's followed by `Input for this run:` and the input. For a scheduled run, it ends with `This is a scheduled run for <time>.`, where the time is the occurrence in UTC, such as `2026-10-01T08:00:00Z`.
* **Sessions show where they came from.** A session an automation started shows **Started by** in its session panel: the automation, the trigger (**Manual**, **Schedule**, **Slack**, **GitHub**, or **Webhook**), and the run, each linked.

## What can go wrong

The most common problems:

* **A run is `failed`.** The session couldn't start. Read the run's `error`. If `retryable` is `true`, run it again; otherwise fix the named `field` or resource.
* **An automation paused on its own.** A run found that something it uses was deleted. Read the failed run's `error`, fix the automation, then resume it.
* **An event starts no runs.** Check the source status and the trigger filters. A Slack message that was edited, deleted, or sent as a direct message never starts a run.
* **Runs still use access you removed from the agent.** The automation runs its pinned agent version. Move it to the new version. See [Keep the agent version current](#keep-the-agent-version-current).

<Accordion title="Every automation error">
  | Code or symptom | Cause | Fix |
  | - | - | - |
  | `422 automation_reference_invalid` on save | The agent version or environment doesn't exist or can't be used. | Choose a live agent version and environment. |
  | `422 automation_event_source_invalid` | A trigger's event source is missing or is the wrong type for the trigger. | Use a Slack source for a Slack trigger, and so on. |
  | `409 conflict` on `triggers` | The save would put an event source over its enabled-trigger limit. See [Limits](/recursion/limits#automations). | Switch off or delete triggers on that source, then save again. |
  | `409 automation_trigger_id_in_use` | Another automation has, or once had, a trigger with that `triggerId`. A trigger id is never reused. | Give the trigger a new UUID, then save again. |
  | A run is `failed` with `environment_not_verified` | The environment's setup script hasn't been verified, or the environment changed since it was. The automation stays active. | Run the environment's setup; later runs then start. See [Verify the setup script](/recursion/environments#verify-the-setup-script). |
  | `422 automation_reference_invalid` on resume | The automation's environment was deleted, or it names an agent or agent version that no longer exists. | Choose a live agent version and environment, save, then resume. |
  | An automation paused on its own | A run found that something it uses was deleted. | Read the failed run's `error`, fix the automation, then resume it. |
  | **This automation changed elsewhere. Nothing was saved.** | Someone saved the automation while you were editing. | Choose **Keep my edits** or **Load the newer version**, then save. |
  | `428 precondition_required` | A replace, delete, pause, or resume sent no `If-Match`. | Read the resource and send its `ETag` as `If-Match`. |
  | `412 precondition_failed` | The automation or source changed after you read it. | Read it again and retry with its new `ETag`. |
  | `400` on `payload` | The run input is over 64 KiB as JSON. | Send less input, or put large data in a file. |
  | `409 idempotency_conflict` | You reused an `Idempotency-Key` with a different request. | Use a new key for a new request. |
  | A source can't be deleted | An automation still has a trigger on it, switched off or not, or the source is managed by its connection. | Delete the automations the refusal names, or remove their triggers on this source, then delete again. An automation whose agent or environment is gone can't be saved until you pick another, so delete it instead. For a managed source, disconnect the account on **Integrations**. |
  | A schedule never runs | The automation or trigger is paused or off, or the time is outside `startAt` and `endAt`. | Resume the automation, and check its schedule status. |
  | A run is `failed` | The session couldn't start. | Read the run's `error`. If `retryable` is `true`, run it again; otherwise fix the named `field` or resource. |
  | Runs still use access you removed from the agent | The automation runs its pinned agent version. | Move it to the new version. See [Keep the agent version current](#keep-the-agent-version-current). |
  | No runs from a Slack message | The message was edited, deleted, or sent as a direct message, the channel isn't routed, or the filters didn't match. | Post a new message in a routed channel, and check the filters. |
  | No runs from a GitHub or webhook event | The source is paused, the signature failed, or the filters didn't match. | Check the source status and trigger filters. |
  | The session has no `web_fetch` | The environment limits internet access, and the session was started by an automation or from Slack. | Allow the hosts the task needs in the environment's network policy. |
  | `health` is `setup_required` | The connected GitHub app needs its event permissions approved. | Approve the permission update in GitHub, then check the connection on **Integrations**. |

  For every error code, see [Errors](/recursion/errors).
</Accordion>

## Limits

Limits on triggers, prompts, names, run input, run defaults, schedules, trigger filters, and custom webhook bodies are in [Limits](/recursion/limits). Two more apply here:

* A page of automations or event sources holds 1 to 100 items, 50 by default.
* Name and prompt limits count UTF-8 bytes, not characters: an emoji or a non-Latin letter takes 2 to 4 bytes, so a name in those scripts fits fewer than 256 characters.

## Next steps

<CardGroup>
  <Card title="Integrations" href="/recursion/integrations">
    Connect Slack and GitHub to get event sources.
  </Card>

  <Card title="GitHub access" href="/recursion/github">
    Give automated runs access to the repositories they work on.
  </Card>

  <Card title="Files" href="/recursion/files">
    Mount the same files into every run.
  </Card>

  <Card title="Sessions" href="/recursion/sessions">
    Follow the sessions automations start.
  </Card>
</CardGroup>


# Billing
Source: https://docs.labelbox.com/recursion/billing

Buy prepaid credits, manage the cards that pay for them, track purchases and receipts, and see what sessions are charged.

Managed Agents uses prepaid credits in US dollars. You buy credits for your tenant from the **Billing** screen, pay by card on Stripe's secure pages, and download a receipt for every purchase.

<Note>
  You need the **billing**, **admin**, **owner**, or **primary owner** tenant role; other roles don't see **Billing**. See [Organizations and roles](/recursion/organizations-and-roles). Billing happens in the console while you're signed in: buying credits, reading the balance, and managing cards are console actions, not API-key operations. Credits belong to the tenant as a whole, not to one organization.
</Note>

## Read your balance

Open the [Recursion console](https://recursion.labelbox.com) and click **Billing** in the sidebar. The top of the screen shows two tiles:

* **Credit balance**: the credit you have, in US dollars. Purchased credit doesn't expire.
* **Payment method**: your default card's brand, last four digits, and expiry, or **No card**.

When you manage billing, **Launch session** warns if your credit balance covers less than about an hour of the selected agent's estimated cost, and says how many minutes it covers. The warning links to **Buy credits**. You can still launch.

When the balance reaches zero, new work is refused and running sessions pause. See [When credits run out](#when-credits-run-out).

## What sessions cost

Sessions are charged by the session hour, covering compute and model spend; see [Pricing](/recursion/pricing). Each session is charged to your credit balance for its whole session tree: the model calls, provider tools, and sandbox compute of the session and every subagent and teammate it started. **Sessions** shows that one cost in its **Cost** column and in each session's header. With the API, read `costUsd` and `costState` on the session; see [Usage and cost](/recursion/usage-and-cost). Reading a session's cost needs an organization role that can read sessions.

Session costs can take up to an hour to settle, so the balance can trail the work. A sandbox can keep running after its session ends, until the environment's idle stop, and that compute is charged too. A cost marked **so far** can still grow; once it is final, it won't change unless you resume the session.

## Buy credits

Choose an amount, and pay with your saved card or on Stripe's page.

<Tabs>
  <Tab title="Console">
    1. In **Billing**, find **Buy credits**.
    2. Choose an amount: **$50**, **$100**, **$250**, or **$500**, or type any whole number of dollars from $5 to $1,000 in **Other**. **\$100** is selected to start with.
    3. Click **Buy \$100** (the button shows your amount).
    4. Confirm the purchase. What you're asked depends on your card:
       * **With a saved card**, the prompt names the card and the balance you'll end up with. Click **Confirm** to charge it now. You'll see **Payment received**.
       * **Without a saved card**, click **Continue**, then **Pay \$100 on Stripe**, and enter a card on Stripe's secure invoice page. The card is saved for later purchases.
       * **If your bank asks for confirmation**, finish it on the same Stripe page.
    5. Come back to the console tab. It watches the purchase and updates on its own.
  </Tab>
</Tabs>

**What success means:** the purchase shows **Paid** in **Purchases**, and the credit appears in **Credit balance** within a few seconds. The balance updates only after Stripe confirms the payment, so it can trail the purchase briefly.

Buying is safe to retry. If a purchase's response is lost and you buy the same amount again, you get the same invoice rather than a second charge.

## Cancel an unpaid purchase

If you started a purchase that goes to Stripe's page and you don't want it, click **Cancel purchase** in the drawer before you open Stripe. After that, including when you close the Stripe tab without paying, open the purchase's actions menu in **Purchases** and click **Cancel purchase**.

An unpaid purchase is canceled automatically 24 hours after it was created. A purchase that was already charged can't be canceled.

## Manage cards

Payments are card only. You enter card details on Stripe's pages; the console never shows a card field.

* To add your first card, click **Add card** on the **Payment method** tile. The new card becomes the default.
* To change the default, remove, or add cards, click **Manage cards** on the **Payment method** tile.

Stripe opens in a new tab. When you come back, the console refreshes your card. If you remove the default card, your next purchase asks for a card on Stripe's page.

## Track purchases and receipts

**Purchases** lists every purchase, newest first, with its date, credits, and status. A paid purchase has a **Receipt** link to its Stripe invoice.

<Accordion title="Every purchase status">
  | Status | Meaning |
  | - | - |
  | **Pending** | The purchase was started but not finished. Buy again, cancel it, or wait for it to be canceled. |
  | **Awaiting payment** | Waiting for a card on Stripe's page. Click **Pay on Stripe**, or cancel it. |
  | **Needs confirmation** | Your bank asked for confirmation. Click **Pay on Stripe** to finish, or cancel it. |
  | **Processing** | The card was charged. The credit arrives in a moment. |
  | **Paid** | The credit was added. The receipt is available. |
  | **Declined** | The card was declined on Stripe's page. Nothing retries it. Cancel it, or wait for it to be canceled. |
  | **Canceled** | Your saved card was declined, or the purchase was abandoned or canceled. Nothing was charged. |
</Accordion>

## When credits run out

When your credit balance reaches zero, new work can't start until you buy credits:

* New sessions, and messages that start a new turn, are refused.
* **Test setup** is refused, because a setup test runs on compute like a session does. The **Setup test** panel says there are no credits and links to **Buy credits**. If you don't manage billing, it asks you to get someone who does to add credits.
* A session that is already running, or a sleeping one that wakes on its own, pauses before its next model request, with `stop_reason: out_of_credit`. Requests already in flight complete and are charged, so the balance can end slightly below zero. A paused session keeps its sandbox until the environment's idle stop, and that compute is billed too. Once your tenant has credit again, from a purchase, an auto-reload, a staff grant, or unlimited mode, a session paused in the last 72 hours continues on its own within about a minute, with its subagents, from where it stopped. Its transcript shows a message saying credit was added. Your tenant runs only as many of them at once as it may run sessions; the rest continue as others finish. A session bound to a Slack thread continues on your next reply there. To continue a session sooner, or one paused longer ago, send it a message.

Saving agents and environments uses no compute, so running out of credit never blocks it: **Save and test setup** saves the environment even when the test is refused. When the **Credit balance** tile says **Unlimited**, credit doesn't limit new work, and the balance can read below zero.

## What can go wrong

The most common problems:

* **Billing isn't in the sidebar.** Your tenant role isn't billing, admin, owner, or primary owner. Ask an owner or admin to change your role.
* **"The card was declined. Replace it in Stripe, then try again."** The bank declined the charge. Click **Replace card** below the message, add or choose another card on Stripe, then come back and buy again.
* **Paid, but the balance hasn't changed.** The credit lands a few seconds after Stripe confirms payment. Wait, or come back to the tab to refresh.
* **The balance kept dropping after a session ended.** Session costs can take up to an hour to settle, and a sandbox's compute is charged until the environment's idle stop.

<Accordion title="Every billing problem">
  | Message or symptom | Cause | Fix |
  | - | - | - |
  | **Billing** isn't in the sidebar | Your tenant role isn't billing, admin, owner, or primary owner. | Ask an owner or admin to change your role. |
  | "The card was declined. Replace it in Stripe, then try again." | The bank declined the charge. | Click **Replace card** below the message, add or choose another card on Stripe, then come back and buy again. |
  | "Something changed since your last try, such as the saved card. Buy again." | The retry no longer matches the earlier attempt. | Buy the credits again to start a new purchase. |
  | **Buy credits** is disabled | The screen is still loading your card. | Wait a moment, or refresh. |
  | The amount field shows an error | The amount isn't a whole number from 5 to 1,000. | Enter a whole-dollar amount in range. |
  | Paid, but the balance hasn't changed | The credit lands a few seconds after Stripe confirms payment. | Wait, or come back to the tab to refresh. |
  | The balance kept dropping after a session ended | Session costs can take up to an hour to settle, and a sandbox's compute is charged until the environment's idle stop. | Wait for the session's cost to read final. See [Usage and cost](/recursion/usage-and-cost). |
</Accordion>

## Limits

| Limit | Value |
| - | - |
| One purchase | $5 to $1,000, whole dollars |
| Unpaid purchase | Canceled about 24 hours after it's created |

## Next steps

<CardGroup>
  <Card title="Pricing" href="/recursion/pricing">
    See what a session hour costs and what that price covers.
  </Card>

  <Card title="Usage and cost" href="/recursion/usage-and-cost">
    See what each session cost, including its helper agents and compute.
  </Card>

  <Card title="Organizations and roles" href="/recursion/organizations-and-roles">
    See who can manage billing.
  </Card>

  <Card title="Console" href="/recursion/console">
    Find every screen in the console.
  </Card>

  <Card title="Security" href="/recursion/security">
    See how payment details and credentials are handled.
  </Card>
</CardGroup>


# Build in the console
Source: https://docs.labelbox.com/recursion/console

Create an agent, prepare its environment, launch a session, and find every screen in the Recursion console.

The [Recursion console](https://recursion.labelbox.com) is where you create agents, prepare environments, connect tools, launch sessions, and follow their work. Sessions started through the API appear here too.

<Note>
  Sign in to the console and choose the organization you want to work in from the switcher near the top of the sidebar. Every write action depends on your role. See [Organizations and roles](/recursion/organizations-and-roles).
</Note>

## Follow Get started

Until you finish it or dismiss it, **Sessions** shows **Get started** under its heading. It numbers the steps to a first session: create an agent, set up an environment, and launch a session. Creating an API key is an optional fourth step. The next step's button is highlighted.

<Accordion title="How Get started works">
  Get started first says how agents, environments, and sessions fit together, then lists the steps.

  * A step shows **Done** once your organization has an agent, an environment, or a session, or once you have an active API key.
  * Get started goes away once your organization has an agent, an environment, and a session, whether or not you have an API key.
  * **Create agent** opens the create form on its starting points. **Open Environments** and **Open API keys** open those pages. **Launch session** opens the launch form once you have an agent and an environment.
  * Links under the steps open the [Quickstart](/recursion/quickstart) and the guides to [connecting a coding agent](/recursion/ai-coding-agents) and [automating your agents](/recursion/automations).
  * **Dismiss** hides it for you in this browser.
  * Members who can view but not create see the same introduction and the automations guide link without the steps, until the organization has an agent.
</Accordion>

## Create an agent

Start from a template, which also creates a paired environment, or build an agent yourself.

<Tabs>
  <Tab title="From a template">
    1. When your organization has no agents and **Get started** is not showing, **Sessions** shows starter cards. **Create agent** in **Get started** offers the same choices.
    2. Choose **Issue fixer** or **PR reviewer** to start with an agent definition and a paired environment.
    3. Coding templates ask you to connect GitHub and select the repositories the agent may use.
    4. Click **Create and launch**. This saves both resources and opens the launch form.
    5. Review the prefilled task, then click **Launch session**.
  </Tab>

  <Tab title="Blank">
    1. Open **Agents**, click **Create agent**, and choose **Blank**.
    2. Set a model and system prompt.
    3. Add tools and integrations as needed. The Skills, Multiagent, Credentials, and Tags sections start as summary rows while they hold defaults; open a row to change it.
    4. Save the agent.
  </Tab>
</Tabs>

Saving an agent publishes a new version. Sessions already running keep their version until you [move them](/recursion/sessions#move-a-session-to-another-version). The [Quickstart](/recursion/quickstart) walks through the same flow without a template, in the console and with cURL.

## Prepare an environment

Templates create a matching environment for you. For a custom one, open **Environments**, click **Create environment**, and set its name, internet access, compute, and optional setup script. A setup script must pass a test run before sessions can use the environment. See [Environments](/recursion/environments).

## Give agents access

* **Integrations**: connect GitHub or another supported app, then grant the connection in an agent's **Integrations** section. See [Integrations](/recursion/integrations).
* **Credential vaults**: store tokens and secrets. Secret values are write-only in the console. See [Vaults](/recursion/vaults).
* **Skills**: add reusable instructions and attach them to an agent. See [Skills](/recursion/skills).

## Launch a session

1. Click **Launch session** on **Sessions** or on an agent's page. The agent page preselects that agent.
2. Under **Configuration**, check the agent and environment.
3. Under **Task**, enter an optional opening message.
4. Expand **Credentials**, **Team**, **Files**, or **Custom metadata** when this session needs a change from its defaults, such as input files to copy into the sandbox. Changes in this form apply only to this session.
5. Click **Launch session** to start the work.

Your tenant needs available credits to launch. If none are available, someone who manages billing can add them and you can retry. See [Sessions](/recursion/sessions) for all launch fields and session controls.

<Accordion title="How the form picks the agent and environment">
  The form remembers your last successful agent and environment. When there is only one choice and nothing remembered, it selects that choice for you.
</Accordion>

## Follow the work

Open a session to read its **Transcript**: messages, tool calls, and results arrive as the agent works. The header shows status and cost. The **Files** tab holds the session's inputs and the outputs it saved, ready to preview or download. Send a follow-up from the composer, or use the session actions to interrupt or stop work. See [Session operations](/recursion/session-operations).

## Automate and use the API

* To run agents on their own, open **Automations** and create one with a schedule or an event trigger. See [Automations](/recursion/automations).
* For API use, open **Settings** and choose **API keys**. A new key is shown only once. See [API keys](/recursion/api-keys).

## Find a screen

The console opens on **Sessions** when no section is named in the link. A link to a section you cannot open lands on the first section available to you. Your account menu at the bottom of the sidebar lets you change tenant, switch color mode, or sign out.

<Accordion title="Every sidebar screen">
  | Screen | Use it to |
  | - | - |
  | **Sessions** | Launch work, search and filter sessions, and read their transcripts and costs. |
  | **Agents** | Create or edit an agent's model, prompt, tools, skills, integrations, and default credential access. |
  | **Environments** | Choose the sandbox, its internet access, compute, and setup script. |
  | **Analytics** | See your organization's usage, completions, and environment startup over a time window. See [Analytics](/recursion/analytics). |
  | **Skills** | Create skills and groups for agents to load. |
  | **Credential vaults** | Store tokens and secrets for agents to use. |
  | **Integrations** | Connect GitHub, Slack, and apps from the built-in catalog, choose their tools, and grant them to agents. |
  | **Automations** | Start sessions on a schedule, from Slack or GitHub events, or by hand. The **Event sources** view lists the sources that deliver events, with their health, and lets you pause or resume them. |
  | **Billing** | Buy prepaid credits, manage cards, and read receipts. Appears only for billing managers. |
  | **Settings** | Open the settings area: **General**, **Members**, **Organizations**, **Enterprise SSO**, **API keys**, and **Audit log**. |
</Accordion>

<Accordion title="Every settings section">
  In **Settings**, **API keys** is open to every member with Managed Agents access. What you can change in the other sections depends on your role.

  | Settings section | Use it to |
  | - | - |
  | **General** | See your tenant and its ID, manage its email domains, sign-in methods, and multi-factor authentication, edit your profile, or leave the tenant. See [Domains](/recursion/domains). |
  | **Members** | Find members, send and track invitations, answer requests to join, and change or remove tenant roles. |
  | **Organizations** | Create, archive, and open organizations, and manage who belongs to each one. See [Organizations and roles](/recursion/organizations-and-roles). |
  | **Enterprise SSO** | Let your people sign in through your company's SAML identity provider. See [Enterprise SSO](/recursion/enterprise-sso). |
  | **API keys** | Create and revoke your personal API keys. See [API keys](/recursion/api-keys). |
  | **Audit log** | Review who changed what in your tenant, filtered by action, person, organization, or date. |
</Accordion>

## Next steps

<CardGroup>
  <Card title="Quickstart" href="/recursion/quickstart">
    Run a first session in the console and with cURL.
  </Card>

  <Card title="Sessions" href="/recursion/sessions">
    Every launch field and session control.
  </Card>

  <Card title="Agents" href="/recursion/agents">
    Configure an agent's model, prompt, and tools.
  </Card>

  <Card title="Environments" href="/recursion/environments">
    Set up the sandbox an agent runs in.
  </Card>
</CardGroup>


# Domains
Source: https://docs.labelbox.com/recursion/domains

Claim your company's email domain, prove it with DNS, and choose whether people at it can find and join your tenant.

Claiming your company's email domain lets colleagues who sign in with an address there find your tenant, instead of each starting in an empty tenant of their own. Verifying it with DNS proves your company controls the domain. That keeps any other tenant from taking it over, and it's required for [Enterprise SSO](/recursion/enterprise-sso).

<Note>
  You need the **Admin** or **Owner** tenant role, or be the tenant's primary owner. See [Organizations and roles](/recursion/organizations-and-roles). You can claim only the domain of your own verified email address: if you sign in as `ada@acme.com`, you can claim `acme.com`.
</Note>

## Claim a domain

<Steps>
  <Step title="Open the Domains card">
    In the [Recursion console](https://recursion.labelbox.com), open **Settings** and choose **General**. The **Domains** card lists your tenant's domains.
  </Step>

  <Step title="Add your domain">
    Under **Add a domain**, enter your domain, such as `acme.com`, and click **Add domain**.
  </Step>
</Steps>

A new claim is **Invite only**, so nothing changes for anyone until you choose who can join. Next, verify it with DNS.

<Accordion title="Which domains you can claim">
  * A claim covers that exact domain. Claiming `acme.com` doesn't cover `eu.acme.com`.
  * Addresses at free email providers, such as `gmail.com` or `outlook.com`, can't be claimed.
</Accordion>

## Verify it with DNS

Your verified address is enough to claim the domain, but until you verify it with DNS, another tenant whose admin has a verified address there can take it over by verifying it first. Verifying also unlocks **Anyone can join**, inviting people who already use Recursion, and Enterprise SSO.

<Steps>
  <Step title="Copy the TXT record">
    On the domain's row, the card shows a TXT record with a **Type**, **Name**, and **Value**. The name is `_recursion-verification`, and the value starts with `recursion-domain-verification=`. Use the copy buttons, since the value is unique to your claim.
  </Step>

  <Step title="Add it at your DNS provider">
    Create a TXT record with that name and value wherever your domain's DNS is managed. Most providers, including Google Cloud DNS and Cloudflare, add your domain to the name themselves. If yours asks for the full name, enter `_recursion-verification.` followed by your domain, as the card's note shows. The record sits on its own name, so it doesn't affect your existing records, such as SPF.
  </Step>

  <Step title="Check it">
    Click **Check DNS record**. New records can take a while to appear. If the check doesn't find it yet, wait a few minutes and check again. If it reports the record at `_recursion-verification.<domain>.<domain>`, your provider added the domain to the full name: edit the record's name to `_recursion-verification` alone and check again.
  </Step>
</Steps>

Once it's verified, the domain shows **Verified**, and no other tenant can take it over. Keep the record in place.

## Choose who can join

Each domain has its own join settings. Pick one under **Who can join**:

| **Who can join** | What happens |
| - | - |
| **Invite only** | Nobody joins through the domain. Invitations stay the only way in. |
| **Request to join** | People with a verified address at the domain can ask to join. An admin approves or denies each request in **Members › Requests**. |
| **Anyone can join** | People with a verified address at the domain join at once, without an invitation or approval. Needs the domain verified with DNS. |

**Default role** is the role people get when they join through the domain: **User** or **Developer**.

To invite everyone with an account at the domain, set **Invite people from your domain who already use Recursion** to **Automatically**. Each person accepts or declines the invitation in **Members › Invitations**. It needs the domain verified with DNS.

<Accordion title="How the default role applies">
  Admin, owner, and billing roles are only given by invitation or a role change. An admin approving a request can pick another role, and members who already joined keep theirs when you change the default.
</Accordion>

## Remove a domain

Click **Remove** on the domain's row and confirm. People at the domain no longer find or join your tenant through it. Members who already joined stay, and invitations already sent stay. Pending requests through the domain are closed.

## What can go wrong

The most common problems:

* **Check DNS record doesn't find the record.** It hasn't appeared yet, or its name or value differs from the one shown. Compare it with the card, copy both again if needed, wait a few minutes, and check again.
* **Check DNS record says the record is at a doubled name.** The provider added your domain to the full name, so it landed at `_recursion-verification.<domain>.<domain>`. Edit the record's name to `_recursion-verification` alone, then check again.
* **The domain shows Held by another tenant.** Another tenant claimed it first. Verify it with DNS: that takes the domain over and ends their claim, unless they already verified it.
* **Anyone can join can't be chosen.** The domain isn't verified with DNS. [Verify it with DNS](#verify-it-with-dns), then choose it.

<Accordion title="Every domain problem">
  | Symptom | Cause | Fix |
  | - | - | - |
  | **Check DNS record** doesn't find the record | The record hasn't appeared yet, or its name or value differs from the one shown. | Compare the record at your DNS provider with the card, copy both again if needed, wait a few minutes, and check again. |
  | **Check DNS record** says the record is at `_recursion-verification.<domain>.<domain>` | The DNS provider added the domain to the full name. | Edit the record's name to `_recursion-verification` alone, then check again. |
  | The domain shows **Held by another tenant** | Another tenant claimed it first. | Verify it with DNS. Verifying takes the domain over and ends their claim, unless they already verified it. |
  | **Anyone can join** can't be chosen | The domain isn't verified with DNS. | [Verify it with DNS](#verify-it-with-dns), then choose it. |
  | You can't claim your company's domain | Your signed-in address is at another domain, or at a free email provider. | Sign in with your company address, then claim its domain. |
</Accordion>

## Next steps

<CardGroup>
  <Card title="Enterprise SSO" href="/recursion/enterprise-sso">
    Let people at your verified domains sign in through your identity provider.
  </Card>

  <Card title="Organizations and roles" href="/recursion/organizations-and-roles">
    See what each role can do.
  </Card>
</CardGroup>


# Enterprise SSO
Source: https://docs.labelbox.com/recursion/enterprise-sso

Connect your SAML 2.0 identity provider, such as Okta, Microsoft Entra ID, or Google Workspace, so people at your verified domains sign in through it.

Enterprise SSO lets people at your company sign in to Recursion through your own identity provider. They enter their work email on the sign-in screen and continue to your provider. The first time they sign in, they join your tenant as members, so there are no invitations to send. You can also require single sign-on, so people at your domains can't sign in any other way.

<Note>
  You need the **Admin** or **Owner** tenant role, or be the tenant's primary owner. See [Organizations and roles](/recursion/organizations-and-roles). At least one of your email domains must be verified with DNS: single sign-on admits only addresses at your verified domains, so your identity provider can vouch for your people and nobody else's. See [Domains](/recursion/domains). Your identity provider must support SAML 2.0 and be able to send each person's email address as the NameID. A tenant has one identity provider connection.
</Note>

## Set up single sign-on

Open the [Recursion console](https://recursion.labelbox.com), open **Settings**, and choose **Enterprise SSO**. The page walks through four steps.

<Steps>
  <Step title="Create a SAML app at your identity provider">
    The first card, **Create a SAML app at your identity provider**, shows the values your provider needs: **Single sign-on URL (ACS)**, **Audience URI (SP entity ID)**, and **SP metadata URL**. Copy them from the console into a new SAML app at your provider. Provider-specific steps are [below](#identity-provider-setup).
  </Step>

  <Step title="Check your verified domains">
    The second card lists your tenant's DNS-verified domains. If none are listed, click **Manage domains** and [verify one](/recursion/domains#verify-it-with-dns). You can save and test the connection before then, but you can't turn it on.
  </Step>

  <Step title="Connect your identity provider">
    In **Connect your identity provider**, choose where the settings come from:

    * **Metadata URL**: paste your provider's metadata URL into **IdP metadata URL**. Recursion reads the sign-in URL and signing certificate from it, and can refresh them later.
    * **Entered by hand**: paste the **IdP single sign-on URL** and the **Signing certificate** in PEM form, starting with `-----BEGIN CERTIFICATE-----`. The certificate is public; no key is stored.

    Leave **Turn on** unchecked for now, and click **Save connection**.
  </Step>

  <Step title="Test it">
    **Test it** shows a **Test sign-in link**. Open it in a private window and sign in at your identity provider. The link works whether single sign-on is on or off. While it's off, it signs in existing members only, so test with your own account or a test account that's already a member.
  </Step>
</Steps>

When the test works, check **Offer single sign-on at sign-in** under **Turn on**, and click **Save changes**. The status changes to **On**, and people at your verified domains can sign in through your provider.

## Identity provider setup

Use the values from the console's first card. Each provider must send the person's email address as the NameID.

<Tabs>
  <Tab title="Okta">
    1. In the Okta Admin Console, go to **Applications › Applications**, click **Create App Integration**, and choose **SAML 2.0**.
    2. Set **Single sign-on URL** to the console's **Single sign-on URL (ACS)**, and **Audience URI (SP Entity ID)** to its **Audience URI (SP entity ID)**.
    3. Set **Name ID format** to **EmailAddress** and **Application username** to **Email**.
    4. Finish creating the app, then assign the people or groups who should sign in on its **Assignments** tab.
    5. On the **Sign On** tab, copy the **Metadata URL** and paste it into **IdP metadata URL** in the console.
  </Tab>

  <Tab title="Microsoft Entra ID">
    1. In the Microsoft Entra admin center, go to **Enterprise applications**, click **New application**, then **Create your own application**, and choose **Integrate any other application you don't find in the gallery**.
    2. Open **Single sign-on** and choose **SAML**.
    3. Under **Basic SAML Configuration**, set **Identifier (Entity ID)** to the console's **Audience URI (SP entity ID)**, and **Reply URL** to its **Single sign-on URL (ACS)**.
    4. Under **Attributes & Claims**, set **Unique User Identifier (Name ID)** to `user.mail`.
    5. Assign the people or groups who should sign in under **Users and groups**.
    6. Under **SAML Certificates**, copy **App Federation Metadata Url** and paste it into **IdP metadata URL** in the console.
  </Tab>

  <Tab title="Google Workspace">
    1. In the Google Admin console, go to **Apps › Web and mobile apps**, click **Add app**, and choose **Add custom SAML app**.
    2. On **Google Identity Provider details**, copy the **SSO URL** and download the **Certificate**. Google doesn't publish a metadata URL, so in the console choose **Entered by hand** and paste the SSO URL and certificate.
    3. On **Service provider details**, set **ACS URL** to the console's **Single sign-on URL (ACS)**, and **Entity ID** to its **Audience URI (SP entity ID)**.
    4. Set **Name ID format** to **EMAIL** and **Name ID** to **Basic Information › Primary email**.
    5. Under **User access**, turn the app on for the people or organizational units who should sign in.
  </Tab>
</Tabs>

## Require single sign-on

Check **Require single sign-on for your verified domains** under **Require it**, and click **Save changes**. The connection must be on first.

While it's required, people at your verified domains can sign in only through your identity provider. Email, Google, and other sign-ins are refused for those addresses, current members included. The sign-in screen tells them to enter their work email and continue through your provider.

<Warning>
  Test the connection before you require it. Your tenant's primary owner is exempt and can still sign in another way, so a broken connection can always be fixed or turned off from **Enterprise SSO**.
</Warning>

## Keep the certificate current

Identity providers rotate their signing certificates. **Enterprise SSO** shows the current certificate's expiry and warns you before it expires.

* **Metadata URL**: after your provider publishes the new certificate, click **Refresh from metadata**.
* **Entered by hand**: paste the new certificate into **Signing certificate** and click **Save changes**. Leave the field empty to keep the current one.

If the certificate expires, single sign-on fails until you replace it.

## Turn off or remove single sign-on

* **Turn off**: uncheck **Offer single sign-on at sign-in** and click **Save changes**. The connection's settings are kept, and the test link still works for existing members. Sign-in from your identity provider's app catalog also keeps working if you allowed it; uncheck that too to stop it.
* **Remove**: click **Remove**, then **Remove single sign-on**. Members who joined through it stay, and can sign in another way.

## How people sign in

* **From the sign-in screen.** A person enters their work email and clicks **Continue**. An address at one of your verified domains goes to your identity provider.
* **From your identity provider, if you allow it.** This stays off until an admin checks **Allow sign-in from your identity provider's app catalog** under **App catalog** and confirms the warning. People can then open Recursion from your provider's app catalog, such as the Okta dashboard, Microsoft My Apps, or the Google app launcher. The warning is there because a sign-in that starts at the identity provider cannot be checked as one the person asked for, so someone can be tricked into a session that belongs to an attacker.
* **New people join automatically.** The first time someone signs in through your provider, they join your tenant at the domain's **Default role**, set on the [Domains](/recursion/domains#choose-who-can-join) card. They don't get a separate tenant of their own.
* **Existing accounts are kept.** If someone already signs in with Google or email at the same address, their first single sign-on asks them to confirm that account once. The two sign-in methods then share one account.
* **Removed members stay removed.** Someone you removed from the tenant can't rejoin through single sign-on.
* **Only verified domains.** Your provider can't sign in an address at a domain your tenant hasn't verified, whatever it asserts.

## What can go wrong

The most common problems:

* **The sign-in screen says your organization's single sign-on could not sign you in.** Your identity provider's app settings don't match the console, most often the audience or entity ID. Compare the app's ACS URL and entity ID with the console's first card, then try the test link again.
* **Sign-in fails even though the ACS URL and entity ID match.** Your provider doesn't send the person's email address as the NameID. Set it as in [Identity provider setup](#identity-provider-setup).
* **Your provider refuses the person before they reach Recursion.** The person isn't assigned to the SAML app at your provider. Assign them, or their group, to the app.

<Accordion title="Every single sign-on problem">
  | Symptom | Cause | Fix |
  | - | - | - |
  | The sign-in screen says your organization's single sign-on could not sign you in | Your identity provider's app settings don't match the console, most often the audience or entity ID. | Compare the app's ACS URL and entity ID with the console's first card, then try the test link again. |
  | Sign-in fails even though the ACS URL and entity ID match | Your provider doesn't send the person's email address as the NameID. | Set the NameID to the person's email address, as in [Identity provider setup](#identity-provider-setup). |
  | Your provider refuses the person before they reach Recursion | The person isn't assigned to the SAML app at your provider. | Assign them, or their group, to the app. |
  | **Offer single sign-on at sign-in** can't be checked | Your tenant has no domain verified with DNS. | [Verify a domain](/recursion/domains#verify-it-with-dns). |
  | The status shows **On, no verified domain** | Your last verified domain was removed, so the connection admits nobody. | Verify a domain again, or turn single sign-on off. |
  | The metadata URL can't be read | The URL isn't public HTTPS, or the document lists no signing certificate or no HTTP-Redirect sign-in location. | Check the URL at your provider, or choose **Entered by hand** and paste the sign-in URL and certificate. |
  | People at your domain can't sign in with Google or email | Single sign-on is required for your verified domains. | Sign in through your identity provider, or ask an admin to turn off **Require single sign-on for your verified domains**. |
</Accordion>

## Next steps

<CardGroup>
  <Card title="Domains" href="/recursion/domains">
    Claim and verify the domains single sign-on admits.
  </Card>

  <Card title="Organizations and roles" href="/recursion/organizations-and-roles">
    See what each role can do.
  </Card>
</CardGroup>


# Set up an environment
Source: https://docs.labelbox.com/recursion/environments

Create an environment, prove its setup script works, control its network access, and keep it current.

An environment is the saved definition of the sandbox a session runs in: its compute size, workspace disk, setup script, and network access. You create it once, prove its setup script works with a setup run, and then name its `environment_id` when you start sessions.

Creating an environment starts no compute. Compute starts only when you run a setup test or start a session. Each top-level session gets its own fresh sandbox, so two sessions on the same environment never share files. Subagents and teammates that a session starts share that session's sandbox. For every field, default, and limit, see [Environment reference](/recursion/environments-reference).

<Note>
  Creating, changing, testing, or deleting environments needs the organization developer or admin role. The organization user role can view environments and their setup runs. See [Organizations and roles](/recursion/organizations-and-roles) and [API keys](/recursion/api-keys). Setup runs start real compute, the same kind a session uses. New environments start with no internet access, so decide first which hosts the sandbox must reach, such as package registries or `github.com`.
</Note>

## Create an environment

This example creates an Agent runner environment with 2 vCPU, 4 GiB of memory, a 20 GiB workspace, and a setup script that installs two pinned Python packages. Its network policy allows only the two Python package hosts the script needs. Creating the environment saves only the configuration, so the example then starts a setup run explicitly.

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Environments**, then click **Create environment**.
    2. Enter a **Name**. The form uses Agent runner automatically. To start from a template instead, see [Start from a template](#start-from-a-template).
    3. Under **Permissions**, set **Internet access** to **Enabled**. The console offers only **Enabled** or **No access**. To allow only specific hosts, use the API as in the cURL tab, or edit the policy later as shown in [Control network access](#control-network-access).
    4. Expand **Compute**, set **Compute kind** to **CPU**, and click **Standard** (2 vCPU, 4 GiB).
    5. Expand **Storage and lifecycle** and set **Workspace disk (GiB)** to `20`.
    6. Paste the script into **Setup script**.
    7. Click **Save and test setup**. The **Setup test** panel streams the run's log.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    environment_id=$(curl -sS -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/environments' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Idempotency-Key: data-analysis-environment-v1' \
      -H 'Content-Type: application/json' \
      -d '{
          "name": "data-analysis",
          "provider": "runs",
          "resources": {"cpu_milli": 2000, "memory_mib": 4096},
          "pvc_size_gi": 20,
          "network_policy": {"version": "v1", "rules": [
            {"match": {"host_glob": "pypi.org"}, "action": {"type": "allow"}},
            {"match": {"host_glob": "files.pythonhosted.org"}, "action": {"type": "allow"}}
          ]},
          "setup": {
            "script": "uv venv --seed /workspace/.venv\nuv pip install --python /workspace/.venv/bin/python \"requests==2.32.3\" \"pandas==2.2.3\"",
            "timeout_seconds": 900
          }
        }' | jq -r '.environment_id')

    curl -X POST "https://api.recursion.labelbox.com/managed-agents/v1/environments/$environment_id/setup-runs" \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Idempotency-Key: data-analysis-setup-v1'
    ```
  </Tab>
</Tabs>

The create call returns `200 OK` with the saved environment. It starts no compute. Because the environment has a setup script but no completed setup run yet, `setup_verification.status` is `never`. The setup-run call returns `202 Accepted` with a separate run whose initial status is `queued`.

Keep `environment_id` to start sessions and `setup_run_id` to [follow the setup run](#wait-for-the-result).

<Accordion title="Create response">
  ```json theme={"theme":"css-variables"}
  {
    "organization_id": "org_01a08a705220724f9a2fe1bcd8638c9d",
    "environment_id": "9d3e7b52-1a4c-4f80-b6e9-2c8a5d0f7e13",
    "name": "data-analysis",
    "scope": "organization",
    "provider": "runs",
    "setup": {
      "script": "uv venv --seed /workspace/.venv\nuv pip install --python /workspace/.venv/bin/python \"requests==2.32.3\" \"pandas==2.2.3\"\n",
      "timeout_seconds": 900
    },
    "resources": {"cpu_milli": 2000, "memory_mib": 4096},
    "network_policy": {
      "version": "v1",
      "rules": [
        {"match": {"host_glob": "pypi.org"}, "action": {"type": "allow"}},
        {"match": {"host_glob": "files.pythonhosted.org"}, "action": {"type": "allow"}}
      ]
    },
    "privileged": false,
    "pvc_size_gi": 20,
    "computer_use": false,
    "setup_verification": {
      "status": "never",
      "stale": false
    },
    "setup_updated_by_user_id": "usr_01a0867d1b0875be9f9a9938ffb90971",
    "setup_updated_at": "2026-09-17T12:04:11Z",
    "created_at": "2026-09-17T12:04:11Z",
    "updated_at": "2026-09-17T12:04:11Z"
  }
  ```
</Accordion>

<Accordion title="Defaults and retries">
  * **Saving and testing are separate.** `createEnvironment` stores the configuration. `createEnvironmentSetupRun` provisions compute and requires its own `Idempotency-Key`.
  * **Create retries can be idempotent.** `Idempotency-Key` is optional on `createEnvironment`. Reuse the same key and exact request on an ambiguous retry to replay the first response instead of creating another environment.
  * **Network access is closed by default.** Omitting `network_policy` stores `{"version":"v1","rules":[]}`, which blocks every host. Send `{}` only when you deliberately want unrestricted outbound access.
  * **Privileged Docker is off by default.** `privileged` defaults to `false`. To set it to `true`, send `network_policy: {}` for unrestricted internet access.
  * **Every environment starts from the Agent runner image.** Add tools and data with the setup script. See [What the sandbox starts with](/recursion/environments-reference#what-the-sandbox-starts-with).
</Accordion>

<Accordion title="Choose a runtime">
  A runtime (the `provider` field) is the kind of sandbox the environment runs on. The console's **Create environment** form uses the catalog's default runtime automatically, and the runtime cannot be changed after creation. API clients should list the available runtimes rather than hard-coding one. Today the list contains `runs`: managed Linux compute with a persistent workspace, network policy, and setup verification.

  ```bash theme={"theme":"css-variables"}
  curl 'https://api.recursion.labelbox.com/managed-agents/v1/sandbox-providers' \
    -H "Authorization: Bearer $RECURSION_API_KEY"
  ```

  ```json theme={"theme":"css-variables"}
  {
    "items": [
      {
        "provider": "runs",
        "displayName": "Agent runner",
        "default": true,
        "requiresCredential": null
      }
    ]
  }
  ```

  The response also carries a `description` for each runtime. The list is the same for every organization and supports `If-None-Match` for caching.
</Accordion>

## Start from a template

Templates are reviewed starting points for common work. A template fills the create form with its compute, network policy, and setup script, if it has one. Review the settings, change what you need, then create the environment. **Base sandbox** and **Browser desktop** have no setup script, so the environment is ready as soon as you create it. A template with a setup script, like any environment with one, needs a passing setup test before sessions can start.

| Template | What it sets up |
| - | - |
| **Base sandbox** | Python, uv, Node, git, and gh, with access to GitHub and the npm and PyPI registries. |
| **Node web** | Node, Corepack, and TypeScript for web projects, with access to GitHub and npm. |
| **Python data** | Pinned scientific Python packages in `/workspace/.venv`, with unrestricted internet access. |
| **Browser desktop** | Chromium on a shared display for [computer use](/recursion/environments-reference#computer-use), with unrestricted internet access. |
| **GPU: PyTorch** | One T4 GPU with CUDA PyTorch and pinned LoRA training tools, with CUDA checked during setup. |

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Environments**, then click **Create environment**.
    2. Under **Choose a starting point**, click a template. The form fills in.
    3. Review the settings. A template that lists hosts keeps its own network policy, limited to the package and source hosts it needs.
    4. If the template has a setup script, click **Save and test setup**. Otherwise, click **Create environment**. The **Setup** column reads **Ready**.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl 'https://api.recursion.labelbox.com/managed-agents/v1/environment-templates' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

`listEnvironmentTemplates` returns each template under `items` with its `id`, `revision`, `name`, `summary`, and a `definition` you can send to `createEnvironment` as the request body. Creating from a template works like any other create: if the definition has a setup script, start a setup run and wait for it to pass before you start sessions. Without one, sessions can start right away.

## Write the setup script

The setup script runs on fresh compute after the sandbox starts and before the agent's first turn. It runs as root, without the setuid and setgid capabilities, in a bash login shell with `set -eo pipefail`. In a setup run, it gets the environment's variables and network policy, but no session credentials such as vault grants.

* **One command per line.** The first failing line ends the run, and the result names that line.
* **Don't wrap lines in `bash -lc`.** The script already runs under bash, and the wrapper's quoting is a common failure.
* **Pin versions.** The run proves the exact script, so unpinned installs can drift from what was proven.
* **Put Python packages in `/workspace/.venv`.** The sandbox selects that virtual environment for the agent's commands automatically. Install through its interpreter, for example `uv pip install --python /workspace/.venv/bin/python`.
* **Don't rely on `export`, `source`, aliases, or shell functions.** The agent's commands run in a new shell, so none of these carry over.
* **Run apt as `apt-get -o APT::Sandbox::User=root ...`.** There is no `sudo`, and apt can't drop privileges. Under a restrictive network policy, point apt at `https` sources: the image's default sources use `http`, which that policy can block.
* **Put files the agent needs under `/workspace`.** They stay with the session after its sandbox stops or is released and can be used when its workspace reopens.

<Accordion title="Size, timeout, and warnings">
  The script can be at most 64 KiB. `setup.timeout_seconds` bounds the whole script; it defaults to 600 and accepts 10 through 3600. A script that runs past the timeout fails with exit code `124`.

  The service saves the script with Windows line endings converted and trailing whitespace removed. `setup_warnings` flags risky lines such as `curl | sh`, unpinned installs, and `bash -lc` wrappers. Warnings are advice; they never block a save or a run.
</Accordion>

## Verify the setup script

An environment without a setup script is usable as soon as you create it. An environment with a setup script is usable only after a setup run passes for its current configuration.

```mermaid theme={"theme":"css-variables"}
flowchart TD
  create["Create environment"] --> hasScript{"Setup script?"}
  hasScript -->|"No"| ready["Usable"]
  hasScript -->|"Yes"| run["Setup run"]
  run -->|"Failed"| fix["Fix from the hint and log"]
  fix --> run
  run -->|"Passed"| verified["Usable"]
  verified -->|"Configuration changes"| run
```

A setup run starts fresh compute from the environment, runs the setup script exactly as a session would, records every line of output, checks the compute, saves a reusable image, and releases the compute. The result becomes the environment's `setup_verification`.

<Accordion title="When a session can start">
  A session can start on the environment when either of these is true:

  * `setup_verification.status` is `not_applicable`, because there is no setup script. In the console, the **Setup** column reads **Ready**.
  * `setup_verification.status` is `verified`, `stale` is `false`, and the service kept a reusable image of the prepared sandbox. In the console, the **Setup** column reads **Verified**.

  Otherwise `startSession` returns `422 environment_not_verified`, and `details.next_action` names the call that fixes it.

  When a setup run passes, the service saves the prepared sandbox as a reusable image. New sessions start from that image, so they don't repeat the setup work. If the runner changes, the platform automatically runs setup on dedicated compute, captures a new image, and keeps new sessions in provisioning until that image is ready. The session page shows the setup-run log while it waits. A failed refresh fails the waiting session and links its setup log from the session failure. The environment retains its prior verified verdict, but sessions needing the new runner image remain blocked until setup is tested again.
</Accordion>

### Start a setup run

`createEnvironmentSetupRun` requires an `Idempotency-Key`. Use one key for each logical setup run and reuse it with the same request if the response is lost. Only one setup run can be in flight per environment, and your organization can have 4 manual setup runs in flight at once.

<Tabs>
  <Tab title="Console">
    1. In **Environments**, open the environment.
    2. In the **Setup test** panel, click **Test setup**.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/environments/9d3e7b52-1a4c-4f80-b6e9-2c8a5d0f7e13/setup-runs' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Idempotency-Key: data-analysis-setup-v1'
    ```
  </Tab>
</Tabs>

The call returns `202 Accepted` with the queued run. Some fields are omitted here.

```json theme={"theme":"css-variables"}
{
  "organization_id": "org_01a08a705220724f9a2fe1bcd8638c9d",
  "environment_id": "9d3e7b52-1a4c-4f80-b6e9-2c8a5d0f7e13",
  "setup_run_id": "3a8e5c10-7d42-4f96-b1a3-9c6e2d5f0b74",
  "kind": "manual",
  "status": "queued",
  "requested_by_user_id": "usr_01a0867d1b0875be9f9a9938ffb90971",
  "message": "Queued; compute will be provisioned shortly.",
  "created_at": "2026-09-17T12:05:02Z"
}
```

If a run is already in flight, the call returns `409 setup_run_in_progress` with that run's id in `details.setup_run_id`. An environment with no setup script returns `400 invalid_request`, because there is nothing to verify.

### Wait for the result

Poll `getEnvironmentSetupRun` until `status` is `succeeded`, `failed`, or `cancelled`. Pass `wait_seconds` to have the server hold the request until the run finishes or the wait ends. The server caps each wait at 5 seconds, so loop until the status is terminal.

<Tabs>
  <Tab title="Console">
    1. Watch the **Setup test** panel. It shows each phase and streams the log.
    2. The status reads **Setup verified.** on success, or shows the hint and failing line on failure.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl 'https://api.recursion.labelbox.com/managed-agents/v1/environments/9d3e7b52-1a4c-4f80-b6e9-2c8a5d0f7e13/setup-runs/3a8e5c10-7d42-4f96-b1a3-9c6e2d5f0b74?wait_seconds=5' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

```json theme={"theme":"css-variables"}
{
  "environment_id": "9d3e7b52-1a4c-4f80-b6e9-2c8a5d0f7e13",
  "setup_run_id": "3a8e5c10-7d42-4f96-b1a3-9c6e2d5f0b74",
  "kind": "manual",
  "status": "succeeded",
  "exit_code": 0,
  "duration_ms": 74210,
  "log_lines": 42,
  "log_bytes": 3180,
  "created_at": "2026-09-17T12:05:02Z",
  "started_at": "2026-09-17T12:05:03Z",
  "finished_at": "2026-09-17T12:06:17Z"
}
```

Some fields are omitted here. `next_action` names the call to make next, such as polling again or reading the log.

**What success means for a setup run.** `status` is `succeeded`, and the environment now reads `setup_verification.status: "verified"` with `stale: false` for the configuration the run tested. Sessions can start. If the script passed but the image step failed, the console shows **Setup passed, but its reusable image failed.** and sessions still return `environment_not_verified`; run setup again. If the run failed, see [Fix a failed setup run](#fix-a-failed-setup-run).

A setup run moves through these statuses:

```mermaid theme={"theme":"css-variables"}
stateDiagram-v2
  [*] --> queued: requested
  queued --> provisioning: compute starts
  provisioning --> running: script runs
  running --> provisioning: retried after a fault
  running --> succeeded: checks passed
  queued --> failed: could not start
  provisioning --> failed: compute failed
  running --> failed: script failed
  queued --> cancelled: cancel
  provisioning --> cancelled: cancel
  running --> cancelled: cancel
```

<Accordion title="Phases, failed and cancelled runs, and re-verifying">
  `phase` tells you where the run is or where it ended: `provision`, `gpu_check`, `setup`, `profile`, `commit` (saving the reusable image), or `cleanup`. Only a failure in `setup` points at your script.

  * **`failed`** records a failed verdict. Sessions can't start until a later run passes.
  * **`cancelled`** records nothing. If the environment was `verified` and unchanged when the run started, it stays `verified`. If it read `running`, it goes back to `never`.
  * **Re-verifying doesn't interrupt sessions.** While a new run is in flight on a verified, unchanged environment, it stays `verified` and sessions keep starting. The console shows **Re-testing**. In every other case, starting a run sets the status to `running`, replacing an earlier `failed` or stale verdict.
</Accordion>

<Accordion title="Read the log">
  The log interleaves four streams: `stdout`, `stderr`, `marker` (the script line about to run), and `system` (notes about phases outside your script). Secrets are redacted before the log is stored.

  In the console, open the environment. The **Setup test** panel streams the log of the selected run. Under **Recent runs**, pick an earlier run to read its log.

  With the API, read from the start with `after=0`. Each page returns up to `limit` lines (default 500, maximum 2000) and a `next_after` cursor. To tail a live run, pass `next_after` as `after` on the next call with `wait_seconds=0`, and wait about two seconds between calls. Stop when `status` is terminal and a page comes back empty.

  ```bash theme={"theme":"css-variables"}
  curl 'https://api.recursion.labelbox.com/managed-agents/v1/environments/9d3e7b52-1a4c-4f80-b6e9-2c8a5d0f7e13/setup-runs/3a8e5c10-7d42-4f96-b1a3-9c6e2d5f0b74/log?after=0&limit=500&wait_seconds=0' \
    -H "Authorization: Bearer $RECURSION_API_KEY"
  ```

  ```json theme={"theme":"css-variables"}
  {
    "lines": [
      {"seq": 8, "stream": "marker", "line": 2, "text": "uv pip install --python /workspace/.venv/bin/python \"requests==2.32.3\" \"pandas==2.2.3\"", "at": "2026-09-17T12:06:01Z"},
      {"seq": 9, "stream": "stderr", "line": 2, "text": "Resolved 7 packages in 412ms", "at": "2026-09-17T12:06:02Z"}
    ],
    "next_after": 9,
    "status": "running"
  }
  ```

  The service keeps up to 4 MiB of log per run. Past that, `truncated` is `true`; the beginning of the log is kept, and the last 2 KiB of stderr stays in the run's `stderr_tail`.
</Accordion>

<Accordion title="List setup runs">
  `listEnvironmentSetupRuns` returns the environment's recent runs, newest first. It includes manual runs (`kind: "manual"`, which you start), automatic runner refreshes (`kind: "runner_refresh"`, shared by sessions that need an updated setup image), and session runs (`kind: "session"`, recorded when a session ran setup on its own compute, with its `session_id`). The newest manual or runner refresh run is the one `setup_verification` describes. `limit` accepts 1 through 50 and defaults to 50. In the console, open the environment and read the **Recent runs** list in the **Setup test** panel.

  ```bash theme={"theme":"css-variables"}
  curl 'https://api.recursion.labelbox.com/managed-agents/v1/environments/9d3e7b52-1a4c-4f80-b6e9-2c8a5d0f7e13/setup-runs?limit=10' \
    -H "Authorization: Bearer $RECURSION_API_KEY"
  ```

  ```json theme={"theme":"css-variables"}
  {
    "setup_runs": [
      {
        "environment_id": "9d3e7b52-1a4c-4f80-b6e9-2c8a5d0f7e13",
        "setup_run_id": "3a8e5c10-7d42-4f96-b1a3-9c6e2d5f0b74",
        "kind": "manual",
        "status": "succeeded",
        "created_at": "2026-09-17T12:05:02Z",
        "finished_at": "2026-09-17T12:06:17Z"
      },
      {
        "environment_id": "9d3e7b52-1a4c-4f80-b6e9-2c8a5d0f7e13",
        "setup_run_id": "1f6b2d90-4e8a-4c37-9a15-7d0c3e8b2a61",
        "kind": "manual",
        "status": "failed",
        "hint_code": "egress_blocked",
        "created_at": "2026-09-17T11:52:40Z",
        "finished_at": "2026-09-17T11:53:31Z"
      }
    ]
  }
  ```

  `setup_runs` is an empty array when no run has been requested.
</Accordion>

<Accordion title="Cancel a setup run">
  A member allowed to update the environment can cancel a manual run or automatic runner refresh that is `queued`, `provisioning`, or `running`. The run stops, its compute is released, and `active_setup_run_id` is cleared. A cancelled run records no verdict. In the console, open the environment while a run is in flight and click **Cancel run** in the **Setup test** panel. The status reads **Cancelled.**

  ```bash theme={"theme":"css-variables"}
  curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/environments/9d3e7b52-1a4c-4f80-b6e9-2c8a5d0f7e13/setup-runs/3a8e5c10-7d42-4f96-b1a3-9c6e2d5f0b74/cancel' \
    -H "Authorization: Bearer $RECURSION_API_KEY"
  ```

  ```json theme={"theme":"css-variables"}
  {
    "environment_id": "9d3e7b52-1a4c-4f80-b6e9-2c8a5d0f7e13",
    "setup_run_id": "3a8e5c10-7d42-4f96-b1a3-9c6e2d5f0b74",
    "kind": "manual",
    "status": "cancelled",
    "message": "Setup run was cancelled.",
    "created_at": "2026-09-17T12:05:02Z"
  }
  ```

  A run that already finished returns `409 setup_run_finished`, and nothing changes. A session run returns `400 invalid_request`; stop the session instead.
</Accordion>

## Fix a failed setup run

On a failed run, read `hint` first. Then read `failed_line`, `failed_command`, `exit_code`, and `stderr_tail`. Fetch the full log only if those don't explain the failure. The same summary appears in the environment's `setup_verification.last_run`.

| `hint_code` | Likely cause | Fix |
| - | - | - |
| `egress_blocked` | The network policy blocks a host the script needs. | Add an allow rule for the host, or choose **Enabled** for **Internet access**. |
| `timeout` | The script ran longer than `setup.timeout_seconds`. Exit code is `124`. | Raise the timeout (up to 3600), or make the slow step faster. |
| `bash_lc_wrapper` | A line wraps its command in `bash -lc` and the quoting broke. | Write the command directly. |
| `command_not_found` | A command isn't installed. Exit code is often `127`. | Install it earlier in the script, or call it by full path. |
| `pip_not_installed` | `pip` isn't on the path. | Create `/workspace/.venv` with uv and install through its interpreter. |
| `apt_lock` | Another process holds apt's lock on fresh compute. | Wait for the lock before installing. |
| `apt_permission` | apt tried to drop privileges, which the sandbox doesn't allow. | Run apt with `-o APT::Sandbox::User=root`. |
| `no_space` | The workspace disk filled up. | Raise `pvc_size_gi`, or download less during setup. |
| `cuda_unavailable` | The script expects a GPU the sandbox doesn't have. | Add an accelerator to the environment's compute, or drop the CUDA step. |

When `phase` is anything other than `setup`, the failure happened outside your script; retry once before you change anything. `failure_code` classifies the failure with the same codes session failures use, such as `environment_setup_failed`, `sandbox_capacity_unavailable`, or `sandbox_provision_timeout`.

<Accordion title="Example failed run">
  ```json theme={"theme":"css-variables"}
  {
    "environment_id": "9d3e7b52-1a4c-4f80-b6e9-2c8a5d0f7e13",
    "setup_run_id": "1f6b2d90-4e8a-4c37-9a15-7d0c3e8b2a61",
    "kind": "manual",
    "status": "failed",
    "phase": "setup",
    "exit_code": 2,
    "failed_line": 2,
    "failed_command": "uv pip install --python /workspace/.venv/bin/python \"requests==2.32.3\" \"pandas==2.2.3\"",
    "hint_code": "egress_blocked",
    "hint": "A host the script reaches for could not be reached. If this environment uses a restricted network_policy, add an allow rule for the host or set network_policy to {} for unrestricted internet access. Otherwise, check the host or registry.",
    "stderr_tail": "error: Failed to fetch: `https://pypi.org/simple/requests/`\n  Caused by: Could not resolve host: pypi.org",
    "failure_code": "environment_setup_failed",
    "created_at": "2026-09-17T11:52:40Z",
    "finished_at": "2026-09-17T11:53:31Z"
  }
  ```
</Accordion>

## Control network access

`network_policy` controls which hosts the sandbox can reach. It applies to the setup script, the agent's commands, package downloads, and websites opened by the sandbox browser. Outbound traffic that no rule allows is blocked.

| Setting | `network_policy` | Console **Internet access** |
| - | - | - |
| No internet (default) | `{"version": "v1", "rules": []}` | **No access** |
| Only listed hosts | `{"version": "v1", "rules": [ ...allow rules... ]}` | Set through the API; the console shows **Custom policy (saved)** |
| Unrestricted | `{}` | **Enabled** |

Rules are checked in order, and the first match wins. For the rule grammar, see [Network policy](/recursion/environments-reference#network-policy).

<Warning>
  Access is granted per host, not per action. Once a host is allowed, sandbox code can send anything to it, including uploads such as `git push`. Allow only the hosts the work needs, especially when agents process untrusted content.
</Warning>

<Accordion title="Traffic the policy doesn't govern">
  Some traffic doesn't leave from the sandbox, so the policy doesn't govern it: model calls, calls to the agent's MCP servers, and the `web_search` and `web_fetch` tools. Control those in the agent's configuration. See [Tools](/recursion/tools).

  * **Automated sessions in a restricted environment don't get `web_fetch`.** A session started by an automation or from Slack, in an environment with limited or no internet access, has no `web_fetch` tool, so outside text can't use it to send data out. Its sandbox commands still reach the hosts the policy allows.
  * **Built-in integration apps reach their own hosts.** The tools of apps from the [Integrations](/recursion/integrations) catalog, such as Jira and Confluence, run inside the sandbox. Recursion allows the hosts they need, so they work with **Enabled** internet access or a policy limited to specific hosts. Only **No access** blocks them.
</Accordion>

<Accordion title="Privileged Docker">
  Privileged Docker requires the **Enabled** mode. A nonempty policy, including **No access** and custom host rules, cannot be saved together with `privileged: true` on create or when access settings change.
</Accordion>

### Allow specific hosts

A policy change must prove you saw the current access settings. Read the environment, then send its current `network_policy` and `privileged` values in `expected_access` along with the new policy. This example allows the hosts that `git` and the `gh` CLI use for GitHub. Attaching a GitHub credential in a vault doesn't open these hosts; the environment's policy must allow them.

<Tabs>
  <Tab title="Console">
    The console can't edit individual host rules. Under **Internet access**, choose **Enabled** for unrestricted access or **No access** to block everything, then click **Save environment**. Choosing either replaces a saved custom policy. Use the API for host rules.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X PATCH 'https://api.recursion.labelbox.com/managed-agents/v1/environments/9d3e7b52-1a4c-4f80-b6e9-2c8a5d0f7e13' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      -d '{
        "network_policy": {"version": "v1", "rules": [
          {"match": {"host_glob": "pypi.org"}, "action": {"type": "allow"}},
          {"match": {"host_glob": "files.pythonhosted.org"}, "action": {"type": "allow"}},
          {"match": {"host_glob": "github.com"}, "action": {"type": "allow"}},
          {"match": {"host_glob": "api.github.com"}, "action": {"type": "allow"}},
          {"match": {"host_glob": "codeload.github.com"}, "action": {"type": "allow"}},
          {"match": {"host_glob": "objects.githubusercontent.com"}, "action": {"type": "allow"}},
          {"match": {"host_glob": "github-cloud.s3.amazonaws.com"}, "action": {"type": "allow"}}
        ]},
        "expected_access": {
          "network_policy": {"version": "v1", "rules": [
            {"match": {"host_glob": "pypi.org"}, "action": {"type": "allow"}},
            {"match": {"host_glob": "files.pythonhosted.org"}, "action": {"type": "allow"}}
          ]},
          "privileged": false
        }
      }'
    ```
  </Tab>
</Tabs>

The PATCH returns the updated environment. Because the policy changed, the earlier verdict is stale. [Start a setup run](#start-a-setup-run) and wait for it to pass before starting sessions.

<Accordion title="Updated environment">
  Some fields are omitted here.

  ```json theme={"theme":"css-variables"}
  {
    "environment_id": "9d3e7b52-1a4c-4f80-b6e9-2c8a5d0f7e13",
    "network_policy": {
      "version": "v1",
      "rules": [
        {"match": {"host_glob": "pypi.org"}, "action": {"type": "allow"}},
        {"match": {"host_glob": "files.pythonhosted.org"}, "action": {"type": "allow"}},
        {"match": {"host_glob": "github.com"}, "action": {"type": "allow"}},
        {"match": {"host_glob": "api.github.com"}, "action": {"type": "allow"}},
        {"match": {"host_glob": "codeload.github.com"}, "action": {"type": "allow"}},
        {"match": {"host_glob": "objects.githubusercontent.com"}, "action": {"type": "allow"}},
        {"match": {"host_glob": "github-cloud.s3.amazonaws.com"}, "action": {"type": "allow"}}
      ]
    },
    "privileged": false,
    "setup_verification": {
      "status": "verified",
      "stale": true,
      "setup_run_id": "cd7a01a8-6f35-43d0-bef9-54c91612de48"
    }
  }
  ```
</Accordion>

<Accordion title="How the access precondition works">
  * `network_policy` in a PATCH replaces the whole policy. Include every rule you want to keep.
  * `expected_access` is required when a PATCH changes a saved policy other than `{}` (including the default no-internet policy), or turns `privileged` on. Without it the API returns `428 precondition_required`. If either value changed since your read, it returns `412 precondition_failed`; read the environment again and retry.
  * Closing access (changing `{}` to a restricted policy, or turning `privileged` off) doesn't need `expected_access`.
  * Omitting `network_policy` or `privileged` in a PATCH keeps the saved value.
  * A PATCH that changes access settings cannot combine `privileged: true` with a nonempty policy; the API returns `400 invalid_request`. An existing environment with both settings can still be viewed and have other settings edited. To start a new session, set `network_policy: {}` or turn off privileged Docker.
</Accordion>

## Update an environment

An update is partial: omitted fields keep their saved values. Changes apply only to sandboxes started afterward; a session already running keeps the configuration it started with. This example raises compute to 4 vCPU and 8 GiB and raises the idle stop to one hour.

<Tabs>
  <Tab title="Console">
    1. In **Environments**, click the environment name, or choose **Edit** from its action menu.
    2. Expand **Compute** and click **Heavy** (4 vCPU, 8 GiB).
    3. Expand **Storage and lifecycle** and set **Idle stop after (seconds)** to `3600`.
    4. Click **Save and test setup**, or **Save environment** to save without testing.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X PATCH 'https://api.recursion.labelbox.com/managed-agents/v1/environments/9d3e7b52-1a4c-4f80-b6e9-2c8a5d0f7e13' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      -d '{"resources": {"cpu_milli": 4000, "memory_mib": 8192}, "idle_stop_after_seconds": 3600}'
    ```
  </Tab>
</Tabs>

The PATCH response is the updated environment. Some environment fields are omitted here.

```json theme={"theme":"css-variables"}
{
  "environment_id": "9d3e7b52-1a4c-4f80-b6e9-2c8a5d0f7e13",
  "resources": {"cpu_milli": 4000, "memory_mib": 8192},
  "idle_stop_after_seconds": 3600,
  "setup_verification": {
    "status": "verified",
    "stale": true,
    "setup_run_id": "5e2f9a47-0c1d-4b83-a6e2-91d7f3b0c85a"
  }
}
```

Changing compute makes the earlier verdict stale. After a verification-affecting change, [start a setup run](#start-a-setup-run) explicitly and wait for it to pass before starting new sessions. Changing only the name, description, metadata, idle stop, retention, or computer use keeps the verdict. The full list is in [Setup verification](/recursion/environments-reference#setup-verification).

<Accordion title="Fields that replace the saved value, and edit conflicts">
  A few fields replace the saved value as a whole:

  * `setup` replaces the whole setup object. Send both `script` and `timeout_seconds` to keep both. `{"script": ""}` removes the script.
  * `resources` replaces the whole compute size.
  * `env_vars` and `mounts` replace the whole map or list. Send `{}` or `[]` to clear them.

  A concurrent edit to the same environment's access or compute can return `409 revision_conflict`. Read the environment again and retry.
</Accordion>

## List and get environments

`listEnvironments` returns every environment in your organization, newest first, with the full configuration. `getEnvironment` returns one environment.

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Environments**. The table shows each environment's **Name**, **ID**, **Resources**, and **Setup** status.
    2. Search by name or ID.
    3. Click a name, or choose **Edit** from its action menu, to open the full configuration.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl 'https://api.recursion.labelbox.com/managed-agents/v1/environments' \
      -H "Authorization: Bearer $RECURSION_API_KEY"

    curl 'https://api.recursion.labelbox.com/managed-agents/v1/environments/9d3e7b52-1a4c-4f80-b6e9-2c8a5d0f7e13' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

<Accordion title="List response and edge cases">
  The list is not paginated, and deleted environments are left out. An unknown, deleted, or other-organization id returns `404 not_found` from `getEnvironment`. `environments` is `null`, not an empty array, when the organization has no environments. Some fields are omitted here.

  ```json theme={"theme":"css-variables"}
  {
    "environments": [
      {
        "environment_id": "9d3e7b52-1a4c-4f80-b6e9-2c8a5d0f7e13",
        "name": "data-analysis",
        "provider": "runs",
        "resources": {"cpu_milli": 2000, "memory_mib": 4096},
        "setup_verification": {"status": "verified", "stale": false},
        "created_at": "2026-09-17T12:04:11Z",
        "updated_at": "2026-09-17T12:04:11Z"
      }
    ]
  }
  ```
</Accordion>

## Delete an environment

Deleting removes the environment from lists and reads, and new sessions can't use it. Sessions that are running keep their sandbox, and past session history stays readable. There is no restore, so create a new environment if you need it back.

Deleting an environment does not delete the [automations](/recursion/automations) that use it. They stay, stop running, and can't be resumed until you point them at another environment. The **Delete environment** dialog names them. Deleting an agent is different: it deletes the automations that run it.

<Tabs>
  <Tab title="Console">
    1. In **Environments**, open the environment's action menu.
    2. Click **Delete**.
    3. In the **Delete environment** dialog, click **Delete**.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X DELETE 'https://api.recursion.labelbox.com/managed-agents/v1/environments/9d3e7b52-1a4c-4f80-b6e9-2c8a5d0f7e13' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

```json theme={"theme":"css-variables"}
{ "deleted": true }
```

## What can go wrong

The most common problems:

* **`422 environment_not_verified` on `startSession`.** The setup script hasn't passed a current setup run. Follow `details.next_action`: start a setup run, or wait for the one in flight.
* **A setup run fails with `egress_blocked`.** The network policy blocks a host the script needs. Allow the host, then run setup again.
* **The agent's `git clone` of a public repository fails.** The policy doesn't allow the GitHub hosts. Allow the hosts shown in [Allow specific hosts](#allow-specific-hosts).
* **`428 precondition_required` or `412 precondition_failed` on a PATCH.** Access changes need the current `network_policy` and `privileged` in `expected_access`. Read the environment and resend.

<Accordion title="Every environment error">
  | Symptom or code | Cause | Fix |
  | - | - | - |
  | `422 environment_not_verified` on `startSession` | The script has never passed, failed last time, is still running, is stale, or has no reusable image. | Follow `details.next_action`: start a setup run, or wait for the one in flight. |
  | `409 setup_run_in_progress` | A setup run is already in flight for this environment. | Poll the run in `details.setup_run_id`. A manual run or automatic runner refresh can be cancelled by a member allowed to update the environment. |
  | `429 setup_run_limit` | Your organization already has 4 manual setup runs in flight. | Wait `Retry-After` seconds, or cancel a run listed in `details.in_flight`. |
  | `409 setup_run_finished` | You cancelled a run that had already finished. | Nothing to do; read the run's result. |
  | `400 invalid_request` on `setup.script` | The script is over 64 KiB or contains a NUL byte, or you requested a run with no script. | Shorten or fix the script, or skip verification for a script-free environment. |
  | `400 invalid_request` on `network_policy` | The policy isn't `v1`, has over 200 rules, or has an unknown field. | Fix the policy; see [Network policy](/recursion/environments-reference#network-policy). |
  | `400 invalid_request` on `resources` | CPU or memory is out of range, or no machine offers that CPU and memory together. | Pick a size from the console shortcuts, or follow the size named in the error message. |
  | `400 invalid_request` on `resources.accelerator.name` | The GPU model is missing or isn't offered. | Use a model listed in the error message. See [Compute](/recursion/environments-reference#compute). |
  | `400 invalid_request` on `idle_stop_after_seconds` | Computer use is on and the idle stop is under 600 seconds. | Use `0` or at least 600. |
  | `400 invalid_request` on `env_vars.PATH` | `PATH` can't be set through `env_vars`. | Call tools by full path, or prepend to `$PATH` inside the command that needs it. |
  | `400 invalid_request` on `env_vars.PYTHONPATH` (or `HOME`, `LD_PRELOAD`, another startup variable) | The sandbox reads it before every command, so it can't be set for the whole environment. | Set it inside the command that needs it, for example `PYTHONPATH=/workspace/src python main.py`. |
  | `428 precondition_required` | A PATCH changed a saved policy other than `{}`, or turned on `privileged`, without `expected_access`. | Read the environment and resend with its `network_policy` and `privileged`. |
  | `412 precondition_failed` | Access settings changed since your read. | Read the environment again and retry. |
  | `409 revision_conflict` | Another edit to access or compute landed first. | Read the environment again and retry. |
  | Setup run failed with `egress_blocked` | The network policy blocks a host the script needs. | Allow the host, then run setup again. |
  | An integration app's tools, such as Jira's, fail with a network error | The environment has **No access** to the internet, and these tools run inside the sandbox. | Set **Internet access** to **Enabled** or limit it to specific hosts, then run setup again. |
  | Agent's `git clone` of a public repository fails | The policy doesn't allow the GitHub hosts. | Allow the GitHub hosts shown in [Allow specific hosts](#allow-specific-hosts). For repositories you granted, the agent uses its GitHub tools instead. |
  | Duplicate environments after a retry | The retry was not covered by the original idempotency receipt—for example, no key reached the API or the receipt had expired. | Reuse the same key and exact request within the approximately 24-hour receipt window. After an ambiguous response, list environments before retrying. Delete any extra environment. |
  | `404 not_found` | The id is wrong, deleted, or belongs to another organization. | Check the id with `listEnvironments`. |
  | `403 forbidden` | Your role can't make this change or start setup runs. | See [Organizations and roles](/recursion/organizations-and-roles). Use an account or key with the developer or admin role. |

  For the full error catalog and retry guidance, see [Errors](/recursion/errors).
</Accordion>

## Limits

An environment name can be up to 256 characters. Setup script, setup-run, log, and network policy limits are in [Limits](/recursion/limits). For compute, disk, and lifecycle limits, see [Environment reference](/recursion/environments-reference).

## Next steps

<CardGroup>
  <Card title="Environment reference" href="/recursion/environments-reference">
    Every field, default, compute size, sandbox path, and network rule.
  </Card>

  <Card title="Start a session" href="/recursion/sessions">
    Combine a usable environment with an agent and start work.
  </Card>

  <Card title="Authenticate with vaults" href="/recursion/vaults">
    Give agents credentials without putting secret values in the environment.
  </Card>

  <Card title="Deliverables and artifacts" href="/recursion/artifacts">
    Where agents save the files you asked for.
  </Card>
</CardGroup>


# Environment reference
Source: https://docs.labelbox.com/recursion/environments-reference

Look up every environment field, default, compute size, sandbox path, network rule, setup state, and limit.

This page lists every environment setting you can use, what each one does inside the sandbox, and the limits the API enforces. For step-by-step tasks, see [Set up an environment](/recursion/environments). For request and response schemas, open **Endpoints** in the sidebar and select **Environments**.

## Environment fields

Send these fields to `createEnvironment` and `updateEnvironment`. An update is partial: omitted fields keep their saved values.

| Field | Default | Description |
| - | - | - |
| `name` | Required | Label shown in lists. Up to 256 characters. |
| `provider` | Required | Runtime. Use a value from `listSandboxProviders`; today that is `runs`. See [Runtimes](#runtimes). |
| `description` | None | Note for people. The agent never sees it. |
| `resources` | Service default size | CPU and memory, or one GPU. See [Compute](#compute). |
| `pvc_size_gi` | Service default | Workspace disk in GiB. See [Workspace and lifecycle](#workspace-and-lifecycle). |
| `setup` | No script | Setup script and timeout. See [Setup](#setup). |
| `network_policy` | No internet | Which hosts the sandbox can reach. See [Network policy](#network-policy). |
| `privileged` | `false` | Lets the agent run Docker containers inside the sandbox. See [Privileged Docker](#privileged-docker). |
| `computer_use` | `false` | Adds a shared browser display for the `computer` tool. See [Computer use](#computer-use). |
| `idle_stop_after_seconds` | 30 minutes | Idle time before compute stops. The workspace is kept. |
| `stopped_delete_after_seconds` | Service default | Time before a stopped sandbox may be released. The setting does not expire the session or its workspace files. |
| `env_vars` | None | Plain-text environment variables. See [Environment variables](#environment-variables). |
| `mounts` | None | Files downloaded into the workspace when the sandbox starts. See [Mounts](#mounts). |
| `metadata` | None | Your own JSON. The service stores and returns it unchanged. |

### Fields the service sets

These appear in responses. The service sets them, so don't send them.

| Field | Description |
| - | - |
| `organization_id` | Organization that owns the environment, taken from your API key. |
| `environment_id` | Id to pass when you start a session. |
| `scope` | `organization` by default. Every member of your organization can see and use the environment, subject to their role. |
| `http_port` | `6901` when computer use is on. |
| `setup_verification` | Whether the setup script has passed for the current configuration. See [Setup verification](#setup-verification). |
| `setup_warnings` | Advice about risky script lines. See [Setup warnings](#setup-warnings). |
| `setup_updated_at` | When the setup script last changed. |
| `setup_updated_by_user_id` | Who last changed the setup script. |
| `created_at`, `updated_at` | RFC 3339 timestamps. |

For an Agent runner environment, responses always include `network_policy` and `privileged`. A `network_policy` of `{}` means unrestricted.

## Runtimes

The runtime (`provider`) decides what kind of sandbox the environment runs on. `listSandboxProviders` returns the runtimes API clients can choose, with a `displayName`, `description`, and `default` flag. The list is the same for every organization.

| `provider` | Console label | What you get |
| - | - | - |
| `runs` | **Agent runner** | Managed Linux compute with a persistent workspace, CPU and memory sizing, network policy, idle stop, setup verification, and a reusable setup image. The default runtime. |

The console uses the catalog's default runtime and does not show a runtime control. API clients still send `provider` when creating an environment; an existing environment's provider cannot be changed.

### What the sandbox starts with

Every environment starts from the Agent runner image: Debian-based Linux with Python 3 and uv, Node.js and npm, git, curl, and apt. There is no `sudo`; setup and sessions already run as root. Add anything else in the setup script. The agent's working directory is `/workspace`.

## Environment templates

`listEnvironmentTemplates` returns reviewed starting points. Each has an `id`, a `revision`, a `name` and `summary` for the console card, and a `definition` in the `createEnvironment` request shape. The console's **Choose a starting point** cards fill the create form from the same list. See [Start from a template](/recursion/environments#start-from-a-template).

## Compute

`resources` sizes the sandbox. Omit it, or choose **Use platform default** in the console, to use the service's default size.

| Field | Range | Description |
| - | - | - |
| `resources.cpu_milli` | 250 to 8000 | CPU in millicores. `1000` is one vCPU. `0` uses the default. |
| `resources.memory_mib` | 1024 to 32768 | Memory in MiB. `0` uses the default. |
| `resources.accelerator` | One GPU | Attaches one exclusive GPU: `type` is `gpu`, `name` is the GPU model (`a100`, `l4`, or `t4`), and `count` must be `1`, the default. `name` is required, and a model that isn't offered is rejected with `400 invalid_request` whose message lists the available models. A GPU sandbox always gets 8,000 millicores (8 vCPU) and 32,768 MiB (32 GiB) of memory, so omit `cpu_milli` and `memory_mib`. Omit `accelerator` for a CPU-only sandbox. |

In the console, expand **Compute**, set **Compute kind** to **GPU**, and pick a GPU model. Each model shows its memory and whether one is free now; availability is checked again when a sandbox starts.

For CPU sizes, the console offers these shortcuts, in 250-millicore and 1024-MiB steps:

| Shortcut | CPU | Memory |
| - | -: | -: |
| Light | 1 vCPU (1000) | 2 GiB (2048) |
| Standard | 2 vCPU (2000) | 4 GiB (4096) |
| Heavy | 4 vCPU (4000) | 8 GiB (8192) |

Choose **Custom** and move the sliders for any other size, up to 24 vCPU and 100 GiB. The API and MCP accept up to 8 vCPU (8000) and 32 GiB (32768).

How sizes are checked:

* **CPU and memory must fit one machine together.** A pair that no machine offers is rejected with `400 invalid_request`, and the message names the most memory available at that CPU size.
* **Available compute can lower the ceiling.** CPU environments never exceed 24 vCPU or 100 GiB in the console, or 8 vCPU or 32 GiB through the API and MCP. The current provider catalog and a restricted network policy, including the default no-internet policy, can impose a smaller maximum; a rejected save returns `400 invalid_request` with the current limit.
* **Changing compute makes the setup verdict stale.** Run setup again after you resize.

## Workspace and lifecycle

| Field | Range | Behavior |
| - | - | - |
| `pvc_size_gi` | 1 to 100; `0` uses the default | Size of the persistent workspace disk. A setup run that fills it fails with hint `no_space`. |
| `idle_stop_after_seconds` | 120 to 7200 (2 hours); `0` uses the default of 30 minutes | After this long with no activity, compute stops and the workspace is kept. With computer use on, use `0` or at least 600. |
| `stopped_delete_after_seconds` | 3600 to 2592000; `0` uses the service default | After this long, a stopped sandbox may be released. This setting does not expire the session or its workspace files. |

Message the same session to reopen its workspace after an idle stop or later release. Its `/workspace` files can be used when the workspace reopens successfully. If it cannot reopen, contact support. Delete the session separately when you no longer need it. The API checks these ranges when you save; run a setup test afterward to verify the complete environment configuration.

### Sandbox paths

| Path | Contents |
| - | - |
| `/workspace` | The agent's working directory and persistent workspace. Put repositories and data here in the setup script. |
| `/workspace/.venv` | Python virtual environment. When the setup script creates it, the agent's commands use it automatically. |
| `/workspace/.managed-agents/outputs` | Deliverables. Files the agent saves here are kept with the session. Agents already know this path, so ask for a named file. See [Deliverables and artifacts](/recursion/artifacts). |
| `/workspace/.managed-agents/files` | Files you attach to the session, read-only. See [Files](/recursion/files). |

### Who shares a sandbox

* Each top-level session gets its own fresh sandbox from the environment. Sessions never share files through an environment.
* Subagents and teammates run in their root session's sandbox, so they can hand work over by file path. See [Multi-agent](/recursion/multi-agent).
* A change to the environment affects only sandboxes started after it. Running sessions keep the configuration they started with.

## Setup

| Field | Default | Description |
| - | - | - |
| `setup.script` | Empty (no setup) | Bash script run on fresh compute before the agent's first turn. Up to 64 KiB. No NUL bytes. |
| `setup.timeout_seconds` | 600 | Time limit for the whole script, 10 to 3600 seconds. A timeout exits with code `124`. |

How the script runs:

* As root without the setuid and setgid capabilities, in a separate bash login shell with `set -eo pipefail`. The first failing line ends the run. There is no `sudo`. Run apt as `apt-get -o APT::Sandbox::User=root ...`, and point it at `https` sources: the image's default apt sources use `http`, and a restrictive network policy can block port 80 and the image's other sources.
* With the environment's `env_vars`, `mounts`, and network policy. In a setup run, session credentials such as vault grants are never available to it.
* Exports, `source`, aliases, and shell functions don't carry over to the agent's commands.

On save, the service converts Windows line endings and trims trailing whitespace. A non-empty script is stored ending in one newline. In an update, `setup` replaces the whole object, and `{"script": ""}` removes the script.

### Setup warnings

`setup_warnings` lists advice. Warnings never block a save or a run.

| `code` | Meaning |
| - | - |
| `bash_lc_wrapper` | A line wraps its command in `bash -lc`. Write the command directly. |
| `curl_pipe_sh` | A line pipes a download into a shell. |
| `unpinned_install` | A package install doesn't pin a version. |
| `rm_rf_root` | A line removes files from `/`. |
| `ambiguous_system_python` | A line uses the system Python instead of `/workspace/.venv`. |
| `path_clobber` | A line replaces `PATH` instead of extending it. |
| `nonpersistent_shell_activation` | A line activates something that won't carry over to the agent's shell. |
| `legacy_setup_discarded` | Line `0`. The saved setup uses an old format that no longer runs. Enter it again as a script. |

Each warning has a `code`, a one-based `line` (`0` when it's about the save), and a `message`.

## Setup verification

`setup_verification` tells you whether the setup script has passed for the environment's current configuration. When a script exists, a session can start only when `status` is `verified`, `stale` is `false`, and the service kept a reusable image from the passing run. Otherwise `startSession` returns `422 environment_not_verified`.

| `status` | Meaning | Console **Setup** label |
| - | - | - |
| `not_applicable` | No setup script. Sessions can start. | **Ready** |
| `never` | A script exists but no run has recorded a verdict. | **Not verified** |
| `running` | A run is in flight and there is no passing verdict for the current configuration. | **Testing** |
| `verified` | The recorded setup verdict passed in a manual run or automatic runner refresh. Check `stale` too. | **Verified**, **Stale**, or **Re-testing**; **Image missing**, **Image failed**, or **Check image** when there is no usable reusable image |
| `failed` | The recorded setup verdict failed. | **Failed** |

| Field | Description |
| - | - |
| `status` | Current verdict, from the table above. |
| `stale` | `true` when a setting that affects setup changed after the recorded run. Always `false` for `not_applicable` and `never`. |
| `setup_run_id` | Run that produced the verdict. |
| `active_setup_run_id` | Manual run or automatic runner refresh in flight, if any. A verified verdict remains visible while it runs; sessions needing a new runner image wait for the refresh. |
| `at` | When the recorded run finished. |
| `verified_by_user_id` | Who requested the recorded run. |
| `fingerprint` | Opaque value identifying the configuration the run tested. Compare only for equality. |
| `last_run` | Summary of the recorded run. See the next table. |

**Changes that make the verdict stale:** `provider`, `setup` (script or timeout), `resources`, `pvc_size_gi`, `env_vars`, `mounts`, `network_policy`, and `privileged`. **Changes that don't:** `name`, `description`, `metadata`, `computer_use`, `idle_stop_after_seconds`, and `stopped_delete_after_seconds`.

### Last-run summary

| Field | Description |
| - | - |
| `last_run.setup_run_id` | Run summarized here. |
| `last_run.status` | `succeeded`, `failed`, or `cancelled`. |
| `last_run.phase` | Phase the run ended in. |
| `last_run.exit_code` | Script exit code. `124` means timeout. Absent when the script never started. |
| `last_run.failed_line` | One-based script line that failed, or `0` when unknown. |
| `last_run.failed_command` | That line as written, before variable expansion. |
| `last_run.stderr_tail` | Last lines of stderr, redacted, up to 2 KiB. |
| `last_run.hint_code` | Stable id of a recognized cause. Absent when the cause isn't recognized. |
| `last_run.hint` | Likely fix for a recognized cause. |
| `last_run.message` | One-sentence summary of the result. |
| `last_run.duration_ms` | Total run time, including starting compute. |

## Setup runs

A setup run executes the setup script on fresh compute, records its output, checks the compute, saves a reusable image, and releases the compute. Saving an environment never starts compute. After the save succeeds, call `createEnvironmentSetupRun` with its own `Idempotency-Key` to start a run.

| Field | Description |
| - | - |
| `setup_run_id` | Id of the run. |
| `kind` | `manual` (you started it to verify the environment), `runner_refresh` (the platform captured a new setup image after the runner changed), or `session` (an older session ran setup on its own compute). |
| `session_id` | The session, for `kind: "session"`. |
| `status` | See [Setup-run statuses](#setup-run-statuses). |
| `phase` | Current or final phase: `provision`, `gpu_check`, `setup`, `profile`, `commit`, or `cleanup`. |
| `exit_code`, `failed_line`, `failed_command`, `stderr_tail` | Same meaning as in `last_run`. |
| `hint_code`, `hint`, `message` | Same meaning as in `last_run`. |
| `failure_code` | Failure class for a run that didn't succeed, using the same codes as session failures, such as `environment_setup_failed`, `sandbox_capacity_unavailable`, or `sandbox_provision_timeout`. |
| `next_action` | The call to make next, as `operation_id`, `method`, and `path`. |
| `requested_by_user_id` | Who started a manual run. |
| `created_at`, `started_at`, `finished_at` | When the run was requested, started compute, and finished. |
| `duration_ms` | Total run time, including starting compute. |
| `log_lines`, `log_bytes` | Size of the stored log. The last line's `seq` equals `log_lines`. |
| `log_truncated` | `true` when output passed 4 MiB. The beginning of the log and `stderr_tail` are kept. |

### Setup-run statuses

| `status` | Meaning | Terminal |
| - | - | - |
| `queued` | Accepted, waiting to start compute. | No |
| `provisioning` | Fresh compute is starting. | No |
| `running` | The script and compute checks are running. | No |
| `succeeded` | The script and checks passed. The environment is verified for this configuration. | Yes |
| `failed` | Compute, the script, or a check failed. The environment records a failed verdict. | Yes |
| `cancelled` | Stopped by `cancelEnvironmentSetupRun`. No verdict is recorded. | Yes |

Only a failure in the `setup` phase points at your script. A run can succeed even if saving the reusable image fails; the console then shows **Setup passed, but its reusable image failed.** and sessions can't start until a later run saves one.

### Setup-run log lines

`getEnvironmentSetupRunLog` returns `lines`, a `next_after` cursor, the run's `status`, and `truncated`.

| Field | Description |
| - | - |
| `seq` | Position of the line in the run. Pass the last one as `after` to continue. |
| `stream` | `stdout`, `stderr`, `marker` (the script line about to run), or `system` (a phase outside your script). |
| `line` | One-based script line this output belongs to, when known. |
| `text` | The line, redacted, without its trailing newline. |
| `at` | When the line was captured. |

### Setup-run parameters

| Operation | Parameter | Range or requirement |
| - | - | - |
| `getEnvironmentSetupRun` | `wait_seconds` | Accepts 0 to 60; the server waits at most 5 seconds. |
| `getEnvironmentSetupRunLog` | `after` | `0` (default) reads from the start. |
| `getEnvironmentSetupRunLog` | `limit` | 1 to 2000, default 500. |
| `getEnvironmentSetupRunLog` | `wait_seconds` | Accepts 0 to 60; the server waits at most 5 seconds. To tail, use `0` and wait about two seconds between calls. |
| `listEnvironmentSetupRuns` | `limit` | 1 to 50, default 50. |
| `createEnvironmentSetupRun` | `Idempotency-Key` | Required. Use one key for each logical run, and reuse it with the same request if the response is lost. |

## Network policy

`network_policy` decides which hosts the sandbox can reach. It governs the setup script, the agent's commands, package managers, `git`, and websites the sandbox browser opens. Model calls, the agent's MCP servers, and the `web_search` and `web_fetch` tools don't go through the sandbox, so the policy doesn't affect them.

* A session started by an automation or from Slack, in an environment with any restricted policy (limited or no internet access), doesn't get the `web_fetch` tool.
* Tools of built-in integration apps, such as Jira and Confluence, run inside the sandbox. Recursion adds the hosts they need to a policy limited to specific hosts; a no-access policy blocks them.

```json theme={"theme":"css-variables"}
{
  "version": "v1",
  "rules": [
    {"match": {"host_glob": "registry.npmjs.org"}, "action": {"type": "allow"}},
    {"match": {"host_glob": "*.example.com"}, "action": {"type": "allow"}}
  ]
}
```

| Field | Description |
| - | - |
| `version` | `v1`. |
| `rules` | Ordered list, up to 200 rules. |
| `rules[].match.host_glob` | Host to match, such as `api.example.com`, `*.example.com`, or `*` for every host. |
| `rules[].action.type` | `allow` or `deny`. Other action types are restricted and rejected with `400 invalid_request`. |

How rules apply:

* Rules are checked in order. The first rule that matches a host decides.
* Outbound traffic that no rule allows is blocked.
* A rule allows or blocks a whole host. Once allowed, sandbox code can send any request to it, including uploads.
* Unknown fields are rejected with `400 invalid_request`.

### Common policies

| Goal | `network_policy` |
| - | - |
| No internet (default for new environments) | `{"version": "v1", "rules": []}` |
| Only npm | `{"version": "v1", "rules": [{"match": {"host_glob": "registry.npmjs.org"}, "action": {"type": "allow"}}]}` |
| Everything except one host | `{"version": "v1", "rules": [{"match": {"host_glob": "blocked.example.com"}, "action": {"type": "deny"}}, {"match": {"host_glob": "*"}, "action": {"type": "allow"}}]}` |
| Unrestricted | `{}` |

### Hosts common tools need

| Tool | Hosts to allow |
| - | - |
| `git` and `gh` with GitHub | `github.com`, `api.github.com`, `codeload.github.com`, `objects.githubusercontent.com`, `github-cloud.s3.amazonaws.com` |
| pip or uv from PyPI | `pypi.org`, `files.pythonhosted.org` |
| npm | `registry.npmjs.org` |

Attaching a credential in a vault never opens the network. If the agent uses a credential from inside the sandbox, the policy must also allow the host. See [Authenticate with vaults](/recursion/vaults) and [GitHub access](/recursion/github).

### Defaults and updates

* **Create:** omitting `network_policy`, or sending `null`, stores `{"version": "v1", "rules": []}`: no internet. Send `{}` for unrestricted access.
* **Update:** omitting `network_policy` keeps the saved policy. A value you send replaces the whole policy.
* **Privileged Docker:** a nonempty policy cannot be combined with `privileged: true` on create or when access settings change. Use `{}` for unrestricted access before enabling privileged Docker.
* **Access precondition:** a PATCH that changes a saved policy other than `{}` (including the default no-internet policy), or turns `privileged` on, must include `expected_access` with the `network_policy` and `privileged` values from your last read. Missing returns `428 precondition_required`; out of date returns `412 precondition_failed`.
* **Console:** **Internet access** sets **Enabled** (`{}`) or **No access** (no rules). A saved host-rule policy shows as **Custom policy (saved)** until you pick a mode, which replaces it.
* **Verification:** any policy change makes the setup verdict stale.

## Privileged Docker

`privileged: true` lets the agent run Docker containers inside the sandbox. It's `false` by default and requires unrestricted internet access (`network_policy: {}`). The console's **Privileged Docker access** checkbox is available after you choose **Internet access → Enabled**. Changing either setting does not silently change the other. Turning privilege on needs `expected_access` in a PATCH and makes the setup verdict stale. If an existing environment has both limited internet access and privileged Docker, you can view it and edit other settings. To start a new session, enable unrestricted internet access or turn off privileged Docker. If no compute is available for the requested settings, saving or starting a session fails.

## Computer use

`computer_use: true` adds a shared browser display to the sandbox, which the agent controls with the `computer` tool. You can follow the recorded screens in the session. The service sets `http_port` to `6901` for the display. When the agent needs a person for a step such as a sign-in, it calls `handoff_to_user` and the session waits in `awaiting_human`; see [Sessions](/recursion/sessions).

* `idle_stop_after_seconds` must be `0` or at least 600.
* The browser follows the network policy. Under a restricted policy, allow each site the agent must visit.
* Turning computer use on or off doesn't make the setup verdict stale.

## Environment variables

`env_vars` is a map of names to plain-text values exported in the sandbox. Anyone who can view the environment can read the values, and so can any code the agent runs. Never put secrets here; use [vaults](/recursion/vaults) instead.

* `PATH` is rejected. The sandbox already selects `/workspace/.venv`; call tools by full path or prepend to `$PATH` inside a command.
* Variables the shell, the loader, or Python read at startup are rejected too: `BASH_ENV`, `HOME`, `SHELLOPTS`, `PS4`, `PYTHONPATH`, `PYTHONHOME`, `PYTHONUSERBASE`, `PYTHONWARNINGS`, `PYTHONPLATLIBDIR`, `LD_PRELOAD`, `LD_AUDIT`, `LD_LIBRARY_PATH`, `GCONV_PATH`, and `BASH_FUNC_*`. Set one inside the command that needs it instead, for example `PYTHONPATH=/workspace/src python main.py`.
* In an update, `env_vars` replaces the whole map. Send `{}` to clear it.
* Changing variables makes the setup verdict stale.

## Mounts

`mounts` downloads files into the workspace when a sandbox starts, so the agent finds input data on disk at its first turn.

```json theme={"theme":"css-variables"}
{
  "mounts": [
    {"source": "https://data.example.com/exports/customers.csv", "mount_path": "data/customers.csv"}
  ]
}
```

| Field | Description |
| - | - |
| `mounts[].source` | HTTPS URL of the file. |
| `mounts[].mount_path` | Where the file goes, relative to `/workspace`. It must name a file: it can't end in `/` or contain `..`. |

* Two mounts can't use the same path or one inside the other.
* Mounts are checked when a sandbox starts, not when you save. An invalid mount makes provisioning fail, so run a setup test after you change mounts.
* Anyone who can view the environment can read each `source` URL. Don't embed long-lived credentials in it.
* In an update, `mounts` replaces the whole list. Send `[]` to clear it.

## Sandbox tools that depend on the environment

The agent's tools run in the sandbox this environment defines. A few depend on its settings. For the full tool list, see [Tools](/recursion/tools).

| Tool | Depends on |
| - | - |
| `run_command`, `read_file`, `write_file`, `share_file` | Available in every Agent runner sandbox. Commands follow the network policy. |
| `computer` | `computer_use: true`. Websites follow the network policy. |
| `release_compute` | Root sessions only. Releases compute early and keeps the workspace. Subagents share their root's compute and can't release it. |

A tool being available never grants network access.

## Environment lifecycle

| Event | Effect |
| - | - |
| Create | Saves the definition. No compute starts. |
| Setup run | Starts fresh compute, runs the script, checks the compute, saves a reusable image, releases the compute, and records the verdict. |
| Start a session | Starts a fresh sandbox from the saved image, or from the runner image when there is no script. If the runner has changed, the platform first captures a matching setup image on dedicated compute; the session waits in provisioning and shows the setup run log. |
| Update | Saves the changes without starting a setup run. The changes apply to sandboxes started afterward; running sessions keep their configuration. |
| Idle stop | Stops compute and keeps the workspace. The next tool call restores it. |
| Stopped retention | Allows a stopped sandbox to be released. A later message can reopen the same session's workspace files. |
| Delete environment | Removes it from lists and reads. New sessions can't use it; running sessions keep their sandbox. There is no restore. |

## Limits

| Limit | Value |
| - | - |
| Name | 256 characters |
| CPU | 250 to 8000 millicores; the available compute or a restricted network policy can impose a lower maximum |
| Memory | 1024 to 32768 MiB; the available compute or a restricted network policy can impose a lower maximum |
| GPU | One per environment: `a100`, `l4`, or `t4` |
| Workspace disk | 1 to 100 GiB |
| Idle stop | 120 to 7200 seconds; 600 minimum with computer use |
| Stopped delete | 3600 to 2592000 seconds |
| Setup script | 64 KiB |
| Setup timeout | 10 to 3600 seconds, default 600 |
| Setup runs in flight | 1 per environment; 4 manual runs per organization |
| Setup-run log | 4 MiB per run; 2 KiB `stderr_tail` |
| Network policy | 200 rules |

For every product limit, see [Limits](/recursion/limits).

## Next steps

<CardGroup>
  <Card title="Set up an environment" href="/recursion/environments">
    Create, verify, restrict, update, and delete environments.
  </Card>

  <Card title="Deliverables and artifacts" href="/recursion/artifacts">
    How agents save files to the outputs folder and how they're collected.
  </Card>

  <Card title="Security" href="/recursion/security">
    What reaches the sandbox and how credentials flow.
  </Card>

  <Card title="Troubleshooting" href="/recursion/troubleshooting">
    Fix common session and environment problems.
  </Card>
</CardGroup>


# Errors
Source: https://docs.labelbox.com/recursion/errors

Every error code the Managed Agents API returns, grouped by area, with its cause, the fix, and whether retrying can help.

Every failed request returns the same JSON shape with a stable `code`. Branch on `code`, never on `message`. This page lists every code you can receive, what causes it, and whether to retry.

## The error shape

```json theme={"theme":"css-variables"}
{
  "code": "idempotency_in_progress",
  "message": "a request with this Idempotency-Key is already in progress",
  "details": {
    "field": "Idempotency-Key",
    "requestId": "0b6f2c1d-9a4e-4b7f-8c3d-5e6a7b8c9d0e"
  }
}
```

| Field | Use |
| - | - |
| `code` | Stable and machine-readable. Branch on this. |
| `message` | For people. It can change at any time. |
| `details` | Code-specific context, such as `field`, measured values, limits, or the next call to make. |
| `details.retryable` | `true` when the same request can succeed later, and `false` when it must not be retried. When omitted, follow the operation's generated retry strategy in **Endpoints**. |
| `details.requestId` | Include this, and the `x-request-id` response header, when you report a problem. |

## How to retry

The operation's generated retry strategy in **Endpoints** decides whether replay is allowed. `Retry-After` only says when to make an allowed retry; it never grants retry permission.

| Situation | What to do |
| - | - |
| `Retry-After` header present | If the operation's strategy permits replay, wait that long before retrying. It's sent with some `409` and `429` responses. |
| `details.retryable: true`, no `Retry-After` | If the operation's strategy permits replay, retry with exponential backoff and jitter, starting around one second. |
| `5xx` on a keyed mutation | Retry only when the operation's strategy permits it, using the same `Idempotency-Key` and exact request bytes. |
| Any other `4xx` | Don't retry unchanged. Fix the request first. |

## Request and validation

| Status and code | Cause | Fix | Retry? |
| - | - | - | - |
| `400 invalid_request` | A field, query parameter, or header is missing, malformed, or out of range. `details.field` names it. Also returned when the body isn't valid JSON, a tenant-scoped key omits `x-organization-id`, `Idempotency-Key` is malformed or repeated, or a `page_token` has expired or doesn't match the query. Session-list tokens last one hour. | Fix the named field. For a `page_token`, restart the list without it. | No |
| `400 invalid_json` | The body is JSON but doesn't match the operation's schema, such as a field with the wrong type. `details.field` names the field when it's known. | Check the body against **Endpoints**. | No |
| `413 payload_too_large` | The body is bigger than the operation accepts, for example a skill bundle over 8 MiB or an `uploadFile` file part over 30 MiB. | Send a smaller body. | After fixing |
| `415 unsupported_media_type` | The body's media type isn't accepted. Skill bundles and `uploadFile` use `multipart/form-data`; other request bodies use JSON. | Use the operation's declared media type. | No |

## Authentication and access

| Status and code | Cause | Fix | Retry? |
| - | - | - | - |
| `401 unauthorized` | No API key, or the key is malformed, expired, revoked, or no longer active. | Check the `Authorization: Bearer` header. See [API keys](/recursion/api-keys). | No |
| `403 forbidden` | The key is valid, but your role in the organization doesn't allow this operation. | See [Organizations and roles](/recursion/organizations-and-roles). Use an account or key with the developer or admin role. | No |
| `403 session_start_not_admitted` | New work was refused. With `details.reason: "out_of_credit"`, your tenant has used its credits, so new sessions and messages that start a turn are refused. A session that is already running pauses before its next model request, with `stop_reason: out_of_credit`. With `details.reason: "tenant_suspended"`, your account is suspended and its running sessions pause with `stop_reason: tenant_suspended`. | Out of credit: buy credits, then start the session or send the message again. See [When credits run out](/recursion/billing#when-credits-run-out). Suspended: contact support. | After a top-up, or once support restores the account |
| `403 setup_run_not_admitted` | A setup test was refused before it was recorded. With `details.reason: "out_of_credit"`, your tenant has used its credits; with `details.reason: "tenant_suspended"`, your account is suspended. | Out of credit: buy credits, then test setup again. See [When credits run out](/recursion/billing#when-credits-run-out). Suspended: contact support. | After a top-up, or once support restores the account |
| `503 setup_run_admission_unavailable` | The setup test's credit check could not be completed; no run was recorded. | Retry the same request with the same `Idempotency-Key`. | Yes |
| `404 not_found` | The resource doesn't exist, or it's in an organization you can't reach. Also returned for an `x-organization-id` you can't use, and for any path the API doesn't serve. The cases look the same on purpose. | Check the id, the path, and the organization. | No |
| `503 service_unavailable` | Your key couldn't be checked right now. The request did not run. | Retry shortly. | Yes |

## Conflicts and concurrency

| Status and code | Cause | Fix | Retry? |
| - | - | - | - |
| `409 conflict` | The request conflicts with the resource's current state, such as a duplicate name, or deleting a session while its hard stop is still finishing. | Read the resource, then change the request. | Only when `details.retryable` is `true`. |
| `409 conflict` on a session's agent version | The session can't be moved to another version: it is a subagent session, a session the service runs itself, a pull request review session, or one that is still stopping or can no longer continue. Reading the version state of a subagent session, or of a session that wasn't started from an agent, answers the same. | Move or read the root session. See [Move a session to another version](/recursion/sessions#move-a-session-to-another-version). | Only when `details.retryable` is `true`. |
| `409 revision_conflict` | Someone published a newer agent or skill version after you read it, or an environment's settings changed while your update was saving. | Read it again, apply your change to the latest version, and retry. | After re-reading |
| `412 precondition_failed` | An environment update sent `expected_access` that no longer matches its network policy or privileged access. | Read the environment and retry with its current access settings. | After re-reading |
| `428 precondition_required` | An update changes an environment's stored network policy, or turns on privileged access, without `expected_access`. | Read the environment and send its current access settings as `expected_access`. | After re-reading |
| `412 precondition_failed` on `If-Match` | An automation, event source, or its public representation contract changed after you read it. | Read it again and retry with its new returned ETag. | After re-reading |
| `428 precondition_required` on `If-Match` | A replace, pause, resume, or delete of an automation or event source sent no `If-Match`. | Send the complete returned `ETag` from the current read as `If-Match`; do not construct it from `revision`. | After fixing |

## Idempotency

| Status and code | Cause | Fix | Retry? |
| - | - | - | - |
| `409 idempotency_conflict` | The `Idempotency-Key` was already used for a different request. | Use a new key for a new request. Resend the exact same bytes to replay. | No |
| `409 idempotency_in_progress` | The first request with this key is still running. | Wait for `Retry-After`, then send the same request again. | Yes |
| `503 idempotency_unavailable` | The key couldn't be checked or reserved. The request did not run. | Retry with the same key. | Yes |

See [Idempotent mutations](/recursion/api#idempotent-mutations).

## Rate limits and capacity

| Status and code | Cause | Fix | Retry? |
| - | - | - | - |
| `429 rate_limit_exceeded` | You sent too many requests. The limit is per allowance: all your API keys share one, and console use has its own. | Wait for `Retry-After`. | Yes |
| `429 rate_limited` | Your organization's request rate limit is exhausted. | Wait for `Retry-After`. | Yes |
| `429 tenant_concurrency_limit` | Your tenant already has as many working root sessions as it may. `details.limit` is the cap. | Wait for `Retry-After`, or for a session to finish its turn. | Yes |
| `429 setup_run_limit` | Your organization has as many setup runs in flight as it may. `details.limit` is the cap and `details.in_flight` lists the runs. | Wait for a run to finish, or cancel one. | Yes |
| `503 service_unavailable` | The service is briefly unavailable, or at event-stream capacity. | Retry with backoff. For streams, close ones you no longer need. | Yes |

## Environments and setup

| Status and code | Cause | Fix | Retry? |
| - | - | - | - |
| `422 environment_not_verified` | The environment's setup script has never passed, failed, changed since it passed, or is still being tested. `details.next_action` names the call to make. | Run a setup test and wait for it to pass. See [Environments](/recursion/environments). | Yes, while a test is running |
| `422 sandbox_provider_disabled` | The request names a runtime this deployment doesn't offer for new work. | Use the **Agent runner** runtime (`runs`). | No |
| `409 setup_run_in_progress` | A setup run is already in flight for this environment. `details.setup_run_id` names it. | Wait for it to finish; a manual run or automatic runner refresh can be cancelled by a member allowed to update the environment. | After it ends |
| `409 setup_run_finished` | You tried to cancel a setup run that had already finished. Nothing changed. | None needed. | No |
| `503 setup_run_not_started` | The setup run was recorded but couldn't start. It's marked failed. | Start another setup run. | Yes |

## Integrations

| Status and code | Cause | Fix | Retry? |
| - | - | - | - |
| `400 invalid_request` on `resources` or `permission` | A tool allow-list is empty, has more than 200 tools, names a tool the app doesn't have, or holds a tool that isn't read-only under the `read` preset. | Send 1 to 200 tool names from the catalog, or use `permission: "write"`. See [Integrations](/recursion/integrations#change-an-apps-tools). | No |
| `400 invalid_request` on `built_in_integrations` | An agent names more than 25 apps, sends `tools`, or names a connection that isn't an active built-in integration with at least one tool. | Fix the entry `message` describes. See [Grant apps to an agent](/recursion/integrations#grant-apps-to-an-agent). | No |
| `422 provider_selection_unsupported` | A tool allow-list change was sent for a native integration, whose access is set per agent. | Set the access on the agent instead. | No |
| `503 built_in_catalog_unavailable` | The app catalog couldn't be loaded. | Retry. | Yes |
| `503 built_in_integrations_unconfigured` | The app catalog couldn't be reached for your organization. | Contact support. | No |
| `503 integrations_unconfigured` | The integration service couldn't be reached. | Retry later. If it persists, contact support. | Yes |
| `429 github_rate_limited` | GitHub is limiting requests made for your connection. | Retry after `Retry-After`. | Yes |
| `503 github_unavailable` | GitHub couldn't be reached. | Retry later. | Yes |

A connection test that fails still returns `200`, with `ok: false` and a `failure` reason. See [Test a connection](/recursion/integrations#test-a-connection).

## Files

| Status and code | Cause | Fix | Retry? |
| - | - | - | - |
| `413 payload_too_large` on `uploadFile` | The file part is over 30 MiB, or the complete multipart request exceeds the file limit plus its 128 KiB framing and form-field allowance. | Upload a file of at most 30 MiB through the API, or use the console for a larger file. | After fixing |
| `413 file_quota_exceeded` | The upload would take your organization's live files past its storage quota. `details` has `quota_bytes`, `used_bytes`, and `incoming_bytes`. | Set expiries on uploads so they stop counting once they expire. | After fixing |
| `404 not_found` on `file_id` | The file doesn't exist in your organization, or it expired. | Use a live file id. | No |
| `409 conflict` on `resources` | An attachment path collides with another, or the session isn't the root of its tree. | Choose another `relative_path`, or attach to the root. | After fixing |

See [Files](/recursion/files).

## Automations and event sources

| Status and code | Cause | Fix | Retry? |
| - | - | - | - |
| `422 automation_reference_invalid` | The automation's agent version or environment doesn't exist or can't be used. | Choose a live agent version and a verified environment. | After fixing |
| `422 automation_event_source_invalid` | A trigger names a missing or incompatible event source. | Use a source of the trigger's type. | After fixing |
| `409 automation_trigger_id_in_use` | A trigger's `triggerId` already belongs to another automation. A trigger id is never reused, even after its trigger or automation is deleted. | Give the trigger a new UUID. | After fixing |
| `422 event_source_verification_credential_invalid` | A signed custom source names a verification credential that isn't a live signing secret in your organization. | Name a live signing secret from your organization, or keep the source's current one. | After fixing |
| `409 conflict` on `eventSourceId` | The source is still used by automations, or it's managed by its connection. | Remove it from triggers first, or manage it through its connection. | After fixing |
| `503 event_admission_unavailable` | A webhook delivery couldn't be recorded. Returned to the sender. | The sender retries. | Yes |

A run that can't start its session ends `failed` with an `error` that has its own `code`, `message`, and `retryable`. See [Automations](/recursion/automations#see-run-history).

## Models and the session analyst

| Status and code | Cause | Fix | Retry? |
| - | - | - | - |
| `503 model_gateway_unreachable` | The model list, or the model a request names, couldn't be checked right now. | Retry later. | Yes |
| `503 session_analyst_busy` | Another request to the same session analyst was still running. | Retry. | Yes |
| `503 session_analyst_model_unavailable` | The session analyst's model isn't available. | Retry later. | Yes |
| `503 session_analyst_unconfigured` | The session analyst couldn't be started for your organization. | Contact support. | No |

## Server errors

| Status and code | Cause | Fix | Retry? |
| - | - | - | - |
| `500 internal_error` | An unexpected failure. Details are withheld. | Retry once with the same `Idempotency-Key` if it was a create. If it persists, report `details.requestId`. | Once |
| `500 invariant_violation` | An internal consistency check failed. Details are withheld. | Retry once. If it persists, report `details.requestId`. | Once |
| `502 bad_gateway` | An internal service returned an invalid response. | Retry with backoff. | Yes |
| `503 managed_agents_unavailable` | Managed Agents is temporarily unavailable. | Retry with backoff. | Yes |
| `504 gateway_timeout` | The request took too long. It may or may not have run. | Retry creates with the same `Idempotency-Key`. Read the resource before retrying other writes. | Yes |
| Any other `503` code | A service dependency is temporarily unavailable. | Retry with backoff. If it persists, report `details.requestId`. | Yes |

## Session failure codes

A session that fails has a `failure` object. These codes are separate from HTTP errors: the request that started the session succeeded, but the work later stopped.

```json theme={"theme":"css-variables"}
{
  "status": "failed",
  "failure": {
    "code": "sandbox_provision_timeout",
    "phase": "provisioning",
    "category": "transient",
    "retryable": true,
    "message": "Agent runner provisioning timed out. The session may succeed when capacity recovers."
  }
}
```

Read `category` first:

| `category` | Meaning | What to do |
| - | - | - |
| `transient` | A temporary condition. | Send a follow-up message to resume, or start again later. |
| `caller_error` | Something in your request or configuration. Retrying as is won't help. | Fix the agent, environment, or grant, then resume or start a new session. |
| `internal` | A problem on our side. | Report the `session_id`. |

| `failure.code` | Cause | Fix |
| - | - | - |
| `sandbox_provision_timeout` | Compute wasn't ready within 5 minutes. | Resume or start again. |
| `sandbox_capacity_unavailable` | Compute capacity was temporarily unavailable. | It's retried automatically. If it persists, try again later. |
| `sandbox_image_pull_failed` | The image saved by the environment's last passing setup test couldn't be loaded. | None. It's retried automatically without rerunning setup. |
| `sandbox_configuration_invalid` | The environment's settings can't be run. | Fix the environment. `message` says what's wrong. |
| `sandbox_provider_unavailable` | The runtime was briefly unavailable. | Resume or start again. |
| `sandbox_gpu_not_ready` | The GPU didn't pass its readiness check. | Report it. Starting again usually reproduces it. |
| `environment_setup_failed` | The environment's setup failed while preparing the session. | Check the linked setup run, fix the script if it failed, and test setup again. See [Environments](/recursion/environments). |
| `credential_resolution_failed` | A granted credential couldn't be opened. | Check the session's vault grants and that the credentials still exist. |
| `model_not_found` | The agent's model isn't available. | Update the agent to a model from `listModels`. |
| `model_authentication_failed` | The model provider rejected the platform's credential for the model. | None on your side. Choose another model or try later, and report the `session_id`. |
| `model_rate_limited` | The model provider throttled the requests. | Resume later. |
| `model_provider_unavailable` | The model provider was unavailable. | Resume later. |
| `model_request_failed` | The model provider rejected a request. | Check the agent's model settings, such as reasoning effort. |
| `model_prompt_cache_invalid` | The agent's prompt-cache retention settings are invalid or unsupported for its model. No model request was sent. | Clear or correct the agent's prompt-cache retention settings, then start a new session. |
| `model_reasoning_capacity_invalid` | The reasoning effort doesn't fit the model's output capacity. | Lower the agent's reasoning effort or choose another model. |
| `model_refused` | The model declined to continue the task. | Change the task or instructions. Resuming repeats the refusal. |
| `roster_agent_unresolved` | A subagent in the roster names an agent, version, or model that doesn't exist. | Fix the roster. See [Multi-agent](/recursion/multi-agent). |
| `worker_replaced` | A service update interrupted the work. | Send a follow-up message to resume if the session is still stopped. |
| `repository_provider_unavailable` | The repository host was unavailable. | Resume later. |
| `internal_error` | An unexpected failure. | Report the `session_id`. |

Setup runs report the same failure vocabulary in their `failure_code`.

## Next steps

<CardGroup>
  <Card title="Troubleshooting" href="/recursion/troubleshooting">
    Match a symptom to its cause and fix.
  </Card>

  <Card title="Limits" href="/recursion/limits">
    See the limit behind each size or rate error.
  </Card>

  <Card title="API conventions" href="/recursion/api">
    Retries, idempotency, and paging.
  </Card>

  <Card title="Event types and states" href="/recursion/reference">
    Look up session statuses and stop reasons.
  </Card>
</CardGroup>


# Events
Source: https://docs.labelbox.com/recursion/events

Send messages to a session, stream and read its events without losing any, and watch many sessions at once.

Everything that happens in a session is recorded as an ordered event: your messages, the agent's replies, tool calls, and status changes. Send events to steer a session, and stream or list them to follow it.

<Note>
  You need a session id; see [Sessions](/recursion/sessions). Sending messages needs the organization developer or admin role. The organization user role can read and stream events. The billing role cannot see sessions. See [Organizations and roles](/recursion/organizations-and-roles). For API calls, create a key under **API keys**; see [API keys](/recursion/api-keys).
</Note>

## Send messages

`sendSessionEvents` adds turns to a session. Send `message` for one plain-text turn, or `events` for typed turns. If you send both, `events` wins and `message` and the top-level `actor` are ignored.

<Tabs>
  <Tab title="Console">
    Open the session, type in **Message the agent…**, and click **Send**. While the agent is working the button reads **Queue**, and **Interrupt & send** appears beside it.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53/events' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      -d '{
        "events": [
          {"type": "user.message", "text": "Use the incident tracker as the source of truth."},
          {"type": "system.message", "content": [{"type": "text", "text": "Reply in British English."}]}
        ]
      }'
    ```
  </Tab>
</Tabs>

```json theme={"theme":"css-variables"}
{ "ok": true, "delivery_state": "queued", "events_accepted": 2 }
```

The response confirms that your turns were stored. It does not include the agent's reply; read that from the stream or the event list. Compare `events_accepted` with the number of events you sent.

| `delivery_state` | Meaning |
| - | - |
| `queued` | The agent is mid-turn. It reads your message at the end of the current turn. [Interrupt](/recursion/sessions) to have it read now. Until then the message appears in pending inputs, not the transcript. See [Session operations](/recursion/session-operations). |
| `signaled` | The session was waiting, and your message woke it. |
| `resumed` | The session was idle or finished, and your message started a new run. Only root sessions resume. |
| `stored` | The message was saved, but nothing is running to read it yet. |

<Accordion title="Event fields and send rules">
  | Event field | Description |
  | - | - |
  | `type` | `user.message` (the default), `system.message`, or `handoff_resolved`. Any other value fails the request with `400` naming that entry, and nothing is appended. |
  | `text` | Shorthand for one text block. Ignored when `content` is set. |
  | `content` | Content blocks, such as `[{"type": "text", "text": "..."}]`. Wins over `text`. A `system.message` uses only `content`. |
  | `actor` | Who a `user.message` is attributed to. Defaults to `human:api`. A `system.message` ignores it. |

  A `user.message` with neither `text` nor `content` is dropped without an error. If every entry is dropped, the request also fails with `400`.

  The top-level `referenced_session_ids` grants the session read access to up to 10 earlier sessions from this turn on. If you cannot read one of them, the whole request fails with `404` and no message is delivered. See [Referenced sessions](/recursion/referenced-sessions).

  A top-level `image` or `document` block can name a file from your library instead of carrying bytes. See [Show a file to the agent in a message](/recursion/files#show-a-file-to-the-agent-in-a-message).

  A message to a session that is still being cancelled returns a retryable `409 conflict`. A message to a subagent session in a terminal status also returns `409 conflict`, because only root sessions resume.
</Accordion>

<Accordion title="Retry a send safely">
  To retry a send after an ambiguous network failure without adding the turn twice, send one stable `Idempotency-Key` header and resend the same request. See [Idempotent mutations](/recursion/api#idempotent-mutations).
</Accordion>

<Accordion title="Hand a browser step back">
  When a session is `awaiting_human`, the agent is waiting for a person to handle a step in its browser display, such as a sign-in. Its `active_handoff` names the request. Send a `handoff_resolved` event with that `handoff_id` to let the agent continue, with an optional `note` of up to 2,000 characters telling it what you did. It must be the only event in the request.

  ```bash theme={"theme":"css-variables"}
  curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53/events' \
    -H "Authorization: Bearer $RECURSION_API_KEY" \
    -H 'Content-Type: application/json' \
    -d '{"events": [{"type": "handoff_resolved", "handoff_id": "7c2e9a41-5b38-4d06-a1f7-3e8c6b2d9f50", "note": "Signed in to the vendor portal."}]}'
  ```

  A `handoff_id` that doesn't match the active hand-off, or a session with none, returns `409 conflict`. Without this event, the agent resumes on its own when the hand-off deadline passes.
</Accordion>

## Stream events

`streamSessionEvents` is a server-sent events (SSE) stream of a session's events as they are recorded. It replays everything after your cursor first, then stays open for new events. The stream closes on its own only when the session reaches a terminal status: `completed`, `failed`, or `cancelled`. An interrupted or sleeping session keeps it open.

<Tabs>
  <Tab title="Console">
    Open the session. The transcript streams live.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -N 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53/events/stream' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

```text theme={"theme":"css-variables"}
: connected

event: agent.message
id: 01997a3d-0b82-7c15-9a43-6e1f2d8c5b09
data: {"type":"agent.message","id":"01997a3d-0b82-7c15-9a43-6e1f2d8c5b09","session_id":"e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53","processed_at":"2026-09-25T14:05:40Z","content":[{"type":"text","text":"Found 12 open incidents."}],"raw_event":{"event_type":"message"}}

: ping
```

Each event is sent as `event: <type>`, `id: <event_id>`, and `data: <json>`. Save the `id` of each frame you handle: it is your position if the connection drops. See [Recover from a dropped stream](#recover-from-a-dropped-stream).

<Accordion title="Stream parameters">
  | Parameter | Description |
  | - | - |
  | `after_event_id` | Start after this event id. Takes precedence over the `Last-Event-ID` header. |
  | `Last-Event-ID` header | Start after this event id when `after_event_id` is not set. Standard SSE clients send it on reconnect. |
  | `types` | Only send these frame types, comma-separated (`types=agent.message,session.status_idle`) or repeated as `types[]`. Filtered-out events still move your position forward. |
</Accordion>

<Accordion title="Every frame type">
  The frame's JSON has `type`, `id`, `session_id`, `session_thread_id`, `processed_at`, `content`, `stop_reason`, `usage`, `error`, `anchor_event_id`, and `raw_event`, which holds the full stored event.

  | Frame `type` | Sent for |
  | - | - |
  | `user.message`, `agent.message`, `system.message` | Messages, by role. |
  | `agent.tool_use`, `agent.tool_result` | Tool calls and their results. |
  | `agent.approval_request`, `user.tool_confirmation` | Approval requests and decisions. |
  | `agent.artifact` | A saved deliverable. |
  | `agent.thread_context_compacted` | A thread's context was compacted. |
  | `session.status_provisioning`, `session.status_queued`, `session.status_running` | The session entered that state. |
  | `session.status_idle` | The session went idle or completed. |
  | `session.status_terminated` | The session failed or was cancelled. |
  | `session.status` | Any other status change. |
  | `session.event` | Everything else: summaries, plan updates, turn commits, advisor notes, and board updates. Read `raw_event.event_type`. |
  | `session.error` | A problem with the stream. See **Stream errors and record size**. |

  The stream also sends frames without an `id`. These are not stored events and do not move your position:

  * `: connected` is the first line. `: ping` follows every 15 seconds. They keep the connection alive; ignore them.
  * When you connect to a running session, a `session.status_running` frame tells you the current state.
  * When the session reaches a terminal status, the stream sends one last `session.status_idle` (completed) or `session.status_terminated` (failed or cancelled), with `stop_reason` and, on failure, `error`. Then it closes.
</Accordion>

<Accordion title="Stream errors and record size">
  | Frame | Meaning | What to do |
  | - | - | - |
  | `session.error` with an `id` and `error.code: event_record_too_large` | One event was too large to send. `error.content_path_template` gives the path to fetch it. The stream continues. | Fetch it with `getSessionEventContent`, or list events with `payloads=refs`. |
  | `session.error` without an `id`, with `error.message` | The server could not read events. The stream closes. | Reconnect from your last event id. |

  The `Content-Type` of the response is `text/event-stream; charset=utf-8; max-record-bytes=<N>`, where `<N>` is the largest record the server may send. Size your SSE parser's buffer to that value. If a record is larger than your client accepts, list events after your last id with `payloads=refs`, then resume the stream after the event you fetched.
</Accordion>

## Recover from a dropped stream

Networks drop connections. The stream is designed so you can reconnect without losing events.

<Steps>
  <Step title="Track the last event you finished processing">
    Save a frame's `id` only after you have handled it. Frames without an `id` never change your position.
  </Step>

  <Step title="Reconnect from that id">
    Reconnect with `Last-Event-ID: <id>`, or with `after_event_id=<id>` in the URL. The server replays every event after it. If you use `after_event_id`, update it on every reconnect; reusing the original URL replays from the original position. A malformed id returns `400`. A well-formed id from another session does not fail; it is treated as a position.
  </Step>

  <Step title="Back off between attempts">
    Retry with exponential backoff and jitter. A `503` means the server is at its stream capacity; retry after a short wait.
  </Step>

  <Step title="Deduplicate by event id">
    Replays can repeat an event you already saw, so keep a set of handled `event_id` values and skip repeats.
  </Step>

  <Step title="Reconcile when the session ends">
    When several agents write at once, an event can be recorded slightly after a later-sorting event was already sent to you, and a cursor reader can miss it. After the stream closes at a terminal status, page through `listSessionEvents` from the start and add any event id you have not seen.
  </Step>
</Steps>

<Tip>
  If your process restarts and loses its place, list events with `listSessionEvents` to rebuild state, then open the stream with `after_event_id` set to the last id you listed.
</Tip>

## List events

`listSessionEvents` returns events oldest first, in pages. Any session id in a tree returns the combined timeline of the whole tree, including subagents.

<Tabs>
  <Tab title="Console">
    Open the session. The transcript shows every event, and the **Events** tab lists them with their raw fields.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53/events?limit=200' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

```json theme={"theme":"css-variables"}
{
  "events": [
    {
      "event_id": "01997a3c-5e21-7b4d-8f10-3c6a2e9d4b71",
      "event_type": "message",
      "actor": "human:api",
      "role": "user",
      "session_id": "e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53",
      "content": {
        "blocks": [{ "type": "text", "text": "Use the incident tracker as the source of truth." }]
      },
      "event_status": "completed",
      "created_at": "2026-09-25T14:05:02Z"
    }
  ],
  "next_page_token": "<id of the last event on this page>"
}
```

To read the next page, pass `next_page_token` as `after_event_id`. An empty token means you have read everything that exists right now.

<Accordion title="List parameters and paging">
  | Parameter | Description |
  | - | - |
  | `after_event_id` | Return events after this id, not including it. Must be a lowercase event id; a malformed value returns `400`. |
  | `limit` | Events per page. Default 100. A page also stops at about 8 MiB of data, so it can hold fewer events than `limit`. |
  | `hydrate` | `images` inlines image data in image blocks. |
  | `image_urls` | `signed` adds a short-lived `url` and `url_expires_at` to image blocks. A URL can be left out when many images are signed at once; fetch those with [Get an image](#get-an-image). |
  | `payloads` | `refs` returns large tool payloads as references instead of inline, which keeps pages small. |

  Any other value for `hydrate`, `image_urls`, or `payloads` returns `400`.

  **Pagination.** `next_page_token` is always present and equals the id of the last event on the page. A short page with a non-empty token does not mean the end. `events` can be `null` on the last page.

  **Event ids.** Event ids sort in the order events were created, so an id works as a position. An `after_event_id` that is well formed but not in this session is not an error: you get the events that sort after it.

  **Oversized events.** An event too large for a page is returned alone, with `content_hydration_status: oversized`. Fetch its full content with [Get full event content](#get-full-event-content).
</Accordion>

<Accordion title="Event fields">
  | Field | Description |
  | - | - |
  | `event_id` | Unique, time-ordered id. Use it as a cursor and to deduplicate. |
  | `event_type` | `message`, `tool_invocation`, `tool_result`, `approval_request`, `approval_decision`, `summary`, `plan_update`, `session_status`, `artifact`, `advisor_intervention`, `board_update`, `thread_context_compacted`, or `turn_committed`. |
  | `actor`, `role` | Who produced the event. `role` is `user`, `assistant`, or `system` on messages. |
  | `content` | `blocks`, plus `stop_reason`, `usage`, `plan`, and `metadata` where they apply. |
  | `event_status` | `pending`, `completed`, `warning`, `error`, or a lifecycle value on `session_status` events. |
  | `session_id`, `thread_id` | Which session and thread produced the event. See [Multi-agent sessions](/recursion/multi-agent). |
  | `tool_name`, `tool_use_id` | On tool calls and results. A result shares `tool_use_id` with its call. |
  | `parent_event_id`, `causal_event_id`, `turn_id` | Links to related events and the turn. |
  | `input_tokens`, `output_tokens`, `cost_micros` | Usage on model events. `cost_micros` is for diagnostics, not what you are charged. See [Usage and cost](/recursion/usage-and-cost). |
  | `content_hydration_status` | Set when content was not returned in full. |
  | `created_at` | When the event was recorded. |
</Accordion>

A [move to another agent version](/recursion/sessions#move-a-session-to-another-version) is recorded as a `session_status` event with `event_status: agent_version_changed`. Its `content.metadata` names the earlier and the new version, who asked, and what changed. A refused move is a `session_status` event with `event_status: warning` and the code `agent_version_change_failed`. The agent also receives one `message` event with `role: system` that lists what changed.

### Get full event content

`getSessionEventContent` returns the full `content` of one event. Use it for oversized events or payload references. The event must belong to the session. The response carries an `ETag`.

```bash theme={"theme":"css-variables"}
curl 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53/events/01997a3c-5e21-7b4d-8f10-3c6a2e9d4b71/content' \
  -H "Authorization: Bearer $RECURSION_API_KEY"
```

### Get an image

Image blocks in events carry an image URI. `getSessionImage` returns the image bytes for it, with the image's media type. Call it with an HTTP client or cURL. The URI must come from an event in the same session tree. In the console, images show inline in the transcript.

```bash theme={"theme":"css-variables"}
curl -G 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53/images' \
  --data-urlencode 'uri=<the image URI from the event>' \
  -H "Authorization: Bearer $RECURSION_API_KEY" \
  -o image.png
```

To skip this call, list events with `image_urls=signed` and download from each block's short-lived `url`.

## Watch many sessions with the gestalt

The gestalt is a live, compact summary of every root session in a time window: its status, execution state, and last activity. It is built for dashboards and alerting. Use it instead of polling `listSessions` or opening one event stream per session. Take one snapshot, draw it, then keep it current with the gestalt stream.

```bash theme={"theme":"css-variables"}
curl 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/gestalt?since=2026-09-18T00:00:00Z&agent_id=b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38' \
  -H "Authorization: Bearer $RECURSION_API_KEY"
```

```json theme={"theme":"css-variables"}
{
  "as_of": "2026-09-25T14:20:03.512Z",
  "sessions": [
    {
      "session_id": "e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53",
      "agent_id": "b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38",
      "status": "active",
      "execution_state": "running",
      "created_at": "2026-09-25T14:02:11Z",
      "updated_at": "2026-09-25T14:19:48Z",
      "last_activity_at": "2026-09-25T14:19:48Z"
    }
  ]
}
```

`since` is required, in RFC 3339 format. It can be at most 30 days ago, or 400 days ago when you also pass `agent_id`. Keep the snapshot's `as_of` for the stream.

<Accordion title="Snapshot fields">
  Sessions are ordered newest first by creation time. A row has `failed: true` when the session ended in a failure. `truncated: true` means only the newest 5,000 were returned; narrow the window or add `agent_id`. Both fields are left out when false.
</Accordion>

### Keep the snapshot current

Open `streamSessionGestalt` with the same `since` and `agent_id`, and `after` set to the snapshot's `as_of`.

```bash theme={"theme":"css-variables"}
curl -N 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/gestalt/stream?since=2026-09-18T00:00:00Z&agent_id=b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38&after=2026-09-25T14:20:03.512Z' \
  -H "Authorization: Bearer $RECURSION_API_KEY"
```

On each `gestalt.change` frame, replace each row by `session_id` and remove rows with `deleted: true`. The gestalt stream never closes on its own otherwise, so close it when your dashboard is done.

<Accordion title="Gestalt frames and reconnecting">
  | Frame | Meaning | What to do |
  | - | - | - |
  | `: connected`, `: ping` | Keep-alive. Nothing is sent while nothing changes. | Ignore. |
  | `gestalt.change` | `data` is `{as_of, sessions}` with the rows that changed. Its `id` is the new `as_of`. | Replace each row by `session_id`. Remove rows with `deleted: true`. |
  | `gestalt.reset` | The server cannot continue from your position. The stream closes. | Take a new snapshot and reconnect from its `as_of`. |
  | `gestalt.error` | The server hit an error. The stream closes. | Reconnect from the last `id` you received. If it repeats, take a new snapshot. |

  To reconnect, send the last frame `id` as `after` or as the `Last-Event-ID` header. One of them is required. A stream position older than 30 days is rejected; take a new snapshot.
</Accordion>

## What can go wrong

The most common problems:

* **The reply never arrives after `delivery_state: queued`.** The agent is still on its current turn. Wait, or interrupt the session.
* **`events_accepted` is lower than the events you sent.** Some entries were dropped. Check each entry's `type` and content.
* **The stream replays the same events after every reconnect.** The URL still has the original `after_event_id`. Update or remove it on reconnect.
* **The stream stays open after the agent stopped.** The session is interrupted or sleeping, not terminal. Close the stream yourself, or cancel the session.

<Accordion title="Every events error">
  | Symptom or code | Cause | Fix |
  | - | - | - |
  | `400 invalid_request` on `events` | Every event was dropped: unknown types or empty messages. | Send `text` or `content` on each `user.message`. |
  | `events_accepted` is lower than the events you sent | Some entries were dropped. | Check each entry's `type` and content. |
  | `404 not_found` when sending | The session or a referenced session is missing or not visible to you. | Check the ids and your access. |
  | `409 conflict` when sending | The session is still being cancelled. | Retry after a few seconds. |
  | The reply never arrives after `delivery_state: queued` | The agent is still on its current turn. | Wait, or interrupt the session. |
  | `409 conflict` when messaging a subagent | Subagent sessions in a terminal status do not resume. | Message the root session. |
  | `400 invalid_request` on `after_event_id` | The id is malformed or not lowercase. | Pass an `event_id` exactly as returned. |
  | The stream replays the same events after every reconnect | The URL still has the original `after_event_id`. | Update or remove it on reconnect. |
  | `503 service_unavailable` when opening a stream | The server is at its stream capacity. | Retry with backoff. |
  | `session.error` with `event_record_too_large` | One event is larger than a stream record. | Fetch it with `getSessionEventContent`. |
  | An event is missing from a long stream | A concurrent write was recorded out of order. | Re-read events after the session reaches a terminal status. |
  | The stream stays open after the agent stopped | The session is interrupted or sleeping, not terminal. | Close the stream yourself, or cancel the session. |
  | `gestalt.reset` | Your gestalt position can no longer be continued. | Take a new snapshot. |
  | `400 invalid_request` on `since` | The window is older than 30 days (400 with `agent_id`), or not RFC 3339. | Shorten the window or add `agent_id`. |

  Every error has required `code` and `message` fields plus optional `details`. When present, details can include `field`, `requestId`, `retryable`, and code-specific context. Use the generated **Endpoints** reference for exact alternatives and [Errors](/recursion/errors) for recovery guidance.
</Accordion>

Limits on referenced sessions, event pages, stream keep-alive, and the gestalt are in [Limits](/recursion/limits).

## Next steps

<CardGroup>
  <Card title="Sessions" icon="play" href="/recursion/sessions">
    Start, interrupt, cancel, and delete sessions.
  </Card>

  <Card title="Session operations" icon="gauge" href="/recursion/session-operations">
    Pending inputs, compute usage, the analyst, and the session tree.
  </Card>

  <Card title="Multi-agent sessions" icon="sitemap" href="/recursion/multi-agent">
    Read events across threads and subagents.
  </Card>

  <Card title="Deliverables and artifacts" icon="box-archive" href="/recursion/artifacts">
    Ask for deliverables and see how they're kept.
  </Card>
</CardGroup>


# Files
Source: https://docs.labelbox.com/recursion/files

Upload input files once, attach them read-only to any session, show them to the agent in a message, and download the deliverables sessions save.

Files are how you give a session input and how you take its results away. Your organization's files are of two kinds:

* **Uploads.** Files you add, such as a data set or a spec. Attach an upload to any number of sessions, read-only.
* **Session outputs.** Deliverables an agent saves in a session. They're kept with the session, and you can preview and download them.

The sidebar **Files** page shows uploads and session outputs together. You can also attach inputs when you launch a session or from its **Files** tab, and find that session's outputs there.

<Note>
  Listing and downloading files needs read permission. Uploading needs create, and attaching to or detaching from a session needs update. Organization developers and admins have all of them. See [Organizations and roles](/recursion/organizations-and-roles). Attaching files needs an environment with a sandbox, which every environment has by default.
</Note>

## Upload a file

<Tabs>
  <Tab title="Console">
    Open **Files** in the sidebar and click **Upload file**, or upload from **Launch session**, the session's **Files** tab, or **Attach from Files** in the message box. Each upload is added to your organization's files and selected for you. The console accepts a file up to 1 GiB.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/files' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -F 'file=@signups.csv;type=text/csv' \
      -F 'expires_in_seconds=604800'
    ```
  </Tab>
</Tabs>

The response is `201` with the file, including its `file_id`. Use that id to attach the file to a session or name it in a message.

To remove a file you no longer need, select it on the sidebar **Files** page and choose **Delete**. This works for uploads and session outputs. Deleting its catalog row removes it from Files and prevents future attachments; a copy already mounted in a running sandbox remains there. Session outputs that you keep are removed when their session is purged.

<Accordion title="Upload fields">
  `uploadFile` takes a `multipart/form-data` body. The `file` part's filename and `Content-Type` become the file's name and media type, and the part can hold at most 30 MiB. `expires_in_seconds` is optional, from 3600 (an hour) to 7776000 (ninety days); omit it for a file that doesn't expire. `metadata` is an optional JSON object sent as a string.
</Accordion>

## Attach files when a session starts

Name files in `resources` when you start the session. They're in the sandbox before the agent's first step. An attached file is copied into the sandbox under `/workspace/.managed-agents/files`, read-only. The agent knows it's there and can read it with `read_file` or from a shell.

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Sessions**, then click **Launch session**.
    2. Choose the agent and environment, and write the opening message.
    3. Under **Files**, click **Add**, choose each file or upload one, and optionally set the path it should have in the sandbox.
    4. Click **Launch session**.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/sessions' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      -H 'Idempotency-Key: signups-summary-2026-09-25' \
      -d '{
        "agent_id": "5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55",
        "environment_id": "0d5e8a2c-6f13-4b97-a4e0-3c7f9b1d5e62",
        "message": "Summarize the weekly trend in data/signups.csv.",
        "resources": [
          { "type": "file", "file_id": "9f8b1c2d-3e4f-5a6b-7c8d-9e0f1a2b3c4d", "relative_path": "data/signups.csv" }
        ]
      }'
    ```
  </Tab>
</Tabs>

The start is all-or-nothing: one unknown, expired, or out-of-organization `file_id` refuses the whole request with `404`, and no session is created.

<Accordion title="Paths and file content">
  `relative_path` places the file under the files directory; it defaults to the file's own name. It must be unique in the session and can't contain `.` or `..` segments. Each attachment takes the file's content as it is when you attach it.
</Accordion>

## Attach and detach files while a session runs

1. Open the session and choose the **Files** tab in the side panel. **Inputs** lists the files attached to the session.
2. Click **Attach files…**, choose or upload the files, and click **Attach**.
3. To detach one, click the **×** on its row in **Inputs**.

**What success means:** the file is placed in the sandbox on the session's next tool call, and the agent can see it from its next turn.

<Accordion title="Attach and detach rules">
  * Attach is all-or-nothing. The console refuses the batch if a file is unavailable or a path collides with or nests inside another attachment.
  * Attach to the root session of a multi-agent tree. The console refuses a subagent or teammate session; everyone in the tree shares the root's sandbox.
  * You can attach to a finished root session. Its next message resumes it with the file already in place.
  * Detach stops listing the file. A copy already in the sandbox stays, because the agent may be reading it.
</Accordion>

## Show a file to the agent in a message

To have the agent look at an image or read a short document as part of a message, name the file in the message instead of attaching it. The file is read when the message is accepted.

<Tabs>
  <Tab title="Console">
    In the session's message box, click **Attach from Files**, choose an image or a text document, or upload one, then send the message.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53/events' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      -d '{"events": [{"type": "user.message", "content": [
        {"type": "text", "text": "Does this chart match the numbers in the CSV?"},
        {"type": "image", "source": {"type": "file", "file_id": "2b6e9d41-8c37-4f05-a1d2-7e9c3b5f8a60"}}
      ]}]}'
    ```
  </Tab>
</Tabs>

Use `"type": "image"` for a PNG, JPEG, or WebP image the model sees, and `"type": "document"` for a text file, which is placed in the message as text. To send an image that isn't in your library, see [Send an image](/recursion/sessions#send-an-image).

## Download session deliverables

Every deliverable an agent saves in `/workspace/.managed-agents/outputs` is kept as a session output after each turn. See [Deliverables and artifacts](/recursion/artifacts) for what's kept and when.

<Tabs>
  <Tab title="Console">
    Open the session and choose the **Files** tab in the side panel. **Outputs** lists each deliverable with its path, size, and when it was captured. Click one to preview it, and download it from the preview.
  </Tab>

  <Tab title="cURL">
    List the session's deliverables, then download each by its `file_id`:

    ```bash theme={"theme":"css-variables"}
    curl 'https://api.recursion.labelbox.com/managed-agents/v1/files?scope_session_id=e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53' \
      -H "Authorization: Bearer $RECURSION_API_KEY"

    curl 'https://api.recursion.labelbox.com/managed-agents/v1/files/7c3d9e1a-4b5f-4a2e-8d61-0f9b2c7e5a38/content' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -o risks.md
    ```
  </Tab>
</Tabs>

Subagent and teammate deliverables are listed under the root session. `getFileContent` returns the bytes as `application/octet-stream` with a `Content-Disposition` naming the file. The recorded media type is in the file's metadata.

## List and find files

`listFiles` returns your organization's files, newest first, with a `next_page_token` while more follow. Filter by `source` to see only uploads or only session outputs.

```bash theme={"theme":"css-variables"}
curl 'https://api.recursion.labelbox.com/managed-agents/v1/files?source=upload&limit=50' \
  -H "Authorization: Bearer $RECURSION_API_KEY"
```

`getFile` returns one file's metadata: `filename`, `media_type`, `byte_size`, `sha256`, `source`, `expires_at`, and, for a session output, `scope_session_id` and the agent it belongs to.

```bash theme={"theme":"css-variables"}
curl 'https://api.recursion.labelbox.com/managed-agents/v1/files/9f8b1c2d-3e4f-5a6b-7c8d-9e0f1a2b3c4d' \
  -H "Authorization: Bearer $RECURSION_API_KEY"
```

<Accordion title="List parameters and expired files">
  | Parameter | Description |
  | - | - |
  | `source` | `upload` or `session_output`. |
  | `scope_session_id` | Only the files this session produced. |
  | `file_ids` | Up to 100 ids, comma-separated or repeated, answered as one page. Can't be combined with `limit` or `page_token`. |
  | `limit`, `page_token` | Page size, 1 to 1,000 with a default of 200, and the cursor from the previous page. Send the same filters with the token. |

  An expired file stays listed, with `expires_at` in the past, until it's removed after a grace period, but it can no longer be attached or downloaded.
</Accordion>

## How files reach a session

```mermaid theme={"theme":"css-variables"}
flowchart TD
  upload["Upload a file"] --> library["Organization files"]
  library -->|"attach"| inputs[".managed-agents/files<br>read-only"]
  library -->|"name in a message"| message["Image or document<br>in the message"]
  agent["Agent saves a deliverable"] --> outputs[".managed-agents/outputs"]
  outputs -->|"kept after each turn"| library
```

Uploads reach a session by attachment or by name in a message. Deliverables the agent saves return to your organization's files after each turn.

## What can go wrong

The most common problems:

* **`404 not_found` when a session request names a `file_id`.** The file doesn't exist in your organization or has expired. Use a live file id from `listFiles`.
* **The agent can't find an attached file.** It was attached mid-turn. It's in place on the next tool call and visible from the next turn. Check `mount_path` on the resource.
* **A deliverable isn't in the outputs.** The agent saved it somewhere else, or it broke a capture rule. See [Deliverables and artifacts](/recursion/artifacts#what-can-go-wrong).

<Accordion title="Every files error">
  | Code or symptom | Cause | Fix |
  | - | - | - |
  | The console refuses a file | It is over 1 GiB. | Split it into smaller files. |
  | `413 payload_too_large` from `uploadFile` | The file part is over 30 MiB. | Upload a file of at most 30 MiB through the API, or use the console for a larger file. |
  | `413 file_quota_exceeded` on upload | The upload would exceed your organization's live-file storage quota. | Set expiries on API uploads so they stop counting once they expire. |
  | `404 not_found` when a session request names a `file_id` | The file doesn't exist in your organization or has expired. | Use a live file id from `listFiles`. |
  | A file can't be attached mid-session | Its path collides with or nests inside another attachment, or the session isn't the root of its tree. | Choose another path, or attach to the root session. |
  | `400 invalid_request` on a message's `source.file_id` | The file isn't an image on an image block, or is binary or too large on a document block. | Send PNG, JPEG, or WebP images, and text documents up to 512 KiB. |
  | The agent can't find an attached file | It was attached mid-turn. | It's in place on the next tool call and visible from the next turn. Check `mount_path` on the resource. |
  | A deliverable isn't in the outputs | The agent saved it somewhere else, or it broke a capture rule. | See [Deliverables and artifacts](/recursion/artifacts#what-can-go-wrong). |

  For every error code, see [Errors](/recursion/errors).
</Accordion>

Limits on uploads, storage, expiry, attachments, list pages, files named in a message, and deliverables are in [Limits](/recursion/limits).

## Next steps

<CardGroup>
  <Card title="Deliverables and artifacts" href="/recursion/artifacts">
    Ask for named deliverables and see how they're kept.
  </Card>

  <Card title="Sessions" href="/recursion/sessions">
    Start sessions, send messages, and attach images.
  </Card>

  <Card title="Automations" href="/recursion/automations">
    Mount the same files into every automated run.
  </Card>
</CardGroup>


# GitHub access
Source: https://docs.labelbox.com/recursion/github

Choose GitHub MCP tools and repositories for each Managed Agent.

Standalone Managed Agents use GitHub MCP tools. An agent's sandbox does not receive authenticated `git` or `gh` credentials. Each session uses a short-lived GitHub App installation token for its MCP requests.

<Note>
  Connecting GitHub and changing an agent's access need the organization developer or admin role. See [Organizations and roles](/recursion/organizations-and-roles). You also need a GitHub account that can install the GitHub App on the target organization or user account. Some organizations require an owner to approve new apps.
</Note>

## Connect GitHub

1. In **Integrations**, click **Add integration**, then choose **GitHub**. Authorize the GitHub user so Recursion can find the App installations that user can access.
2. Choose an existing installation, or choose **Install on another account** and select the account and repositories in GitHub. If none are available, Recursion starts installation automatically. For an existing installation, use **Manage repositories on GitHub** to change its repository access in a separate tab. Return to Recursion and click **Continue** to connect with the updated access. Choosing an already installed account on GitHub's App installation page opens its settings without returning to Recursion.
3. Back in **Integrations**, select the tools available to the organization and save. The picker describes each tool and groups the current GitHub tools as **Read-only** or **Changes data**. Merging a pull request changes data; the **Destructive** category is reserved for tools that delete data, and none of the current GitHub tools do so. Use the group checkboxes or **Select read-only**, **Select all**, and **Clear** to choose several at once. You can change this selection later with **Edit tools** on the connection.

The App's installed permissions are fixed by its configuration. Choosing tools in Recursion narrows which MCP calls agents can make and which permissions their short-lived tokens request. It does not change the App installation's permissions in GitHub. New connections start with the supported read tools; connections created before tool selection have all supported tools enabled.

The tool list comes from the production App's permissions. The connection's tool selection is a ceiling for every agent that uses it. Workflow-file editing is unavailable.

To change repository access after connecting, open the installation's settings in GitHub, save the new selection, then use **Reconnect** on the GitHub connection in Recursion. Reconnecting refreshes the repositories available through that connection while preserving agent grants.

## Assign GitHub to an agent

1. In the agent's **Integrations** section, click **Add integration access** and select the GitHub connection.
2. In **Tools**, review each tool's description and read-only or write classification. Use **Select read-only**, **Select all**, **Clear**, or the group checkboxes to choose a subset of the connection's tools.
3. In **Repositories**, choose specific repositories or **All authorized repositories**.
4. Click **Use this access**, then save the agent version.

The agent's GitHub token requests the permission categories required by its selected tools. There is no separate read/write token setting in standalone Managed Agents. Existing agent grants without a tool list retain their former read or write tool scope, subject to the connection's current tool ceiling. Repository selections can only narrow the repositories installed in GitHub.

If an administrator removes a tool from the connection, the agent's saved selection still shows it as unavailable. Edit the agent's GitHub access and uncheck the unavailable tool. To keep the access, select at least one available tool; otherwise, use **Remove access**. New sessions can use only tools still enabled for the connection.

When using the API, set `nativeIntegrations[].tools` to the selected tool names. The `permission` field remains in the version contract for compatibility; the server derives its `read` or `write` value from the tools. You can discover supported tools through `listIntegrationProviders` and the connection's enabled tools through `listIntegrationConnections`.

A new agent version or connection selection applies to new sessions. A new agent version also applies to a running session you [move to it](/recursion/sessions#move-a-session-to-another-version). Running sessions may keep credentials and tool lists they already received until they refresh or end.

Name repositories as `owner/name`, up to 500, where `owner` is the connection's own account. A repository of another owner is refused; grant it through the connection for that owner. A new agent version replaces the whole definition, so include every field you want to keep.

## Verify access

Open the connection's actions menu and click **Test connection**. Then start a new agent session and ask it to use one of the selected GitHub tools. A passing connection test proves GitHub issued a token; a successful tool call also proves that the tool, repository, and permission were available.

GitHub MCP calls run outside the sandbox, so GitHub host allowlists for sandbox `git` and `gh` are unnecessary for these calls.

## More information

See [Integrations](/recursion/integrations) for connection states and [Tools](/recursion/tools) for MCP tool use.


# Glossary
Source: https://docs.labelbox.com/recursion/glossary

Short definitions of every term used in the Managed Agents docs, with a link to where each is explained.

Terms are in alphabetical order. Field names appear in code format.

| Term | Definition |
| - | - |
| **Advisor** | A roster entry that gives a second opinion without doing the work. A roster can have at most one. See [Multi-agent](/recursion/multi-agent). |
| **Agent** | A reusable definition of a model, system prompt, tools, skills, MCP servers, granted apps, and default credential grants. See [Agent setup](/recursion/agents). |
| **Agent version** | An immutable snapshot of an agent. Every update publishes a new one. A session runs the version it started with until you [move it to another](/recursion/sessions#move-a-session-to-another-version). |
| **API key** | A personal `rma_` secret that authenticates API requests as you. Scoped to a tenant or one organization. See [API keys](/recursion/api-keys). |
| **Artifact** | An output the agent produces, such as a saved file or an image, recorded in the event log. See [Artifacts](/recursion/artifacts). |
| **Automation** | A pinned agent version, environment, prompt, and run defaults that starts a session when a trigger fires or you click **Run now**. See [Automations](/recursion/automations). |
| **Automation run** | The record of one automation start: its trigger, the frozen inputs, the opening message, and the session it created. |
| **Built-in integration** | An integration with one of the 164+ apps in the catalog, such as Salesforce or Zendesk. An admin connects it once and chooses its tools. See [Integrations](/recursion/integrations). |
| **Child session** | A session a parent session starts to delegate work. It shares its root session's sandbox. Also called a subagent. |
| **Console** | The web interface for Managed Agents. See [Build in the console](/recursion/console). |
| **Credential** | One secret inside a vault: an MCP bearer token, an environment-variable secret, or a native integration grant. See [Vaults](/recursion/vaults). |
| **`credential_refs`** | A list that narrows a session to specific credentials inside its granted vaults. |
| **Default organization** | The organization every tenant has, named **Default**. Every member can reach it, and the console works in it. |
| **Deliverable** | A file the agent saves as the result of its work. Deliverables are collected at the end of every turn. See [Artifacts](/recursion/artifacts). |
| **Environment** | The configuration a session's sandbox runs with: runtime, compute, workspace disk, setup script, internet access, and idle policy. See [Environments](/recursion/environments). |
| **Event** | One recorded step in a session, such as a message, tool call, or tool result. See [Events](/recursion/events). |
| **Event source** | Where automation triggers get their events. Connecting Slack or GitHub on **Integrations** creates one; a custom source receives webhook deliveries. Each source verifies what it receives and can be paused or resumed. See [Receive events](/recursion/automations#receive-events). |
| **Execution state** | `execution_state`: what the agent loop is doing now, such as `running` or `idle`. See [Event types and states](/recursion/reference). |
| **File** | An upload you can attach to sessions, or a deliverable a session saved. A session's **Files** tab shows both. See [Files](/recursion/files). |
| **Hard stop** | Cancelling a session: its work stops and its sandbox is torn down. See [Session operations](/recursion/session-operations). |
| **Idempotency key** | A value sent in `Idempotency-Key` so a retried create can't happen twice. See [API conventions](/recursion/api#idempotent-mutations). |
| **Integration** | An organization's connection to an outside app or service, which you grant to agents. See [Integrations](/recursion/integrations). |
| **Interrupt** | Stopping the current turn without ending the session. The session stays resumable. |
| **MCP server** | A server that offers tools over the Model Context Protocol. Agents can use MCP servers you configure. See [MCP servers](/recursion/mcp-servers). |
| **Native integration** | An integration connected by signing in to the service directly, such as GitHub, Slack, or Jira, granted per agent or through a vault. See [Native integrations](/recursion/integrations#native-integrations). |
| **Network policy** | The rules for what a sandbox can reach on the internet. New environments have no internet access unless you allow it. See [Environment reference](/recursion/environments-reference). |
| **Organization** | A separate space in a tenant. Every resource belongs to exactly one organization. See [Organizations and roles](/recursion/organizations-and-roles). |
| **Primary owner** | The person who owns a tenant. There's exactly one. They can't be removed or demoted, and they can transfer ownership to another member. |
| **Referenced session** | An earlier session a new session may read. See [Referenced sessions](/recursion/referenced-sessions). |
| **Role** | What a member may do. Each member has a tenant role, which gives them roles in organizations. See [Organizations and roles](/recursion/organizations-and-roles). |
| **Root session** | The session you started, at the top of a session tree. |
| **Roster** | The list of agents a coordinating agent may delegate to. Up to 20 entries. |
| **Sandbox** | The isolated workspace where a session tree runs commands and tools. |
| **Session** | One run of an agent version in an environment. See [Sessions](/recursion/sessions). |
| **Session analyst** | A conversation that answers your questions about a session. Open it with **Ask** in the console. |
| **Session cost** | What your balance is charged for a whole session tree: model calls, tools, and sandbox compute. Read it as `costUsd` on the session. See [Usage and cost](/recursion/usage-and-cost). |
| **Session tree** | A root session and all the child sessions under it. They share one sandbox, and cost rolls up to the root. |
| **Setup run** | A test of an environment's setup script on real compute. Sessions can't use a script until a run passes. See [Environments](/recursion/environments). |
| **Setup script** | A shell script that prepares an environment's software before the agent starts. |
| **Skill** | A reusable procedure, in the Agent Skills format, that an agent loads when it needs it. See [Skills](/recursion/skills). |
| **Skill group** | A named set of skills you attach together. |
| **Stop reason** | `stop_reason`: why the agent loop isn't running, such as `end_turn` or `sleeping`. See [Event types and states](/recursion/reference). |
| **Session resource** | A file attached to a session, read-only in its sandbox under `/workspace/.managed-agents/files`. See [Files](/recursion/files). |
| **Tag** | A label on an agent, used to filter agents and sessions. |
| **Task board** | A team's shared list of tasks, owners, notes, and decisions. See [Teams](/recursion/teams). |
| **Team** | Copies of an agent that claim tasks from a shared board and work in parallel. See [Teams](/recursion/teams). |
| **Tenant** | The top level of an account. It holds members and organizations. |
| **Thread** | One agent's line of work in a multi-agent session. |
| **Trigger** | What starts an automation run: a schedule, a Slack message, a GitHub event, or a delivery to a custom webhook source. |
| **Tool** | An action the agent can take, such as running a command, searching the web, or calling an MCP server. See [Tools](/recursion/tools). |
| **Tool allow-list** | The tools every agent granted a built-in integration may call in that app. Set on the **Integrations** page. See [Choose which tools agents may use](/recursion/integrations#choose-which-tools-agents-may-use). |
| **Turn** | One stretch of agent work, from a message until the agent stops. Deliverables are collected at the end of each turn. |
| **Vault** | A named group of credentials that agents and sessions receive by grant. See [Vaults](/recursion/vaults). |
| **Workspace** | The agent's working directory in the sandbox, `/workspace`. |


# How it works
Source: https://docs.labelbox.com/recursion/how-it-works

The concepts behind Managed Agents and how a session moves from start to finished work, including the lifecycle and how credentials reach the agent.

Managed Agents has a small set of building blocks. Once you know what each one is for and what it guarantees, every other page in these docs reads as a variation on the same flow.

## The building blocks

| Concept | What it is | Key guarantee |
| - | - | - |
| **Agent** | A reusable definition: model, system prompt, tools, skills, MCP servers, granted apps, and default credential grants. | Editing an agent never changes a session that already started. |
| **Agent version** | One immutable snapshot of an agent. Every update publishes a new version. | A session runs one version at a time, and changes only when you [move it](/recursion/sessions#move-a-session-to-another-version). |
| **Environment** | Where sessions run: the runtime, compute size, workspace disk, setup script, internet access, and idle policy. | A setup script must pass a verification run before sessions can use it. |
| **Session** | One run of an agent version in an environment, working on a task. Child sessions share their root's sandbox. | Every step is recorded, and a stopped session can be continued with a follow-up message. |
| **Event** | One recorded step: a message, tool call, tool result, or status change. | Events are append-only and can be listed, streamed, and resumed from any point. |
| **Vault** | A named group of credentials: MCP server tokens and environment-variable secrets. | Secret values are write-only and never returned by the API. |
| **Integration** | Your organization's connection to an app, such as Zendesk or Snowflake, with the list of tools agents may use in it. | An agent reaches only the apps it's granted, and only the tools on each app's list. |
| **Skill** | A reusable procedure in the Agent Skills format that an agent loads when it needs it. | Skills are fixed in the session at start, and again when the session moves to another agent version. |
| **Deliverable** | A file the agent saves as a result of its work. | Deliverables are collected at the end of every turn, and you can preview and download them. |
| **File** | An upload you attach read-only to sessions. | A session keeps the content it attached, even if the file changes later. |
| **Automation** | A pinned agent version, environment, prompt, and run defaults, started by a schedule, a Slack or GitHub event, or by hand. | Every run records exactly what it was given and the session it started. |

See the [Glossary](/recursion/glossary) for every other term.

## The session lifecycle

A session has two fields that describe where it is. `status` is the coarse state, and `execution_state` says what the agent loop is doing right now. [Sessions](/recursion/sessions#how-a-session-runs) lists every state and what moves a session between them.

What each state tells you:

* **queued**: The session is waiting to start, because the agent already has as many root sessions running as its `max_concurrent_sessions` allows or because no sandbox capacity is free yet. Child sessions never count toward the cap.
* **provisioning**: The sandbox is being prepared. This normally takes under a minute. If compute isn't ready within five minutes, the session fails with `sandbox_provision_timeout`.
* **running**: The agent or one of its tools is working.
* **idle**: Nothing is running. `stop_reason` says why: `end_turn`, `sleeping`, `awaiting_subagents`, `interrupted`, `requires_action`, `out_of_credit`, `error`, and others. When the agent finishes its work, `stop_reason` is `end_turn` and `status` becomes `completed`. An idle session keeps its sandbox until the environment's idle policy stops it.
* **failed** and **cancelled**: The session stopped on an error or a hard stop. `failure` explains a failure.

A follow-up message starts a fresh turn on a stopped root session. Completed and failed sessions reuse their sandbox when it still exists. A hard stop tears the sandbox down, so a follow-up after one gets a new sandbox with a clean workspace.

The full list of values is in [Event types and states](/recursion/reference).

## What happens when you start a session

1. You call `startSession`, or click **Launch session** in the console, with an agent, an environment, and a task. The request returns `202 Accepted` right away with a `session_id`.
2. The session freezes the current agent version, the environment, and the credentials you granted. The version, and the credentials its defaults grant, change only if you [move the session to another version](/recursion/sessions#move-a-session-to-another-version).
3. The sandbox is provisioned and prepared with the environment's verified setup, if it has a setup script. Session credentials are withheld from the setup script.
4. The agent works through model turns and tool calls. Each step is recorded as an event you can read or stream.
5. At the end of every turn, the agent's deliverables are collected.
6. You read the transcript, the deliverables, and the cost from the same session record.

## How credentials reach the agent

Credentials never go in a prompt. You store them in a vault, or connect an app on the **Integrations** page, grant them to an agent or a session, and each type is delivered in its own way.

| Credential | Where it goes | Can the agent read the value? |
| - | - | - |
| MCP bearer token | Added to requests to the matching MCP server, outside the sandbox. | No. The model and the sandbox never receive it. |
| Environment-variable secret | Set as an environment variable in the sandbox, under the name you choose. | Yes. Any command the agent runs can read and print it. |
| Built-in integration access | Available to commands in the sandbox, renewed while the session runs. | Yes, but it works only for the granted apps and the tools on each app's list, and it expires within an hour. |

Grant only what the task needs. A session can narrow its credentials to a list of specific items with `credential_refs`. Prefer an MCP credential whenever the service speaks MCP, because its value never enters the sandbox. See [Vaults](/recursion/vaults) and [Security](/recursion/security).

## Where the work happens

Each session tree gets its own sandbox. The agent's working directory is `/workspace`. Agents already know where to save deliverables, so you only need to name the file you want in the task. Subagents and teammates share their root session's sandbox and hand work over by file path. See [Artifacts](/recursion/artifacts) and [Multi-agent](/recursion/multi-agent).

## Next steps

<CardGroup>
  <Card title="Quickstart" href="/recursion/quickstart">
    Run the whole flow once: key, environment, agent, session.
  </Card>

  <Card title="Agent setup" href="/recursion/agents">
    Choose the model, prompt, tools, skills, and credentials.
  </Card>

  <Card title="Sessions" href="/recursion/sessions">
    Look up every start field and how to follow a session.
  </Card>

  <Card title="Security" href="/recursion/security">
    See how sandboxes, network policy, and credentials are isolated.
  </Card>
</CardGroup>


# Managed Agents
Source: https://docs.labelbox.com/recursion/index

Run agents in managed sandboxes with tools and credentials, follow every step, and collect the files they produce.

Recursion Managed Agents runs agents that need a sandbox, tools, and more than one model turn. Each session keeps a durable record that you can inspect, continue, or audit.

<Warning>
  Recursion Managed Agents is designed around persistent, stateful sessions. Agent sessions may remain active over extended periods, resume after interruptions, and retain conversation history, sandbox state, and generated outputs on the server.

  Because Managed Agents relies on this server-side state, it is currently not supported for customers requiring Zero Data Retention (ZDR) or HIPAA Business Associate Agreement (BAA) coverage. Customers with an active BAA or ZDR arrangement, or with a use case that requires either, cannot use Recursion Managed Agents at this time.
</Warning>

## What do you want to do?

<CardGroup>
  <Card title="Run your first session" icon="play" href="/recursion/quickstart">
    Create an API key, an environment, and an agent, then start a session and read the reply. About 10 minutes.
  </Card>

  <Card title="Understand the model" icon="diagram-project" href="/recursion/how-it-works">
    See how agents, environments, sessions, vaults, and integrations fit together.
  </Card>

  <Card title="Build in the console" icon="display" href="/recursion/console">
    Find the screen for each setup, launch, and inspection task.
  </Card>

  <Card title="Call the API" icon="code" href="/recursion/api">
    Authenticate REST requests and learn the retry and paging rules.
  </Card>

  <Card title="Connect your apps" icon="plug" href="/recursion/integrations">
    Let agents work in 164+ apps, choose their tools, or add any MCP server.
  </Card>

  <Card title="Split work across agents" icon="users" href="/recursion/multi-agent">
    Delegate to subagents or let a team share a task board.
  </Card>

  <Card title="Automate sessions" icon="clock" href="/recursion/automations">
    Run agents on a schedule, by hand, or when Slack or GitHub sends an event.
  </Card>

  <Card title="Connect a coding agent" icon="terminal" href="/recursion/ai-coding-agents">
    Let your AI coding agent read these docs and call the API for you.
  </Card>

  <Card title="Fix a problem" icon="wrench" href="/recursion/troubleshooting">
    Match an error code or symptom to its cause and fix.
  </Card>
</CardGroup>

## What you can do

| Capability | What it gives you |
| - | - |
| Versioned agents | A model, system prompt, tools, skills, MCP servers, and credential grants, saved as immutable versions. See [Agent setup](/recursion/agents). |
| Managed sandboxes | A workspace per session tree, with a setup script you verify once, sized compute, and an egress policy. See [Environments](/recursion/environments). |
| Durable sessions | Every message, tool call, and result is recorded. Stream it live, resume after it ends, or send a follow-up. See [Sessions](/recursion/sessions). |
| Work in your apps | Connect GitHub, Slack, Jira, Confluence, and 164+ catalog apps such as Salesforce, Zendesk, HubSpot, and Snowflake, choose which tools agents may use, and grant them to agents. See [Integrations](/recursion/integrations) and [GitHub access](/recursion/github). |
| Files in and out | Upload inputs once and attach them to any session, show images and documents in a message, and download every deliverable. See [Files](/recursion/files). |
| Automations | Start sessions on a schedule, from Slack or GitHub events, or by hand, with the same pinned agent version and credentials every run. See [Automations](/recursion/automations). |
| Credentials without pasting secrets | Vaults hold tokens and keys. MCP tokens are attached outside the sandbox. See [Vaults](/recursion/vaults). |
| Multi-agent work | Delegate to subagents, consult an advisor, or run a team with a shared board, all in one sandbox. See [Teams](/recursion/teams). |
| Deliverables | Agents save named files that are kept at the end of each turn, ready to preview and download. See [Artifacts](/recursion/artifacts). |
| Usage and cost | One cost per session, covering its helper agents, tools, and compute. See [Usage and cost](/recursion/usage-and-cost). |
| API keys and roles | Personal API keys scoped to your tenant or one organization, with role-based access checked on every request. See [API keys](/recursion/api-keys). |

## How it fits together

```mermaid theme={"theme":"css-variables"}
flowchart TD
  agent["Agent version"] --> session["Session: your task"]
  environment["Environment"] --> session
  grants["Vaults and<br>integrations"] --> session
  session --> sandbox["Sandbox"]
  session --> events["Event log"]
  session --> deliverables["Deliverables"]
```

You start a session with an agent and an environment. The session freezes both, so later edits never change work in progress unless you [move the session to another agent version](/recursion/sessions#move-a-session-to-another-version). The agent works in its sandbox, every step lands in the event log, and the files it saves are kept as deliverables. [How it works](/recursion/how-it-works) explains each piece.

## When to use Managed Agents

Use Managed Agents when the work needs one or more of these:

* Several model turns and tool calls.
* A shell, files, package installation, or a browser in a sandbox.
* Work in other systems, such as Salesforce, Jira, or Snowflake, an API with a token, or an MCP server.
* A durable record you can review, audit, or continue later.

For a single model response with no tools or sandbox, call the model directly instead.

## Next steps

<CardGroup>
  <Card title="Quickstart" href="/recursion/quickstart">
    Run your first session in the console or with cURL.
  </Card>

  <Card title="How it works" href="/recursion/how-it-works">
    Learn the concepts and see the session and credential flows.
  </Card>

  <Card title="API conventions" href="/recursion/api">
    Set up a REST client and learn the shared request rules.
  </Card>

  <Card title="Glossary" href="/recursion/glossary">
    Look up any term used in these docs.
  </Card>
</CardGroup>


# Integrations
Source: https://docs.labelbox.com/recursion/integrations

Connect your organization to 164+ apps, choose which tools agents may use in each one, and grant apps to agents so sessions can work in them.

Integrations let agents work in the apps your team already uses. An admin connects an app once for the whole organization and chooses which of its tools agents may use. Then you grant the app to any agent, and its sessions can read and act in that app within those limits.

There are three ways to give an agent access to another service:

| Kind | What it covers | How access is controlled |
| - | - | - |
| **Built-in integrations** | 164+ apps, from Salesforce and Zendesk to Snowflake and Jira, picked from a catalog. | An admin chooses each app's tools once. Agents get only those tools. |
| **Native integrations** | GitHub, Slack, Jira, Confluence, Loom, and LaunchDarkly, connected by signing in to them directly. | An access level for most services; for GitHub, organization tools narrowed by agent tools and repositories. |
| **MCP servers** | Any remote server that speaks the Model Context Protocol. | A token you store in a vault. See [Connect MCP servers](/recursion/mcp-servers). |

Most of this page covers built-in integrations. [Native integrations](#native-integrations) and [MCP servers](#mcp-servers) are at the end.

```mermaid theme={"theme":"css-variables"}
flowchart TD
  add["Admin adds the app and picks its tools"] --> active["Admin signs in: connection active"]
  active --> grant["App granted on the agent"]
  grant --> session["Session calls the allowed tools"]
  session --> transcript["Every call and result in the transcript"]
```

<Note>
  Organization users, developers, and admins can view the **Integrations** page, the catalog, and connections. Adding an app, signing in to it, reconnecting or checking it, changing its tools, disconnecting it, and granting apps to an agent need the organization developer or admin role. Console messages that say "ask an admin" mean anyone with the developer or admin role. The billing role has no access to integrations. See [Organizations and roles](/recursion/organizations-and-roles). You also need an account in the app you're connecting, with the access you want agents to have. Agents act in the app as that account, so use one whose permissions fit the work.
</Note>

## The catalog

The built-in catalog has 164 apps, grouped by name below. The console always shows the current catalog, which can grow. Search it from **Add integration** on the **Integrations** page.

<AccordionGroup>
  <Accordion title="A–C (34 apps)">
    ActiveCampaign, Adobe PDF Services, Ahrefs, Airtable, Amadeus, Amazon S3, Amplitude, Anaplan, Apollo, Arize, Articulate Reach 360, Asana, Ashby, Attio, Avalara, Aviationstack, Axiom, BambooHR, Basecamp, bioRxiv, Bitly, Box, Brex, Calendly, Clay, ClickUp, ClinicalTrials.gov, Cloudflare, CMS Coverage, Coda, ComplianceQuest, Confluence, Contentful, Crustdata
  </Accordion>

  <Accordion title="D–G (29 apps)">
    Databricks, Datadog, Deel, Discord, DoorDash, Dropbox, Duffel, Dynamics 365 Sales, Exa, Expensify, Figma, Firecrawl, Fireflies, FireHydrant, FreshBooks, Freshdesk, Freshservice, Front, Gamma, GitLab, Goldcast, Gong, Google BigQuery, Google Maps, Google Tasks, Grafana, Granola, Greenhouse, Guru
  </Accordion>

  <Accordion title="H–M (22 apps)">
    Hex, HiBob, HubSpot, Intercom, Jenkins, Jira, Jira Data Center & Server, Jira Service Management, Kintone, Klaviyo, Linear, Looker, Lucidchart, Luma, Make, Metabase, Microsoft Excel, Microsoft Planner, Microsoft Teams, Microsoft Teams GCC High, Miro, Monday.com
  </Accordion>

  <Accordion title="N–R (30 apps)">
    n8n, Nextech Practice+, Nooks, Notion, NPI Registry, Okta, OneDrive, OneDrive GCC High, OneNote, Oracle HCM, Oracle Sales Cloud, Oracle SCM, Ordinal, Oura, Outlook, PagerDuty, PayPal, Peec AI, Pendo, Pipedrive, Plaud, PostHog, PubMed, Pylon, Quartr, Ramp, ReadMe, Render, Reo.dev, Rootly
  </Accordion>

  <Accordion title="S–Z (49 apps)">
    Sabre, Salesforce, Sanity, SAP SuccessFactors, SendGrid, Sentry, ServiceNow, SharePoint, SharePoint GCC High, Shopify, Sigma Computing, SmartAdvocate, SmartRecruiters, Smartsheet, Snowflake, Sprout Social, Square, Statsig, Straker, Stripe, Supabase, TeamCity, Teamwork.com, TinyFish, Trello, TripAdvisor, UKG Pro, UKG Pro WFM, Unify, Unily, Vercel, Vestaboard, VisualPing, Warp, Weather, Webflow, WHOOP, Wikipedia, Wix, WordPress, Workday, X, Yelp, YouTrack, Zendesk, Zendesk Sell, Zoho CRM, Zoho Desk, Zoom
  </Accordion>
</AccordionGroup>

When your organization can connect Jira or Confluence as a [native integration](#native-integrations), **Add integration** offers the native connection for that service instead of the built-in one. An app that cannot be configured is listed for reference and isn't offered under **Add integration**. The console shows whether its sign-in uses OAuth, a token or key, or either.

## Connect an app

Add the app with the tools agents may use, then sign in to it. Connecting is done in the console. The API can read, check, change, and disconnect connections, but it can't add one.

<Steps>
  <Step title="Open Add integration">
    In the sidebar, click **Integrations**. Under **Integration connections**, click **Add integration**.
  </Step>

  <Step title="Pick the app">
    Search for the app and click it. An app you already added shows **Already added**. Each app can be added once per organization; to change it later, use its row.
  </Step>

  <Step title="Choose its tools">
    The **Configure** drawer lists every tool the app offers. Each one is marked **Read-only.**, **Changes service data.**, or **Destructive: updates or deletes service data.** Tick the tools agents may use, or use **Select read-only**, **Select all**, or **Clear**. See [Choose which tools agents may use](#choose-which-tools-agents-may-use).
  </Step>

  <Step title="Review and create">
    Click **Review selection**. The review shows how many tools you picked and how many of them can change or delete data in the app. Click **Create 1 connection**.
  </Step>

  <Step title="Sign in to the app">
    The app appears in the table with the state **Pending setup** and the account **Not connected yet**. Click **Connect** on its row. On the app's sign-in page, sign in with the account agents should act as, and approve access.
  </Step>

  <Step title="Confirm the connection">
    You come back to **Integrations**, which shows **Finishing connection…** and then **Connected. Assign access in the agent's Integrations section.**
  </Step>
</Steps>

The app's row now shows the state **Active**, the **Account** column shows the app's name, and the tool count appears beside it, for example "6 tools, 2 can change data". Only an active connection can be granted to an agent, so next, [grant the app to an agent](#grant-apps-to-an-agent).

Each sign-in started from **Connect** is single-use and short-lived. If you close the sign-in page or take too long, nothing is connected and the row stays **Pending setup**; click **Connect** again.

## Choose which tools agents may use

Each connection carries a **tool allow-list**: the tools the organization permits agents to use, chosen on the **Integrations** page. An agent can't call a tool that isn't on the list. In standalone Managed Agents, you can narrow this list when assigning the connection to an agent.

* **Start with read-only.** **Select read-only** is the safest starting point. Add write tools only for tasks that need them.
* **The organization list is the ceiling.** Agents inherit all enabled tools unless you select a smaller set for an assignment. An agent's selection never adds tools outside the organization list.
* **The list can't be empty.** A connection always names at least one tool, and at most 200.
* **Changes reach agents on their own.** New sessions always use the new list. A session that's already running may pick it up when its access is next renewed, so don't count on a running session keeping the old list.

<Accordion title="How tools are classified">
  | Console label | Classification | What the tool can do |
  | - | - | - |
  | **Read-only.** | `read_only: true`, `destructive: false` | Read data in the app. |
  | **Changes service data.** | `read_only: false`, `destructive: false` | Create or change data in the app. |
  | **Destructive: updates or deletes service data.** | `destructive: true` | Update or delete data in the app. |

  A tool marked both read-only and destructive is treated as destructive.
</Accordion>

### Change an app's tools

1. In the sidebar, click **Integrations**.
2. Open the app's row menu and click **Tools**.
3. Tick or clear tools. The list applies to every agent using the app.
4. Click **Save tools**. The console confirms that the app's tools were updated for every agent using it.

Changing tools doesn't change the connection's state, and it isn't allowed on a revoked connection; reconnect it first.

<Accordion title="When the catalog changes">
  The catalog can reclassify or remove a tool after you pick it. A tool you saved as read-only that's no longer classified read-only, or a tool that's no longer in the catalog, blocks the whole app: sessions can't use it until the tool is removed from the list.

  The **Tools** drawer shows these tools with a warning and a **Remove** button. Click it, then **Save tools**.
</Accordion>

## Grant apps to an agent

A connection does nothing until an agent is granted it.

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Agents**, then open the agent.
    2. On **Configuration**, find the **Integrations** section.
    3. Click **Add integration access**, search for the app, and click **Select**. A connection that isn't active shows **Reconnect first** instead.
    4. Repeat for each app, and click **Remove** to take one away.
    5. Click **Save new version**.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/agents' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      -d '{
        "name": "Support triage",
        "model": "<model-id>",
        "system": "You triage new support tickets. Read each ticket, find the customer account, and add an internal comment with a suggested priority.",
        "built_in_integrations": [{ "connection_id": "3c9e1f72-5a4b-4d08-9e6c-1b7f2a8d0e45" }]
      }'
    ```
  </Tab>
</Tabs>

A `200` response is the agent. It lists the granted connections. Some fields are left out here.

```json theme={"theme":"css-variables"}
{
  "agent_id": "8d2f4b61-0c3e-4a97-b5d8-7e1a9c6f3b20",
  "latest_agent_version_id": "f4a7c2e9-1b3d-4e68-9a05-6c8e2d7b1f39",
  "name": "Support triage",
  "built_in_integrations": [
    { "connection_id": "3c9e1f72-5a4b-4d08-9e6c-1b7f2a8d0e45" }
  ]
}
```

Grants live on the agent version in `built_in_integrations`, so every change publishes a new version. Running sessions keep their version until you [move them](/recursion/sessions#move-a-session-to-another-version), and [automations](/recursion/automations#keep-the-agent-version-current) keep their pinned version. A grant you remove still applies to a running session until you move it, and to an automation's runs until you move the automation to the new version.

<Accordion title="Rules for built_in_integrations">
  * Each entry names a connection by `connection_id`. Get ids from `listIntegrationConnections`, or from the row on the **Integrations** page.
  * Every connection must be a built-in integration in your organization, be **Active**, and have at least one tool on its allow-list.
  * An agent can name at most 25 entries, counted before duplicates are removed. A repeated id is kept once, and the order you send is kept.
  * Leave out `tools` to inherit the connection's current allow-list. To narrow access, supply 1 to 200 tool names from that list. New sessions intersect the saved selection with the current organization list, so removing a tool at the organization level removes it from agents too.
  * The list is part of the definition, so send it on every `createAgentVersion`. Leaving it out publishes a version with no apps. See [Update rules](/recursion/agents#update-rules).
  * Built-in integrations aren't vault grants. They never appear under an agent's credential access, and `default_vault_ids` and `credential_refs` don't affect them.
</Accordion>

### What a session receives

A session takes the list of apps from its agent version when it starts, and keeps it until you [move the session to another version](/recursion/sessions#move-a-session-to-another-version). For each app, the session gets:

* **Access to the allowed tools only.** The agent can call the tools on the connection's allow-list and nothing else in that app.
* **Short-lived, scoped access.** Access is limited to the apps the agent was granted, expires within an hour, and renews on its own while the session runs. You never handle a token. The access is available to commands in the sandbox, so treat it like anything else the agent can read: it works only for the granted apps and allowed tools, and only until it expires.
* **A skill for using apps.** Sessions of an agent with built-in integrations also load a skill that shows the agent how to find and call app tools. It takes one of the session's 100 skill slots, so if the agent's own skills already fill all 100, the last one is left out.

An app whose connection is no longer active when the session starts is left out of that session. The session still starts.

<Note>
  Built-in apps need a way out of the sandbox. The agent calls a built-in app's tools with commands inside the sandbox. Recursion allows the hosts a built-in app needs, so its tools work when **Internet access** is **Enabled** or limited to specific hosts. With **No access**, calls to the app fail with a network error. MCP servers, native integrations' MCP tools, `web_search`, and `web_fetch` run outside the sandbox and don't need that. See [Control network access](/recursion/environments#control-network-access).
</Note>

## Use apps in a session

Agents already know how to find the right tool in each granted app, check the inputs it needs, and call it. You don't list tools in the prompt. Describe the outcome you want, and name the app when it matters:

```text theme={"theme":"css-variables"}
Read the Zendesk tickets opened since yesterday. For each one, find the customer's
account in Salesforce and add an internal comment on the ticket with a suggested
priority. Don't change ticket status.
```

Each call to an app appears in the transcript as a command the agent runs, with the app's result below it. The **Tools** tab counts every call. A call to an app you didn't grant, or to a tool that isn't on the allow-list, fails. If an app's sign-in stops working, calls to that app fail and the connection becomes **Revoked** on the **Integrations** page until you reconnect it.

<Warning>
  A write or destructive tool on the allow-list can change real data in the app. Say in the task what the agent may and may not change, and keep write tools off the list when the work is read-only.
</Warning>

## Test a connection

A check tries a fresh sign-in with the app, the same way a session does, and reports whether it worked. Use it after connecting, or when an agent can't reach an app.

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Integrations**.
    2. Open the app's row menu and click **Test connection**. **Test connection** appears only for an active connection.
    3. Read the **Connection check** column: **Connection check passed.**, or the reason it failed.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/integrations/connections/3c9e1f72-5a4b-4d08-9e6c-1b7f2a8d0e45/probe' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

`probeIntegrationConnection` answers `200` even when the check fails, so read `ok`, not the status code.

```json theme={"theme":"css-variables"}
{
  "ok": true,
  "state": "active",
  "token_minted": true,
  "resource_count": 3
}
```

When `ok` and `token_minted` are both `true`, a session using the app will get access. For a built-in integration, `resource_count` is the number of tools on the allow-list. A failed check returns `ok: false` and a `failure` reason. If the app no longer accepts the sign-in, `state` comes back `revoked`; [reconnect the app](#reconnect-an-app).

<Accordion title="List and read connections">
  `listIntegrationConnections` returns every connection in your organization, native and built-in, including pending and revoked ones. Disconnected connections are left out. `getIntegrationConnection` returns one.

  ```bash theme={"theme":"css-variables"}
  curl 'https://api.recursion.labelbox.com/managed-agents/v1/integrations/connections' \
    -H "Authorization: Bearer $RECURSION_API_KEY"
  ```

  This example shows one built-in connection. Some fields are left out here.

  ```json theme={"theme":"css-variables"}
  {
    "connections": [
      {
        "connection_id": "3c9e1f72-5a4b-4d08-9e6c-1b7f2a8d0e45",
        "external_id": "zendesk",
        "account_login": "Zendesk",
        "account_type": "Zendesk",
        "state": "active",
        "resource_selection": "selected",
        "permissions": { "add_ticket_comment": "write", "get_ticket": "read", "search_tickets": "read" },
        "connected_by": "usr_01a0867d1b0875be9f9a9938ffb90971",
        "usage": { "agents": 2 },
        "created_at": "2026-09-25T17:41:02Z",
        "updated_at": "2026-09-25T18:04:11Z"
      }
    ]
  }
  ```

  For a built-in connection:

  * `external_id` is the app's slug from the catalog, and `account_type` is its name. `account_login` is the app's name too, followed by the sign-in method when the app names one.
  * `permissions` is the tool allow-list, each tool with its level.
  * `usage.agents` counts the agents whose current version grants the app.
  * `provider` identifies the kind of connection. Every built-in connection shares one provider, which `listIntegrationProviders` lists as **Built-in integrations**.
</Accordion>

## Reconnect an app

Reconnect when a connection is **Revoked**, or to sign in to the app again. Reconnecting keeps the same `connection_id`, so the agents granted the app and its tool allow-list stay as they are.

1. In the sidebar, click **Integrations**.
2. Click **Reconnect** on the app's row. For an active app, it's in the row menu.
3. In the **Reconnect** dialog, click **Continue to sign-in**.
4. Sign in to the app and approve access.

Signing in with a different account changes the account every agent using the app acts as. To keep two accounts separate, disconnect the app and add it again instead.

## Disconnect an app

Disconnecting removes the app's sign-in and stops all access through it. Every agent granted the app loses it from its next session.

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Integrations**.
    2. Open the app's row menu and click **Disconnect**.
    3. In the **Disconnect** dialog, click **Disconnect**.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X DELETE 'https://api.recursion.labelbox.com/managed-agents/v1/integrations/connections/3c9e1f72-5a4b-4d08-9e6c-1b7f2a8d0e45' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

```json theme={"theme":"css-variables"}
{
  "connection_id": "3c9e1f72-5a4b-4d08-9e6c-1b7f2a8d0e45",
  "deleted": true,
  "grants_revoked": 0
}
```

Agents that still name a disconnected connection show it as **Unavailable connection** in their **Integrations** section. Remove it and save a new version; a version that still names it returns `404 not_found` on `built_in_integrations`. Connecting the app again starts from the catalog and creates a new connection, so agents must be granted the new one.

`grants_revoked` counts vault grants that used the connection, which applies to native integrations. It's `0` for a built-in integration, because those are granted on the agent.

## Connection states

```mermaid theme={"theme":"css-variables"}
stateDiagram-v2
  [*] --> pending: app added
  pending --> active: signed in
  active --> revoked: sign-in stops working
  revoked --> active: reconnect
  pending --> [*]: disconnect
  active --> [*]: disconnect
  revoked --> [*]: disconnect
```

A new connection starts **Pending setup**, becomes **Active** once someone signs in, and moves to **Revoked** if the sign-in stops working.

<Accordion title="What each state means">
  | State | Console label | What it means |
  | - | - | - |
  | `pending` | **Pending setup** | Added, but nobody has signed in to the app yet. Agents can't use it. |
  | `active` | **Active** | Signed in and usable. Agents granted the app can use its allowed tools. |
  | `revoked` | **Revoked** | The sign-in stopped working, for example because the account's access was removed in the app. Agents can't use it until you reconnect. |

  A connection moves to **Revoked** when a session or a check finds that the app no longer accepts the sign-in. A temporary problem reaching the app doesn't change the state.
</Accordion>

## Native integrations

Native integrations connect a service by signing in to it directly. Each session gets a short-lived token limited to the access you grant, and the connection's own permissions at the service are always the ceiling.

| Service (`id`) | What agents get | Access levels | Where you grant it |
| - | - | - | - |
| GitHub (`github`) | GitHub's MCP tools, called outside the sandbox. No GitHub credential is placed in the sandbox. See [GitHub access](/recursion/github). | Organization tool selection, then an agent subset for all or up to 500 repositories | On the agent, or through a vault |
| Slack (`slack`) | Bot tools for reading channels and threads, looking up users, and posting messages, plus Slack events for [automations](/recursion/automations). | `trigger_thread` (originating thread only), `workspace_tools` (the installed bot's reachable channels) | On the agent, or through a vault |
| Jira (`jira`), Confluence (`confluence`), Loom (`loom`) | The service's MCP tools. | `read`, `write` | On the agent, or through a vault |
| LaunchDarkly (`launchdarkly`) | LaunchDarkly's MCP tools. | `reader`, `writer`, `observability` | On the agent, or through a vault |

`listIntegrationProviders` returns the same list, with `configured: true` for each service your organization can connect. Jira, Confluence, Loom, and LaunchDarkly fix the access level when you connect (`preset_fixed_at_authorization`); to change it, reconnect with the new level. GitHub tools are selected on the connection and narrowed per agent. Slack levels are chosen per grant. An existing Slack installation can be granted either level without reconnecting.

<Accordion title="Test, reconnect, or disconnect a native connection">
  Testing, reconnecting, and disconnecting work the same for native and built-in connections; see [Test a connection](#test-a-connection), [Reconnect an app](#reconnect-an-app), and [Disconnect an app](#disconnect-an-app). A native connection's test also reports `resource_count`, and `missing_permissions` the service hasn't granted yet. For GitHub, `resource_count` is the number of repositories in [the repository list](#list-github-repositories-and-pull-requests). Disconnecting GitHub doesn't uninstall the app from your GitHub account.
</Accordion>

### Connect a service

<Steps>
  <Step title="Start the connection">
    In the sidebar, click **Integrations**, click **Add integration**, and click the service. The **Connect** drawer opens. If it shows **Access level**, choose it now. Click **Continue**. For GitHub, see [Connect GitHub](/recursion/github#connect-github) for the existing-installation flow.
  </Step>

  <Step title="Approve at the service">
    Sign in and approve the request. For GitHub, choose the organization or user account to install the app on, and the repositories it can reach. For Slack, choose the workspace.
  </Step>

  <Step title="Choose an account if asked">
    If the service returns more than one account you can connect, the console shows **Choose an account**. Pick one and click **Continue**.
  </Step>

  <Step title="Confirm">
    Back on **Integrations**, the console shows **Connected. Assign access in the agent's Integrations section.** The row's **State** is **Active**.
  </Step>
</Steps>

The connection appears in `listIntegrationConnections` with `state: "active"`. Connecting grants nothing to any agent; next, [grant it to an agent](#grant-a-native-integration-to-an-agent). Connecting GitHub or Slack also adds an [event source](/recursion/automations#receive-events) that automations can listen to.

<Accordion title="Start a connection from the API">
  `startIntegrationInstall` returns the service's authorization `url` and a single-use `state`. Open the `url` in a browser where the same person is signed in to the console; after approval the console completes the connection. The `state` expires after 15 minutes.

  ```bash theme={"theme":"css-variables"}
  curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/integrations/jira/install' \
    -H "Authorization: Bearer $RECURSION_API_KEY" \
    -H 'Content-Type: application/json' \
    -d '{"option": "read"}'
  ```

  ```json theme={"theme":"css-variables"}
  {
    "url": "https://auth.atlassian.com/authorize?client_id=...&state=...",
    "state": "4c1f0e9a2b7d4e6f8a3c5b1d9e7f2a60"
  }
  ```

  `option` is the access level for a service that fixes it at connect time. For a new GitHub installation, send `{"option":"install"}` to open GitHub's App installation page; omit `option` to connect an existing installation. Send `connection_id` to re-authorize an existing connection in place, keeping its id so agents that use it keep working.
</Accordion>

### Grant a native integration to an agent

Access is saved with the agent version, so saving it creates a new version.

<Tabs>
  <Tab title="Console">
    1. Open the agent. On **Configuration**, find **Integrations** and click **Add integration access**.
    2. Click **Select** next to the connection. A connection that isn't active shows **Reconnect first**.
    3. In **Configure access**, choose the **Access level** for services that offer one. For GitHub, select the agent's tools and keep **All authorized repositories** or pick repositories.
    4. Click **Use this access**. For services other than GitHub, the checkbox named for the connection's MCP tools controls whether sessions get them.
    5. Click **Save new version**.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/agents/5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55/versions' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      -d '{
        "base_agent_version_id": "c1a94e07-2f6b-4d18-b3a5-0e7d8c6f4a21",
        "name": "Release coordinator",
        "model": "<model-id>",
        "system": "Track the release and report blocked work.",
        "nativeIntegrations": [
          {"connectionId": "31e6c9a4-2d75-48b0-a1f3-7c5e9d2b6a84", "permission": "write", "tools": ["get_file_contents", "create_pull_request"], "resources": ["acme/api"]},
          {"connectionId": "a8d2f4c1-6b3e-4f97-8c20-5e1d7b9a3f46", "permission": "read"}
        ]
      }'
    ```
  </Tab>
</Tabs>

Sessions already running keep their access until you [move them to the new version](/recursion/sessions#move-a-session-to-another-version), and automations keep their version until you move them, so removed access still applies to their runs until then. See [Keep the agent version current](/recursion/automations#keep-the-agent-version-current). A new version replaces the whole definition, so send every field you want to keep.

<Accordion title="nativeIntegrations fields and MCP tools">
  | `nativeIntegrations[]` field | Rules |
  | - | - |
  | `connectionId` | Required. An active connection in your organization, listed once. |
  | `permission` | Required for compatibility. For standalone GitHub, the server derives `read` or `write` from `tools`; other services use their access levels. |
  | `tools` | Standalone GitHub only. The agent's subset of the connection's enabled MCP tools. |
  | `resources` | GitHub only. Leave it out to reach every repository the installation reaches, including ones added later, or list 1 to 500 as `owner/name`. |

  `disabled_integration_mcp_providers` lists services whose MCP tools sessions don't get while the grant still works for everything else. GitHub can't be listed there: its MCP tools are how a GitHub grant works, so `github` in the list is ignored. On a new version, leaving it out turns every service's MCP tools on. See [Tools](/recursion/tools#withhold-integration-tools).
</Accordion>

### Grant through a vault

Grant a connection per session through a [vault](/recursion/vaults) with an `integration` credential. Use it when different sessions of one agent need different accounts or repositories.

```bash theme={"theme":"css-variables"}
curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/vaults/26d4b1a8-73c9-4f60-8a15-9e2c7b5d4f31/credentials' \
  -H "Authorization: Bearer $RECURSION_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "credential_type": "integration",
    "integration_connection_id": "31e6c9a4-2d75-48b0-a1f3-7c5e9d2b6a84",
    "integration_permission": "read",
    "integration_resources": ["acme/api"]
  }'
```

Adding the same connection to the same vault again replaces the earlier grant. If `integration_permission` is left out, the service's least privileged level applies. For Slack, the agent's **Integrations** section grants `trigger_thread` or `workspace_tools` access without a vault credential; a vault grant is also available when access must vary by session.

<Accordion title="How grants combine in a session">
  * A session gets the agent version's `nativeIntegrations` plus any `integration` credentials in its granted vaults.
  * Each connection is used once. If the agent and a vault both grant it, the agent's grant wins. Among vaults, the first in `vault_ids` wins.
  * Several connections to one service, such as two GitHub organizations, all work in one session.
  * A grant whose connection isn't active is skipped, and the session starts without it.
</Accordion>

### List GitHub repositories and pull requests

For a GitHub connection, list the repositories it can reach and their pull requests, for example to fill a repository picker. For an installation on selected repositories, the list is the installation's own repositories, the same list the console's repository pickers show. Results come from GitHub, 100 per page, with the 1-based `page` query parameter.

```bash theme={"theme":"css-variables"}
curl 'https://api.recursion.labelbox.com/managed-agents/v1/integrations/connections/31e6c9a4-2d75-48b0-a1f3-7c5e9d2b6a84/repositories?page=1' \
  -H "Authorization: Bearer $RECURSION_API_KEY"
curl 'https://api.recursion.labelbox.com/managed-agents/v1/integrations/connections/31e6c9a4-2d75-48b0-a1f3-7c5e9d2b6a84/repositories/acme/api/pull-requests?state=open&page=1' \
  -H "Authorization: Bearer $RECURSION_API_KEY"
```

```json theme={"theme":"css-variables"}
{
  "repositories": [
    { "id": 812345671, "name": "api", "full_name": "acme/api", "private": true, "default_branch": "main" }
  ],
  "total_count": 12,
  "page": 1
}
```

The pull-request path names the repository owner and repository name separately. `state` is `open` (the default), `closed`, or `all`. Use `full_name` values in `resources`.

## MCP servers

For a service that isn't in the catalog, or when you want the token kept out of the sandbox entirely, add its MCP server to the agent. The token lives in a vault and is attached to requests outside the sandbox. See [Connect MCP servers](/recursion/mcp-servers) and [Vaults](/recursion/vaults).

## What can go wrong

The most common problems:

* **The row stays Pending setup, or the sign-in couldn't be confirmed.** Nobody finished signing in, or the sign-in link expired or was already used. Click **Connect** on the row again, and finish the sign-in within a few minutes.
* **The row shows Revoked, or the agent says the app must be reconnected.** The app no longer accepts the sign-in. [Reconnect the app](#reconnect-an-app).
* **The agent says a tool isn't available.** The tool isn't on the app's allow-list, or the app isn't granted to the agent. Add the tool on **Integrations**, or grant the app, then move the session to the new version or start a new session.
* **A built-in app's tool fails with a network error.** The environment has **No access** to the internet. Set its **Internet access** to **Enabled**, or limit it to specific hosts.

<Accordion title="Every integration error">
  | Symptom or code | Cause | Fix |
  | - | - | - |
  | An app isn't in **Add integration** | You already added it (it shows **Already added**), it's listed with `configurable: false`, or your organization connects it as a native integration. | Use its existing row, pick another app, or add the native connection. |
  | **Add integration** is missing | Your role can view integrations but not add them. | Ask someone with the developer or admin role. |
  | "The vendor sign-in was closed before it finished. Nothing was connected." | You left the app's sign-in page. | Click **Connect** on the row again. |
  | "the vendor sign-in could not be confirmed" | The sign-in link or its code expired, or was already used. | Click **Connect** on the row again, and finish the sign-in within a few minutes. |
  | The row stays **Pending setup** | Nobody finished signing in to the app. | Click **Connect** on the row. |
  | The row shows **Revoked**, or the agent says the app must be reconnected | The app no longer accepts the sign-in. | [Reconnect the app](#reconnect-an-app). |
  | **Reconnect first** in the agent's **Add integration access** | The connection isn't active. | Connect or reconnect it on **Integrations**. |
  | The **Tools** drawer warns that stored tools are no longer read-only or no longer in the catalog | The catalog changed after you chose the tools. Sessions can't use the app until they're removed. | Click **Remove**, then **Save tools**. |
  | The agent says a tool isn't available | The tool isn't on the app's allow-list, or the app isn't granted to the agent. | Add the tool on **Integrations**, or grant the app, then move the session to the new version or start a new session. |
  | **Test connection** reports "the provider could not mint a token for this connection" | The app refused a fresh sign-in, or couldn't be reached. | Run **Test connection** again. If `state` is `revoked`, reconnect. |
  | A built-in app's tool list can't be saved | The list is empty, has more than 200 tools, names a tool the app doesn't have, or includes a write tool under **Read only** access. | Choose 1 to 200 tools from the catalog, or select **Read and write** access. |
  | A revoked app's tools can't be changed | The connection is no longer active. | Reconnect it first. |
  | A shared tool list isn't available | The connection is a native integration, whose access is set per agent. | Set the access level on the agent instead. |
  | `400 invalid_request` on `built_in_integrations` | More than 25 apps, more than 200 selected tools, a blank or no longer enabled tool name, an entry with no `connection_id`, or a connection that isn't active, has no tools, or isn't a built-in integration. | Read `message`, then fix that entry. |
  | `404 not_found` on `built_in_integrations` or `connection_id` | The connection doesn't exist, was disconnected, or belongs to another organization. | List connections and use a current id. |
  | `403 forbidden` | Your role can't make this change. | See [Organizations and roles](/recursion/organizations-and-roles). |
  | The built-in catalog doesn't load | The catalog is temporarily unavailable. | Click **Retry**. |
  | Built-in apps aren't available | The app catalog couldn't be reached for your organization. | Contact support. |
  | `503 integrations_unconfigured` | The integration service couldn't be reached. | Retry later. If it continues, contact support. |
  | "This organization has to approve the app before it can be connected." | A GitHub organization requires an owner to approve new apps. | After an owner approves, start the connection again. |
  | "This install link is no longer valid." (`400` on `state`) | The `state` was already used, is older than 15 minutes, or belongs to another person. | Start the connection again. |
  | `400 invalid_request` on `nativeIntegrations` or `integration_permission` | A connection is listed twice, the level isn't one the service offers, or it differs from the level fixed at connect time. | Use a level from `permissions`, or reconnect with the level you need. |
  | `400 invalid_request` on `integration_resources` | Resources for a service other than GitHub, or more than 500 repositories. | Leave `resources` out, or name fewer repositories. |
  | `400 invalid_request`: "repository … is owned by …, but this connection is for …" | A GitHub repository in `resources` or `integration_resources` belongs to a different owner than the connection's account. | Name repositories of the connection's own account, or use the connection for that owner. |
  | A built-in app's tool fails with a network error | The environment has **No access** to the internet. | Set the environment's **Internet access** to **Enabled**, or limit it to specific hosts. |
  | `429 github_rate_limited` or `503 github_unavailable` listing repositories | GitHub is limiting requests or unavailable. | Retry later. |
  | Slack tools are missing in a session | The Slack connection isn't granted on the agent or through an accessible vault, or `trigger_thread` was granted but the session didn't start from that Slack installation's thread. | Grant `workspace_tools` for manual, scheduled, or other event-triggered sessions. Use `trigger_thread` for a Slack-origin session. |

  For all error codes, see [Errors](/recursion/errors).
</Accordion>

Limits on connections per app, allow-list size, tool names, apps granted to one agent, catalog search, and session access to an app are in [Limits](/recursion/limits).

## Next steps

<CardGroup>
  <Card title="Agents" href="/recursion/agents">
    Publish agent versions and keep grants across updates.
  </Card>

  <Card title="Connect MCP servers" href="/recursion/mcp-servers">
    Add any remote MCP server with a token from a vault.
  </Card>

  <Card title="Security" href="/recursion/security">
    See what reaches the sandbox for each kind of access.
  </Card>

  <Card title="Sessions" href="/recursion/sessions">
    Start a session and follow its tool calls.
  </Card>
</CardGroup>


# Use the Labelbox MCP server
Source: https://docs.labelbox.com/recursion/labeling-mcp

Connect the Labelbox MCP server to the same MCP client as Recursion, so one conversation can read your labeling projects and run your agents.

The Labelbox MCP server gives an AI assistant tools for your Labelbox workspace: it can inspect projects and their data, and summarize labeling progress and quality. It works with Recursion. Add it to the MCP client that already uses the [Recursion MCP endpoint](/recursion/ai-coding-agents), and one conversation can read your labeling work and start agent sessions on it.

```mermaid theme={"theme":"css-variables"}
flowchart TD
  client["Your MCP client"] -->|"browser sign-in"| labeling["Labelbox MCP"]
  client -->|"API key"| recursion["Recursion MCP"]
  labeling --> workspace["Your workspace"]
  recursion --> sessions["Agent sessions"]
```

| Server | What your client does with it | How it signs in |
| - | - | - |
| **Labelbox MCP server** | Reads projects, data, queues, progress, and quality. | You sign in to your Labelbox account in your browser and approve access. No API key. |
| **Recursion MCP endpoint** | Finds and runs Managed Agents operations, such as starting a session. | An API key. |

<Note>
  For the Labelbox MCP server, you need a Labelbox account with access to a workspace where external MCP access is available and allowed by an administrator, and an MCP client that supports Streamable HTTP with OAuth sign-in, such as Claude Code, Codex, or Cursor. For Recursion, you need an API key, and starting a session needs an [agent](/recursion/agents), an [environment](/recursion/environments), and the Developer or Admin role. See [Connect to the Recursion MCP](/recursion/ai-coding-agents) and [Sessions](/recursion/sessions).
</Note>

## Connect the Labelbox MCP server

Add this server address to your MCP client:

```text theme={"theme":"css-variables"}
https://api.labelbox.com/v1/agent-tools/mcp
```

<Tabs>
  <Tab title="Claude Code">
    ```bash theme={"theme":"css-variables"}
    claude mcp add --transport http --scope user labelbox https://api.labelbox.com/v1/agent-tools/mcp
    ```

    Start Claude Code, enter `/mcp`, select the server, and authenticate.
  </Tab>

  <Tab title="Codex">
    ```bash theme={"theme":"css-variables"}
    codex mcp add labelbox --url https://api.labelbox.com/v1/agent-tools/mcp
    ```

    If sign-in doesn't start by itself, run `codex mcp login labelbox`.
  </Tab>

  <Tab title="Cursor">
    Add the server to `~/.cursor/mcp.json`, next to any servers already there:

    ```json theme={"theme":"css-variables"}
    {
      "mcpServers": {
        "labelbox": {
          "url": "https://api.labelbox.com/v1/agent-tools/mcp"
        }
      }
    }
    ```

    Open Cursor's MCP controls, find the server, and start its sign-in.
  </Tab>
</Tabs>

Your browser opens. Sign in to your Labelbox account, choose the workspace this connection uses, review the requested access, and choose **Allow access**. Keep the client open until it reports the server as connected.

The connection works with the permissions your Labelbox account already has; approving it adds none. A connection may be read-only. Whether you can also allow project changes depends on the access your client requests and on what your workspace permits.

For Claude, ChatGPT, and other clients, the access options, and how to disconnect a client, see the [Labelbox MCP server guide](https://docs.labelbox.com/horizon/guides/labelbox-mcp).

## Check both connections

Start a new session in your client and ask:

```text theme={"theme":"css-variables"}
Use the Labelbox MCP server to show my connected workspace and access
restrictions. Then use the recursion MCP server to list my agents.
```

The first answer names the workspace you chose. The second lists the agents in your Recursion organization. If either is missing, see [What can go wrong](#what-can-go-wrong).

## Use them together

With both servers connected, your client can read from one and act in the other. For example:

```text theme={"theme":"css-variables"}
Use the Labelbox MCP server to summarize progress, throughput, and benchmarks
for project <project-id>, and show its queues. Then use the recursion MCP
server to start a session with my "Labeling review" agent in my "Default"
environment. Put the summary in its first message and ask it to draft a plan
for the queues that need attention.
```

Your client calls the Labelbox MCP server's tools first, then starts the session through the Recursion MCP endpoint. Open **Sessions** in the console to follow the agent's work. See [Sessions](/recursion/sessions).

Another prompt to try:

* "Which datasets and batches are attached to this project? Start a session that checks them against our data checklist."

What your client passes into a session is kept with that session's history. See [Data handling](/recursion/security#data-handling) before you send workspace content.

## Agents that run in Recursion

The Labelbox MCP server signs you in through your browser, so you connect it to your MCP client, as on this page. It isn't a server you add to an agent's own MCP servers: an agent that runs in Recursion reaches a server with a token stored in a vault. See [MCP servers](/recursion/mcp-servers).

To give such an agent context from your Labelbox workspace, have your client read it and pass it in the session's first message, as in the prompt above.

## What can go wrong

The most common problems:

* **No workspace is offered during sign-in.** You signed in to a different account, or the workspace doesn't allow external MCP connections. Ask a workspace administrator to allow them. If they're allowed and the workspace is still missing, contact support as the guide below describes.
* **The tools don't appear after you approve.** The client stopped waiting before authorization finished. Keep the client open, start its sign-in again, and check its MCP status.
* **A tool is missing or access is denied.** Your permissions are checked on every call, and a connection limited to selected projects offers fewer tools. Check your role and the access you approved, then connect again if you need more.
* **The Recursion tools don't appear, or calls return `401 unauthorized`.** The API key isn't set where the client started, or it expired. See [Connect to the Recursion MCP](/recursion/ai-coding-agents#what-can-go-wrong).

For every other connection problem with the Labelbox MCP server, see the troubleshooting section of the [Labelbox MCP server guide](https://docs.labelbox.com/horizon/guides/labelbox-mcp).

## Next steps

<CardGroup>
  <Card title="Labelbox MCP server guide" href="https://docs.labelbox.com/horizon/guides/labelbox-mcp">
    Set up other clients, choose access options, and manage connections.
  </Card>

  <Card title="Connect to the Recursion MCP" href="/recursion/ai-coding-agents">
    Connect the docs server and the authenticated endpoint with an API key.
  </Card>

  <Card title="Sessions" href="/recursion/sessions">
    Start, steer, and follow the sessions your client creates.
  </Card>

  <Card title="MCP servers" href="/recursion/mcp-servers">
    Give an agent that runs in Recursion tools from a server that takes a token.
  </Card>
</CardGroup>


# Limits
Source: https://docs.labelbox.com/recursion/limits

Every size, count, rate, and time limit you can hit in Managed Agents, with the error you get and where each is explained.

This page collects every limit you can run into, grouped by area. When a request goes over one, the error's `details` usually reports the limit and the measured value, so your code can react without parsing the message.

## Account and access

| Limit | Value | Details |
| - | - | - |
| API key name | 100 characters | [API keys](/recursion/api-keys) |
| API key lifetime | 365 days. Every key expires. | [API keys](/recursion/api-keys) |
| Live organizations per tenant | 100 | [Organizations and roles](/recursion/organizations-and-roles) |
| Organization name | 40 characters | [Organizations and roles](/recursion/organizations-and-roles) |
| Tenant name | 100 characters | [Organizations and roles](/recursion/organizations-and-roles) |

## Requests

| Limit | Value | When exceeded |
| - | - | - |
| Request rate per allowance | 300 requests per 60 seconds. All your API keys share one allowance, and console use has its own. | `429 rate_limit_exceeded` with `Retry-After`. |
| Open event streams | Bounded by service capacity. | `503 service_unavailable`. |
| `Idempotency-Key` | 1 to 256 visible ASCII characters, one header value | `400 invalid_request`. |
| Idempotency key retention | About 24 hours for a completed request. An abandoned in-progress key can be reused after about 1 hour. | After that, the same key starts a new request. |
| Session list page token | Valid for 1 hour, with the same filters and `limit` | `400 invalid_request` on `page_token`. |

See [API conventions](/recursion/api#rate-limits) for retry guidance.

## API version

The API's version is below 1.0.0; the **Endpoints** reference shows the current one. Branch on error `code` values and each operation's published schema, not on `message` text.

## Agents

| Limit | Value | Details |
| - | - | - |
| Agent name | 256 characters | [Agent setup](/recursion/agents) |
| Skills on one agent or session | 100, after groups are expanded and duplicates removed | [Skills](/recursion/skills) |
| `max_concurrent_sessions` | 1 to 10,000, or unset for no cap. Extra root sessions wait as `queued`. | [Usage and cost](/recursion/usage-and-cost) |
| Tags on one agent | 32 | [Agent setup](/recursion/agents) |
| Tags per organization | 200 | [Agent setup](/recursion/agents) |
| Tag label | 32 characters | [Agent setup](/recursion/agents) |
| Tag description | 280 characters | [Agent setup](/recursion/agents) |
| Tag ids in one list filter | 32 | [Agent setup](/recursion/agents) |

## Sessions and events

| Limit | Value | Details |
| - | - | - |
| Custom metadata on a session | 32 pairs. Keys up to 64 characters of letters, digits, `_`, `.`, and `-`. Values 1 to 512 characters. Can't be changed after start. | [Sessions](/recursion/sessions) |
| Metadata filters in one list request | 32 `metadata` pairs and 32 `metadata_key` values | [Session operations](/recursion/session-operations) |
| Referenced sessions | 10 per request. A session keeps the 10 most recent. | [Referenced sessions](/recursion/referenced-sessions) |
| Session list page | 100 by default, 1,000 at most | [Session operations](/recursion/session-operations) |
| Event list page | 100 events by default. A page stops at about 8 MiB, so it can hold fewer. | [Events](/recursion/events) |
| One stream record | The stream's `Content-Type` states the largest record it sends. A larger event arrives as `event_record_too_large`; fetch it with `getSessionEventContent`. | [Events](/recursion/events) |
| Stream keep-alive | A ping every 15 seconds | [Events](/recursion/events) |
| Live snapshot (gestalt) | 5,000 sessions, over a 30-day window (400 days when filtered by `agent_id`) | [Events](/recursion/events) |
| Resource-sample page | 120 buckets by default, 720 at most | [Session operations](/recursion/session-operations) |
| Working root sessions per tenant | A new top-level session is refused with `429 tenant_concurrency_limit` while your tenant already has as many working root sessions as it may (provisioning, queued, or running a turn). `details.limit` gives the cap. Resting sessions, messages, resumes and subagents don't count. | [Sessions](/recursion/sessions) |
| Sandbox provisioning | Fails with `sandbox_provision_timeout` if compute isn't ready within 5 minutes | [Sessions](/recursion/sessions) |

## Deliverables

| Limit | Value | Details |
| - | - | - |
| Files kept per turn | 200 | [Artifacts](/recursion/artifacts) |
| Bytes kept per turn | 512 MiB | [Artifacts](/recursion/artifacts) |
| One deliverable file | 1 GiB on managed computes; 64 MiB on self-hosted workers | [Artifacts](/recursion/artifacts) |
| Files considered in the outputs folder | 10,000 per turn | [Artifacts](/recursion/artifacts) |
| Deliverable file name | 255 characters, no control characters | [Artifacts](/recursion/artifacts) |
| Image shown with `share_file` | PNG, JPEG, or WebP; 5 MiB, 8,192 pixels on a side, and 40 megapixels | [Artifacts](/recursion/artifacts) |
| Images per session | 500 images and 100 MiB, across every tool that returns images | [Artifacts](/recursion/artifacts) |

A file over the per-file size, or with a name that breaks the rules above, isn't kept, and the session records a warning. Files past a turn's file or byte cap wait for the next turn. Download kept files from the session's **Files** tab; see [Files](/recursion/files).

## Tools

| Limit | Value | Details |
| - | - | - |
| MCP server request | 60-second timeout; responses up to 8 MiB | [MCP servers](/recursion/mcp-servers) |

## Integrations

| Limit | Value | Details |
| - | - | - |
| Connections per app | 1 per organization | [Integrations](/recursion/integrations#connect-an-app) |
| Tools on one app's allow-list | 1 to 200. Over the cap or empty, `400 invalid_request` on `resources`. | [Integrations](/recursion/integrations#choose-which-tools-agents-may-use) |
| Tool name | 128 characters | [Integrations](/recursion/integrations#choose-which-tools-agents-may-use) |
| Apps granted to one agent | 25 entries in `built_in_integrations`. Over the cap, `400 invalid_request`. | [Integrations](/recursion/integrations#grant-apps-to-an-agent) |
| Catalog `search` | 200 characters | [Integrations](/recursion/integrations#the-catalog) |
| Session access to an app | Expires within an hour, renewed while the session runs | [Integrations](/recursion/integrations#what-a-session-receives) |
| Skill slots a session with apps uses | 1 of the session's 100 | [Integrations](/recursion/integrations#what-a-session-receives) |
| GitHub repositories in one grant | 500 | [GitHub access](/recursion/github) |
| Native integration connections on one agent | 25 | [Integrations](/recursion/integrations#grant-a-native-integration-to-an-agent) |
| Install link (`state`) | Valid for 15 minutes, once | [Integrations](/recursion/integrations#connect-a-service) |
| Repository and pull request pages | 100 per page | [Integrations](/recursion/integrations#list-github-repositories-and-pull-requests) |
| Slack channel list | 1,000 channels the Slack app is a member of | [Automations](/recursion/automations#slack) |

## Files

| Limit | Value | Details |
| - | - | - |
| One console upload | 1 GiB | Files over 30 MiB are sent in parts. Managed-compute output capture uses the same 1 GiB per-file cap. [Files](/recursion/files) |
| One public REST `uploadFile` file part | 30 MiB | The complete multipart request has a separate 128 KiB framing and form-field allowance. [Files](/recursion/files) |
| Live files per organization | 10 GiB in total. Over it, `413 file_quota_exceeded`. | [Files](/recursion/files) |
| Upload expiry | 1 hour to 90 days, or none | [Files](/recursion/files) |
| Files attached in one request | 500 | [Files](/recursion/files) |
| File list page | 200 by default, 1,000 at most; 100 ids in a `file_ids` lookup | [Files](/recursion/files) |
| Documents named in one message | 512 KiB each and 1 MiB per message | [Files](/recursion/files#show-a-file-to-the-agent-in-a-message) |
| Library files named in one message | 100 blocks and 32 MiB | [Files](/recursion/files#show-a-file-to-the-agent-in-a-message) |

## Automations

| Limit | Value | Details |
| - | - | - |
| Automation name | 256 bytes of UTF-8 text | [Automations](/recursion/automations) |
| Triggers per automation | 20 | [Automations](/recursion/automations) |
| Enabled triggers per event source | 100 across all automations, counting triggers on paused automations. Over it, `409 conflict` on `triggers`. | [Automations](/recursion/automations#receive-events) |
| Opening prompt | 65,536 bytes of UTF-8 text | [Automations](/recursion/automations) |
| Run with input | JSON input up to 64 KiB | [Automations](/recursion/automations) |
| Run defaults | 50 vaults, 500 credentials, 500 files | [Automations](/recursion/automations#set-run-defaults) |
| Run metadata | 16 pairs. Keys up to 64 bytes, values up to 512 bytes. | [Automations](/recursion/automations#set-run-defaults) |
| Schedule catch-up window | 10 seconds to one year, default 15 minutes | [Automations](/recursion/automations#schedule) |
| Schedule jitter | 0 to 540 seconds, default 10 | [Automations](/recursion/automations#schedule) |
| Trigger filters | 100 Slack channels, 100 Slack authors, 100 GitHub repositories, 20 webhook field conditions | [Automations](/recursion/automations#choose-triggers) |
| Custom webhook body | 1 MiB. Larger bodies get `413 payload_too_large`. | [Automations](/recursion/automations#send-a-custom-webhook-delivery) |

## Skills

| Limit | Value | Details |
| - | - | - |
| Skill name | 64 characters | [Skills](/recursion/skills) |
| Skill description | 1,024 characters | [Skills](/recursion/skills) |
| Skill instructions body | 64,000 bytes of UTF-8 after escaping | [Skills](/recursion/skills) |
| Compressed bundle upload | 8 MiB | [Skills](/recursion/skills) |
| Unpacked bundle | 32 MiB | [Skills](/recursion/skills) |
| One file in a bundle | 4 MiB | [Skills](/recursion/skills) |
| Files in a bundle | 500 | [Skills](/recursion/skills) |
| Folder depth in a bundle | 3 levels below the skill root | [Skills](/recursion/skills) |
| Skill list page | 100 by default, 500 at most | [Skills](/recursion/skills) |
| Skill search term | 2 to 256 characters | [Skills](/recursion/skills) |

## Environments

| Limit | Value | Details |
| - | - | - |
| Setup script | 64 KiB | [Environments](/recursion/environments) |
| Setup timeout | 10 to 3,600 seconds. `0` or unset uses 600. | [Environment reference](/recursion/environments-reference) |
| CPU | 250 to 8,000 millicores. Available compute or network policy can impose a lower maximum. | [Environment reference](/recursion/environments-reference) |
| Memory | 1,024 to 32,768 MiB. Available compute or network policy can impose a lower maximum. | [Environment reference](/recursion/environments-reference) |
| GPU sandbox | One GPU, with 8,000 millicores and 32,768 MiB set for you | [Environment reference](/recursion/environments-reference#compute) |
| Sandbox lifetime | 10 to 86,400 seconds when set | [Environment reference](/recursion/environments-reference) |
| Workspace disk | 1 to 100 GiB. Unset uses the default. | [Environment reference](/recursion/environments-reference) |
| Idle stop | 120 to 7,200 seconds (2 hours) when set; `0` uses the default of 30 minutes. With computer use, `0` or at least 600. | [Environment reference](/recursion/environments-reference) |
| Stopped sandbox retention | 3,600 to 2,592,000 seconds when set | [Environment reference](/recursion/environments-reference) |
| Network policy rules | 200 | [Environment reference](/recursion/environments-reference) |
| Setup runs in flight | 1 per environment, and 4 per organization by default. Over the organization cap, `429 setup_run_limit` reports the cap in `details.limit`. | [Environments](/recursion/environments) |
| Setup-run list page | 50 | [Environments](/recursion/environments) |
| Setup-run log kept | 4 MiB per run, plus a 2 KiB `stderr_tail` | [Environments](/recursion/environments) |
| Setup-run log page | 500 lines by default, 2,000 at most | [Environments](/recursion/environments) |
| Setup-run wait | `wait_seconds` accepts 0 to 60, but waits at most 5 seconds | [Environments](/recursion/environments) |
| Setup-run stderr tail | 2 KiB | [Environments](/recursion/environments) |

## Multi-agent

| Limit | Value | Details |
| - | - | - |
| Roster entries | 20, with at most 1 advisor | [Multi-agent](/recursion/multi-agent) |
| `max_depth` | 1 by default, 3 at most | [Multi-agent](/recursion/multi-agent) |
| `max_concurrent_threads` | 8 by default, 25 at most | [Multi-agent](/recursion/multi-agent) |
| `max_threads` | 25 by default, 100 at most | [Multi-agent](/recursion/multi-agent) |

Child sessions don't count toward an agent's `max_concurrent_sessions`.

## Next steps

<CardGroup>
  <Card title="Errors" href="/recursion/errors">
    Every error code, its cause, and whether to retry.
  </Card>

  <Card title="API conventions" href="/recursion/api">
    Retries, idempotency, and paging.
  </Card>

  <Card title="Troubleshooting" href="/recursion/troubleshooting">
    Match a symptom to its cause and fix.
  </Card>

  <Card title="Event types and states" href="/recursion/reference">
    Look up every status, event type, and result value.
  </Card>

  <Card title="Pricing" href="/recursion/pricing">
    See what a session hour costs and the purchase amounts credits come in.
  </Card>
</CardGroup>


# Connect MCP servers
Source: https://docs.labelbox.com/recursion/mcp-servers

Add a remote MCP server's tools to an agent, authenticate it with a vault credential, and test the connection before a session depends on it.

An MCP server is a remote service that offers tools over the Model Context Protocol. When you add one to an agent, each session connects to the server, discovers its tools, and lets the agent call them. Calls are made outside the sandbox, so the server's token never enters the sandbox or the model's context.

This page covers agents calling out to MCP servers. To connect your coding assistant to these docs or to the Managed Agents MCP endpoint, see [Use with AI coding agents](/recursion/ai-coding-agents).

<Note>
  Changing an agent or testing a server needs the organization developer or admin role. Testing a server needs the create permission, so the organization user role can't run a test. See [Organizations and roles](/recursion/organizations-and-roles). The server must use the Streamable HTTP transport at a public `http` or `https` URL. Servers on private, loopback, or link-local addresses are refused. Local servers that run over stdio aren't supported. If the server needs a token, have it ready; the token goes in a vault.
</Note>

<Tip>
  If the service is one of the 164+ apps in [Integrations](/recursion/integrations), you can connect it there instead and choose its tools, without handling a token.
</Tip>

<Accordion title="About the Managed Agents MCP endpoint">
  On the Managed Agents MCP endpoint, operation arguments name path parameters in camelCase, such as `vaultId`, while request bodies keep their snake\_case fields, such as `vault_id`. Create skills with scripts, and read a skill version's content, in the console or with the REST API.
</Accordion>

## Add an MCP server to an agent

Each server entry has a `name` and a `url`. MCP servers belong to the agent version, so adding one creates a new version. Running sessions keep the tools they have until you [move them to the new version](/recursion/sessions#move-a-session-to-another-version).

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Agents** and open the agent.
    2. On **Configuration**, find **Tools** and click **Add MCP server**.
    3. Enter a **Server name**, then choose a server from the list or type its **Server URL**. The list suggests common servers, such as GitHub, Notion, Linear, Sentry, Context7, and Slack, and says what kind of token each needs.
    4. Click **Save new version**.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/agents/5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55/versions' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      -d '{
        "base_agent_version_id": "c1a94e07-2f6b-4d18-b3a5-0e7d8c6f4a21",
        "name": "Release coordinator",
        "model": "<model-id>",
        "system": "Track the release and report blocked work.",
        "mcp_servers": [{"name": "project-tracker", "url": "https://mcp.example.com/mcp"}],
        "default_vault_ids": ["26d4b1a8-73c9-4f60-8a15-9e2c7b5d4f31"]
      }'
    ```
  </Tab>
</Tabs>

The response is the agent at its new version:

```json theme={"theme":"css-variables"}
{
  "agent_id": "5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55",
  "latest_agent_version_id": "7a4e1c95-6d2f-4380-b915-3e7a0c6f2d84",
  "mcp_servers": [{ "name": "project-tracker", "url": "https://mcp.example.com/mcp" }],
  "default_vault_ids": ["26d4b1a8-73c9-4f60-8a15-9e2c7b5d4f31"]
}
```

If the server `project-tracker` offers `list_issues`, the agent sees `project-tracker__list_issues`. The prefix keeps two servers from clashing. A new version replaces the whole definition, so send `skills`, tool settings, vault defaults, and every other field you want to keep. See [Agents](/recursion/agents). Next, [authenticate the server](#authenticate-the-server).

<Accordion title="Server entry fields">
  | Field | Description |
  | - | - |
  | `name` | The label the server's tools are grouped under. Characters other than letters, digits, `_`, and `-` become `-` in tool names. |
  | `url` | The server's full Streamable HTTP endpoint, `http` or `https`, with no user name or password in it. Use `https` whenever a token is sent. |
</Accordion>

## Authenticate the server

Store the server's token in a [vault](/recursion/vaults) as a **Token for an MCP server** (`bearer_token`) whose server URL matches the agent's `url`. Then grant the vault to the agent with `default_vault_ids`, or to a session with `vault_ids`. If you narrow grants with credential refs, include this credential.

Tokens that you paste are stored as-is and aren't refreshed. If the server issues short-lived OAuth access tokens, [rotate the credential](/recursion/vaults#rotate-a-credential) before it expires.

<Warning>
  **A granted token also adds its server.** A token for an MCP server in a granted vault adds its server's tools to the session even if the agent doesn't list that server. Those tools get a generated `vault-...` prefix. To keep a server out of a session, don't grant the vault that holds its token, or leave the credential out of the credential refs.
</Warning>

<Accordion title="How matching works">
  * URLs match after lowercasing the scheme and host, dropping a default port (`:443` or `:80`), and ignoring a trailing slash. A different path, subdomain, or port is a different server, so the token isn't sent there.
  * The token is sent as `Authorization: Bearer <token>` by default. If the credential sets `secret_name` (API only), the token is sent as-is in a header with that name instead, for servers that read a header such as `X-Api-Key`.
  * If two granted vaults hold a token for the same server, the vault listed first in the session's grants wins.
  * If no token matches, the session connects without one. That works for public servers and fails for servers that need authorization.
</Accordion>

## Test a server before you use it

`probeMcpServer` makes one live connection using the same URL matching and vault lookup as a session. It creates and changes nothing, but it needs the Developer or Admin role, like creating resources. See [Organizations and roles](/recursion/organizations-and-roles). Omit `vault_ids` to test without a token.

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Credential vaults** and open the vault.
    2. Expand the server's **Token for an MCP server** row. It shows whether the server is reachable and how many tools it offers.
    3. Click **Test again** to run a new test.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/mcp/probe' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      -d '{
        "server_url": "https://mcp.example.com/mcp",
        "vault_ids": ["26d4b1a8-73c9-4f60-8a15-9e2c7b5d4f31"]
      }'
    ```
  </Tab>
</Tabs>

```json theme={"theme":"css-variables"}
{
  "ok": true,
  "reachable": true,
  "credential_matched": true,
  "deployment_authorized": false,
  "protocol_version": "2025-06-18",
  "server_name": "project-tracker",
  "tools": [
    { "name": "list_issues", "description": "List issues and their current status." }
  ]
}
```

A failed test still returns `200`. Read the body:

| Result | Meaning |
| - | - |
| `ok: true` | The server works with this token. |
| `ok: false`, `reachable: true`, no `protocol_version` | The server refused the token with `401` or `403`. It may be wrong or expired; check or rotate it. |
| `ok: false`, `reachable: true`, `protocol_version` set | The connection worked, but listing tools failed. Read `error`. |
| `ok: false`, `fault: egress_blocked` | The server URL points to a private, loopback, link-local, or metadata address, which Recursion refuses to connect to. Nothing was sent. Use a public server URL. |
| `ok: false`, `fault: token_invalid` | The stored token contains a line break, so it can't be sent. Save it again without the line break. |
| `ok: false`, `reachable: false`, `credential_matched: true` | A token was sent, and the server couldn't be reached or failed. Read `error`. |
| `ok: false`, `credential_matched: false` | No token was sent. Check the credential's server URL and the vaults you named. |

**What success means:** a passing test proves the server is reachable and that the token lets you list tools. It doesn't prove that every tool call will succeed; permissions, arguments, and the server's own state still matter.

<Accordion title="Every test response field">
  | Field | Meaning |
  | - | - |
  | `ok` | The server completed the handshake and returned its tool list. |
  | `reachable` | The server answered. Either the handshake succeeded, even if listing tools then failed, or the server refused the token with `401` or `403`; in that case `protocol_version` is absent. |
  | `credential_matched` | A token from the named vaults was attached. `false` means the test ran without a token. |
  | `credential_unopenable` | A token matched the URL but couldn't be used, so none was sent. `material_missing` means the credential has no stored value; re-enter it. `external_reference` means its value is held outside Recursion; re-enter it as a value. |
  | `deployment_authorized` | Always `false` for servers you add in Recursion. You can ignore it. |
  | `server_name` | What the server called itself. Use it to confirm you reached the right service. |
  | `protocol_version` | The Model Context Protocol version the test used. |
  | `tools` | The tools offered on this attempt, as the agent would see them before the name prefix. |
  | `error` | Why the test failed. Never contains the token. |
  | `fault` | Set when the test never reached the server for a reason other than the network. `egress_blocked` means the server URL points to a private or internal address; use a public URL. `token_invalid` means the stored token contains a line break; save it again without it. Absent otherwise, including real network failures. |

  A test gives up after 15 seconds. A server on a private, loopback, or link-local address is refused before anything is sent, so the test returns `reachable: false` and `fault: egress_blocked` with the reason in `error`.
</Accordion>

## How a session uses an MCP server

```mermaid theme={"theme":"css-variables"}
---
config:
  sequence:
    width: 110
    actorMargin: 40
---
sequenceDiagram
  participant S as Session
  participant M as MCP server
  participant A as Agent
  S->>M: Connect with token
  M-->>S: Tools
  S->>A: Offer tools
  A->>M: Call a tool
  M-->>A: Result
```

1. When a session starts, or moves to another agent version, it reads the MCP servers on that version.
2. For each server, it looks for a **Token for an MCP server** in the session's granted vaults whose server URL matches.
3. It connects, lists the server's tools, and fixes that list and the tool schemas until the session moves to another agent version.
4. The agent calls a tool as `<server name>__<tool name>`. The token is attached to every call outside the sandbox.

Tools, prompts, and resources: sessions use the server's tools. Text and images that a tool returns are passed to the agent. Server prompts and standalone resources aren't offered, and resource links in a result aren't fetched.

## Choose which tools an agent gets

A session receives every tool the server lists for its token. You can't hide individual tools of a server. To limit what an agent can do, give it a token whose scopes allow only what it needs, or point it at a server that offers fewer tools.

## What happens when a server fails

* **At session start:** an unreachable server or rejected token doesn't stop the session. Other servers still load, and the session's events record a warning naming the servers: `mcp_auth_rejected` when a server refused the token, and `mcp_discovery_failed` for any other failure. Fix the token, or the URL or server, then start a new session. A running session lists a server's tools again only when it [moves to another agent version](/recursion/sessions#move-a-session-to-another-version); moving it to the version it already runs changes nothing.
* **During a call:** each request to the server has a 60-second timeout, and a response can be at most 8 MiB. A failed call returns an error to the agent as the tool result, and the agent can decide what to do next.

## What can go wrong

The most common problems:

* **A session has an `mcp_auth_rejected` warning.** The server refused the token with `401` or `403`. Check or rotate the token in the vault, then start a new session.
* **A session has an `mcp_discovery_failed` warning.** The server was unreachable, failed the handshake, or is on a private address. Run a test with the same URL and vault, fix the cause, and start a new session.
* **Unexpected `vault-...` tools appear.** A granted vault holds a token for an MCP server the agent doesn't list. Remove the credential from the session's grants, or add the server to the agent with a name you choose.

<Accordion title="Every MCP server problem">
  | Symptom or code | Cause | Fix |
  | - | - | - |
  | `400 invalid_request` on `server_url` | The URL is missing, isn't `http` or `https`, has no host, or includes a user name or password. | Use the server's full endpoint URL, and put the token in a vault. |
  | `404 not_found` from a test | A vault in `vault_ids` doesn't exist in your organization. | Check the vault id. |
  | `403 forbidden` from a test | Your role doesn't have the create permission, which a test needs. | See [Organizations and roles](/recursion/organizations-and-roles). Use an account or key with the developer or admin role. |
  | A session has an `mcp_auth_rejected` warning | The server refused the token with `401` or `403`. | Check or rotate the token in the vault, then start a new session. |
  | A session has an `mcp_discovery_failed` warning | The server was unreachable, failed the handshake, or is on a private address. | Run a test with the same URL and vault, fix the cause, and start a new session. |
  | The server's tools are missing without a warning | The server isn't on the agent version the session used, or the session started before you saved it. | Check the session's agent version, then [move the session](/recursion/sessions#move-a-session-to-another-version) to a version that has the server, or start a new session. |
  | Unexpected `vault-...` tools appear | A granted vault holds a token for an MCP server the agent doesn't list. | Remove the credential from the session's grants, or add the server to the agent with a name you choose. |
  | Tools worked yesterday and now fail with 401 | A short-lived access token expired. | Rotate the credential. |
  | A tool call times out | The server took longer than 60 seconds. | Make the tool faster, or have it return a job id the agent can check later. |
</Accordion>

For all error codes, see [Errors](/recursion/errors). For limits, see [Limits](/recursion/limits).

## Next steps

<CardGroup>
  <Card title="Vaults and credentials" href="/recursion/vaults">
    Store tokens for MCP servers and control which sessions receive them.
  </Card>

  <Card title="Agents" href="/recursion/agents">
    Publish a new agent version with its MCP servers attached.
  </Card>

  <Card title="Tools" href="/recursion/tools">
    See the built-in tools every session can use.
  </Card>

  <Card title="Events" href="/recursion/events">
    Read discovery warnings and MCP tool calls in the session timeline.
  </Card>
</CardGroup>


# Multi-agent
Source: https://docs.labelbox.com/recursion/multi-agent

Choose between one agent, delegation, a team, or an advisor, and control exactly what each child agent can use.

One session can split work across several agents. The agent you start, called the root, can hand a focused task to a child agent, run a team of copies on a shared task board ([Teams](/recursion/teams)), or ask an advisor for a second opinion. Every agent in the tree works in the same sandbox, and cost rolls up to the root.

<Note>
  You need an [agent](/recursion/agents) and an [environment](/recursion/environments). Creating agents and starting sessions needs the Developer or Admin role; the User role can read session trees and threads. See [Organizations and roles](/recursion/organizations-and-roles). Every session can delegate to copies of its own agent without any setup. You need a roster only to delegate to other agents or to add an advisor.
</Note>

## Choose a pattern

Start with one agent. Add agents only when the work splits into pieces that don't depend on each other.

| Pattern | Use it when | Example | Avoid it when |
| - | - | - | - |
| One agent | The steps are short, or each step feeds the next. | Fix a failing test, then update the changelog. | Never; it's the default. |
| Delegate | One hand-off is self-contained and clearly defined. | "Read the three RFCs in `/workspace/rfcs` and list every breaking change." | The child would need most of the parent's conversation to do the work. |
| Team | There are two or more independent, substantial pieces, or the work benefits from independent review. | Compare three caching strategies, each with its own benchmark, then pick one. | Each piece takes a few minutes, or the pieces must happen in order. |
| Advisor | You want a second opinion on a decision, not more hands. | Check a migration plan before the agent starts it. | You need the advisor to read files or run commands. It can't. |

A piece is worth a separate agent when it has its own build-and-test loop, its own sources, or its own approach to try. Splitting a short, sequential task costs more and usually finishes later, because each child starts without the parent's context. Delegation and teams combine: a team member can still delegate a focused sub-task, within the depth limit.

## How delegation works

The root delegates with the `delegate_subagent` tool, which you see in the transcript. Each child is a separate session with its own conversation, called a thread.

```mermaid theme={"theme":"css-variables"}
---
config:
  flowchart:
    nodeSpacing: 20
    rankSpacing: 35
---
flowchart LR
  subgraph shared["One shared sandbox"]
    root["Root session"] --> self["Copy of the root"]
    root --> roster["Roster agent"]
    root -.-> advisor["Advisor"]
    roster --> nested["Nested child"]
  end
```

* **The task must stand alone.** A child receives only the task text, not the parent's conversation. Name the files, the goal, and what a good result contains.
* **Hand work over by file path.** Children share the root's sandbox and environment, so each side can read what the other wrote.
* **Children report a result.** A child finishes with `submit_result`: a status of `done`, `partial`, or `blocked`, a summary, and the paths it produced. The parent reads exactly that summary.

<Accordion title="Follow-ups, failures, and inspection">
  - **Threads stay reachable.** After a child reports, the parent can send it a follow-up with `send_to_agent`, and the child continues with its earlier context. The parent can also wait for children with `wait_for`, stop one with `interrupt_agent`, and close an idle one with `archive_thread`.
  - **A silent child doesn't go unnoticed.** A parent that ends its turn while children are running waits for their reports. When every child it's waiting on has been silent for 30 minutes, the service tells the parent which ones, what each is doing, and for how long. Silent means the child and anything it delegated has written nothing, so it includes a child that is asleep or inside one long command. The parent then stops a child it no longer needs with `interrupt_agent`, or keeps waiting with `wait_for`. The service never stops a child itself, and tells the parent about the same silence at most twice, 30 minutes apart, unless the parent waits on the child or messages it in between. A child that already finished or closed without its report arriving no longer holds the session open; this applies in every mode. A child that is only paused (the account is out of credit or suspended, or a person has been asked to review it) is different: the parent stops waiting on it only when it's told about the pause, which needs `active` mode. `multiagent.interventionist.mode` controls the notices: `shadow` only records them, and `off` turns them off.
  - **Failures stay contained.** A child that fails, or a roster agent that can't be resolved, returns an error to the parent. Its siblings keep working.
  - **Everything is inspectable.** The console session page and the API show every child session, its thread, and a merged timeline. See [Inspect the session tree](#inspect-the-session-tree).
</Accordion>

## Configure a roster

A roster lists the agents a coordinator can delegate to, and at most one advisor. It's part of the agent version, so a running session keeps its roster until you [move it to another version](/recursion/sessions#move-a-session-to-another-version). This coordinator can delegate to a researcher agent and consult one advisor.

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Agents**, then open the coordinator agent.
    2. On the **Configuration** tab, find **Multiagent**.
    3. Click **Add helper agent**, then choose the agent under **Helper agent**.
    4. Click **Add advisor**, choose the **Advisor model**, then click **Add**.
    5. Click **Save new version**.

    The console doesn't edit delegation limits or entry names. Use the API for those.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/agents' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      -d '{
        "name": "Release coordinator",
        "model": "<model-id>",
        "system": "Plan the release review. Delegate independent research, then write one recommendation.",
        "multiagent": {
          "type": "coordinator",
          "agents": [
            {"type": "agent", "id": "2b8c4d16-7a95-4e30-b1f2-6d9c0a5e3b74", "name": "researcher"},
            {"type": "advisor", "model": "<model-id>"}
          ],
          "limits": {"max_depth": 2, "max_concurrent_threads": 4}
        }
      }'
    ```
  </Tab>
</Tabs>

The response is the agent with its first version, and the roster under `multiagent`. Some fields are omitted.

```json theme={"theme":"css-variables"}
{
  "agent_id": "5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55",
  "latest_agent_version_id": "c1a94e07-2f6b-4d18-b3a5-0e7d8c6f4a21",
  "name": "Release coordinator"
}
```

To change the roster later, create a new agent version. A version request replaces the whole definition, so resend every field you want to keep. See [Agents](/recursion/agents).

<Accordion title="Roster fields">
  | Field | Description |
  | - | - |
  | `multiagent.type` | Required. Always `coordinator`. |
  | `multiagent.agents` | Roster entries, at most 20 including the advisor. |
  | `multiagent.team.mode` | `auto` (default), `on`, or `off`. See [Teams](/recursion/teams#choose-a-team-mode). |
  | `multiagent.limits` | Delegation limits. See [Set delegation limits](#set-delegation-limits). |
  | `multiagent.interventionist.mode` | What happens when the service notices a session may be stuck. `shadow` records an `advisor_intervention` event only. `active` also sends the agent a short notice it reads on its next turn. `off` turns both off. Omit it to use the default, `active`. |

  Each roster entry has these fields:

  | Field | Description |
  | - | - |
  | `type` | Required. `agent` for another agent, `advisor` for the advisor, or `self` to rename copies of the coordinator. |
  | `id` | Required for `agent`. The agent's id. |
  | `version` | Pins an agent version. Omit it to use the agent's current version at the moment of delegation. When the session moves to another agent version, a changed `version` moves the agents already working, and an entry without `version` moves them to that agent's latest version. |
  | `name` | The name the coordinator delegates to. Omit it to use the agent id. Names are unique, ignoring case. |
  | `model` | Required for `advisor`. A model id from the [models list](/recursion/agents). |
</Accordion>

### Use the advisor

The advisor answers questions the agent asks with `consult_advisor`. It sees a digest of the session's recent activity and its own earlier advice, and it returns guidance only. It has no tools, can't read files, and doesn't do work. Ask the agent to consult it at decision points, for example: "Before you change the schema, consult the advisor with your migration plan."

## Set delegation limits

Set limits in the coordinator's roster under `multiagent.limits`, not in a session start request. A child carries the root's limits down the tree. A child agent's own roster can tighten them but never widen them.

When a limit is reached, the delegation is refused with a tool error that the agent reads, such as "3 threads are already running, the most allowed at once". The agent then waits for a child to report, reuses an idle child, or archives one. The session itself keeps running.

<Accordion title="Limit fields">
  | Field | Default | Allowed | What it bounds |
  | - | - | - | - |
  | `max_depth` | `1` | `0` to `3` | The deepest child level. The root is depth 0. `1` means children can't delegate further; `0` forbids delegation. |
  | `max_concurrent_threads` | `8` | `1` to `25` | Children one coordinator can run at once. It can't exceed `max_threads`. It is also the team size. |
  | `max_threads` | `25` | `1` to `100` | Live children, running or idle, that one coordinator can hold. Archiving a child frees its slot. |
</Accordion>

## Inspect the session tree

The tree view shows every session in the tree, every thread, and the first page of one merged timeline. The thread list is the complete list of children, ordered by position in the tree.

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Sessions**, then open the root session.
    2. The transcript shows each child's work in its own lane.
    3. Open the **Work** tab to see members, the board, and messages between agents.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38/tree' \
      -H "Authorization: Bearer $RECURSION_API_KEY"

    curl 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38/threads' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

Both calls accept any session id in the tree and resolve to its root. A root with `stop_reason` `awaiting_subagents` is waiting on its children, not stuck; if the children go silent, the service tells it, as described under [How delegation works](#how-delegation-works). When `next_event_id` is present, read the rest of the timeline with the events API. See [Events](/recursion/events).

<Accordion title="Example tree response">
  `events` is shortened.

  ```json theme={"theme":"css-variables"}
  {
    "root_session_id": "b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38",
    "sessions": [
      {
        "session_id": "b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38",
        "root_session_id": "b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38",
        "kind": "api_call",
        "status": "active",
        "execution_state": "idle",
        "stop_reason": "awaiting_subagents"
      },
      {
        "session_id": "4a9f2c61-0e3b-4d87-b5a2-7c1e6f9d3b40",
        "parent_session_id": "b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38",
        "root_session_id": "b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38",
        "agent_id": "2b8c4d16-7a95-4e30-b1f2-6d9c0a5e3b74",
        "kind": "subagent",
        "status": "active",
        "execution_state": "running"
      }
    ],
    "threads": [
      {
        "thread_id": "4a9f2c61-0e3b-4d87-b5a2-7c1e6f9d3b40",
        "session_id": "4a9f2c61-0e3b-4d87-b5a2-7c1e6f9d3b40",
        "root_session_id": "b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38",
        "parent_session_id": "b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38",
        "agent_id": "2b8c4d16-7a95-4e30-b1f2-6d9c0a5e3b74",
        "name": "researcher",
        "role": "subagent",
        "status": "running",
        "session_path": "/4a9f2c61-0e3b-4d87-b5a2-7c1e6f9d3b40",
        "thread_path": "/4a9f2c61-0e3b-4d87-b5a2-7c1e6f9d3b40",
        "created_at": "2026-09-25T17:02:11Z",
        "updated_at": "2026-09-25T17:04:37Z"
      }
    ],
    "events": [],
    "next_event_id": "0192d4e7-5b1a-7c3e-9f20-6a8b4c1d2e37"
  }
  ```
</Accordion>

## Track cost across the tree

Every child's cost counts toward the root session. A session's `costUsd` is one total for the whole tree, and a child session returns the same figure as its root. See [Usage and cost](/recursion/usage-and-cost). Child sessions don't take a slot under the agent's `max_concurrent_sessions`; ordinary root sessions do.

## What a child inherits

A copy of the root acts as the root under another name. A different agent from the roster is its own principal and gets only what its own definition grants.

| | Copy of the root (`self`, any name not on the roster, and automatic teammates) | Different agent from the roster |
| - | - | - |
| Sandbox, environment, and workspace files | Shared with the root | Shared with the root |
| [Referenced sessions](/recursion/referenced-sessions) | Inherited | Inherited |
| Delegation limits | Inherited; can only be tightened | Inherited; can only be tightened by its own roster |
| Model | The root's model | Its own model |
| System prompt | The root's prompt for `self`; a general worker prompt for any other name | Its own prompt |
| Skills | The root's skills | Its own skills |
| MCP servers | The root's servers | Its own servers |
| Vaults and credentials | Exactly the same grants as the root | Only its own default vaults and credentials, never the root's |
| Roster | The root's roster | Its own roster |

The table describes a child when it starts. When you move the root session to another agent version, each child follows before its own next model call, after its parent. See [Move a session to another version](/recursion/sessions#move-a-session-to-another-version).

Two rules apply to credentials across the whole tree:

* **Environment-variable credentials come only from the vaults granted to the root session.** They're set as plaintext environment variables in the shared sandbox when it starts, every agent in the tree can read them, and nothing adds to them later. To make a variable available, grant its vault to the root session.
* **MCP server credentials work for each agent individually.** A roster agent's MCP servers authenticate with its own grants, and the root's credentials never authenticate a roster agent's servers.

```mermaid theme={"theme":"css-variables"}
---
config:
  flowchart:
    nodeSpacing: 20
    rankSpacing: 35
---
flowchart LR
  rootVaults["Root session's vaults"] -->|"env vars"| sandbox["Shared sandbox"]
  rootVaults --> rootMcp["Root's<br>MCP servers"]
  childVaults["Roster agent's vaults"] --> childMcp["Roster agent's<br>MCP servers"]
  childVaults -.->|"not added"| sandbox
```

If a roster agent's own vaults hold environment-variable credentials that the root doesn't have, the child's log shows a `session_status` warning with the code `delegated_env_credentials_unavailable` and the missing variable names.

<Warning>
  The shared sandbox holds the root's environment-variable credentials, and every agent in the tree runs its commands in that sandbox. Don't rely on a roster agent to keep a sandbox credential away from other agents in the same tree. To share a credential on purpose, attach the same vault to both agents.
</Warning>

## What can go wrong

The most common problems:

* **A child did the wrong thing.** Its task depended on context it never received. Write tasks that stand alone: goal, input paths, output path, and what done means.
* **"threads are already running" or "live threads" in a tool result.** A thread limit was reached. This is expected under load: the agent waits, reuses, or archives children. Raise the limit if the work needs more.
* **`delegated_env_credentials_unavailable` warning in a child.** A roster agent expected environment-variable credentials that the root's vaults don't provide. Grant that vault to the root session.

<Accordion title="Every multi-agent error">
  | Symptom or code | Cause | Fix |
  | - | - | - |
  | `400 invalid_request` on create | The roster has more than 20 entries, two advisors, a duplicate name, an `agent` entry without `id`, or a limit out of range. | Correct the named field. |
  | `409 revision_conflict` when you save a new version | Someone else saved a version after the one you based yours on. | Read the agent, rebuild the full definition from its latest version, and retry. |
  | A delegation fails with "could not be resolved" | The roster names an agent or pinned version that was deleted or is outside your organization. | Fix the roster entry and save a new version. Then move the session to it, or start a new session. |
  | "may not delegate" in a child's tool result | The child is at `max_depth`. | Raise `max_depth` on the root's roster, or have the child do the work itself. |
  | "threads are already running" or "live threads" in a tool result | A thread limit was reached. | Expected under load. The agent waits, reuses, or archives children. Raise the limit if the work needs more. |
  | `delegated_env_credentials_unavailable` warning in a child | A roster agent expected environment-variable credentials that the root's vaults don't provide. | Grant that vault to the root session. |
  | A roster agent can't reach an MCP server the root can | Roster agents use only their own servers and grants. | Add the server and grant to the roster agent's own definition. |
  | A child did the wrong thing | Its task depended on context it never received. | Write tasks that stand alone: goal, input paths, output path, and what done means. |
</Accordion>

Roster, depth, and thread limits are in [Limits](/recursion/limits).

## Next steps

<CardGroup>
  <Card title="Teams" href="/recursion/teams">
    Run copies of an agent on a shared task board.
  </Card>

  <Card title="Deliverables and artifacts" href="/recursion/artifacts">
    Where children write outputs and how they're kept.
  </Card>

  <Card title="Research team tutorial" href="/recursion/use-cases/research-team">
    Run a research team end to end.
  </Card>

  <Card title="Vaults" href="/recursion/vaults">
    Grant credentials to the root and to roster agents.
  </Card>
</CardGroup>


# Organizations and roles
Source: https://docs.labelbox.com/recursion/organizations-and-roles

How tenants, organizations, and roles decide what each person and API key can see and do in Managed Agents.

Everything you create in Managed Agents belongs to one organization, and your role decides what you can do there. This page explains the model, so you can predict exactly what a person or an API key can reach.

## Tenants and organizations

* **Tenant.** The top level of an account. The first time you sign in, Recursion creates a tenant for you and makes you its **primary owner**.
* **Organization.** A separate space inside a tenant. Agents, environments, sessions, vaults, skills, and tags each belong to exactly one organization and are never visible from another.
* **Default organization.** Every tenant has one, named **Default**. Every member of the tenant can reach it, so it has no member list of its own.
* **Other organizations.** Tenant admins and owners create more under **Settings › Organizations**, up to 100 active ones per tenant, and add tenant members to each with a role.

The console works in one organization at a time. Pick it from the organization menu at the top of the sidebar. API keys follow the same model: an organization-scoped key works in its own organization, and a tenant-scoped key names one with `x-organization-id`, using an organization id or `default`. See [API keys](/recursion/api-keys).

## Roles at a glance

Each member has one **tenant role**. It decides what they can do in the Default organization and across the tenant:

* **User** views agents, sessions, and their results, and creates their own API keys.
* **Developer** builds and runs agents: everything a User can do, plus creating, changing, and deleting them. A Developer doesn't manage people or tenant settings.
* **Billing** pays for the tenant: credits, cards, and invoices. A Billing member's console shows **Billing** and **Settings**, with no Managed Agents access.
* **Admin** runs the tenant: everything a Developer can do, plus members, organizations, tenant settings, Domains, Enterprise SSO, Billing, and the audit log.
* **Owner** is an Admin who can also manage other Admins and Owners.
* **Primary owner** is the one Owner the tenant was created for, and the only person who can transfer ownership.

In other organizations an admin can give a member a different role there; see [Roles in each organization](#roles-in-each-organization).

## What each role can do

| Task | User | Developer | Billing | Admin or Owner |
| - | :-: | :-: | :-: | :-: |
| **Managed Agents** | | | | |
| View agents and sessions | ✓ | ✓ | – | ✓ |
| Build and run agents | – | ✓ | – | ✓ |
| Connect integrations | – | ✓ | – | ✓ |
| Own [API keys](/recursion/api-keys) | ✓ | ✓ | – | ✓ |
| **People and organizations** | | | | |
| Invite and manage members | – | – | – | ✓ |
| Manage Admins and Owners | – | – | – | Owner only |
| Organizations | – | – | – | ✓ |
| **Tenant settings** | | | | |
| Rename the tenant and require multi-factor authentication | – | – | – | ✓ |
| [Domains](/recursion/domains) | – | – | – | ✓ |
| [Enterprise SSO](/recursion/enterprise-sso) | – | – | – | ✓ |
| [Billing](/recursion/billing) | – | – | ✓ | ✓ |
| [Audit log](/recursion/console) | – | – | – | ✓ |

**Build and run agents** means creating, changing, and deleting agents, environments, vaults, and skills, and starting sessions. **Invite and manage members** covers User, Developer, and Billing members; only Owners and the primary owner manage Admins and Owners.

The Managed Agents rows apply in the Default organization. In other organizations, the role you have there decides them; see [Roles in each organization](#roles-in-each-organization).

The primary owner is the person the tenant was created for, until they transfer ownership. There's exactly one, and nobody can remove or demote them, including themselves. On **Members**, they can transfer ownership to another member. They become an owner, and that member becomes the primary owner. The page warns them first: they cannot undo it, and only the new primary owner can transfer it again.

## Roles in each organization

Your role in each organization comes from your tenant role:

| Tenant role | In the Default organization | In other organizations |
| - | - | - |
| **User** | Organization user | The role an admin gave them there, or no access |
| **Developer** | Organization developer | The role an admin gave them there, or no access |
| **Billing** | Organization billing | Organization billing, or Organization admin if an admin made them one |
| **Admin**, **Owner**, **Primary owner** | Organization admin | Organization admin |

An Organization admin can do everything an Organization developer can, and also manages that organization's members. An Organization user can only read, and an Organization billing member has no Managed Agents access.

<Accordion title="What read, create, and update cover">
  * **Read** covers listing and viewing agents, sessions, transcripts, events, environments, vaults (never secret values), cost, and [Analytics](/recursion/analytics).
  * **Create** covers creating resources and starting sessions. Testing an environment's setup script also needs create, because it provisions compute.
  * **Update** covers changing resources that exist, such as publishing a new agent version or cancelling a session.
  * **Moving a session needs create and update.** [Moving a session to another agent version](/recursion/sessions#move-a-session-to-another-version) can give it the vaults a new session on that version would receive, so it needs both. Organization developers and admins have both. Reading a session's version state needs only read.
  * **Testing a connection needs create too.** Testing an MCP server (the server's test in the console, or `probeMcpServer`) and **Test connection** on an integration are open to Organization developers and admins, not to Organization users.
</Accordion>

The role is checked on every request, for people in the console and for API keys alike. A request your role doesn't allow returns `403 forbidden`. A request for an organization you can't reach returns `404 not_found`, so its existence isn't revealed.

## What can go wrong

The most common problems:

* **`403 forbidden` on create, update, or delete.** The role is **Organization user**, which can only read. Use an account or key with the developer or admin role.
* **`403 forbidden` on every Managed Agents request.** The role is **Organization billing**, which has no Managed Agents access. Use an account or key with the user, developer, or admin role.
* **`404 not_found` naming an organization.** The `x-organization-id` header names an organization you can't reach, an archived one, or a different one than an organization-scoped key's. Send an organization you belong to, or `default`, or omit the header for an organization-scoped key. See [API keys](/recursion/api-keys).

<Accordion title="Every role and organization error">
  | Code or symptom | Cause | Fix |
  | - | - | - |
  | `403 forbidden` on create, update, or delete | The role is **Organization user**, which can only read. | Use an account or key with the developer or admin role. |
  | An MCP server test says **Testing requires create access.** | The role is **Organization user**. | Ask an Organization developer or admin to run the test, or to change your role. |
  | `403 forbidden` on every Managed Agents request | The role is **Organization billing**, which has no Managed Agents access. | Use an account or key with the user, developer, or admin role. |
  | `404 not_found` naming an organization | The `x-organization-id` header names an organization you can't reach, an archived one, or a different organization than an organization-scoped key's. | Send an organization you belong to, or `default`, or omit the header for an organization-scoped key. See [API keys](/recursion/api-keys). |
</Accordion>

## Next steps

<CardGroup>
  <Card title="API keys" href="/recursion/api-keys">
    Create a key scoped to one organization or the whole tenant.
  </Card>

  <Card title="Billing" href="/recursion/billing">
    Buy prepaid credits and manage cards.
  </Card>

  <Card title="Security" href="/recursion/security">
    See how organizations, sandboxes, and credentials are isolated.
  </Card>

  <Card title="Console" href="/recursion/console">
    Find the screen for each task.
  </Card>
</CardGroup>


# Write outcome rubrics
Source: https://docs.labelbox.com/recursion/outcomes

Write a rubric a grader can check, launch a session against it, and read the verdict for each criterion.

An outcome is a session's definition of done: an objective and a rubric. Each time the agent finishes its work, a grader checks it against every criterion in the rubric. It either accepts the work or sends the agent back with what fell short, and the agent keeps revising until the rubric is met or the passes run out.

Use an outcome when "the agent stopped" isn't proof enough that the work is right. How well this works depends mostly on the rubric, so this page starts with how to write one.

<Note>
  Launching a graded session and saving an agent's default rubric need the Developer or Admin role in the organization. The User role can read outcomes. See [Organizations and roles](/recursion/organizations-and-roles). You need an [agent](/recursion/agents) to do the work and an [environment](/recursion/environments).
</Note>

## Write a rubric

A rubric is Markdown. Each list item is one criterion. The grader owes a verdict on every list item and on nothing else.

| You write | How it's read |
| - | - |
| A list item (`-`, `*`, `+`, `1.`, or `1)`) | One criterion. A task box in front of the text, as in `- [ ] The file exists`, is dropped and the text is kept. Indented items are criteria too. |
| A heading (`#`, `##`, and so on) | The section for the criteria below it. The agent sees its criteria grouped by section, and each criterion in the result shows its section. |
| A paragraph | Context. It isn't scored. The grader reads it; the agent doesn't receive it. |
| A fenced code block | Ignored, including list items inside it. |
| A horizontal rule (`---`) | Ignored. |

```markdown theme={"theme":"css-variables"}
# Release notes for v4.12.0

## Deliverable
- The notes are saved as a deliverable named `release-notes.md`.

## Coverage
- Every pull request merged between the tags v4.11.0 and v4.12.0 appears exactly once.
- No entry describes a pull request outside that range.

## Format
- Breaking changes are listed first, under a heading named "Breaking changes".
- Each entry is one sentence and ends with the pull request number in parentheses.
- No entry contains an internal tracker id.
```

This rubric has six criteria in three sections. Every criterion counts the same: you can't set weights, and a strong result on one criterion never makes up for a failure on another.

### Make each criterion stand on its own

The agent receives the criteria, grouped by section, and nothing else from the rubric. The grader receives the objective and the whole rubric, but it owes one verdict for each criterion, so each criterion has to carry everything needed to meet it and to check it.

* **Make it checkable.** "The CSV has a numeric `price` column" can be checked. "The data looks good" can't, and a criterion like that sends the work back until the passes run out.
* **Give each list item one requirement.** When one item holds two requirements, a failure doesn't say which one the work missed.
* **Keep each criterion on one line.** Text on a second line isn't part of the criterion.
* **Say what counts as evidence in the criterion.** Write "The test suite passes when run with `make test`", not "Tests pass" followed by a paragraph about how to run them.
* **Ask for work that leaves evidence.** A file, a passing test, or a record in another system can be verified. A claim in the agent's last message can't.
* **Put shared background in the objective.** The opening message is the objective, and both the agent and the grader read it. Facts both need, such as which repository to use, belong there.
* **Start from a good example.** Take a result you would accept, list what makes it good, and turn each point into a criterion.

### Name the deliverable

Only files the agent saves as deliverables are kept with the session. Agents already know where to save them, so you don't need to give a path. Name the file in the opening message and in the rubric, for example "Save the notes as `release-notes.md`" and "- The notes are saved as a deliverable named `release-notes.md`." Deliverables are saved before every grading pass, so the grader checks the version the agent just finished. See [Deliverables and artifacts](/recursion/artifacts).

### Name your criteria

A list item written as `- key: text` gives the criterion a name:

```markdown theme={"theme":"css-variables"}
- breaking-first: Breaking changes are listed first, under a heading named "Breaking changes".
- one-sentence: Each entry is one sentence and ends with the pull request number in parentheses.
```

A key starts with a lowercase letter and uses lowercase letters and digits, with single hyphens or underscores between them. A name is for the people who write and review the rubric. The agent, the grader's verdicts, and the result show the text after the colon without the name, so that text has to make sense by itself. Name every criterion or none: a rubric that mixes named and unnamed items is refused.

<Warning>
  A list item that starts with a lowercase word, a colon, and a space is read as a named criterion. `- output: saved as report.csv` gets the name `output`, and the agent is shown "saved as report.csv" without it. If your other items have no name, the rubric is refused. To keep an item unnamed, start it with a capital letter or reword it.
</Warning>

<Accordion title="Rules a rubric must pass">
  A rubric is refused when:

  * It has no list item, or more than 200.
  * A list item is empty, such as a lone `-` or `- [ ]`, or a named item has no text, such as `- todo:`.
  * A line starts with a dash and no space, such as `-The file exists`.
  * It mixes named and unnamed criteria, uses the same key twice, or has a key longer than 64 characters.
  * It is larger than 256 KiB, or one criterion is larger than 4 KiB.
</Accordion>

## Who grades

Grading runs inside the session, and the rubric is all you provide. When the agent finishes, a grader starts as a subagent of the session, in the same sandbox, with the session's model, tools, and credentials.

* **It judges only against the rubric.** The grader has its own transcript. It reads the objective and the rubric, and it can't see the agent's conversation or reasoning.
* **It checks the real work.** It can open files, run the project's tests, and check systems the agent changed. It works in the agent's own sandbox, not a copy, and it's instructed not to modify files. Don't write a criterion that asks the grader to change something.
* **Its guidance comes from the rubric.** How strict to be, what counts as evidence, and what to ignore belong in the rubric's paragraphs, which the grader reads.

## Grade a session against a rubric

<Steps>
  <Step title="Open the launch form">
    In the sidebar, click **Sessions**, then click **Launch session**. In **Launch a session**, choose the **Agent** and **Environment**.
  </Step>

  <Step title="Turn on grading">
    Under **Task**, select **Grade this session against a rubric**. It's already selected when the agent has a stored rubric.
  </Step>

  <Step title="Write the rubric">
    Write your criteria in **Rubric**. The field starts with the agent's default rubric, or with a short template. Replace the template's lines with your own. The paragraphs under its `## Evidence` and `## Ignore` headings are notes, not criteria.
  </Step>

  <Step title="Write the objective and launch">
    Write the task in **Opening message**. It's the objective the grader checks the work against, and a graded session needs one. Click **Launch session**.
  </Step>
</Steps>

Grading is on from the first turn. A grading pass runs inside the session, so the grader's model use and its time in the sandbox are part of the session's cost. See [Pricing](/recursion/pricing).

## Grade every session of an agent

Store a rubric on the agent, and its sessions are graded without anyone turning grading on.

1. Open the agent and stay on the **Configuration** tab.
2. In **Outcome**, click **Add** if the section is closed.
3. Write the rubric in the text box.
4. Click **Save new version**.

The rubric is saved with that version. A session that runs a version with a rubric and starts with an opening message is graded against it, however the session starts: from the launch form, through the API, or by an automation or Slack. The opening message is the objective, so a session that starts without one runs ungraded.

<Warning>
  A stored rubric also grades sessions nobody launches by hand. Runs of an automation that uses the agent are graded, for up to five passes, and each pass adds to that session's cost. See [Pricing](/recursion/pricing).
</Warning>

In **Launch a session**, an agent with a stored rubric has **Grade this session against a rubric** already selected and shows the rubric. Clear the box to run that session ungraded. An edit to the rubric in the form applies to that session only.

An automation keeps the agent version it was saved with, and its runs are graded against that version's rubric. A version saved with the **Outcome** section empty runs ungraded. An older version that has no rubric setting at all is graded against the agent's current rubric. After you add or change a rubric, [move the automation to the new version](/recursion/automations#keep-the-agent-version-current) so you know which rubric its runs use.

## Read the result

1. In the sidebar, click **Sessions** and open the session.
2. Read the **Outcome** line above the transcript: the result so far, the number of unmet criteria if any, and the objective.
3. Click **Details** to see the objective, the pass limit, the **Rubric**, and one block per pass, newest first. Each pass shows how many criteria were met and every criterion with its verdict. A failed criterion also shows the grader's reason.
4. To read what the grader checked, open its **Grader** lane in the session. Each pass has its own.

| The outcome shows | When | What happens next |
| - | - | - |
| **Pending** | No pass has returned a verdict yet. The agent is still working, the first pass is running, or a pass returned no result. | A running pass ends with a verdict. Otherwise the next pass starts when the agent finishes. |
| **Running** | A pass sent the work back. The agent is revising, or the next pass is running. | The next pass ends with a verdict. |
| **Satisfied** | No criterion failed and at least one passed. | Grading ends with the work accepted. |
| **Max iterations reached** | A criterion still failed on the last pass. | Grading ends. The last pass keeps its verdicts. |
| **Not applicable** | The grader judged every criterion not applicable. | Grading ends. Check that the rubric fits the task. |
| **Failed** | The work couldn't be graded. | Grading ends. Open **Details** to read the reason. |

After a final result, the session still accepts messages, but those turns aren't graded. To grade more work, launch a new session.

## How grading works

```mermaid theme={"theme":"css-variables"}
flowchart TD
  work["Agent works"] --> done["Turn ends"]
  done --> grade["Grader checks the rubric"]
  grade -->|"a criterion failed"| revise["Agent revises"]
  revise --> done
  grade -->|"none failed"| accepted["Outcome satisfied"]
```

1. The agent works until it ends a turn with nothing left to do. A turn you interrupt or cancel isn't graded.
2. The agent's deliverables are saved, and a **grading pass** starts. The grader runs inside the session, in the agent's own sandbox, with the same tools and credentials. It can open files, run the project's tests, and check systems the agent changed. It can't read the agent's transcript.
3. The grader gives every criterion one verdict: **Pass**, **Fail**, or **Not applicable**. One **Fail** is enough to send the work back.
4. The agent receives the whole scorecard, with the grader's reason for each failed criterion, and revises. Then the next pass starts with a new grader.

For a session launched from the console, grading stops after five passes, the first one included. The outcome is **Satisfied** when no criterion failed and at least one passed.

## What can go wrong

The most common problems:

* **The work keeps coming back, then the outcome reads Max iterations reached.** A criterion is vague or can't be met. Read the grader's reason on the failed criterion, rewrite it so it can be checked, and launch again.
* **A deliverable criterion fails.** The agent saved the file under another name, or didn't save it as a deliverable. Name the file in both the opening message and the rubric.
* **A session wasn't graded.** The box was cleared in the launch form, or the session started with no opening message. For a session an automation started, the automation may run an agent version saved without the rubric: move it to the new version.
* **The transcript says the outcome check returned no result.** The grader didn't return a usable verdict, so the work wasn't judged. Send a message to continue. Grading runs again when the agent next finishes.

## Limits

| Limit | Value |
| - | - |
| Criteria in one rubric | 1 to 200 |
| Rubric size | 256 KiB |
| One criterion's text | 4 KiB |
| Grading passes for one outcome | 5 for a session launched from the console, the first one included |
| Criterion weights | None. Every criterion counts the same. |

Other limits are on [Limits](/recursion/limits).

## Next steps

<CardGroup>
  <Card title="Deliverables and artifacts" href="/recursion/artifacts">
    Ask for named deliverables the grader can check.
  </Card>

  <Card title="Sessions" href="/recursion/sessions">
    Launch, steer, and follow a session while it's graded.
  </Card>

  <Card title="Agents" href="/recursion/agents">
    Store a default rubric on an agent.
  </Card>

  <Card title="Pricing" href="/recursion/pricing">
    See what a session hour covers.
  </Card>
</CardGroup>


# Pricing
Source: https://docs.labelbox.com/recursion/pricing

What a session hour costs, what that price covers, and how prepaid credits pay for it.

Recursion charges by the session hour. One price covers everything a session uses, the sandbox it runs on and its model spend, and sessions are paid from a prepaid credit balance in US dollars. There are no plans, seats, or subscription fees.

Before you launch, every agent shows its estimated price per session hour, so you can decide what to run before anything starts.

<Note>
  Anyone who can open an agent can read its estimate. Buying the credits sessions run against needs the **billing**, **admin**, **owner**, or **primary owner** tenant role. See [Organizations and roles](/recursion/organizations-and-roles) and [Billing](/recursion/billing).
</Note>

## What a session hour covers

A session hour is one hour of an agent session. Its price covers the two things a session uses:

* **Compute**: the sandbox the agent runs on, sized by its CPU, memory, and any GPU.
* **Model spend**: the model's reading, thinking, and output, including tools the model provider runs, such as the provider's own web search, and the web searches and page reads the agent makes, charged per use.

Both are in the session-hour price, and nothing is billed on top of it.

One session's cost covers its whole session tree: the session and every subagent and teammate it started. A team therefore costs more per hour than a single agent. See [Multi-agent](/recursion/multi-agent) and [Teams](/recursion/teams).

### What isn't charged

* **Storage.** Transcripts and session files aren't charged. They also aren't stored permanently and are cleaned up over time, so download anything you want to keep long-term. See [Artifacts](/recursion/artifacts).
* **Saving agents and environments.** Saving uses no compute, so it works even when your balance is empty.

## What changes the price

* **Your setup.** A bigger sandbox or a more capable model costs more per hour. Lighter setups cost less.
* **How busy the agent is.** An hour in which the agent works the whole time costs more than one in which it mostly waits for you. That is why the estimate is a range rather than one figure.

Where your model usage is billed by your own model provider, the estimate says so, and the session-hour price covers compute only.

## Compare models and machines

What a session hour costs for each model and machine, from a light hour, when the agent mostly waits, to a busy one. Prices differ by whether the agent works alongside you or on its own.

<SessionHourPrices />

The table lists the models with a published baseline. For any other model, open an agent in the console to see its estimate. Once you run an agent, its own estimate reflects how that agent actually works, so it can be lower or higher than this table.

## See the estimate before you launch

<Tabs>
  <Tab title="Console">
    <Steps>
      <Step title="Open the agent">
        In **Agents**, open the agent you want to price. Its model has to be set; until it is, the estimate asks you to pick one.
      </Step>

      <Step title="Choose the compute to price on">
        In **Estimated cost per session hour**, set **Price on compute** to a preset such as **Standard · 2 CPU / 4 GiB**, a GPU, or one of your environments.
      </Step>

      <Step title="Read the estimate">
        You get the typical price per session hour, the typical range, and what the hour costs if the agent stays busy throughout.
      </Step>
    </Steps>
  </Tab>
</Tabs>

If you manage billing and your balance covers less than about an hour of the agent's typical cost, **Launch session** tells you how many minutes it covers and links to **Buy credits**. You can still launch.

<Accordion title="What each figure means">
  | Figure | Meaning |
  | - | - |
  | The figure per session hour | What an hour of this agent typically costs on the compute you selected. |
  | **Typical** | The range most hours land in. |
  | **up to ... if busy all hour** | Compute plus the upper model rate, for an hour in which the agent never waits. |

  The estimate is drawn from how comparable sessions have actually run, so it sharpens as you run more. A model with no price history shows a **from** figure covering compute only. A GPU with no price is left out of the estimate, which understates the cost.
</Accordion>

## Pay with prepaid credits

Sessions run against your credit balance.

* **Prepaid, in US dollars.** Credits belong to your tenant as a whole, not to one organization, and purchased credit doesn't expire.
* **Bought by card.** Choose $50, $100, $250, or $500, or enter any whole-dollar amount from $5 to $1,000. Payments go through Stripe, and every purchase has a receipt.
* **Who can buy.** Members with the billing, admin, owner, or primary owner tenant role. Buying is a console action, not an API-key operation.
* **Before your first session.** A new tenant has no credit, so buy credits before you launch.

[Billing](/recursion/billing) has the purchase flow, card management, and every purchase status.

## When credits run out

When your balance reaches zero, new sessions, messages that start a new turn, and setup tests are refused until you buy credits. A session that is already running, or a sleeping one that wakes on its own, pauses before its next model request. Requests already in flight complete and are charged, so the balance can end slightly below zero. A paused session keeps its sandbox until the environment's idle stop, and that compute is billed too. Buy credits, then send the paused session a message to continue it.

See [When credits run out](/recursion/billing#when-credits-run-out) for what each role sees.

## Track what you spent

**Sessions** shows what each run cost, and `getSession` returns `costUsd` with a `costState` of `so_far` or `final`. Costs can take up to an hour to settle, and a sandbox that keeps running after its session ends is charged until the environment's idle stop. See [Usage and cost](/recursion/usage-and-cost).

## Common questions

<AccordionGroup>
  <Accordion title="What is a session hour?">
    One hour of an agent session. Its price covers the compute and model spend the session uses in that hour.
  </Accordion>

  <Accordion title="Why do two agents cost different amounts per hour?">
    Because the price follows the setup. An agent on a larger sandbox or a more capable model costs more per hour than a lighter one.
  </Accordion>

  <Accordion title="Why is the estimate a range?">
    Because an hour costs more when the agent is working than when it is waiting. The range shows where typical hours land, and the upper figure shows an hour in which the agent works the whole time.
  </Accordion>

  <Accordion title="Am I charged while the agent is idle?">
    Model spend, no. Compute, yes, while the sandbox is up: until the environment's idle stop, or until you close the session.
  </Accordion>

  <Accordion title="Do I pay separately for storage, compute, or model usage?">
    No. Compute and model spend make up the session-hour price, and storage isn't charged. Nothing is billed on top.
  </Accordion>

  <Accordion title="Can I cap what a session spends?">
    Your credit balance is the cap. Sessions pause when it reaches zero, and **Launch session** warns you when the balance covers less than about an hour of the agent's typical cost.
  </Accordion>

  <Accordion title="How do I add credit?">
    Buy credits on **Billing**. Once the balance is positive, you can start new sessions and continue paused ones straight away.
  </Accordion>
</AccordionGroup>

## Limits

| Limit | Value |
| - | - |
| One purchase | $5 to $1,000, whole dollars |
| Time for a session's cost to settle | Up to an hour |

Every other limit is on [Limits](/recursion/limits).

## Next steps

<CardGroup>
  <Card title="Billing" href="/recursion/billing">
    Buy credits, manage cards, and keep receipts.
  </Card>

  <Card title="Usage and cost" href="/recursion/usage-and-cost">
    Read what each session actually cost.
  </Card>

  <Card title="Environments" href="/recursion/environments">
    Size the compute a session-hour price is based on.
  </Card>

  <Card title="Limits" href="/recursion/limits">
    Check quotas and request limits.
  </Card>
</CardGroup>


# Quickstart
Source: https://docs.labelbox.com/recursion/quickstart

Create an API key, an environment, and an agent, start a session with a task, and read the agent's reply, in the console or with cURL.

This quickstart runs the whole flow once: an API key, an environment, an agent, and a session that writes a short file. Each step shows the console and the matching cURL request, so you can follow either path. It takes about 10 minutes.

<Note>
  Sign in to the [Recursion console](https://recursion.labelbox.com). If this is your first sign-in, Recursion creates your tenant and its **Default** organization. You need the organization developer or admin role to create agents and environments and to start sessions; see [Organizations and roles](/recursion/organizations-and-roles). Your tenant needs credits to start a session. If it has none, a billing manager can add them on **Billing**; [Pricing](/recursion/pricing) explains what a session hour costs. Creating the agent and environment does not start paid work.
</Note>

The sign-in page links to the [Terms of Service](https://docs.labelbox.com/page/terms-of-service), [Privacy Policy](https://docs.labelbox.com/page/privacy-notice), [Copyright Dispute Policy](https://docs.labelbox.com/page/copyright-dispute-policy), and [Fees Policy](/recursion/pricing).

## 1. Create an API key

Skip this step if you use only the console.

1. In the [Recursion console](https://recursion.labelbox.com), open **Settings** and choose **API keys**, then click **Create key**.
2. Enter a **Name**, such as `quickstart`, and keep the default **Expires** value.
3. Set **Scope** to the **Default** organization, then click **Create key**.
4. Copy the key. It starts with `rma_` and is shown only once.

Store it in an environment variable for the requests below:

```bash theme={"theme":"css-variables"}
export RECURSION_API_KEY='rma_...'
```

An organization-scoped key needs no other header. See [API keys](/recursion/api-keys) for scopes and rotation.

## 2. Create an environment

An environment is where the session's sandbox runs. This one uses the default compute and has no internet access, which is enough for a task that only writes a file.

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Environments**, then click **Create environment**.
    2. Enter a **Name**, such as `quickstart`.
    3. Click **Create environment**.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/environments' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      -d '{"name": "quickstart", "provider": "runs"}'
    ```
  </Tab>
</Tabs>

The response is the environment. Save its `environment_id`. Some fields are left out here:

```json theme={"theme":"css-variables"}
{
  "organization_id": "org_01a08a705220724f9a2fe1bcd8638c9d",
  "environment_id": "4c8e2a61-9b3f-4d70-a5e2-1f6b8c9d0e47",
  "name": "quickstart",
  "scope": "organization",
  "provider": "runs",
  "computer_use": false,
  "created_at": "2026-09-25T14:02:11Z",
  "updated_at": "2026-09-25T14:02:11Z"
}
```

See [Environments](/recursion/environments) to add internet access, a setup script, or more compute.

## 3. Create an agent

An agent is a model with a system prompt, saved as a version that sessions run.

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Agents**, then click **Create agent**.
    2. Under **Choose a starting point**, click **Blank**.
    3. Under **General**, enter a **Name**, choose a **Model** such as Claude Sonnet 5, and write a **System prompt**, such as `You are a concise writing assistant.`
    4. Click **Create agent**.
  </Tab>

  <Tab title="cURL">
    List the models you can use, and pick one's `modelId`:

    ```bash theme={"theme":"css-variables"}
    curl 'https://api.recursion.labelbox.com/managed-agents/v1/models' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```

    Create the agent with it in place of `<model-id>`:

    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/agents' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      -d '{
        "name": "Quickstart writer",
        "model": "<model-id>",
        "system": "You are a concise writing assistant."
      }'
    ```
  </Tab>
</Tabs>

The response is the agent with its first version. Save its `agent_id`. Some fields are left out here:

```json theme={"theme":"css-variables"}
{
  "agent_id": "5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55",
  "latest_agent_version_id": "c1a94e07-2f6b-4d18-b3a5-0e7d8c6f4a21",
  "name": "Quickstart writer"
}
```

Your organization can use only the models that list returns. See [Agent setup](/recursion/agents) to add tools, skills, and credentials.

## 4. Start a session

A session runs the agent in the environment on the task in its opening message.

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Sessions**, then click **Launch session**.
    2. Under **Configuration**, select your **Agent** and **Environment**.
    3. Under **Task**, write the **Opening message**: `Write a four-line poem about sandboxes and save it as poem.md.`
    4. Click **Launch session**. The session opens while its sandbox starts.
  </Tab>

  <Tab title="cURL">
    Replace the ids with the ones from steps 2 and 3.

    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/sessions' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      -H 'Idempotency-Key: quickstart-session-1' \
      -d '{
        "agent_id": "5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55",
        "environment_id": "4c8e2a61-9b3f-4d70-a5e2-1f6b8c9d0e47",
        "message": "Write a four-line poem about sandboxes and save it as poem.md."
      }'
    ```
  </Tab>
</Tabs>

The request returns `202 Accepted` right away. The agent works in the background:

```json theme={"theme":"css-variables"}
{
  "session_id": "e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53",
  "status_path": "/managed-agents/v1/sessions/e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53"
}
```

`Idempotency-Key` is required, so a retried request never starts a second session. Use a new key for each new session. If launch is refused because no credits are available, ask someone who manages billing to add credits, then try again. Your agent and environment remain saved.

## 5. Read the reply

<Tabs>
  <Tab title="Console">
    Read the **Transcript** as the agent works. It shows messages, tool calls, and results. The session header shows its status and cost. Open the **Files** tab to preview or download `poem.md`.
  </Tab>

  <Tab title="cURL">
    Stream the agent's messages as they arrive. The stream closes when the session finishes.

    ```bash theme={"theme":"css-variables"}
    curl -N 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53/events/stream?types=agent.message' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```

    Or list every recorded event after the session finishes:

    ```bash theme={"theme":"css-variables"}
    curl 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53/events' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

In the event list, the agent's reply is a `message` event with `role` set to `assistant`. Some fields are left out here:

```json theme={"theme":"css-variables"}
{
  "events": [
    {
      "event_id": "01997a3d-0b82-7c15-9a43-6e1f2d8c5b09",
      "event_type": "message",
      "role": "assistant",
      "session_id": "e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53",
      "content": {
        "blocks": [{ "type": "text", "text": "I saved the poem as poem.md." }]
      },
      "created_at": "2026-09-25T14:05:40Z"
    }
  ],
  "next_page_token": ""
}
```

When the agent finishes, the session's `status` is `completed` and `stop_reason` is `end_turn`. To continue the work, send a follow-up message. See [Sessions](/recursion/sessions) and [Events](/recursion/events).

<Tip>
  In the console, you can also start from a template. On **Agents**, click **Create agent** and choose **Issue fixer** or **PR reviewer**. Enable the template's required tools on the organization GitHub connection, then choose the agent's tools and repositories. The template creates the agent and a matching environment, and opens the launch form with a suggested task.
</Tip>

## Next steps

<CardGroup>
  <Card title="Build in the console" href="/recursion/console">
    Learn where agents, environments, credentials, and sessions live.
  </Card>

  <Card title="API conventions" href="/recursion/api">
    Learn the shared rules for retries, paging, and errors.
  </Card>

  <Card title="Connect your apps" href="/recursion/integrations">
    Let agents work in GitHub, Slack, and other apps.
  </Card>

  <Card title="Automate sessions" href="/recursion/automations">
    Run the same agent on a schedule or when an event arrives.
  </Card>
</CardGroup>


# Event types and states
Source: https://docs.labelbox.com/recursion/reference

Look up every session status, execution state, stop reason, event type, and setup-run state in Managed Agents.

Use this page to decode the values you read from sessions, events, and setup runs. For how these fit together, see [How it works](/recursion/how-it-works#the-session-lifecycle).

## Session status

`status` is the coarse state of a session.

| `status` | Meaning |
| - | - |
| `active` | The session can work or accept another turn. |
| `awaiting_human` | The agent asked a person to handle a step in its browser display, which needs computer use. It resumes when you send a `handoff_resolved` event, or on its own when the hand-off deadline passes, 30 minutes by default. |
| `completed` | The work finished. A follow-up message on a root session starts a new turn. |
| `failed` | The work stopped on an error. Read `failure`. A follow-up message on a root session starts a new turn. |
| `cancelled` | The session was hard-stopped and its sandbox torn down. A follow-up message on a root session starts a new turn in a new sandbox. |

## Execution state

`execution_state` says what the agent loop is doing right now. It's the authoritative signal: the event stream is written separately and can lag it.

| `execution_state` | Meaning |
| - | - |
| `queued` | The agent is at its `max_concurrent_sessions` cap. The session waits for a slot. |
| `provisioning` | The sandbox is being prepared. Normally under a minute, and never more than 5 minutes before the session fails with `sandbox_provision_timeout`. |
| `running` | The agent or one of its tools is working. |
| `idle` | Nothing is running. `stop_reason` says why. |

## Stop reasons

`stop_reason` is set whenever `execution_state` is `idle`.

| `stop_reason` | Meaning | What to do |
| - | - | - |
| `end_turn` | The agent finished its turn with nothing left to do. This isn't proof the work is correct. | Read the transcript and the deliverables, then send guidance if needed. |
| `requires_action` | The agent is blocked on something only a person can settle, such as a repeated failure. | Fix the cause, then reply to continue. |
| `sleeping` | The agent chose to wait for a person, an external system, or a timer. It resumes on the next message or at `wake_at`. | Nothing, or send a message to wake it. |
| `awaiting_subagents` | A coordinating agent is waiting for delegated work to report back. | Nothing. If the delegated work goes silent for 30 minutes, the service tells the agent. See [Multi-agent](/recursion/multi-agent#how-delegation-works). |
| `out_of_credit` | Your tenant ran out of credit, so the session paused before its next model request. Requests already in flight for the tenant complete and are charged, so the balance can end slightly below zero. | Buy credits. A session paused in the last 72 hours continues on its own within about a minute; send a message to continue it sooner or after that. |
| `tenant_suspended` | Your account is suspended, so the session paused without finalizing its work. | Contact support to restore the account, then send a message to continue. |
| `interrupted` | A person interrupted the turn. | Send a message to continue. |
| `error` | The loop failed after its retries. The session stays resumable. | Read `failure`, then send a follow-up to retry. |

## Event types

Each event has an `event_type`. Events are append-only and ordered.

| `event_type` | What it records |
| - | - |
| `message` | A message from a person, the agent, or the system. |
| `tool_invocation` | A tool call the agent made, with its arguments. |
| `tool_result` | The result of a tool call. |
| `summary` | A summary of earlier work, kept so the agent retains context in long sessions. |
| `plan_update` | A change to the agent's current plan. |
| `session_status` | A change to `status` or `execution_state`, a warning, or a [move to another agent version](/recursion/sessions#move-a-session-to-another-version). |
| `artifact` | A file, image, or other output the agent produced. |
| `turn_committed` | A checkpoint at the end of a turn. |
| `thread_context_compacted` | The point where older context was summarized for future turns. The original events stay in the log. |
| `approval_request` | The agent asked a person to take over its browser display. The session shows `awaiting_human` until the hand-off ends. |
| `approval_decision` | How an `approval_request` was resolved. |
| `advisor_intervention` | The service noticed the session may be stuck and recorded a notice, with the events or child sessions that prompted it. Controlled by `multiagent.interventionist.mode`: `shadow` only records the event, `active` also sends the agent a notice it reads on its next turn, and `off` records nothing. Omitted, it uses `active`. See [Configure a roster](/recursion/multi-agent#configure-a-roster). |
| `board_update` | A change to a team's shared task board: a task posted, claimed, updated, or released, or a note or decision posted. Written to the root session. |

See [Events](/recursion/events) to list, stream, and resume events.

## Session failures

A failed session has a `failure` object.

| Field | Meaning |
| - | - |
| `code` | Stable failure code. See [Session failure codes](/recursion/errors#session-failure-codes). |
| `phase` | The stage that failed, such as `provisioning`, `setup`, `credentials`, or `model`. |
| `category` | `transient`, `caller_error`, or `internal`. Tells you who can act. |
| `retryable` | Whether a retry may succeed on its own. |
| `message` | A short, safe explanation. |
| `at` | When the failure was recorded. |

## Environment setup

### Setup verification

An environment's `setup_verification.status` says whether sessions can use it.

| `status` | Meaning |
| - | - |
| `not_applicable` | No setup script, so no test is needed. |
| `never` | A script exists but has never been tested. Sessions can't start yet. |
| `running` | A setup test is in progress, and no earlier test passed. Re-testing a `verified` environment keeps it `verified` until the new test finishes. |
| `verified` | The latest test passed. Sessions can start unless `stale` is `true`. |
| `failed` | The latest test failed. Fix the script and test again. |

`stale: true` means the environment changed after its last test, so the result no longer applies. Changing the name, description, or metadata doesn't make it stale.

### Setup-run status and phase

| `status` | Meaning |
| - | - |
| `queued` | Waiting to start. |
| `provisioning` | Compute is being prepared. |
| `running` | The script is running. |
| `succeeded` | The script passed. |
| `failed` | The run failed. Read `failure_code`, `phase`, and `stderr_tail`. |
| `cancelled` | The run was cancelled. |

`phase` is the stage the run reached: `provision`, `gpu_check`, `setup`, `profile`, `commit`, or `cleanup`. A failure in `setup` points at your script. A failure in another phase is usually temporary, so retry once before changing the script.

## Account states

| Resource | Values | Details |
| - | - | - |
| API key `status` | `active`, `inactive`, `expired`, `revoked` | [API keys](/recursion/api-keys) |

## Next steps

<CardGroup>
  <Card title="Errors" href="/recursion/errors">
    Every HTTP error code and session failure code.
  </Card>

  <Card title="Limits" href="/recursion/limits">
    Every size, count, and time limit.
  </Card>

  <Card title="Events" href="/recursion/events">
    List, stream, and resume session events.
  </Card>

  <Card title="Glossary" href="/recursion/glossary">
    Definitions of every term.
  </Card>
</CardGroup>


# Reference prior sessions
Source: https://docs.labelbox.com/recursion/referenced-sessions

Give a session read-only access to earlier sessions so its agent can search their transcripts, costs, and deliverables.

A referenced session is an earlier session that a new or running session may read. Pass its id in `referenced_session_ids`, and the agent gets read-only tools to search that session's transcript, review its cost, and open its deliverables. Use references to debug a failed run, pick up where earlier work stopped, or compare several attempts at the same task, without pasting transcripts into prompts.

<Note>
  Starting a session or sending it messages needs the organization developer or admin role. See [Organizations and roles](/recursion/organizations-and-roles) and [API keys](/recursion/api-keys). You must be able to read every session you reference, and it must be in the same organization as the session that reads it. References never give an agent access that you do not have. There is no console action to add a reference, so use the API; the console shows references that were added.
</Note>

## Start a session with references

Send `referenced_session_ids` in the start request. The agent can use the read tools from its first turn.

<Tabs>
  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/sessions' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Idempotency-Key: triage-failed-run-2026-09-17' \
      -H 'Content-Type: application/json' \
      -d '{
        "agent_id": "5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55",
        "environment_id": "9d3e7b52-1a4c-4f80-b6e9-2c8a5d0f7e13",
        "message": "The referenced session failed its test step. Find the first failing command, explain the cause, and propose a fix.",
        "referenced_session_ids": ["b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38"]
      }'
    ```
  </Tab>
</Tabs>

The API answers `202 Accepted` with the new session's id:

```json theme={"theme":"css-variables"}
{
  "session_id": "e3a1f7c2-9b4d-4e60-8c15-2d7f0a6b9e48",
  "status_path": "/managed-agents/v1/sessions/e3a1f7c2-9b4d-4e60-8c15-2d7f0a6b9e48"
}
```

Name the task in the message. A reference only makes the earlier session readable; the agent decides what to read based on what you ask.

## Add references to a running session

Send `referenced_session_ids` with a user message on `sendSessionEvents`. The reference is recorded before the message, so the turn that the message starts can already read it. `interruptAndSendSessionMessage` accepts the same field.

<Tabs>
  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/e3a1f7c2-9b4d-4e60-8c15-2d7f0a6b9e48/events' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      -d '{
        "events": [
          {
            "type": "user.message",
            "content": [{"type": "text", "text": "Compare your fix with the one in the second referenced session."}]
          }
        ],
        "referenced_session_ids": ["0c6f2b9e-4d17-4a38-b5e2-9f1a3c7d8e06"]
      }'
    ```
  </Tab>
</Tabs>

```json theme={"theme":"css-variables"}
{
  "ok": true,
  "delivery_state": "signaled",
  "events_accepted": 1
}
```

See [Session operations](/recursion/session-operations) for `delivery_state` and the other message options.

**What success means:** the session's timeline gets a status event that names the new references. In the console, open the session and look under **Referenced sessions** on the **Session** tab of the details panel.

<Accordion title="Reference status events">
  One new reference reads, for example, "This session was given read access to prior session b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38." A request that adds several says how many, such as "This session was given read access to 2 prior sessions." Naming a session that is already referenced adds nothing and records no event.
</Accordion>

## What the agent can read

When a session has at least one reference, its agent gets read-only tools to list the referenced sessions, outline, search, and read their events, and read their team boards, costs, deliverables, and skills. A session with no references does not have them. When a session has more than one reference, the agent names the one it wants on each call. `list_referenced_sessions` gives it the ids.

<Accordion title="Every read tool">
  | Tool | What it reads |
  | - | - |
  | `list_referenced_sessions` | The referenced sessions: each one's agent, status, and task, when it started, and when it was referenced. Marks any that can no longer be read. |
  | `session_overview` | Every member of the tree, including subagents and teammates: status, stop reasons, event and tool counts, and the tree's cost. |
  | `session_outline` | A compact timeline of the tree, up to 200 entries (80 by default). |
  | `search_events` | A case-insensitive text search over events, filtered by member session, tool name, or event type. Up to 50 matches (20 by default). |
  | `read_events` | A page of events in order, up to 40 at a time (20 by default). |
  | `get_event` | One event in full, with an offset for reading long content in pieces. |
  | `view_event_images` | An image attached to an event. |
  | `get_team_state` | A team's shared board and each member's scratchpad. |
  | `get_costs` | The tree's cost in US dollars: one total for every member's model calls, tools, and compute. |
  | `list_session_files` | The deliverables each member saved and the skills each member used. |
  | `read_session_file` | One deliverable, read in pieces for large files. |
  | `read_session_skill` | A file from a skill that a member used, exactly as that member saw it. |
</Accordion>

<Warning>
  Referenced transcripts can contain text written by users, tools, and websites. Treat what the agent concludes from them with the same care as any other untrusted input, and only reference sessions whose contents you are willing to show the agent.
</Warning>

## How references work

* **The whole tree is granted.** Each id grants the session tree it belongs to: the root session and every subagent and teammate under it. You can pass the id of any session in the tree.
* **All or nothing.** If you cannot read one of the ids, the whole request fails and nothing is recorded: no reference, no message, no new session.
* **Read-only and live.** The agent cannot change a referenced session. It reads current data, so a referenced session that is still running shows its latest events.
* **Checked on every read.** If a referenced session is deleted, or its access is restricted after it was added, the agent's reads of it stop working.
* **Inherited, never widened.** Subagents and teammates of the reading session can read the same sessions. They cannot add references of their own.
* **Kept for the session.** You can't remove a specific reference. Once a session holds 10, adding another drops the oldest. To drop a reference on purpose, start a new session.

## What can go wrong

The most common problems:

* **`404 not_found` on `referenced_session_ids`.** One of the ids does not exist, is in another organization, or you cannot read it. Nothing was recorded. Check each id and your API key's organization, remove ids you cannot read, then resend.
* **`400 invalid_request` with more than 10 ids.** One request can name at most 10 sessions. Send the most relevant 10, and start a separate session for the rest.
* **The agent says a reference can no longer be read.** The referenced session was deleted, or its access was restricted after it was added. Start a new session with a different reference.

<Accordion title="Every referenced-session error">
  | Symptom or code | Cause | Fix |
  | - | - | - |
  | `404 not_found` on `referenced_session_ids` | One of the ids does not exist, is in another organization, or you cannot read it. Nothing was recorded. | Check each id and your API key's organization. Remove ids you cannot read, then resend. |
  | `400 invalid_request` on `referenced_session_ids`, more than 10 ids | One request can name at most 10 sessions. | Send the most relevant 10. Start a separate session for the rest. |
  | `400 invalid_request` on `referenced_session_ids`, empty id | The list contains an empty string. | Remove it. |
  | `400 invalid_request` on `referenced_session_ids`, the session references itself | A session cannot reference its own tree. | Reference a different session. |
  | `400 invalid_request` on `referenced_session_ids`, other sessions | Session analyst conversations and sessions with restricted access cannot be referenced. | Reference the session you analyzed, not the analyst conversation. |
  | `403 forbidden` | Your role can only read sessions. | See [Organizations and roles](/recursion/organizations-and-roles). Use an account or key with the developer or admin role. |
  | The agent says a reference can no longer be read | The referenced session was deleted, or its access was restricted after it was added. | Start a new session with a different reference. |
  | The agent cannot find an early reference | The session holds at most 10 references. Adding more drops the oldest. | Keep the set small. Start a new session for a different set. |
  | The agent does not have the read tools | The session has no references. | Send `referenced_session_ids` with a message. |

  The full list of codes is in [Errors](/recursion/errors).
</Accordion>

## Limits

Duplicate ids, or two ids from the same tree, count once. A reference must be in the same organization. The caps of 10 ids per request and 10 session trees held per session are in [Limits](/recursion/limits).

## Next steps

<CardGroup>
  <Card title="Session operations" href="/recursion/session-operations">
    Send messages, interrupt, and resume sessions.
  </Card>

  <Card title="Usage and cost" href="/recursion/usage-and-cost">
    Compare what earlier sessions cost.
  </Card>

  <Card title="Deliverables and artifacts" href="/recursion/artifacts">
    See which files a session keeps.
  </Card>
</CardGroup>


# Security
Source: https://docs.labelbox.com/recursion/security

How Managed Agents isolates organizations and sandboxes, controls network access, decides which credential values agents can read, and handles your data.

Agents run code, reach the network, and use credentials on your behalf. This page describes the boundaries that contain them, what you control, and what your agents can and can't see.

## Organization isolation

* Every agent, environment, session, vault, skill, tag, and integration connection belongs to exactly one organization.
* Each request runs in one organization, chosen by your API key's scope or the `x-organization-id` header, and is checked against your membership on every request.
* A resource in an organization you can't reach returns `404 not_found`, the same as one that doesn't exist, so its existence isn't revealed.
* Your role is checked for every operation. A read-only role can't create, change, or delete anything. See [Organizations and roles](/recursion/organizations-and-roles).

## Authentication

* API keys are personal, act with your current role, and always expire, after 365 days at most.
* The full key is shown once and can't be retrieved later. Revoking a key takes effect immediately.
* If you're removed from the tenant, or a key's organization is archived, the key stops working.
* API keys can't create or revoke API keys or buy credits. Those are console actions that need a signed-in session. See [API keys](/recursion/api-keys).
* People at your DNS-verified domains can sign in through your own SAML identity provider, and you can require it for them. See [Enterprise SSO](/recursion/enterprise-sso).

## Sandbox isolation

* **One sandbox per session tree.** A root session and its child sessions share one sandbox. Other sessions can't see into it.
* **Setup tests carry no credentials.** A setup test runs without any session's credentials. When a session runs setup on its own compute, environment-variable secrets are removed from the script's shell, but a program the script starts can still read them. Only use setup scripts you trust.
* **No privileged containers by default.** Privileged Docker access is off unless you turn it on for an environment.
* **Sandboxes don't stay open forever.** A hard stop closes the sandbox. An idle sandbox stops after the environment's idle period and may later be released. After an idle stop or release, send another message to reopen the same session. Its workspace files can be used when the workspace reopens successfully.

## Network access

New environments have no internet access unless you allow it. Grant only the destinations the task needs.

* In the console, **Internet access** defaults to **No access**. **Enabled** allows unrestricted access.
* In the API, omitting `network_policy` stores a deny-all policy. Send `{}` for unrestricted access, or up to 200 ordered rules to allow specific destinations.

<Accordion title="Privileged Docker and concurrent changes">
  Privileged Docker requires unrestricted internet access (`network_policy: {}`). A nonempty policy and privileged Docker cannot be combined on create or when access settings change.

  Changing a stored network policy, or turning on privileged access, requires `expected_access` from your latest read, so two people can't silently overwrite each other's access settings. See [Environment reference](/recursion/environments-reference).
</Accordion>

Model calls, MCP servers, and the `web_search` and `web_fetch` tools don't go through the sandbox, so the network policy doesn't govern them; control them in the agent's configuration. A session started by an automation or from Slack in an environment with limited or no internet access gets no `web_fetch` tool, so event text can't use it to send data out. See [Tools](/recursion/tools).

## Credentials

Credentials live in vaults, and secret values are write-only. The API and the console show how a credential is stored, never its value. Stored secrets are encrypted at rest with a key specific to your organization.

What reaches the agent depends on the credential type:

```mermaid theme={"theme":"css-variables"}
flowchart TD
  vault["Vault"] --> bearer["MCP bearer token"]
  vault --> envvar["Environment-variable secret"]
  bearer -->|"added outside the sandbox"| mcp["MCP server"]
  envvar -->|"set as plaintext"| sandbox["Sandbox, shared by the tree"]
```

| Credential | Can the agent read the value? | Recommendation |
| - | - | - |
| MCP bearer token | No. It's added to requests to the MCP server and never enters the sandbox or the model's context. | Prefer this whenever the service speaks MCP. |
| Environment-variable secret | Yes. It's a plaintext environment variable in the sandbox, so any command any agent in the session tree runs can read it. The console asks you to acknowledge this before storing one. | Use only for tools that need a variable, with the narrowest-scoped token the service offers. |

Tool output masks an environment-variable secret's current value, but a transformed, truncated, very short or since-rotated value can still appear in output or the transcript.

You control what each session gets:

* An agent's default vaults apply to its sessions. A session can replace them, or narrow them to specific credentials with `credential_refs`.
* A subagent that's a different agent from the roster gets only its own default vaults, never its parent's. To share a credential, grant the same vault to both. See [Multi-agent](/recursion/multi-agent#what-a-child-inherits).
* Environment-variable secrets are the exception. They come only from the vaults granted to the root session, and every agent in the session tree, including roster agents, can read them, because the tree shares one sandbox.

## Integrations

Built-in integrations don't use vaults. An admin signs in to each app once on the **Integrations** page, and that sign-in is never returned by the API or shown in the console.

* **Scoped by the allow-list.** An agent reaches only the apps it's granted, and only the tools on each app's allow-list. Read-only tools can't change data; write and destructive tools can, so add them only when a task needs them.
* **Short-lived access in the sandbox.** A session gets access that works only for its granted apps and allowed tools, and expires within an hour. It's renewed while the session runs. Commands in the sandbox can use it, so an agent could read it, but it's no use outside that scope or after it expires.
* **Acts as one account.** Agents act in the app as the account that signed in. Choose an account whose own permissions fit the work.
* **Stops when you disconnect.** Disconnecting removes the app's sign-in, and no session gets new access through it. Access already issued lasts at most until it expires.

Native integrations such as GitHub, Slack, and Jira give sessions short-lived access bounded by the connection and agent grant. Standalone GitHub MCP requests use a token whose permissions come from the selected tools and repositories; sandbox `git` and `gh` receive no GitHub credential. See [GitHub access](/recursion/github) and [Integrations](/recursion/integrations).

## Automations and event sources

* **Deliveries are verified.** Slack and GitHub event sources check every delivery's signature. Connecting the account on **Integrations** creates and manages the signing secret, which never reaches a sandbox. A custom webhook source that doesn't verify signatures accepts any request that reaches its URL, so use one only for data that's safe to act on.
* **Event data is input, not instructions you wrote.** An event run's opening message includes the event as JSON after your prompt. Treat event text, such as a Slack message or an issue body, as untrusted, and grant automation sessions only what their task needs. In an environment with unrestricted internet access, that text reaches a session that can use the open internet.
* **A running session keeps a removed grant until you move it.** Removing a grant from the agent publishes a new version, and a session that is already running keeps its version. [Move the session](/recursion/sessions#move-a-session-to-another-version) to the new version to remove the grant; the session's events record the move and who asked for it. A session whose trigger set its own credentials keeps them through a move. An environment-variable secret stays in a running sandbox until that sandbox is replaced, so rotate the secret to revoke it.
* **Moving a session can add access.** A move gives the session the vaults and credentials the new version's defaults add, so it needs both create and update access, and agents can't move a session through the Recursion MCP tools.
* **Automations run a pinned agent version.** Removing a grant from the agent doesn't reach an automation until you move it to the new version.
* **Runs are recorded.** Every run keeps its frozen configuration, trigger, event, and opening message, including after the automation is deleted. A webhook run records only the sender's own headers.

## Data handling

* **Sessions keep their history.** Transcripts, events, sandbox state, and outputs are kept on the server so sessions can resume. Session messages and events are encrypted at rest with your organization's key. Because of this server-side state, Managed Agents can't be used for workloads that require Zero Data Retention or HIPAA BAA coverage; see the notice on the [overview](/recursion).
* **Files.** Files you attach to a session and the deliverables agents save stay in your organization, in the session's **Files** tab. See [Files](/recursion/files).
* **Deleting a session isn't erasure.** Deleting stops the session, requests sandbox teardown, and removes it from lists and reads, but the transcript is retained. It can't be undone through the API.
* **Secrets stay out of history when you use vaults.** Never put a secret in a system prompt, message, metadata, environment `env_vars`, or a file. Those values are stored with the session or environment and returned by reads.
* **Failure messages are sanitized.** A session's `failure.message` never contains raw provider responses or credentials, and setup-run `stderr_tail` is redacted and limited to 2 KiB.

## Your responsibilities

* Grant each session only the vaults and credentials its task needs, and each agent only the apps and tools its work needs.
* Keep environments on **No access**, or on specific allowed destinations, unless the task needs the open internet.
* Review what an agent produces before you merge, deploy, or send it.
* Rotate credentials and API keys regularly, and revoke them at once if you suspect exposure.
* Treat anything an agent can read as something it might print.

## Next steps

<CardGroup>
  <Card title="Vaults" href="/recursion/vaults">
    Store credentials and grant them to sessions.
  </Card>

  <Card title="Environment reference" href="/recursion/environments-reference">
    Configure network policy, compute, and lifecycle.
  </Card>

  <Card title="API keys" href="/recursion/api-keys">
    Scope, rotate, and revoke keys.
  </Card>

  <Card title="Organizations and roles" href="/recursion/organizations-and-roles">
    Decide who can do what.
  </Card>
</CardGroup>


# Session operations
Source: https://docs.labelbox.com/recursion/session-operations

Search sessions by metadata, see queued messages, measure compute, ask the session analyst, and read the board, tree, and threads.

These operations help you find, inspect, and understand sessions after you start them. Starting and stopping sessions is covered in [Sessions](/recursion/sessions), and the transcript in [Events](/recursion/events).

<Note>
  The organization user role can use every read on this page. Opening or questioning the session analyst needs the organization developer or admin role, because it starts a session. The billing role cannot see sessions. See [Organizations and roles](/recursion/organizations-and-roles). For API calls, create a key under **API keys**; see [API keys](/recursion/api-keys).
</Note>

## Find sessions by metadata

Metadata is the set of key/value pairs you attach when you [start a session](/recursion/sessions), such as `customer_id` or `run`. List the keys and values in use, so you can build filters without knowing the values in advance. Then filter `listSessions` with them.

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Sessions**.
    2. Click the **Metadata** filter and choose a key from the suggestions.
    3. Type a value, or press Enter to match any value for that key. To combine filters, pick another key in the same field, which then reads **Add another**.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/metadata-keys' \
      -H "Authorization: Bearer $RECURSION_API_KEY"

    curl 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/metadata-values?key=customer_id&q=acme-&limit=50' \
      -H "Authorization: Bearer $RECURSION_API_KEY"

    curl 'https://api.recursion.labelbox.com/managed-agents/v1/sessions?metadata=customer_id:acme-042&metadata_key=run&root_only=true' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

```json theme={"theme":"css-variables"}
{ "values": ["acme-017", "acme-042"], "truncated": false }
```

When `truncated` is `true`, there are more matches than returned. Narrow `q`; there is no page token. Subagent sessions inherit their root's match, so add `root_only=true` to get one row per run.

<Accordion title="Metadata keys, values, and filters">
  * `listSessionMetadataKeys` returns `{ keys, truncated }`: every key in use in your organization, sorted, up to 200.
  * `listSessionMetadataValues` returns `{ values, truncated }` for one `key`. `q` is a case-sensitive prefix of up to 512 characters. `limit` defaults to 50 and can be at most 200. An unknown key returns an empty list, not an error.
  * In `listSessions`, each `metadata=key:value` must match, and each `metadata_key` must be present. See [List sessions](/recursion/sessions) for the other filters.
</Accordion>

## See queued messages

A message sent while the agent is mid-turn is queued until the turn ends. `listSessionPendingInputs` shows those accepted messages that are not in the transcript yet, in the order they were accepted. Nothing needs answering: these are messages you or a teammate already sent.

<Tabs>
  <Tab title="Console">
    Open the session. Queued messages appear under the last event, marked **Queued**. Click **Interrupt to read now** to have the agent read them immediately.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53/pending-inputs' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

```json theme={"theme":"css-variables"}
{
  "pending_inputs": [
    {
      "input_id": "01997a41-2c9e-7f03-b8d4-5a1e6c3f9d27",
      "kind": "message",
      "event": {
        "event_type": "message",
        "actor": "human:api",
        "role": "user",
        "content": { "blocks": [{ "type": "text", "text": "Also check last week's incidents." }] }
      },
      "created_at": "2026-09-25T14:11:30Z"
    }
  ]
}
```

`kind` is `message` or `interrupt`. When the agent reads a queued message, it moves into the transcript with its `input_id` as the `event_id`, so you can match the two. To have the agent read them now instead of at the end of the turn, interrupt the session.

## Check compute usage

`listSessionResourceSamples` returns CPU, memory, disk, and GPU usage of the sandboxes in a session tree, in one-minute buckets. Use it to right-size an environment or to see why a run was slow. Any session id in the tree works.

<Tabs>
  <Tab title="Console">
    Open the session and click the **Compute** tab.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53/resource-samples?from=2026-09-25T14:00:00Z&to=2026-09-25T15:00:00Z&resolution=minute&limit=120' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

```json theme={"theme":"css-variables"}
{
  "session_id": "e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53",
  "root_session_id": "e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53",
  "resolution": "minute",
  "live": true,
  "sandboxes": [
    {
      "sandbox_id": "7a3c9e15-2b84-4d60-9f71-0e6d2c8b5a43",
      "attached_at": "2026-09-25T14:02:40Z",
      "sample_interval_ms": 5000
    }
  ],
  "buckets": [
    {
      "sandbox_id": "7a3c9e15-2b84-4d60-9f71-0e6d2c8b5a43",
      "bucket_start": "2026-09-25T14:05:00Z",
      "complete": true,
      "sample_count": 12,
      "cpu_millicores_avg": 840,
      "cpu_millicores_peak": 1910,
      "cpu_capacity_millicores": 2000,
      "memory_working_set_avg_bytes": 1610612736,
      "memory_working_set_peak_bytes": 2147483648,
      "memory_capacity_bytes": 4294967296,
      "disk_used_avg_bytes": 3221225472,
      "disk_capacity_bytes": 21474836480
    }
  ],
  "as_of": "2026-09-25T14:06:12Z"
}
```

Keep polling while `live` is `true`. A bucket with `complete: false` is still filling and changes on the next read.

<Accordion title="Parameters and metrics">
  | Parameter | Description |
  | - | - |
  | `from` | Earliest bucket start, inclusive. RFC 3339. Omit to start from the beginning of the tree. |
  | `to` | Latest bucket start, exclusive. RFC 3339. Omit for now. |
  | `resolution` | `minute` (default) returns per-minute averages and peaks. `raw` also returns each sample, about every 5 seconds, in `offsets_ms` and parallel arrays such as `cpu_millicores` and `memory_working_set_bytes`. |
  | `limit` | Buckets per page, 1 to 720. Default 120. |
  | `page_token` | The `next_page_token` from the previous page. It is present only when more buckets remain. |

  | Metric | Fields |
  | - | - |
  | CPU | `cpu_millicores_avg`, `cpu_millicores_peak`, `cpu_capacity_millicores`, `cpu_limit_millicores`. 1000 millicores is one fully busy core. Divide by capacity for a percentage. |
  | Memory | `memory_working_set_avg_bytes`, `memory_working_set_peak_bytes`, `memory_capacity_bytes`, `memory_limit_bytes`. |
  | Disk | `disk_used_avg_bytes`, `disk_used_peak_bytes`, `disk_capacity_bytes` for the workspace volume. |
  | GPU | `gpu_count`, `gpu_utilization_avg`, `gpu_utilization_peak`, `gpu_memory_used_peak_bytes`, `gpu_memory_total_bytes`. Absent without GPUs. |
</Accordion>

<Accordion title="Buckets and sandboxes">
  Buckets are ordered by sandbox, then time. Each names its `sandbox_id`; `sandboxes` lists every sandbox the tree has had, oldest first, with `attached_at`, `detached_at`, and `detach_reason` (`released`, `closed`, `replaced`, or `lost`). An empty `sandboxes` list means the tree never had a sandbox or sampling is off.
</Accordion>

## Ask the session analyst

The session analyst is an assistant that answers questions about a session tree: what the agents were asked to do, what they did, which tools failed, how they coordinated, and what it cost. It reads the whole tree. It is read-only: it has no sandbox and no credentials, and nothing it does changes the session.

<Tabs>
  <Tab title="Console">
    1. Open the session and click **Ask** in the header.
    2. Type a question in **Ask about this session…**, or pick a suggested one.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53/analyst' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      -d '{"question": "Why did this session end the way it did?"}'
    ```
  </Tab>
</Tabs>

```json theme={"theme":"css-variables"}
{
  "created": true,
  "session": {
    "session_id": "c4f82d19-6a37-4e5b-8d01-9b2e7f3a6c58",
    "kind": "session_analyst",
    "status": "active",
    "execution_state": "running",
    "created_at": "2026-09-25T14:21:07Z"
  }
}
```

Read the analyst's answers from its session's [events or event stream](/recursion/events), using the returned `session_id`. Ask follow-ups with `sendSessionEvents` on that id.

<Accordion title="What each analyst request does">
  | Request | Result |
  | - | - |
  | Empty body | Returns your existing analyst conversation for this tree, or `created: false` with no `session` if you have none. Starts nothing. |
  | `question`, no conversation yet | Starts the analyst with your question as its first turn. Returns `201` and `created: true`. |
  | `question`, conversation exists | Delivers the question to it. Returns `200` and `created: false`. |
  | `reset: true` | Ends your current conversation. It stays readable. Your next question starts a new one. |

  Each person has one analyst conversation per tree, and any session id in the tree opens the same one. Analyst sessions are left out of `listSessions` unless you filter with `kind=session_analyst`.
</Accordion>

<Note>
  The analyst runs on a fixed model, and its model usage is recorded for your organization like any other session's. Each question adds usage, so reuse a conversation rather than resetting it for every question. See [Usage and cost](/recursion/usage-and-cost).
</Note>

## Read the team board

When a session works as a team, agents coordinate through a shared board of tasks and posts. `getSessionBoard` returns its current state: `tasks`, `posts`, `members`, and `as_of`.

<Tabs>
  <Tab title="Console">
    Open the session and click the **Work** tab.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53/board' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

The board is the source of truth for team state. `board_update` events can be missed by a reader following events by cursor, so read the board when you need the current picture. See [Teams](/recursion/teams).

<Accordion title="Board tasks and team modes">
  Each task has a `seq` shown as `T1`, `T2`, and so on, a `kind` (`task`, `explore`, or `review`), a `status` (`open`, `claimed`, `blocked`, `done`, or `dropped`), and an `owner_session_id` once claimed.

  A session started with team mode `off` has no board and returns `404`. Under `auto` or `on`, a board with nothing posted yet comes back empty.
</Accordion>

## Read the session tree and threads

A session that delegates work becomes a tree: the root session and the subagent sessions it starts. Each agent's work runs in a thread. `getSessionTree` returns the whole tree in one call: `root_session_id`, every session in `sessions`, every thread in `threads`, and the first page of the combined timeline in `events`. Any session id in the tree works. See [Multi-agent sessions](/recursion/multi-agent) for how trees form. After a [move to another agent version](/recursion/sessions#move-a-session-to-another-version), `getSessionAgentVersion` lists which agents in the tree have followed.

<Tabs>
  <Tab title="Console">
    Open the session and click the **Threads** tab. The tab appears when the session has more than one thread.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53/tree' \
      -H "Authorization: Bearer $RECURSION_API_KEY"

    curl 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53/threads' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

```json theme={"theme":"css-variables"}
{
  "threads": [
    {
      "thread_id": "f1d6a82c-4e97-4b35-a0c8-7e2b9d5f3a16",
      "session_id": "f1d6a82c-4e97-4b35-a0c8-7e2b9d5f3a16",
      "root_session_id": "e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53",
      "parent_session_id": "e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53",
      "agent_id": "b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38",
      "name": "researcher",
      "role": "subagent",
      "status": "idle",
      "stop_reason": { "type": "end_turn" },
      "thread_path": "/f1d6a82c-4e97-4b35-a0c8-7e2b9d5f3a16",
      "created_at": "2026-09-25T14:06:55Z",
      "updated_at": "2026-09-25T14:14:20Z"
    }
  ]
}
```

The tree's `events` is only the first page. When `next_event_id` is set, continue with `listSessionEvents` and `after_event_id` set to it.

<Accordion title="Tree and thread details">
  * `getSessionTree` accepts `hydrate`, `image_urls`, and `payloads`, like [List events](/recursion/events).
  * `listSessionThreads` returns every thread, sorted by `thread_path`, including one for each session in the tree. A subagent's `thread_id` is its session id. `thread_path` is the chain of session ids below the root, such as `/<child>/<grandchild>`; the root's is `/`.
  * Thread `role` is `primary` for the root and `subagent` for a delegated session. Thread `status` is `running`, `idle`, or `terminated`.
  * `getSessionThread` returns `404` for a thread outside the tree.
</Accordion>

## Attach and detach files

Open the session and choose the **Files** tab in the side panel. Under **Inputs**, click **Attach files…** to add files, or click the **×** on a row to detach it. Attach to the root session of a multi-agent tree; everyone in the tree shares its sandbox. See [Attach and detach files while a session runs](/recursion/files#attach-and-detach-files-while-a-session-runs).

## What can go wrong

The most common problems:

* **A metadata filter returns subagent sessions.** Subagents match their root's metadata. Add `root_only=true`.
* **`403 forbidden` opening the analyst.** Your role can read but not start sessions. Use an account or key with the developer or admin role. See [Organizations and roles](/recursion/organizations-and-roles).
* **`404 not_found` reading the board.** The session runs with team mode `off`. Use the tree and events instead.

<Accordion title="Every session operations error">
  | Symptom or code | Cause | Fix |
  | - | - | - |
  | `400 invalid_request` on `key` | The key has characters other than letters, digits, `_`, `.`, and `-`, or is longer than 64 characters. | Use the key exactly as returned by `listSessionMetadataKeys`. |
  | `400 invalid_request` on `limit` | `limit` is out of range: above 200 for metadata values, or outside 1 to 720 for resource samples. | Use a value in range. |
  | `truncated: true` on metadata values | More values match than one response holds. | Narrow `q`. |
  | A metadata filter returns subagent sessions | Subagents match their root's metadata. | Add `root_only=true`. |
  | `pending_inputs` is empty but the agent has not replied | The message was already read, or the session is not running. | Read the transcript; send a message to resume an idle session. |
  | `sandboxes` is empty | The tree never had a sandbox, or sampling is off. | Nothing to fix. |
  | Latest resource bucket keeps changing | It has `complete: false`. | Treat only complete buckets as final. |
  | `403 forbidden` opening the analyst | Your role can read but not start sessions. | See [Organizations and roles](/recursion/organizations-and-roles). Use an account or key with the developer or admin role. |
  | `503` `session_analyst_busy` | Another open of the same conversation is in progress. | Retry after a moment. |
  | `503` `session_analyst_unconfigured` | The analyst couldn't be started for your organization. | Contact support. |
  | A file can't be attached mid-session | The path collides with another attachment, or the session isn't the root of its tree. | Choose another path, or attach to the root session. |
  | `503` `session_analyst_model_unavailable` | The analyst's model is not available right now, so a new conversation cannot start. | Retry later. |
  | `404 not_found` reading the board | The session runs with team mode `off`. | Use the tree and events instead. |
  | Board state differs from the events you followed | A `board_update` event was missed. | Read the board again. |
  | `404 not_found` on a thread | The thread is not in this session's tree. | List threads first. |

  Every error is a flat object with required `code` and `message` fields and optional `details`, which can include `field`, `requestId`, and `retryable`. See [Errors](/recursion/errors).
</Accordion>

## Limits

* Metadata keys: up to 200 per response. Metadata values: up to 200 per response.
* Session analyst: one conversation per person per session tree.

Resource-sample pages and every other limit are in [Limits](/recursion/limits).

## Next steps

<CardGroup>
  <Card title="Sessions" icon="play" href="/recursion/sessions">
    Start sessions, check success, and stop them.
  </Card>

  <Card title="Events" icon="wave-pulse" href="/recursion/events">
    Read and stream the transcript.
  </Card>

  <Card title="Teams" icon="users" href="/recursion/teams">
    Let copies of an agent share work through the board.
  </Card>

  <Card title="Usage and cost" icon="chart-line" href="/recursion/usage-and-cost">
    See what each session cost.
  </Card>
</CardGroup>


# Sessions
Source: https://docs.labelbox.com/recursion/sessions

Start a session, watch it work, and continue, stop, or clean it up.

A session is one run of an agent on a task. It runs in its own sandbox, starts on the agent's current version in the environment you choose, and keeps a transcript you can stream, read, and continue.

<Note>
  You need an [agent](/recursion/agents) and an [environment](/recursion/environments). If the environment has a setup script, it must have passed a current setup run. Starting, messaging, moving, and stopping sessions needs the Developer or Admin role; the User role can list and read them, and can read a session's version state. API calls use a key from [API keys](/recursion/api-keys).
</Note>

## Start a session

Send the agent, the environment, and the task. Name the files you want back, so the agent saves them as deliverables.

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Sessions**, then click **Launch session**.
    2. Under **Configuration**, select an **Agent** and an **Environment**.
    3. Under **Task**, write the **Opening message**: the task, the deliverables you want by name, and what done looks like.
    4. Optionally, choose vaults under **Credentials**, a mode under **Team**, and add **Custom metadata**.
    5. Click **Launch session**.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/sessions' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      -H 'Idempotency-Key: weekly-risk-review-2026-09-25' \
      -d '{
        "agent_id": "b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38",
        "environment_id": "4c8e2a61-9b3f-4d70-a5e2-1f6b8c9d0e47",
        "message": "Review the open incidents and write a risk summary named risks.md.",
        "metadata": {"customer_id": "acme-042", "run": "weekly"}
      }'
    ```
  </Tab>
</Tabs>

The session is accepted right away and starts in the background. The response is `202 Accepted`:

```json theme={"theme":"css-variables"}
{
  "session_id": "e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53",
  "status_path": "/managed-agents/v1/sessions/e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53"
}
```

Keep `session_id`. `202` means the session was accepted, not that it is running. `status_path` is relative to `https://api.recursion.labelbox.com` and already starts with `/managed-agents`, as does the `Location` response header.

The `Idempotency-Key` header is required. Use a value that identifies the work, such as a job id, and send it again on every retry: the same request with the same key returns the same session instead of starting a second one.

<Accordion title="All start fields">
  | Field | Required | Description |
  | - | - | - |
  | `agent_id` | Yes | The agent to run. The session starts on the agent's current version and reports it as `agent_version_id`. You cannot choose an older version here. To change the version later, see [Move a session to another version](#move-a-session-to-another-version). |
  | `environment_id` | Yes | The environment the sandbox is built from. If it has a setup script without a current passing setup run, the request fails with `422 environment_not_verified`. |
  | `message` | No | The opening message: the task and what done looks like. It stays in the agent's context for the whole run, so context compaction cannot summarize it away. A message of only whitespace returns `400`. |
  | `resources` | No | Up to 500 files to mount read-only in the sandbox before the first turn, as `[{ "type": "file", "file_id" }]`. One unknown file id refuses the whole request with `404`. See [Files](/recursion/files). |
  | `vault_ids` | No | Vaults to grant. Omit to use the agent's default vaults. `[]` grants none. See [Vaults](/recursion/vaults). |
  | `credential_refs` | No | `[{ "vault_id", "credential_id" }]`. Narrows access to exactly these credentials across the granted vaults. `[]` with non-empty `vault_ids` is rejected. |
  | `referenced_session_ids` | No | Up to 10 earlier sessions the agent may read. If you cannot read one of them, the request fails with `404` and nothing starts. See [Referenced sessions](/recursion/referenced-sessions). |
  | `metadata` | No | Your own string key/value pairs, such as a customer id. Up to 32 entries. Keys are up to 64 characters of letters, digits, `_`, `.`, and `-`. Values are 1 to 512 characters without control characters. Set only at start: it's returned on the root session by `getSession` and `listSessions`, you can filter on it, and there is no call to change it. See [Session operations](/recursion/session-operations) for metadata search. |
  | `outcome` | No | A definition of done for this session, as `{ "description", "rubric" }` with an optional `max_iterations`. The work is graded against the rubric, and the agent keeps revising until the rubric is met or the passes run out. Omit to use the agent's `default_rubric`, graded against `message`. See [Outcome rubrics](/recursion/outcomes). |
  | `skip_default_outcome` | No | `true` starts the session ungraded when the agent has a `default_rubric`. |
  | `team` | No | `{ "mode": "auto" \| "on" \| "off" }`. Omit to keep the agent's setting, which is `auto` unless the agent sets another. See [Teams](/recursion/teams). |
</Accordion>

<Accordion title="Retry a start safely">
  | You send | Result |
  | - | - |
  | No key | `400 invalid_request`. |
  | A key that is empty, longer than 256 characters, has non-visible characters, or is sent more than once | `400 invalid_request`. |
  | The same key, exact HTTP method, escaped path, raw query, and raw body bytes | The original `202` response again, with `Idempotency-Replayed: true`. No second session starts. The first response carries `Idempotency-Replayed: false`. |
  | The same key with any different request-identity bytes | `409 idempotency_conflict`. |
  | The same key while the first request is still being processed | `409 idempotency_in_progress` with `Retry-After`. Wait, then resend the same request byte for byte. |
  | The same key after the first attempt failed with a non-2xx response | The key is released. The retry runs as a new attempt. |
  | The key cannot be checked or reserved | `503 idempotency_unavailable`. The request did not run, so resend the same request with the same key. |

  Keys share one namespace across keyed mutations in your organization, so include the operation and work identity. Completed receipts last about 24 hours, pending claims can be reclaimed after about 1 hour, and deduplication is not guaranteed after expiry. The session id is derived deterministically from the durable receipt, so an exact replay returns the same `session_id`. See [Idempotent mutations](/recursion/api#idempotent-mutations).
</Accordion>

## Watch it work

Stream the session's events to follow along as the agent works. The stream replays what already happened, then stays open for new events, and closes when the session reaches a terminal status.

<Tabs>
  <Tab title="Console">
    Open the session from **Sessions**. The transcript updates live, and the status badge shows the current state.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -N 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53/events/stream' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

You see the agent's steps as they happen. This output is abridged; each `data` line carries the full event.

```text theme={"theme":"css-variables"}
event: session.status_running
event: agent.tool_use
event: agent.tool_result
event: agent.message
data: {"type":"agent.message","content":[{"type":"text","text":"Found 12 open incidents."}]}
event: agent.artifact
event: session.status_idle
data: {"type":"session.status_idle","stop_reason":"end_turn"}
```

[Events](/recursion/events) lists every frame type and shows how to resume a dropped stream. To poll instead, read the session with `view=summary` and stop on any terminal status, not only `completed`, and also when the session needs you.

<Warning>
  An interrupted or sleeping session stays `active`. A loop that waits only for a terminal status keeps waiting until someone sends a message, the wake time arrives, or the session is cancelled.
</Warning>

## What success means

When the agent loop finishes, `status` becomes `completed` whether or not the task succeeded. Read the final message and open the files it saved to judge the work.

To have the work checked against a rubric before the session is done, see [Write outcome rubrics](/recursion/outcomes).

| You see | Result |
| - | - |
| `status: completed`, `execution_state: idle`, `stop_reason: end_turn` | The agent finished. Check its final message and deliverables against your task. |
| `status: failed` | The session hit an error. Read `failure`. |
| `status: cancelled` | The session was cancelled. Events and deliverables stay readable. |
| `status: active`, `execution_state: idle`, `stop_reason: interrupted` or `sleeping` | Not finished. The session is paused and waits for a message or its wake time. |
| `status: active`, `execution_state: idle`, `stop_reason: requires_action` | Not finished. The session needs your review before it continues. |
| `status: active`, `execution_state: idle`, `stop_reason: out_of_credit` | Not finished. Your tenant ran out of credit. Buy credits; it continues on its own within about a minute if it paused in the last 72 hours. |

When a session fails, `failure` holds `phase`, `code`, `message`, `retryable`, `category`, and `at`. `category` is `transient` (retry), `caller_error` (fix the request or setup), or `internal`.

Deliverables appear in the session's **Files** tab. Files the agent leaves elsewhere stay in `/workspace` for later turns, even after a stopped sandbox is released, and can be used when the workspace reopens successfully, but they aren't captured as deliverables. Deleting the session removes access to its workspace. See [Deliverables and artifacts](/recursion/artifacts) and [Download session deliverables](/recursion/files#download-session-deliverables). To give a session input files, attach them in **Launch session**, with `resources` at start, or in the session's **Files** tab while it runs.

<Accordion title="Read a session's details">
  In **Sessions**, click a session. The **Session** tab shows its state, agent, environment, and **Custom metadata**. A session an automation started also shows **Started by**: the **Automation**, the **Trigger** that fired, and the **Run**, each linking back to the automation. See [Automations](/recursion/automations).

  With the API, `view=full` (the default) returns the whole session, including the agent and model snapshots and the resolved configuration. `view=summary` omits `agent_snapshot`, `model_snapshot`, and `config`, which makes polling cheaper. Any other `view` value returns `400`.

  ```bash theme={"theme":"css-variables"}
  curl 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53?view=summary' \
    -H "Authorization: Bearer $RECURSION_API_KEY"
  ```

  ```json theme={"theme":"css-variables"}
  {
    "session_id": "e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53",
    "root_session_id": "e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53",
    "session_path": "/",
    "kind": "chat",
    "status": "completed",
    "execution_state": "idle",
    "stop_reason": "end_turn",
    "agent_id": "b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38",
    "agent_version_id": "9d2f7b1e-6c43-4a85-b0e9-3a7c5d8f2e16",
    "environment_id": "4c8e2a61-9b3f-4d70-a5e2-1f6b8c9d0e47",
    "metadata": { "customer_id": "acme-042", "run": "weekly" },
    "costUsd": "0.418204",
    "costState": "final",
    "created_at": "2026-09-25T14:02:11Z",
    "updated_at": "2026-09-25T14:19:48Z",
    "last_activity_at": "2026-09-25T14:19:48Z"
  }
  ```

  | Field | Description |
  | - | - |
  | `root_session_id`, `parent_session_id`, `session_path` | Where the session sits in its tree. A root has `session_path` `/` and no parent. |
  | `kind` | `chat` for sessions you start, `subagent` for sessions an agent starts, `session_analyst` for the [session analyst](/recursion/session-operations). |
  | `agent_version_id` | The agent version the session runs. It changes only when you [move the session to another version](#move-a-session-to-another-version). |
  | `metadata` | Set at start. Present on root sessions only. |
  | `wake_at` | When a sleeping session wakes on its own. |
  | `concurrency_slot_held` | Whether the session counts against the agent's `max_concurrent_sessions`. |
  | `costUsd`, `costState` | What the whole session tree cost, and whether that is `final` or `so_far`. See [Usage and cost](/recursion/usage-and-cost). |
  | `failure` | Present only after a failure. |
</Accordion>

## Continue the conversation

Send a follow-up to add to the task or ask for changes. A running session reads it at its next turn. An idle or finished root session picks the work up again, and applies a [version move](#move-a-session-to-another-version) that is waiting before it does. The full send options, including the delivery states in the response, are in [Events](/recursion/events).

<Tabs>
  <Tab title="Console">
    Open the session, type in **Message the agent…**, and click **Send**. While the agent is working the button reads **Queue**.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53/events' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      -d '{"message": "Also add a mitigation for each high-severity item."}'
    ```
  </Tab>
</Tabs>

```json theme={"theme":"css-variables"}
{ "ok": true, "delivery_state": "resumed", "events_accepted": 1 }
```

Only root sessions resume. A message to a subagent session in a terminal status returns `409 conflict`; message the root session instead.

To change direction mid-turn, interrupt instead:

* **Interrupt** stops the current model call and gives a running tool a short grace period to stop. The session waits, with its sandbox, for your next message.
* **Interrupt and send** stops the current turn and starts a new one with your message, in one call.

<Accordion title="Interrupt, and interrupt and send">
  An interrupt stops the current model call right away and gives a running tool a short grace period to stop. If messages are queued, the newest starts a fresh turn. Otherwise the session stays `active` and goes `idle` with `stop_reason: interrupted`, ready for your next message. The sandbox stays up. In the console, click **Interrupt** under the last event while the agent is working.

  ```bash theme={"theme":"css-variables"}
  curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53/interrupt' \
    -H "Authorization: Bearer $RECURSION_API_KEY" \
    -H 'Content-Type: application/json' \
    -d '{}'
  ```

  The response is `{ "interrupted": true }`. A session with nothing running (`idle` and not sleeping or waiting on its subagents, or already finished) answers `409 conflict`, and nothing is recorded. `interrupted: false` means the stop was recorded but reaching the running work was not confirmed, usually because the run ended as the request arrived.

  `interruptAndSendSessionMessage` stops the current turn and starts a fresh one with your message in one call. Send `message` for plain text, or `content` for content blocks; `content` wins if you send both. In the console, type your message and click **Interrupt & send**, which appears while the agent is working.

  ```bash theme={"theme":"css-variables"}
  curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53/interrupt-and-send' \
    -H "Authorization: Bearer $RECURSION_API_KEY" \
    -H 'Content-Type: application/json' \
    -d '{"message": "Stop. Only cover incidents opened this week."}'
  ```

  The response has the same shape as a send, with `events_accepted: 2`: the interrupt and your message.
</Accordion>

### Send an image

To show the agent a screenshot or chart, attach it to a message. The model sees the image when its model accepts images. In the session's message box, click **Attach image**, choose the image, write your message, and click **Send**. To use an image from your library, click **Attach from Files**. Send PNG, JPEG, or WebP; the type is detected from the bytes. An image the session's image limits refuse returns `400` with the reason. See [Limits](/recursion/limits).

## Move a session to another version

A session runs the agent version it started on until you move it. Move it to give work in progress new instructions, tools, skills, or access without starting over: the transcript, sandbox, and files stay.

<Tabs>
  <Tab title="Console">
    1. Open the session. When its agent has a newer version, a notice under the header names both, for example "This session runs Version 2 of backend-fixer. The agent is now on Version 3".
    2. Click **Use Version 3**.
    3. Read the comparison of the two versions, then click **Use Version 3** to confirm.

    The notice changes to "Moving to Version 3" and says when the move applies. Click **Withdraw** to cancel a move that hasn't applied yet. If the move is refused, the notice says why and offers **Try again**.

    To move several sessions at once, open the agent and choose the **Sessions** tab. Select the sessions marked **Older version**, click **Use Version 3**, and confirm.

    The console moves a session to its agent's latest version. To move to any other version, including an older one, use the API.
  </Tab>

  <Tab title="cURL">
    Send the version the session should run with `replaceSessionAgentVersion`:

    ```bash theme={"theme":"css-variables"}
    curl -X PUT 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53/agent-version' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      -d '{"agentVersionId": "5a0c8e37-1f94-4b2d-a6c3-7e9d2b4f8a15"}'
    ```
  </Tab>
</Tabs>

The request records the version to move to. It doesn't interrupt the session, wake it, or start any work. The session changes before its next model call, so the response is `200` with `state: pending`:

```json theme={"theme":"css-variables"}
{
  "agentVersionId": "5a0c8e37-1f94-4b2d-a6c3-7e9d2b4f8a15",
  "currentAgentVersionId": "9d2f7b1e-6c43-4a85-b0e9-3a7c5d8f2e16",
  "currentVersion": 2,
  "state": "pending",
  "pending": {
    "agentVersionId": "5a0c8e37-1f94-4b2d-a6c3-7e9d2b4f8a15",
    "version": 3,
    "requestedAt": "2026-09-25T15:04:09Z",
    "requestedByUserId": "usr_01a0867d1b0875be9f9a9938ffb90971"
  },
  "lastFailure": null,
  "latestAgentVersion": {
    "agentVersionId": "5a0c8e37-1f94-4b2d-a6c3-7e9d2b4f8a15",
    "version": 3
  },
  "members": [
    {
      "sessionId": "e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53",
      "name": "",
      "role": "root",
      "adoption": "pending",
      "agentVersionId": "9d2f7b1e-6c43-4a85-b0e9-3a7c5d8f2e16",
      "version": 2
    }
  ]
}
```

The version must belong to the session's agent, and it can be older or newer than the one the session runs. The request needs no `Idempotency-Key`:

* **Repeating a move that is pending, or that the session already applied, changes nothing.** You get the current state back.
* **Repeating a move after it failed asks again.**
* **The later of two different requests wins.**
* **Sending the version the session runs now withdraws a pending move.**

This operation's fields are camelCase. The session itself still reports its version as `agent_version_id`.

<Accordion title="Which sessions you can move, and who can move them">
  * **Root sessions that run an agent.** That includes sessions you start in the console or through the API, and sessions an automation or Slack starts. A subagent session follows its root, so move the root. Sessions the service runs itself, such as the session analyst, stay on the version they started with, and so do pull request review sessions.
  * **Live sessions, and finished ones that can continue.** A session that a follow-up message can start again can be moved. A session that is still stopping, or that can no longer continue, can't.
  * **To any version of the same agent.** A version of another agent returns `404 not_found`.
  * **With create and update access.** A move can give the session the vaults a new session on that version would receive, so it needs both, and reading the state needs read access. The Developer and Admin roles have all three; the User role can read the state and can't move a session. See [Organizations and roles](/recursion/organizations-and-roles).
  * **Not through the MCP server.** An agent that calls the API through the [Recursion MCP server](/recursion/ai-coding-agents) can read a session's version state with `getSessionAgentVersion`. It can't move a session: `replaceSessionAgentVersion` isn't among the MCP tools.
</Accordion>

### Check the move

`getSessionAgentVersion` returns the version the session runs, the move that is waiting, the agent's latest version, and how far each agent in the session's tree has followed. The state changes as the session works, so read it again instead of keeping a copy. A subagent session, or a session that wasn't started from an agent, answers `409 conflict`.

In the console, the transcript shows "Now on Version 3" once the session has moved, and the **Session** tab of the details panel shows the version in the **Agent version** row. For a session with subagents or teammates, **Agent version by member** on the same tab lists how far each one has followed.

```bash theme={"theme":"css-variables"}
curl 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53/agent-version' \
  -H "Authorization: Bearer $RECURSION_API_KEY"
```

After the session has moved, the read shows the new version and nothing waiting:

```json theme={"theme":"css-variables"}
{
  "agentVersionId": "5a0c8e37-1f94-4b2d-a6c3-7e9d2b4f8a15",
  "currentAgentVersionId": "5a0c8e37-1f94-4b2d-a6c3-7e9d2b4f8a15",
  "currentVersion": 3,
  "state": "current",
  "pending": null,
  "lastFailure": null,
  "latestAgentVersion": {
    "agentVersionId": "5a0c8e37-1f94-4b2d-a6c3-7e9d2b4f8a15",
    "version": 3
  },
  "members": [
    {
      "sessionId": "e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53",
      "name": "",
      "role": "root",
      "adoption": "current",
      "agentVersionId": "5a0c8e37-1f94-4b2d-a6c3-7e9d2b4f8a15",
      "version": 3
    }
  ]
}
```

| `state` | Meaning | What to do |
| - | - | - |
| `current` | The session runs `currentAgentVersionId`. Nothing is waiting. | Nothing. |
| `pending` | A move is recorded and takes effect before the session's next model call. | Nothing. Send a message if the session is finished or you want it to move sooner. |
| `failed` | The last move was refused. The session still runs `currentAgentVersionId`, and `lastFailure` says why. | Fix the cause, then send the request again. |

<Accordion title="Response fields">
  | Field | Description |
  | - | - |
  | `agentVersionId` | The version the session is set to run: the pending version while a move is waiting, otherwise the version it runs now. The only field you send. |
  | `currentAgentVersionId`, `currentVersion` | The version the session runs now, as an id and a number. `currentVersion` is left out when the number can no longer be read. |
  | `state` | `current`, `pending`, or `failed`. |
  | `pending` | The move that is waiting: `agentVersionId`, `version`, `requestedAt`, and `requestedByUserId` when a user asked. `null` when nothing is waiting. |
  | `lastFailure` | Why the most recent move was refused: `agentVersionId`, `version`, `code`, `message`, and `at`. `null` when there is none, and while a newer move is pending. |
  | `latestAgentVersion` | The agent's newest version, as `agentVersionId` and `version`. `null` when the agent was deleted. |
  | `members` | The root session and every open thread in its tree, root first. Each has `sessionId`, `name`, `role`, and `adoption`, with `agentVersionId` when it runs an agent of its own, `version` when that agent is the root's, and `failure` when its move was refused. `name` is the name the thread was given, and is empty for the root. |

  | `role` | Where the member's configuration comes from |
  | - | - |
  | `root` | The session itself. |
  | `self` | A copy of its parent's own agent. Teammates are copies of the leader. |
  | `worker` | A thread with no agent of its own. It takes its parent's model and tools. |
  | `roster` | Another agent, named by its parent's roster. |

  | `adoption` | Meaning |
  | - | - |
  | `current` | Nothing to adopt. |
  | `pending` | The member is running and adopts before its next model call. |
  | `idle` | The member isn't running. It adopts when it is next given work. |
  | `failed` | The member's move was refused, and it keeps the configuration it had. `failure` says why. |
  | `retained` | The member keeps the configuration it has, because its parent is closed or couldn't adopt, or its roster entry was removed. |

  On the root, `adoption` follows `state`: `pending` while a move waits, and `failed` after a refused one.
</Accordion>

### When the move applies

A move never cuts into a model call. The session finishes the call it is making and the tool calls that came from it, then changes version before its next model call.

* **A working session** moves before its next model call.
* **A sleeping session** moves when it next runs. Moving it doesn't wake it.
* **A finished session** moves when you send its next message.
* **A session retrying a failing model call** doesn't change model between retries of the same call. A move to a version with a different model applies before the session's next model call, once the retried call succeeds or you send the session a message. A move that keeps the model can apply at the next retry.

Moving a session to the version it already runs changes nothing: it doesn't resolve skills or list MCP tools again.

### What changes and what stays

| Follows the new version | Stays as it is |
| - | - |
| System prompt | Transcript and events |
| Model and model settings | Sandbox, `/workspace`, and running processes |
| Tool settings, such as web search, and custom tools | Environment |
| MCP servers, with their tools listed again | Attached files and referenced sessions |
| Skills, resolved again | Custom metadata |
| Granted apps and integrations | Team mode |
| Roster and delegation limits | Cost so far |
| [File history](/recursion/agent-file-history) setting | |
| Default vaults and credentials. See **Credentials after a move**. | |

The agent receives one notice about the move, a `message` event with `role: system` that you can read in the session's events. It names what changed: that its instructions were replaced, whether the model changed, and the skills, tools, MCP servers, integrations, credentials, and roster agents that were added or removed. It lists names only, never secret values. A skill that kept its name and changed its content is listed as updated, and the agent activates it again before relying on it. A session that hasn't done any work yet gets no notice, because it simply starts on the new version.

When the new version uses a different model, the earlier conversation is summarized for the new model before the session continues. A message the earlier model hadn't read yet stays as you wrote it, and the original events stay in the log.

<Accordion title="Credentials after a move">
  A move carries the new version's default credential changes onto the session:

  * **Vaults and credentials the new version adds are added.**
  * **Vaults and credentials the new version removes are removed.**
  * **What the person who launched the session changed is kept.** A vault they added stays, and a credential they unchecked stays out.

  Two cases differ:

  * **A session whose trigger set its own credentials keeps them.** When Slack or an automation started the session with its own grant, including an empty one, the move leaves that grant alone. The record of the move and the agent's notice both say that the version's defaults changed and the session kept its own.
  * **A session granted whole vaults gains an added vault whole.** A session started with `vault_ids` and no `credential_refs` holds every credential in each vault, so it receives every credential in a vault the new version adds, including any added to that vault later.

  A vault or credential the new version names that no longer exists is skipped, and the session's events record a warning.

  When each kind of credential takes effect:

  | Credential | After the move |
  | - | - |
  | Token for an MCP server | Immediately. An added token is sent from the session's next MCP request, and a removed one is no longer sent. |
  | Granted apps and integrations | At the move. Access is issued for the new version's apps. |
  | Environment-variable secret, added | On the session's next sandbox. A running sandbox doesn't receive it. |
  | Environment-variable secret, removed | Still present in a running sandbox until that sandbox is replaced. Its value stays masked in tool output while it is there, as long as the credential is still in its vault with the same value. |

  <Warning>
    Removing an environment-variable secret from a version doesn't take it out of a sandbox that is already running. To revoke it, rotate the secret at its provider. See [Rotate a credential](/recursion/vaults#rotate-a-credential).
  </Warning>
</Accordion>

<Accordion title="Subagents and teams">
  Every open thread in the session's tree follows the move before its own next model call, after its parent has followed. A thread that is running finishes its current model call first, and one that isn't running follows when it is next given work. Each receives its own notice.

  | Member | After the move |
  | - | - |
  | A copy of its parent's agent, including teammates | Its parent's new configuration in full: prompt, model, skills, tools, MCP servers, and grants. |
  | A general worker with no agent of its own | Its parent's new model, skills, tools, MCP servers, and grants. It keeps the general worker prompt. |
  | A different agent from the roster | What the new roster says. A changed `version` moves it to that version, and an entry without `version` moves it to that agent's latest version. |
  | An agent the new roster no longer lists, or lists under the same name for another agent | Keeps the configuration it has and finishes its work. |

  A thread whose parent is closed, or whose parent couldn't adopt, also keeps the configuration it has. `members` reports it as `retained`.

  Team mode and seats are fixed when the session starts. A move never adds seats or removes a teammate, and the board keeps its tasks, claims, and notes. Seats can shrink: when the new version's `max_concurrent_threads` is lower than the team size, the team fills only that many seats from then on, and teammates already working stay.
</Accordion>

### If a move fails

A move that can't be applied never stops the session. The session keeps the version it has and keeps working, `state` becomes `failed`, and `lastFailure` gives a `code` and one fixed sentence in `message`. Fix the cause and send the request again.

<Accordion title="Failure codes">
  | `code` | Meaning |
  | - | - |
  | `agent_unavailable` | The session's agent no longer exists. |
  | `agent_version_unavailable` | The requested version no longer exists. |
  | `agent_changed` | The agent changed while the session was moving. Send the request again. |
  | `session_not_eligible` | The session can no longer move to another version. |
  | `evaluation_toolset` | The version uses a toolset this kind of session can't run. |
  | `model_not_allowed` | The version's model isn't available to this session. |
  | `model_unavailable` | The version's model couldn't be found. |
  | `model_config_invalid` | The version's model settings aren't valid for its model. |
  | `project_required` | The version's model requires a project, and the session has none. |
  | `configuration_invalid` | The version's configuration can't run in this session. |
  | `temporarily_unavailable` | A temporary problem stopped the move. Send the request again. |

  The list can grow, so accept codes that aren't in it.
</Accordion>

A temporary problem doesn't fail a move at once. The session tries again before its next model calls, and after three attempts records `temporarily_unavailable`. An MCP server that is new in the version and can't be reached, or that refuses its token, doesn't fail the move either: the session moves without that server's tools and records a warning, as it does at start.

Each move is recorded in the session's events as a `session_status` event:

| Outcome | `event_status` | `content.metadata` |
| - | - | - |
| Moved | `agent_version_changed` | `fromAgentVersionId`, `toAgentVersionId`, `fromVersion`, `toVersion`, `requestId`, `requestedByUserId` when a user asked, a `message`, and `changes`: the names of what was added, updated, and removed. |
| Refused | `warning` | `code` is `agent_version_change_failed`, `reason` is the failure code, and `message` is its sentence, with the same version ids. |
| A thread kept its configuration | `agent_version_changed` | `retained` is `true`. |

A thread's events carry `applyId` in place of `requestId`.

<Accordion title="Limits of a move">
  * **Team mode and seats are fixed at start.** A move doesn't turn team mode on or off, and seats can shrink but never grow.
  * **A running sandbox keeps its environment variables.** Environment-variable secrets, and the network destinations a sandbox may reach for granted apps, are set when a sandbox is created. A move changes them on the session's next sandbox.
  * **A thread adopts after its parent.** A thread whose parent hasn't followed yet waits for it.
  * **A session doesn't change model between retries of one model call.** While a session retries a failing model call, a move to a version with a different model waits for its next model call, which comes once the retried call succeeds or you send a message.
  * **Moving to the current version does nothing.** It isn't a way to load a newer skill version or list an MCP server's tools again.
  * **The environment never changes.** To use another environment, start a new session.
</Accordion>

## Stop or delete a session

| Action | What happens | Console |
| - | - | - |
| Cancel | Stops the session now. In-flight work is abandoned, the sandbox is torn down, and the status becomes `cancelled` with `stop_reason: interrupted`. Events and deliverables stay readable, and a follow-up message to a cancelled root session starts it again in a new sandbox. | **Session actions** › **Force stop…** › **Stop now** |
| Delete | Removes the session and all its subagent sessions from lists and reads, stops any running work, and tears down the sandbox. The transcript is retained, not erased. There is no undo. | **Session actions** › **Delete session…** › **Delete** |

<Tabs>
  <Tab title="Cancel">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53/cancel' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>

  <Tab title="Delete">
    ```bash theme={"theme":"css-variables"}
    curl -X DELETE 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/e3a91f5c-7d24-4b68-9c10-2f8e6b4d7a53' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

Cancel answers `{ "cancelled": true }` and delete `{ "deleted": true }`. You can cancel any session that is not completed, failed, or already cancelled, including an idle one. The cancel response is the same when the session was already finished, so it is safe to repeat. Sandbox teardown can finish shortly after the response.

## Find sessions

`listSessions` returns sessions in your organization, most recently updated first. Filter by status, kind, agent, agent tags, or your metadata.

<Tabs>
  <Tab title="Console">
    In the sidebar, click **Sessions**. The **Status**, **Agent**, **Agent tags**, and **Metadata** filters cover every session. The search box matches id, agent, state, or metadata on the page shown; paste a full session id to find a session on any page. Click a column header to sort the page.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl 'https://api.recursion.labelbox.com/managed-agents/v1/sessions?status=failed&metadata=customer_id:acme-042&root_only=true&limit=50' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

`root_only=true` returns one row per run, without the subagent sessions under it. List rows are summaries: they leave out snapshots, `config` is empty, and token totals aren't included. Each row carries the session tree's `costUsd` and `costState` (see [Usage and cost](/recursion/usage-and-cost)).

<Accordion title="List parameters and paging">
  | Parameter | Description |
  | - | - |
  | `status` | One status, such as `failed`. |
  | `kind` | Comma-separated kinds, matched as OR: `chat`, `subagent`, `session_analyst`. Analyst sessions appear only when you name `session_analyst`. |
  | `agent_id` | Sessions of one agent. |
  | `tag_ids` | Comma-separated agent tag ids, up to 32. A session matches only if its agent currently has every tag. An unknown tag id returns `404`. |
  | `metadata` | `key:value`, repeatable up to 32 times. All pairs must match. The value is everything after the first colon. Child sessions of a matching root also match, so add `root_only=true` for one row per run. |
  | `metadata_key` | A key that must be present, repeatable up to 32 times. |
  | `root_only` | `true` returns root sessions only. |
  | `limit` | Page size. Default 100, maximum 1000. Larger values are reduced to 1000. |
  | `page_token` | The `next_page_token` from the previous page. |

  A page token is valid for one hour and only with the same filters and `limit`. Changing either returns `400 invalid_request`; start again without a token. Because the list is ordered by last update, a session that changes while you page can appear twice or be skipped. Deduplicate by `session_id`, or filter on a value that does not change.
</Accordion>

## How a session runs

Two fields describe a session. `status` says whether it's still live. While it's `active`, `execution_state` says what the agent loop is doing. When the loop isn't running, `stop_reason` says why.

| `stop_reason` | Meaning | What to do |
| - | - | - |
| `end_turn` | The agent finished its turn with nothing left to do. | Read the final message and deliverables. |
| `requires_action` | The session was stopped for review after failing the same way repeatedly, or it is waiting on a browser hand-off (`status: awaiting_human`). | After a review stop, fix what blocked it, then reply to continue. |
| `sleeping` | The agent chose to wait. The session stays `active`. | Nothing, or send a message to wake it before `wake_at`. |
| `awaiting_subagents` | A coordinator is waiting for subagents to report back. | Nothing. See [Multi-agent sessions](/recursion/multi-agent). |
| `out_of_credit` | Your tenant ran out of credit. The session paused before its next model request. Requests already in flight for the tenant complete and are charged, so the balance can end slightly below zero. Its subagents pause the same way. | Buy credits. A session paused in the last 72 hours continues on its own within about a minute, subagents included; send a message to continue it sooner or after that. See [Billing](/recursion/billing#when-credits-run-out). |
| `tenant_suspended` | Your account is suspended. The session paused without finalizing its work. | Contact support to restore the account, then send a message to continue. |
| `interrupted` | Someone interrupted or cancelled the run. | Send a message to continue. |
| `error` | The loop stopped on an error. | Read `failure`. |

<Accordion title="Statuses and execution states">
  | `status` | Meaning |
  | - | - |
  | `active` | The session is live. Check `execution_state` for what it is doing. |
  | `completed` | The agent loop finished. This does not by itself mean the task succeeded. See [What success means](#what-success-means). |
  | `failed` | The session stopped on an error. The `failure` object explains it. |
  | `cancelled` | Someone cancelled the session. Its sandbox is torn down. |
  | `awaiting_human` | The agent asked a person to handle a step in its browser display, such as a sign-in, which can happen only when the environment has computer use on. `active_handoff` describes the request. The agent resumes when you send a `handoff_resolved` event (see [Events](/recursion/events)), or on its own when the hand-off deadline passes: 30 minutes unless the agent set another, at most 4 hours. Interrupting or cancelling the session also ends the hand-off. `execution_state` is `idle` with `stop_reason: requires_action`. |

  | `execution_state` | Meaning |
  | - | - |
  | `provisioning` | The sandbox is starting. This normally takes under a minute. After 5 minutes the session fails with `sandbox_provision_timeout`. |
  | `queued` | Waiting for a free slot under the agent's `max_concurrent_sessions`, or for sandbox capacity. |
  | `running` | The agent is working. |
  | `idle` | The loop is not running. `stop_reason` is always set. |

  `provisioning`, `queued`, `running`, and `idle` are execution states of a session whose `status` is `active`. `completed`, `failed`, and `cancelled` are terminal statuses: the session does not change on its own after reaching one. A follow-up message to a root session in a terminal status starts it again from `provisioning`, or from `queued` when the agent is at its cap. After a cancel, the session gets a new sandbox. After a completed or failed run, the sandbox is reused when possible.

  When a root session is stopped for review and no subagent is working, its sandbox is stopped with `/workspace` preserved, and restored from `/workspace` before the agent's next tool call; processes it started are gone. A session whose setup ran on its own compute keeps its sandbox until the environment's idle stop.
</Accordion>

<Accordion title="Every state transition">
  | From | When | To |
  | - | - | - |
  | Start | The start request is accepted | `provisioning`, or `queued` when the agent is at its `max_concurrent_sessions` cap |
  | `queued` | A slot or sandbox capacity frees up | `provisioning` |
  | `provisioning` | The sandbox is ready | `running` |
  | `provisioning` | No sandbox capacity is free yet | `queued` |
  | `provisioning` | The sandbox isn't ready within 5 minutes | `failed` |
  | `running` | The run is interrupted, the agent sleeps or waits for subagents, the session is stopped for review, or your tenant runs out of credit | `idle`, with a `stop_reason` |
  | `running` | The agent finishes its work | `completed` |
  | `running` | The agent asks a person for a browser hand-off | `awaiting_human` |
  | `running` | An unrecoverable error | `failed` |
  | `idle` (`sleeping` or `awaiting_subagents`) | A message, the wake time, a subagent's report, or a notice that every subagent it's waiting on has gone silent | `running` |
  | `idle` (`awaiting_subagents`) | Every subagent it was waiting on turns out to have finished or closed already, and `multiagent.interventionist.mode` is `shadow` or `off`, so there is no notice to read | `completed` |
  | `idle` (any other `stop_reason`) | A follow-up message | `provisioning`, or `queued` at the cap |
  | `awaiting_human` | A `handoff_resolved` event or the hand-off deadline | `running` |
  | Any live state | Cancel | `cancelled` |
  | `completed`, `failed`, `cancelled` | A follow-up message to a root session | `provisioning`, or `queued` at the cap |
</Accordion>

## What can go wrong

The most common problems:

* **The session stays `queued`.** The agent is at `max_concurrent_sessions`, or a slot freed while your tenant was out of credit. A queued session starts only when a slot is free and credit is available. Wait, cancel other sessions, raise the agent limit, or add credits.
* **`422 environment_not_verified` on start.** The environment's setup script has not passed a current setup run. Run environment setup, then start again. See [Environments](/recursion/environments).
* **A version move stays `pending`.** A move applies before the session's next model call, and a finished or sleeping session makes none until it gets a message or wakes. Send a message. See [When the move applies](#when-the-move-applies).
* **`status: completed` but the task is not done.** The loop finishing does not mean the task succeeded. Read the final message, then send a follow-up with what is missing.

<Accordion title="Every session error">
  | Symptom or code | Cause | Fix |
  | - | - | - |
  | `400 invalid_request` on `Idempotency-Key` | The header is missing, malformed, or sent twice. | Send exactly one key of 1 to 256 visible characters. |
  | `409 idempotency_conflict` | The key was used for a different request. | Use a new key for different work. |
  | `409 idempotency_in_progress` | The first request with this key is still running. | Wait for `Retry-After`, then retry with the same key. |
  | `404 not_found` on start | An agent, environment, vault, or referenced session is missing or not visible to you. | Check each id and your access. |
  | `400 invalid_request` on `metadata` | Too many entries, a bad key, or an empty or too-long value. | Keep to 32 entries and the key and value rules in **All start fields**. |
  | `400 invalid_request` on `page_token` | The token expired or the filters changed. | List again without a token. |
  | The session stays `provisioning`, then fails with `sandbox_provision_timeout` | The sandbox did not start within 5 minutes. | Start again. If it repeats, check the environment. |
  | A poller never returns | The session is interrupted or sleeping and stays `active`. | Stop on `idle` states you care about, or send a message. |
  | `409 conflict` when messaging or deleting | A cancel is still finishing. | Retry after a few seconds. |
  | `409 conflict` when messaging a subagent | Subagent sessions in a terminal status do not resume. | Message the root session. |
  | `404 not_found` when moving a session | The session or the version is missing, deleted, or not visible to you, or the version belongs to another agent. Any `agentVersionId` that isn't a version of the session's agent answers this, including a value that isn't an id at all. | Send a version of the session's own agent. |
  | `409 conflict` on `session_id` when moving a session | The session is a subagent session, a session the service runs itself, a pull request review session, or one that is still stopping or can no longer continue. A session whose agent was deleted has no other version to move to. | Move the root session. A session that can't continue can't be moved. |
  | `409 conflict` on `session_id` when reading a session's version state | The session is a subagent session, or it wasn't started from an agent. | Read the root session. A session that wasn't started from an agent has no version to read. |
  | `409 conflict` on `agentVersionId` | The version uses a toolset a running session can't adopt. | Send another version. |
  | `409 conflict` with `details.retryable: true` when moving a session | Other requests kept changing the session's pending move while yours was recorded. | Send the request again. |
  | `400 invalid_request` on `agentVersionId` | The field is missing or empty, or the version's model runs only for a named project and the session has none. | Send one version id from `listAgentVersions`. |
  | `400 invalid_request` on `model` when moving a session | The version's model isn't available to this session. | Send a version whose model the session can run. |
  | `403 forbidden` when moving a session | Your role or key doesn't have both create and update access. | Use the Developer or Admin role. |
  | A move stays `pending` | The session hasn't made its next model call yet: it is finished or sleeping, or it is retrying a failing model call and the version changes the model. | Send the session a message, or wait for the retried call to succeed. |
  | A move shows `failed` | The version can't run in this session. The session keeps its earlier version. | Read `lastFailure`, fix the cause, and send the request again. |
  | `429 rate_limit_exceeded` | Too many requests. | Wait for `Retry-After`. Retry only when the operation's retry strategy in **Endpoints** permits replay. |

  Every error is a flat object with required `code` and `message` fields and optional `details`, which can include `field`, `requestId`, and `retryable`. See [Errors](/recursion/errors).
</Accordion>

Limits on metadata, referenced sessions, concurrency, and provisioning are in [Limits](/recursion/limits).

## Next steps

<CardGroup>
  <Card title="Events" icon="wave-pulse" href="/recursion/events">
    Stream, send, and read every kind of session event.
  </Card>

  <Card title="Session operations" icon="gauge" href="/recursion/session-operations">
    Search metadata, inspect compute, ask the analyst, and read the tree.
  </Card>

  <Card title="Files" icon="file" href="/recursion/files">
    Give a session input files and download its outputs.
  </Card>

  <Card title="Deliverables and artifacts" icon="box-archive" href="/recursion/artifacts">
    Ask for deliverables and see how they're kept.
  </Card>
</CardGroup>


# Skills
Source: https://docs.labelbox.com/recursion/skills

Write reusable procedures as SKILL.md, version them, group them, and attach them to agents.

A skill is a reusable procedure stored as `SKILL.md`, optionally with scripts and reference files. Attach skills to an agent instead of putting every procedure in its system prompt: the agent sees each skill's short description on every turn and loads the full instructions only when a skill fits the task.

<Note>
  Creating, changing, or deleting skills and skill groups needs the organization developer or admin role. The organization user role can view them. See [Organizations and roles](/recursion/organizations-and-roles). For the API, create a key on [API keys](/recursion/api-keys) and export it as `RECURSION_API_KEY`.
</Note>

## Write SKILL.md

`SKILL.md` starts with YAML frontmatter on the first line, followed by Markdown instructions. `name` and `description` are required. Write the description as a condition for using the skill, because it is what the model routes on.

```markdown theme={"theme":"css-variables"}
---
name: release-notes
description: Use when drafting customer-facing notes from merged pull requests.
---

# Draft release notes

1. Read every merged pull request in the requested range.
2. Group the changes by product area.
3. Write one plain sentence for each change.
4. Save the result with write_artifact as release-notes.md.
```

<Accordion title="Frontmatter keys and document rules">
  | Frontmatter key | Rule |
  | - | - |
  | `name` | Required. Lowercase letters, digits, and single hyphens, at most 64 characters. The agent activates the skill by this name. |
  | `description` | Required. At most 1,024 characters. Write it as a condition for using the skill, because it is what the model routes on. |
  | `license`, `compatibility`, `metadata`, `allowed-tools` | Optional. Stored with the version. `compatibility` is at most 500 characters, and the agent sees it when it activates the skill. `allowed-tools` isn't enforced. |

  Any other key has no effect. The whole `SKILL.md` can be at most 256 KiB, and it must have instructions below the frontmatter.

  When a session loads a skill, it removes HTML comments and escapes `<`, `>`, and `&` in the name, description, and instructions. The agent reads `&lt;`, `&gt;`, and `&amp;` in their place, so write procedures that don't depend on those characters or on hidden comments. Instructions longer than 64,000 bytes of UTF-8 after this escaping are cut off, and the agent is told they were cut off. The catalog keeps your text as you wrote it.
</Accordion>

## Create a skill

Creating a skill also creates version 1. Versions are immutable.

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Skills**.
    2. Click **Create skill**.
    3. Enter the document in **SKILL.md**, and optionally a **Display title** and a **Group**.
    4. Click **Create skill**.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/skills' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      -d '{
        "display_title": "Release notes",
        "document": "---\nname: release-notes\ndescription: Use when drafting customer-facing notes from merged pull requests.\n---\n\n# Draft release notes\n\nRead the merged pull requests, group them by product area, and write one sentence for each change."
      }'
    ```
  </Tab>
</Tabs>

A `200` response is the catalog entry. Some fields are left out here.

```json theme={"theme":"css-variables"}
{
  "skill_id": "4e7a2c91-6d35-4f80-b8a1-9c3e5d7f2a64",
  "latest_skill_version_id": "8b1f5d73-2c94-467a-a0e6-4d9c7b3f1e52",
  "latest_version_number": 1,
  "latest_instruction_chars": 104,
  "name": "release-notes",
  "display_title": "Release notes",
  "description": "Use when drafting customer-facing notes from merged pull requests.",
  "organization_id": "org_01a08a705220724f9a2fe1bcd8638c9d",
  "created_at": "2026-09-17T09:30:11Z",
  "updated_at": "2026-09-17T09:30:11Z"
}
```

Keep `skill_id` to attach the skill and `latest_skill_version_id` to publish the next version. Next, [attach it to an agent](#attach-skills-to-an-agent).

<Accordion title="Create fields">
  | Field | Description |
  | - | - |
  | `document` | Required. The complete `SKILL.md`, frontmatter included. |
  | `display_title` | Title shown in the catalog, at most 256 characters. Defaults to the frontmatter `name`. |
  | `skill_group_id` | The [group](#organize-skills-in-groups) to file the skill under. Omit it to leave the skill ungrouped. |
  | `metadata` | Your own JSON object. The API returns it unchanged. |

  Skill names are unique in your organization. A name that another live skill already uses returns `409 conflict` on `name`.
</Accordion>

### Upload a bundle

To include scripts and reference files, upload a zip archive of the skill's directory instead. Put `SKILL.md` at the root of the archive, or inside one wrapper directory. Put runnable helpers in `scripts/`, supporting documents in `references/`, and other files in `assets/`. Files under `scripts/` are made executable in the sandbox; all other files are not.

<Tabs>
  <Tab title="Console">
    1. In **Skills**, click **Create skill**.
    2. Drop the zip on **Upload a bundle**, or pick it. Optionally enter a **Display title**.
    3. Click **Review and upload**, check the file list, then click **Create skill** in the **Upload bundle** dialog.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/skills/upload' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -F 'bundle=@release-notes.zip' \
      -F 'display_title=Release notes'
    ```
  </Tab>
</Tabs>

The document, name, and description come from the `SKILL.md` inside the archive.

<Accordion title="Bundle rules and limits">
  If you use a wrapper directory, its name must match the skill's `name`, and every file must be inside it. Symbolic links and other non-regular files are refused.

  The multipart form also accepts `skill_group_id`, and `metadata` as a JSON string.

  | Bundle limit | Value |
  | - | - |
  | Compressed archive | 8 MiB |
  | Unpacked contents | 32 MiB |
  | One file | 4 MiB |
  | Files | 500 |
  | Directory depth below the skill root | 3 |
</Accordion>

## Attach skills to an agent

Skills are part of an agent version, in the `skills` field. The list is ordered: the API stores and returns it in the order you send it, and the model reads the skills in that order. Each entry is one of these:

| Entry | What the session gets |
| - | - |
| `{ "skill_id": "..." }` | The skill's latest version when the session starts or moves to another agent version. |
| `{ "skill_id": "...", "version": "<skill_version_id>" }` | That exact version, every time. `"latest"` also works and means the same as leaving `version` out. |
| `{ "type": "group", "skill_group_id": "..." }` | The group's current skills when the session starts or moves to another agent version. A skill added to the group later reaches the agent without an agent update. |

<Tabs>
  <Tab title="Console">
    1. Open the agent and stay on the **Configuration** tab.
    2. In **Skills**, pick a skill or group in **Attach a skill or group**. Reorder the cards so the most important skills come first.
    3. Click **Save new version**.

    The console attaches the latest version. To pin a version, use the API.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/agents/5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55/versions' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      -d '{
        "base_agent_version_id": "c1a94e07-2f6b-4d18-b3a5-0e7d8c6f4a21",
        "name": "Release notes writer",
        "model": "<model-id>",
        "system": "Follow the attached procedures.",
        "disabled_integration_mcp_providers": ["jira"],
        "skills": [
          {"skill_id": "4e7a2c91-6d35-4f80-b8a1-9c3e5d7f2a64"},
          {"type": "group", "skill_group_id": "6c2e9a47-1f83-45d0-b7a6-3e8c5f2d9a14"}
        ]
      }'
    ```
  </Tab>
</Tabs>

A `201` response is the agent at its new version. A version is a full replacement, so resend every other setting you want to keep. See [Update an agent](/recursion/agents#update-an-agent).

```json theme={"theme":"css-variables"}
{
  "agent_id": "5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55",
  "latest_agent_version_id": "7a4e1c95-6d2f-4380-b915-3e7a0c6f2d84",
  "skills": [
    { "skill_id": "4e7a2c91-6d35-4f80-b8a1-9c3e5d7f2a64" },
    { "type": "group", "skill_group_id": "6c2e9a47-1f83-45d0-b7a6-3e8c5f2d9a14" }
  ]
}
```

Entry order matters, because descriptions past the disclosure budget are dropped from the end. A group's skills take the group's place in the list, in the order they were created, oldest first. You can't change that order; to order skills one by one, attach each skill instead of the group. Next, [check the saved agent's skills](#check-the-skills-of-a-saved-agent) to see that the descriptions fit.

<Accordion title="Entry rules">
  An agent can have at most 100 entries. The API checks every id when you save the version; an unknown skill, version, or group returns `400 invalid_request` naming the entry, such as `skills[0].skill_id`.
</Accordion>

### Reorder skills

To change the order, save a new agent version with the same entries in the new order. A version is a full replacement, so the [update rules](/recursion/agents#update-rules) apply.

<Tabs>
  <Tab title="Console">
    1. Open the agent and stay on the **Configuration** tab.
    2. In **Skills**, drag a card to its new place, or open the card's menu and click **Move to top** or **Move to bottom**.
    3. Click **Save new version**.
  </Tab>

  <Tab title="cURL">
    Get the agent first, as in [Find an agent](/recursion/agents#find-an-agent). Then send its `skills` in the new order, with its `latest_agent_version_id` as `base_agent_version_id`. This request moves the group ahead of the skill.

    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/agents/5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55/versions' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      -d '{
        "base_agent_version_id": "7a4e1c95-6d2f-4380-b915-3e7a0c6f2d84",
        "name": "Release notes writer",
        "model": "<model-id>",
        "system": "Follow the attached procedures.",
        "disabled_integration_mcp_providers": ["jira"],
        "skills": [
          {"type": "group", "skill_group_id": "6c2e9a47-1f83-45d0-b7a6-3e8c5f2d9a14"},
          {"skill_id": "4e7a2c91-6d35-4f80-b8a1-9c3e5d7f2a64"}
        ]
      }'
    ```
  </Tab>
</Tabs>

A `201` response returns `skills` in the new order.

## Check the skills of a saved agent

Every turn lists the session's skills within a character budget. The budget depends on the agent's model: a model with a larger context window allows a longer skill list. When the descriptions don't fit, the ones at the end of the list are dropped and the model sees only those skills' names, which makes it less likely to pick them. `getAgentSkillDisclosure` reports the list the agent's next session gets, in one read.

<Tabs>
  <Tab title="Console">
    1. Open the agent and stay on the **Configuration** tab.
    2. In **Skills**, read the cards from the top. That is the order the model reads the skills. Click **Show skills** on a group's card to see the skills it adds.
    3. Read the budget meter. Cards past the line marked "Past this line the model sees only skill names." are flagged **Name only**.

    The meter is a [preview](#preview-prompt-disclosure) of the cards on the page. It doesn't count the [skill for using apps](/recursion/integrations#what-a-session-receives) that sessions of an agent with built-in integrations list first, and for a model the model list doesn't include it measures against the smallest budget and says so. The API read is exact in both cases.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl 'https://api.recursion.labelbox.com/managed-agents/v1/agents/5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55/skill-disclosure' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

A `200` response is the list. This agent's latest version is the one saved in [Reorder skills](#reorder-skills), so the group's skill comes first.

```json theme={"theme":"css-variables"}
{
  "budget_chars": 16000,
  "used_chars": 291,
  "entries": [
    {
      "name": "release-checklist",
      "description_included": true,
      "attachmentIndex": 0,
      "skillId": "b3d81e6a-4c07-4f52-9a1d-7e2c5f8a6b90"
    },
    {
      "name": "release-notes",
      "description_included": true,
      "attachmentIndex": 1,
      "skillId": "4e7a2c91-6d35-4f80-b8a1-9c3e5d7f2a64"
    }
  ],
  "warnings": []
}
```

`entries` is in the order the model reads the skills. The first entry with `description_included: false` is where the line falls, and every entry after it is name only too. Those skills can still be activated, but the model sees only their names. `attachmentIndex` is the position, counted from 0, of the entry in the agent's `skills` that a skill came from, so it tells you which entry to move. [Move important skills earlier](#reorder-skills), or shorten descriptions, until they fit.

<Accordion title="What the read measures">
  The read answers for the agent's latest version, the way a session starting now would list its skills:

  * **The agent's own model.** The budget comes from the model the agent is saved with, including a model the model list no longer offers. If that model's context window can't be read when you ask, the response measures against the smallest budget, 8,000 characters, and says so in `warnings`.
  * **The skill for using apps.** Sessions of an agent with built-in integrations list [a skill for using apps](/recursion/integrations#what-a-session-receives) first. The response lists it first too and counts it in `used_chars`. Its entry has no `attachmentIndex` and no `skillId`.
  * **Groups and versions as they are now.** Each group expands to the skills it holds now, and each entry without a pinned version resolves to the skill's latest version.

  An agent that doesn't exist, is deleted, or belongs to another organization returns `404 not_found`. An agent whose model can no longer start sessions returns `400 invalid_request` on `model`. Save a new agent version with a model from the model list, then read again.
</Accordion>

<Accordion title="Response fields">
  | Field | Description |
  | - | - |
  | `budget_chars` | Characters the model allows the skill list on each turn. |
  | `used_chars` | Characters the list uses. |
  | `entries[].name` | The skill's name, as the model sees it. |
  | `entries[].description_included` | `false` when the skill is listed by name only. |
  | `entries[].attachmentIndex` | The position, counted from 0, of the entry in `skills` that the skill came from. Every skill of a group has the group's position. Absent for a skill the platform adds, such as the skill for using apps. |
  | `entries[].skillId` | The catalog skill's `skill_id`. Absent for a skill the platform adds, and for a skill written inline in its entry. |
  | `warnings` | Sentences that say what the list lost and why: an entry that could not be resolved, such as a deleted skill or group, a budget measured against the smallest because the model's context window could not be read, and each skill listed by name only. |
</Accordion>

## Preview prompt disclosure

To check a list before you save it, send it to the preview. The preview resolves groups exactly as a session start does and reports what fits, in the same shape as [the read for a saved agent](#check-the-skills-of-a-saved-agent). It saves nothing.

<Tabs>
  <Tab title="Console">
    1. Open the agent and stay on the **Configuration** tab.
    2. In **Skills**, attach the skills or groups you plan to use.
    3. Read the budget meter. It measures against the model selected on the agent. For a model the model list doesn't include, it measures against the smallest budget and says so. Cards past the line marked "Past this line the model sees only skill names." are flagged **Name only**. When the list resolves to a different number of skills than it has cards, as with a group that holds several skills, the meter names the first affected skills instead of drawing the line.
  </Tab>

  <Tab title="cURL">
    Send the model's context window as `context_window_tokens`. It is `contextWindow` on the model's entry from `listModels`, the request in [REST client setup](/recursion/api#rest-client-setup). Without it, the preview measures against the smallest budget any model gets.

    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/skills/disclosure-preview' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      -d '{"context_window_tokens":200000,"skills":[{"skill_id":"4e7a2c91-6d35-4f80-b8a1-9c3e5d7f2a64"},{"type":"group","skill_group_id":"6c2e9a47-1f83-45d0-b7a6-3e8c5f2d9a14"}]}'
    ```
  </Tab>
</Tabs>

```json theme={"theme":"css-variables"}
{
  "budget_chars": 16000,
  "used_chars": 291,
  "entries": [
    {
      "name": "release-notes",
      "description_included": true,
      "attachmentIndex": 0,
      "skillId": "4e7a2c91-6d35-4f80-b8a1-9c3e5d7f2a64"
    },
    {
      "name": "release-checklist",
      "description_included": true,
      "attachmentIndex": 1,
      "skillId": "b3d81e6a-4c07-4f52-9a1d-7e2c5f8a6b90"
    }
  ],
  "warnings": []
}
```

Read the response as you read [the one for a saved agent](#check-the-skills-of-a-saved-agent): `entries` is in the order the model reads the skills, and the first entry with `description_included: false` is where the line falls. Here `attachmentIndex` is the position, counted from 0, of the entry in the `skills` you sent. [Move important skills earlier](#reorder-skills), or shorten descriptions, until they fit.

<Accordion title="Preview details">
  Here the group holds one skill, `release-checklist`, so the response lists it after `release-notes`, with the group's `attachmentIndex`, 1. `skillId` is the catalog skill each entry resolved to. `warnings` also reports entries that could not be resolved, such as a deleted skill or group. `context_window_tokens` is optional. Without it, or when the model's entry has no `contextWindow` to send, the preview measures against the smallest budget, 8,000 characters.

  The preview measures only the list and the context window you send. The model list holds the models you can choose for an agent today, so for an agent on a model it no longer lists there is no `contextWindow` to send. Sessions of an agent with built-in integrations also list [a skill for using apps](/recursion/integrations#what-a-session-receives) first, which the preview does not count. For a saved agent, [`getAgentSkillDisclosure`](#check-the-skills-of-a-saved-agent) is exact in both cases.
</Accordion>

## Publish a new version

Get the skill first, then send its current `latest_skill_version_id` as `base_skill_version_id`. The request is a full replacement: a `display_title` you leave out resets to the frontmatter `name`, and `metadata` you leave out is cleared. You cannot change the group here; use [Move a skill to a group](#move-a-skill-to-a-group).

<Tabs>
  <Tab title="Console">
    1. In **Skills**, open the skill.
    2. Edit **SKILL.md** or **Display title**, or drop a new zip on **Upload a bundle**. A skill with bundled files keeps its instructions locked here; change them by uploading a new zip.
    3. Click **Save new version**. For a bundle, confirm in the **Upload bundle** dialog.
  </Tab>

  <Tab title="cURL — document">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/skills/4e7a2c91-6d35-4f80-b8a1-9c3e5d7f2a64/versions' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      -d '{
        "base_skill_version_id": "8b1f5d73-2c94-467a-a0e6-4d9c7b3f1e52",
        "document": "---\nname: release-notes\ndescription: Use when drafting customer-facing notes from merged pull requests.\n---\n\n# Draft release notes\n\nCheck every pull request before writing the notes."
      }'
    ```
  </Tab>

  <Tab title="cURL — bundle">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/skills/4e7a2c91-6d35-4f80-b8a1-9c3e5d7f2a64/versions/upload' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -F 'base_skill_version_id=8b1f5d73-2c94-467a-a0e6-4d9c7b3f1e52' \
      -F 'bundle=@release-notes.zip' \
      -F 'display_title=Release notes'
    ```
  </Tab>
</Tabs>

The JSON `createSkillVersion` operation returns the new version with `201`; the multipart `publishSkillVersionBundle` action returns it with `200`.

```json theme={"theme":"css-variables"}
{
  "skill_id": "4e7a2c91-6d35-4f80-b8a1-9c3e5d7f2a64",
  "skill_version_id": "13d8a6f4-5b27-49e1-8c30-7a2f9d6b4e85",
  "version_number": 2,
  "name": "release-notes",
  "description": "Use when drafting customer-facing notes from merged pull requests.",
  "created_at": "2026-09-18T14:02:37Z"
}
```

Agents that follow the latest version pick up version 2 in their next session. Sessions already running keep the skill version they have: a session resolves its skills again only when it [moves to another agent version](/recursion/sessions#move-a-session-to-another-version). If another writer published first, the API returns `409 revision_conflict` and saves nothing. Get the skill again, apply your change, and retry.

## Find a skill

List the catalog, or get one skill by id. Neither includes the instructions; [get a version](#list-and-get-versions) for those.

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Skills**.
    2. Search by name, description, or id, or expand a group in the catalog.
    3. Click the skill to open its latest version.
  </Tab>

  <Tab title="cURL — list">
    ```bash theme={"theme":"css-variables"}
    curl 'https://api.recursion.labelbox.com/managed-agents/v1/skills?search=release&limit=100' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>

  <Tab title="cURL — get">
    ```bash theme={"theme":"css-variables"}
    curl 'https://api.recursion.labelbox.com/managed-agents/v1/skills/4e7a2c91-6d35-4f80-b8a1-9c3e5d7f2a64' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

```json theme={"theme":"css-variables"}
{
  "skills": [
    {
      "skill_id": "4e7a2c91-6d35-4f80-b8a1-9c3e5d7f2a64",
      "latest_skill_version_id": "8b1f5d73-2c94-467a-a0e6-4d9c7b3f1e52",
      "name": "release-notes",
      "description": "Use when drafting customer-facing notes from merged pull requests."
    }
  ]
}
```

The list is newest first and leaves out deleted skills. That is a browse order, unrelated to the order of an agent's `skills`. Follow `next_page_token` to read the whole catalog; it is absent on the last page. A get returns one skill in the same shape as the create response.

<Accordion title="List parameters">
  | Query parameter | Description |
  | - | - |
  | `search` | 2 to 256 characters. Matches part of the name, title, or description, ignoring case, or a complete skill id. |
  | `skill_group_id` | Only skills in this group. An unknown group returns `404 not_found`. |
  | `ungrouped` | `true` for only skills in no group. Cannot be combined with `skill_group_id`. |
  | `limit` | Page size. Defaults to 100, at most 500. |
  | `page_token` | The `next_page_token` from the previous page. Send the same filters and `limit` with it. |
</Accordion>

## List and get versions

`listSkillVersions` returns every version, newest first, without pages and without instructions. `getSkillVersion` returns one version with its `instructions`.

<Tabs>
  <Tab title="Console">
    1. In **Skills**, open the skill.
    2. Use **Compare with** under **Changes** to see what changed between versions.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl 'https://api.recursion.labelbox.com/managed-agents/v1/skills/4e7a2c91-6d35-4f80-b8a1-9c3e5d7f2a64/versions/13d8a6f4-5b27-49e1-8c30-7a2f9d6b4e85' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

Every version has a bundle, and `bundle.entries` always includes `SKILL.md`. A skill created from a document alone has a bundle that holds only `SKILL.md`. Some fields are left out of this sample.

```json theme={"theme":"css-variables"}
{
  "skill_id": "4e7a2c91-6d35-4f80-b8a1-9c3e5d7f2a64",
  "skill_version_id": "13d8a6f4-5b27-49e1-8c30-7a2f9d6b4e85",
  "version_number": 2,
  "name": "release-notes",
  "description": "Use when drafting customer-facing notes from merged pull requests.",
  "instructions": "# Draft release notes\n\nCheck that every pull request is merged before writing the notes. Run scripts/collect.sh to list them.",
  "entrypoint_path": "SKILL.md",
  "file_manifest": ["scripts/collect.sh", "references/style.md"],
  "bundle": {
    "bytes": 1843,
    "entries": [
      { "path": "SKILL.md", "bytes": 214, "mode": 420 },
      { "path": "references/style.md", "bytes": 2210, "mode": 420 },
      { "path": "scripts/collect.sh", "bytes": 512, "mode": 493 }
    ]
  },
  "created_at": "2026-09-18T14:02:37Z"
}
```

A version with bundled files lists them in `bundle.entries`. `mode` is `493` (octal 0755) for files under `scripts/` and `420` (octal 0644) for everything else.

<Accordion title="Download a version's files">
  `getSkillVersionContent` returns the version's files as a gzipped tarball, exactly as a session receives them: `SKILL.md` plus any bundled files.

  ```bash theme={"theme":"css-variables"}
  curl 'https://api.recursion.labelbox.com/managed-agents/v1/skills/4e7a2c91-6d35-4f80-b8a1-9c3e5d7f2a64/versions/13d8a6f4-5b27-49e1-8c30-7a2f9d6b4e85/content' \
    -H "Authorization: Bearer $RECURSION_API_KEY" \
    -o release-notes-v2.tar.gz
  ```
</Accordion>

## Organize skills in groups

A skill group is a folder in the catalog and a way to attach a whole set of skills to an agent in one entry. The model never sees the group. Filing a skill in a group never creates a skill version.

### Create a group

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Skills**.
    2. Click **New group**.
    3. Fill in **Name**, and optionally **Display title** and **Description**.
    4. Click **Save group**.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/skill-groups' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      -d '{
        "name": "release-workflows",
        "display_title": "Release workflows",
        "description": "Procedures used while preparing a release."
      }'
    ```
  </Tab>
</Tabs>

A `201` response is the group. Keep `skill_group_id` to file skills under it and to attach it to an agent.

```json theme={"theme":"css-variables"}
{
  "skill_group_id": "6c2e9a47-1f83-45d0-b7a6-3e8c5f2d9a14",
  "organization_id": "org_01a08a705220724f9a2fe1bcd8638c9d",
  "name": "release-workflows",
  "display_title": "Release workflows",
  "description": "Procedures used while preparing a release.",
  "skill_count": 0,
  "created_at": "2026-09-17T09:41:52Z",
  "updated_at": "2026-09-17T09:41:52Z"
}
```

<Accordion title="Group fields">
  `name` is required, follows the same rule as a skill name, and must be unique among your organization's groups; a taken name returns `409 conflict` on `name`. `display_title` is at most 256 characters and defaults to `name`. `description` is for people browsing the catalog, at most 1,024 characters.
</Accordion>

### Move a skill to a group

`updateSkill` files a skill under a group. Send an empty `skill_group_id` to take the skill out of its group. The skill's versions don't change.

<Tabs>
  <Tab title="Console">
    1. In **Skills**, open the skill.
    2. In the header, pick a group in **Group**, or **Ungrouped**. The change saves at once.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X PATCH 'https://api.recursion.labelbox.com/managed-agents/v1/skills/4e7a2c91-6d35-4f80-b8a1-9c3e5d7f2a64' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      -d '{"skill_group_id":"6c2e9a47-1f83-45d0-b7a6-3e8c5f2d9a14"}'
    ```
  </Tab>
</Tabs>

<Accordion title="List, get, and update groups">
  `listSkillGroups` returns `skill_groups` newest first, with `limit` (default 100, at most 500) and `page_token`. `getSkillGroup` returns one group. `updateSkillGroup` changes only the fields you send, so renaming a group is safe: agents attach it by id.

  In the console, open the group's menu in the catalog in **Skills**, click **Edit group**, change the fields, and click **Save group**.

  ```bash theme={"theme":"css-variables"}
  curl -X PATCH 'https://api.recursion.labelbox.com/managed-agents/v1/skill-groups/6c2e9a47-1f83-45d0-b7a6-3e8c5f2d9a14' \
    -H "Authorization: Bearer $RECURSION_API_KEY" \
    -H 'Content-Type: application/json' \
    -d '{"display_title":"Release procedures"}'
  ```
</Accordion>

<Accordion title="List the skills in a group">
  `listSkillsInGroup` returns the group's skills in the order a group attachment expands them, oldest first. That order decides whose description is dropped first when the disclosure budget runs out. It takes `limit` and `page_token` like `listSkills`, and the response has the same shape as the list in [Find a skill](#find-a-skill). In the console, open an agent that attaches the group and click **Show skills** on the group's card to see the same order. The catalog in **Skills** sorts a group's skills by name.

  ```bash theme={"theme":"css-variables"}
  curl 'https://api.recursion.labelbox.com/managed-agents/v1/skill-groups/6c2e9a47-1f83-45d0-b7a6-3e8c5f2d9a14/skills' \
    -H "Authorization: Bearer $RECURSION_API_KEY"
  ```
</Accordion>

<Accordion title="Delete a group">
  In the console, open the group's menu in the catalog in **Skills**, click **Archive group**, then confirm.

  ```bash theme={"theme":"css-variables"}
  curl -X DELETE 'https://api.recursion.labelbox.com/managed-agents/v1/skill-groups/6c2e9a47-1f83-45d0-b7a6-3e8c5f2d9a14' \
    -H "Authorization: Bearer $RECURSION_API_KEY"
  ```

  A delete returns `{ "deleted": true }`. The group's skills stay in the catalog. Agents that attach the group keep the entry, and their next session runs without the group's skills.
</Accordion>

## Delete a skill

A delete is a soft delete. The skill leaves the catalog and stops resolving into new sessions, and its versions can no longer be read through the API. A session that already started keeps the version it loaded until it moves to another agent version. Agents that reference the skill keep the entry, and their next session runs without it. A deleted skill cannot be restored.

<Tabs>
  <Tab title="Console">
    1. In **Skills**, open the skill.
    2. At the bottom of the page, click **Archive skill**, then confirm **Archive skill**.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X DELETE 'https://api.recursion.labelbox.com/managed-agents/v1/skills/4e7a2c91-6d35-4f80-b8a1-9c3e5d7f2a64' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

```json theme={"theme":"css-variables"}
{ "deleted": true }
```

## How an agent uses a skill

1. When a session starts, it resolves the agent's skills and skill groups and fixes the version of each skill. It does so again only when the session [moves to another agent version](/recursion/sessions#move-a-session-to-another-version).
2. Every turn lists each skill's name and, while the budget allows, its description. See [Check the skills of a saved agent](#check-the-skills-of-a-saved-agent).
3. When a skill fits, the agent calls `activate_skill` with the skill's `name`. The full instructions and the list of bundled files enter the conversation and stay in effect for the rest of the session. After a move to another agent version, the agent activates a skill whose content changed again to read the new instructions.
4. Bundled files are placed under `/workspace/.managed-agents/skills` in the sandbox. The agent reads or runs a file only when the instructions point it there.

`activate_skill` is offered only when the session has at least one usable skill.

## What can go wrong

The most common problems:

* **The agent never activates a skill.** Its description was dropped by the disclosure budget, or doesn't say when to use the skill. [Check the saved agent's skills](#check-the-skills-of-a-saved-agent), move the skill earlier, and write the description as a condition.
* **A new skill version doesn't reach a running session.** Sessions fix skill versions when they start, and again only when they move to another agent version. Start a new session, or save a new agent version and [move the session](/recursion/sessions#move-a-session-to-another-version) to it. Pinned entries never move; update `version` to change them.
* **`400 invalid_request` on `document` or `bundle`.** The frontmatter, instructions, or archive layout is wrong, or the document or bundle is over a limit. Read the full cause below.

<Accordion title="Every skills problem">
  | Symptom or code | Cause | Fix |
  | - | - | - |
  | The agent never activates a skill | Its description was dropped by the disclosure budget, or doesn't say when to use the skill. | [Check the saved agent's skills](#check-the-skills-of-a-saved-agent), move the skill earlier, and write the description as a condition. |
  | A session runs without a skill the agent attaches | The skill or group was deleted after the agent attached it. [`getAgentSkillDisclosure`](#check-the-skills-of-a-saved-agent) reports it in `warnings`. | Remove the entry in a new agent version, or attach a replacement. |
  | A new skill version doesn't reach a running session | Sessions fix skill versions when they start, and again only when they move to another agent version. | Start a new session, or save a new agent version and [move the session](/recursion/sessions#move-a-session-to-another-version) to it. Pinned entries never move; update `version` to change them. |
  | `400 invalid_request` on `document` or `bundle` | The frontmatter is missing, `name` or `description` is missing or malformed, there are no instructions below the frontmatter, the bundle has no `SKILL.md` at the root or one directory down, the wrapper directory's name doesn't match `name`, or the document or bundle is over a limit. | Fix the frontmatter or the archive layout, and check the [limits](#limits). |
  | `400 invalid_request` on a list filter | `skill_group_id` and `ungrouped` are combined, `search` is shorter than 2 characters, or `page_token` was sent with different filters. | Send one filter, and reuse the same filters with a page token. |
  | `400 invalid_request` on `skills` or `skills[n]` | An agent has more than 100 entries, or an entry names a skill, version, or group that doesn't exist, is deleted, or belongs to another organization. | Use groups to shorten the list, and check the ids in the named entry. |
  | `400 invalid_request` on `model` from `getAgentSkillDisclosure` | The agent's model can no longer start sessions. | Save a new agent version with a model from the model list. |
  | `404 not_found` | A skill, version, or group doesn't exist, is deleted, or belongs to another organization. | Check the id. |
  | `409 conflict` on `name` | Another skill or group already uses the name. | Pick another name, or publish a new version of the existing skill. |
  | `409 revision_conflict` | `base_skill_version_id` is no longer the latest version. | Get the skill again, apply your change, and retry. |
  | `413 payload_too_large` | The whole request is larger than the upload limit. | Shrink the bundle to fit the [limits](#limits). |
  | `429 rate_limit_exceeded` | Your organization is over its request rate. | Wait for the time in `Retry-After`. Retry only when the operation's retry strategy in **Endpoints** permits replay. |

  The full error catalog is on [Errors](/recursion/errors).
</Accordion>

## Limits

Skill description, bundle, entry, list page, and search limits are in [Limits](/recursion/limits), with request sizes and rates. These apply too:

| Limit | Value |
| - | - |
| Skill and group `name` | 64 characters |
| `compatibility` in frontmatter | 500 characters |
| `SKILL.md` | 256 KiB |
| Instructions a session loads | 64,000 bytes of UTF-8 after escaping, then cut off |
| Skill and group `display_title` | 256 characters |
| Group `description` | 1,024 characters |
| `listSkillGroups` and `listSkillsInGroup` page size | 500, default 100 |

## Next steps

<CardGroup>
  <Card title="Agents" href="/recursion/agents">
    Publish an agent version with skills and groups attached.
  </Card>

  <Card title="Tools" href="/recursion/tools">
    See what else the agent can call once a skill is active.
  </Card>

  <Card title="Sessions" href="/recursion/sessions">
    Start a session and watch the agent activate a skill.
  </Card>

  <Card title="Artifacts" href="/recursion/artifacts">
    Collect the files a skill tells the agent to produce.
  </Card>
</CardGroup>


# Teams
Source: https://docs.labelbox.com/recursion/teams

Run copies of an agent on a shared task board so independent pieces of work happen in parallel, get reviewed, and end in one decision.

A team is a session tree that coordinates through a shared task board. The root agent, called the leader, posts tasks. Teammates, which are copies of the leader, join automatically, claim tasks, do the work in the shared sandbox, and mark tasks done. The leader reads the results, records a decision, and assembles the final answer. You can watch the board live and read it through the API.

<Note>
  You need an agent and an environment. No roster is required: every agent can run as a team of its own copies. Starting sessions needs the Developer or Admin role; the User role can read the board. See [Organizations and roles](/recursion/organizations-and-roles). Teammates follow the same delegation, inheritance, and limit rules as any copy of the root; see [Multi-agent](/recursion/multi-agent).
</Note>

## When to use a team

Use a team when the work has two or more independent pieces that each take real effort, or when the result should be checked by someone other than its author. For one precise hand-off, use [delegation](/recursion/multi-agent#how-delegation-works) instead.

Every teammate is a full agent that runs its own model calls, so a team of four costs roughly four times as much per minute as one agent. It pays off when the pieces are substantial and would otherwise run one after another.

<Accordion title="Good and poor fits">
  Good fits:

  * **Competing approaches.** Try three ways to speed up a query, each with its own benchmark, then pick the best one.
  * **Separate components.** Build an API client, a CLI, and their docs, each with its own tests.
  * **Separate questions.** Research three vendors, each from its own sources.
  * **Independent review.** Have a second teammate check each result before the decision.

  Poor fits:

  * Steps that must happen in order, where each step needs the last one's output.
  * Pieces that take a few minutes each. The coordination costs more than it saves.
  * One precise hand-off. Use [delegation](/recursion/multi-agent#how-delegation-works) instead.
</Accordion>

## Choose a team mode

The team mode decides whether a session forms a team.

| Mode | Console label | Behavior |
| - | - | - |
| `auto` | **When the work calls for it** | The default. The agent works alone and forms a team only if it decides the work warrants one, by posting its first task. |
| `on` | **Always** | A team from the first turn. The leader posts the work rather than doing it itself. |
| `off` | **Never** | No board. The agent can still delegate to children and wait for them. |

Set the agent's default under **Multiagent** on the agent's **Configuration** tab, or with `multiagent.team.mode`. Override it for one session with `team.mode` when you start the session:

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Sessions**, then click **Launch session**.
    2. Choose the **Agent** and **Environment**.
    3. Under **Task**, write the brief in **Opening message**.
    4. Under **Team**, set **Work as a team** to **Always**.
    5. Click **Launch session**.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/sessions' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Idempotency-Key: caching-comparison-2026-09-25' \
      -H 'Content-Type: application/json' \
      -d '{
        "agent_id": "5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55",
        "environment_id": "9d3e7b52-1a4c-4f80-b6e9-2c8a5d0f7e13",
        "message": "Compare in-process LRU, Redis, and CDN caching for the catalog API and recommend one.",
        "team": {"mode": "on"}
      }'
    ```
  </Tab>
</Tabs>

The API answers `202 Accepted` with the new session id. Next, watch the board.

```json theme={"theme":"css-variables"}
{
  "session_id": "b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38",
  "status_path": "/managed-agents/v1/sessions/b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38"
}
```

## Write a good team brief

The brief is your opening message. The leader turns it into tasks, and each teammate sees only its own task body, so the brief has to make the split obvious.

1. **Name the independent pieces.** "Compare three approaches: an in-process LRU, Redis, and CDN edge caching."
2. **Say where each piece writes.** "Put each prototype in `/workspace/proto/<approach>/` and its findings in `/workspace/research/<approach>.md`." Separate paths keep teammates from overwriting each other.
3. **Give the shared rules once.** The test command, the data set, and the format every piece must follow.
4. **Ask for review when it matters.** "Have a different teammate check each result's numbers."
5. **Say how results will be judged and what the final deliverable is.** "Pick one approach by p95 latency under the load in `/workspace/bench/run.sh`. Save the recommendation as `caching-recommendation.md` with the numbers for all three."

The [research team tutorial](/recursion/use-cases/research-team) shows a complete example.

## Read the board

The board shows every task with its status, owner, notes, and result summary, plus notices, recommendations, and the decision. It's the authoritative state; refresh from it rather than rebuilding the board from events.

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Sessions**, then open the root session.
    2. Open the **Work** tab. The **Board** lists tasks by status with their owners.
    3. Click a task to see its body, result summary, output paths, and notes.
    4. In the transcript, choose the **Team** filter to see board changes in order.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38/board' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

Any session id in the tree works; the read resolves to the root. Read `tasks[].status` and `owner_session_id` for who holds what, `tasks[].outcome` for finished work, and the `decision` post for the leader's call.

<Accordion title="Example board">
  A board midway through a round looks like this:

  ```json theme={"theme":"css-variables"}
  {
    "as_of": "2026-09-25T17:41:08Z",
    "members": [
      {"session_id": "b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38", "name": "leader", "leader": true, "state": "running"},
      {"session_id": "4a9f2c61-0e3b-4d87-b5a2-7c1e6f9d3b40", "name": "self-1", "leader": false, "state": "idle"},
      {"session_id": "e3c7a915-6b2d-4f08-9a41-2d5b8c0f7e66", "name": "self-2", "leader": false, "state": "running"}
    ],
    "tasks": [
      {
        "task_id": "7d2e4b19-3a5c-4e81-b6f0-9c1a2d7e5b43",
        "seq": 1,
        "kind": "explore",
        "title": "Prototype and benchmark an in-process LRU cache",
        "body": "Build an LRU cache in /workspace/proto/lru/. Run /workspace/bench/run.sh and write p50 and p95 latency to /workspace/research/lru.md.",
        "status": "done",
        "owner_session_id": "4a9f2c61-0e3b-4d87-b5a2-7c1e6f9d3b40",
        "claim_generation": 1,
        "artifacts": ["/workspace/research/lru.md"],
        "outcome": {
          "summary": "p95 fell from 182 ms to 41 ms. RAM use grew by 310 MB per instance.",
          "confidence": "high",
          "artifacts": ["/workspace/research/lru.md"]
        },
        "created_by": "b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38",
        "created_at": "2026-09-25T17:08:02Z",
        "updated_at": "2026-09-25T17:33:47Z"
      },
      {
        "task_id": "a18f6c02-9e4b-4d3a-8c75-1b0e2f9d6a38",
        "seq": 4,
        "kind": "review",
        "title": "Check the LRU benchmark numbers",
        "body": "Rerun /workspace/bench/run.sh against /workspace/proto/lru/ and confirm the p95 in /workspace/research/lru.md. Leave your findings as a note on T1.",
        "status": "claimed",
        "owner_session_id": "e3c7a915-6b2d-4f08-9a41-2d5b8c0f7e66",
        "claim_generation": 1,
        "blocked_on": ["7d2e4b19-3a5c-4e81-b6f0-9c1a2d7e5b43"],
        "created_by": "b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38",
        "created_at": "2026-09-25T17:08:02Z",
        "updated_at": "2026-09-25T17:35:10Z"
      }
    ],
    "posts": [
      {
        "post_id": "c5b1e8a4-2f73-4d96-a0e7-8d3c6b1f4a29",
        "kind": "notice",
        "text": "The load test reads /workspace/bench/catalog-sample.json. Don't edit it.",
        "created_by": "b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38",
        "created_at": "2026-09-25T17:09:15Z"
      }
    ]
  }
  ```
</Accordion>

<Accordion title="What each board field tells you">
  | Field | What to read |
  | - | - |
  | `tasks[].status` and `owner_session_id` | Who holds what. A task claimed for a long time with no update may be stuck. |
  | `tasks[].outcome` | The owner's summary, confidence (`low`, `medium`, or `high`), and output paths, once the task is done. |
  | `tasks[].notes` | Reviewers' findings, blockers, and hand-over details, in order. |
  | `posts[]` with `kind` `recommendation` | A teammate's pick and rationale. |
  | `posts[]` with `kind` `decision` | The leader's call: `task_ids` names the winners, `text` explains why. |
  | `members[]` | Everyone on the board. The leader is named `leader`. Teammates are named after the leader's copies, such as `self-1`. |
</Accordion>

## What success means

A team session is finished when the leader has integrated the results and ended its work, the same as any session. Along the way, a round is over when no task is `open` or `claimed`, and an exploratory round ends with a `decision` post. Deliverables that teammates write count as the root session's deliverables. See [Deliverables and artifacts](/recursion/artifacts).

## How a team works

The leader posts tasks, teammates claim and finish them, and the leader records one decision. As tasks become claimable, copies of the leader join to take them, up to the team size: the roster's `max_concurrent_threads`, which defaults to 8 and can be at most 25.

Teammates are copies of the leader, so they have the leader's model, prompt, skills, MCP servers, and exactly the same vault and credential grants. See [What a child inherits](/recursion/multi-agent#what-a-child-inherits). When you [move the leader's session to another agent version](/recursion/sessions#move-a-session-to-another-version), each teammate follows before its own next model call. Team mode and seats are fixed when the session starts: a move never adds seats, seats can shrink with the new version's `max_concurrent_threads`, and the board keeps its tasks and claims.

<Accordion title="Tasks, statuses, and team rules">
  * **Tasks.** Each task has a label such as `T1`, a kind, a self-contained body, and the output paths it will write. Kinds are `task` (work with a named output), `explore` (one approach to an open question), and `review` (check another task's output and leave findings in a note). A task can wait on others with `blocked_on`.
  * **Statuses.** `open` (posted and not held; claimable once every task in its `blocked_on` is done), `claimed` (in progress), `blocked` (the owner can't proceed and says why in a note), `done` (finished, with a result summary), and `dropped` (withdrawn).
  * **One owner per task.** A claim belongs to one teammate. A teammate that joined from the board can change the shared sandbox only while it holds a claimed task. A teammate can't claim the review of a task it did.
  * **Shared state.** Notices are facts every member must know, such as a moved path; the latest 10 stay in front of every member. Teammates can message each other directly, up to 20 messages per assignment, and put what they agree on the board.
  * **One decision stands.** Teammates post recommendations. Only the leader records a decision, which names the winning tasks and the reason. If the leader records another, the newest one stands.
</Accordion>

## What can go wrong

The most common problems:

* **The board is empty under `auto`.** The agent decided to work alone. Use `on` when you want a team regardless.
* **Two teammates overwrote one file.** The brief didn't give each piece its own path. Name a distinct output path for every piece.
* **The cost is higher than expected.** Every teammate runs its own model calls. Use a team only for substantial parallel pieces, and check the session's cost in [Usage and cost](/recursion/usage-and-cost).

<Accordion title="Every team error">
  | Symptom or code | Cause | Fix |
  | - | - | - |
  | `404 not_found` from the board | The session runs with team mode `off`. | Start the session with `team.mode` `auto` or `on`. |
  | The board is empty under `auto` | The agent decided to work alone. | Use `on` when you want a team regardless. |
  | Two teammates overwrote one file | The brief didn't give each piece its own path. | Name a distinct output path for every piece. |
  | Teammates report conflicting formats | The shared rules weren't in the brief. | State the format, test command, and data once in the brief. |
  | A task stays `claimed` with no progress | The owner is stuck on a long step or waiting on something. | Read its notes in the **Work** tab. The leader can message the owner or post an alternative. |
  | Fewer teammates than tasks | The team is at its size, `max_concurrent_threads`. | Expected. Teammates take the rest as they finish. Raise the limit if the pieces are all substantial. |
  | The cost is higher than expected | Every teammate runs its own model calls. | Use a team only for substantial parallel pieces, and check the session's cost. See [Usage and cost](/recursion/usage-and-cost). |
</Accordion>

## Limits

| Limit | Value |
| - | - |
| Team size | The roster's `max_concurrent_threads`: default 8, maximum 25 |
| Notices kept in front of every member | The latest 10 |
| Direct messages from a teammate to other teammates | 20 per assignment |
| Tasks on one board | 200 |
| Notices and recommendations on one board | 200 |

See [Limits](/recursion/limits) for every product limit.

## Next steps

<CardGroup>
  <Card title="Research team tutorial" href="/recursion/use-cases/research-team">
    Run a research team session end to end.
  </Card>

  <Card title="Multi-agent" href="/recursion/multi-agent">
    Delegation, rosters, inheritance, and limits.
  </Card>

  <Card title="Usage and cost" href="/recursion/usage-and-cost">
    See what a team session cost.
  </Card>

  <Card title="Deliverables and artifacts" href="/recursion/artifacts">
    How team outputs are kept.
  </Card>
</CardGroup>


# Tools
Source: https://docs.labelbox.com/recursion/tools

See which tools an agent gets in a session, turn off web search, and withhold integration tools.

Tools let an agent run commands, work with files, read the web, and coordinate with other agents. You don't list built-in tools on an agent. A session receives them from its agent version, its environment, and the features the agent uses, and the set is fixed when the session starts. It changes only when you [move it to another agent version](/recursion/sessions#move-a-session-to-another-version).

<Note>
  Creating, changing, or deleting agents and their tool settings needs the organization developer or admin role. The organization user role can view them. See [Organizations and roles](/recursion/organizations-and-roles). For the API, create a key on [API keys](/recursion/api-keys) and export it as `RECURSION_API_KEY`. Tool settings belong to an agent version, and every change is a new version, so read [Update an agent](/recursion/agents#update-an-agent) first.
</Note>

## How a session gets its tools

* **The agent version** decides web search, remote [MCP servers](/recursion/mcp-servers), [skills](/recursion/skills), the apps it's granted through [integrations](/recursion/integrations), and [multi-agent](/recursion/multi-agent) roles.
* **The environment** decides the sandbox tools. `computer` needs an environment with computer use turned on. See [Environments](/recursion/environments).
* **The model** decides `view_image`, which is offered only to models that accept images.

A running session keeps its tool set. A new agent version reaches sessions that start afterward, and a running session only when you [move it to that version](/recursion/sessions#move-a-session-to-another-version). A new tool on an MCP server reaches sessions that start afterward, and a running session when it next moves to another agent version.

## Turn off web search

`web_search_enabled` defaults to `true`. Set it to `false` to remove `web_search`. The agent can still read a known URL with `web_fetch`, except in sessions started by an automation or a Slack thread in an environment that limits internet access.

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Agents**, then open the agent.
    2. On **Configuration**, find **Tools**.
    3. Clear **Web search**.
    4. Click **Save new version**.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/agents/5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55/versions' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      -d '{
        "base_agent_version_id": "c1a94e07-2f6b-4d18-b3a5-0e7d8c6f4a21",
        "name": "Release notes writer",
        "model": "<model-id>",
        "system": "Write release notes from the repository.",
        "web_search_enabled": false,
        "disabled_integration_mcp_providers": ["jira"]
      }'
    ```
  </Tab>
</Tabs>

A `201` response is the agent at its new version. Some fields are left out here.

```json theme={"theme":"css-variables"}
{
  "agent_id": "5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55",
  "latest_agent_version_id": "7a4e1c95-6d2f-4380-b915-3e7a0c6f2d84",
  "web_search_enabled": false,
  "disabled_integration_mcp_providers": ["jira"]
}
```

A version is a full replacement, so resend `mcp_servers`, `skills`, vault defaults, and anything else you want to keep. In a multi-agent roster, a subagent can turn web search off but not on.

## Withhold integration tools

`disabled_integration_mcp_providers` lists the [native integration](/recursion/integrations#native-integrations) providers whose MCP tools an agent's sessions do not receive, even when the agent has access to the connection. In the console, it's the MCP tools checkbox on each native connection in the agent's **Integrations** section. The `jira` entry in the example above withholds Jira's MCP tools.

<Accordion title="Field rules">
  * Supported values are `confluence`, `github`, `jira`, `launchdarkly`, and `loom`. An unknown value returns `400 invalid_request`.
  * Omitting the field means an empty list.
  * `github` is accepted but ignored: a GitHub grant works only through GitHub's MCP tools.
  * The field never affects servers you list in `mcp_servers`.
  * It doesn't affect [built-in integrations](/recursion/integrations), whose tools are controlled by each app's allow-list.
</Accordion>

## Read a session's tool set

A session records the tool settings of the version it runs in `config`. Read the session to see them.

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Sessions**, then open the session.
    2. Each tool call in the transcript shows the tool's name and input.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

```json theme={"theme":"css-variables"}
{
  "session_id": "b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38",
  "status": "active",
  "execution_state": "running",
  "sandbox_provider": "runs",
  "config": {
    "available_builtin_tools": [
      "web_fetch",
      "write_artifact",
      "read_artifact",
      "get_events",
      "delegate_subagent"
    ]
  }
}
```

This session's agent has web search turned off, so `web_search` is missing. `config` is left out when you request `view=summary`.

`config.available_builtin_tools` lists which of `web_search`, `web_fetch`, `write_artifact`, `read_artifact`, `get_events`, and `delegate_subagent` the session may use. The session's top-level `sandbox_provider` names the sandbox its environment used. Sandbox and feature tools follow the rules in [Built-in tools](#built-in-tools).

## Built-in tools

A session's tools come in three groups. Tools from [MCP servers](/recursion/mcp-servers) are added next to these. Their names and schemas come from the server and are fixed when the session starts, and listed again when the session moves to another agent version.

**Sandbox tools** run inside the session's sandbox and see its `/workspace`: `run_command`, `write_file`, `read_file`, `share_file`, `write_artifact`, `read_artifact`, `release_compute`, `screenshot`, `computer`, and `handoff_to_user`.

<Accordion title="Every sandbox tool">
  | Tool | What it does |
  | - | - |
  | `run_command` | Runs a shell command in the session's persistent shell. The working directory and exported variables carry over between calls. A command whose plain `sleep` calls add up to five minutes or more is refused; the agent should call `sleep` instead. |
  | `write_file` | Writes a file at an absolute path, creating parent directories. `append: true` adds to the file instead of replacing it. |
  | `read_file` | Reads a text file. PNG, JPEG, and WebP files come back as images for models that accept images. |
  | `share_file` | Shows an image under `/workspace` (PNG, JPEG, or WebP) to the person watching the session, with an optional `caption`. The image is not sent back to the model. |
  | `write_artifact` | Writes a deliverable at a path relative to `/workspace/.managed-agents/outputs`. Everything in that directory is kept as a [deliverable](/recursion/artifacts) when the turn ends. Empty content is refused. |
  | `read_artifact` | Reads a deliverable back by the same relative path. |
  | `release_compute` | Gives the compute back while the root session waits for a message or a timer. Files under `/workspace` survive; running processes and anything installed outside `/workspace` do not. Root sessions only. |
  | `screenshot` | Renders a `url` or a local HTML `path` to a PNG with a headless browser. Width is 200 to 4096 pixels (default 1280), height is 200 to 8192 (default 800), and `wait_ms` is at most 30000 (default 1500). Offered when the sandbox can launch the browser. |
  | `computer` | Drives a shared 1280 by 800 browser display. Offered when the environment has computer use turned on and the display started. Subagents share the same display. |
  | `handoff_to_user` | Asks a person to handle a step in the browser display the agent can't do, such as a sign-in. The session shows `awaiting_human`. The agent resumes when you send a `handoff_resolved` event, or when the hand-off deadline passes: 30 minutes by default, at most 4 hours. Interrupting or cancelling the session also ends the wait. Root sessions with computer use only. |

  Under a restricted [network policy](/recursion/environments-reference), browser tools reach an external site only when the destination is allowed.
</Accordion>

**Tools that run outside the sandbox** are not subject to the environment's network policy: `web_search`, `web_fetch`, `get_events`, `delegate_subagent`, `sleep`, `update_scratchpad`, and `view_image`.

<Accordion title="Every tool that runs outside the sandbox">
  To protect restricted environments, a session started by an [automation](/recursion/automations) or a Slack thread doesn't get `web_fetch` when its environment has a network policy that limits internet access.

  | Tool | What it does |
  | - | - |
  | `web_search` | Searches the public web and returns ranked sources with excerpts, at most 10 results. Offered when `web_search_enabled` is `true`. |
  | `web_fetch` | Fetches one public `https` URL and returns readable text. Plain `http`, private, loopback, and link-local addresses are refused. Turning off web search does not remove it. |
  | `get_events` | Reads older parts of the session's own event log that no longer fit in the model's context. |
  | `delegate_subagent` | Hands a focused task to a subagent: a copy of the agent, or a member of its [multi-agent](/recursion/multi-agent) roster. Removed once the session reaches the delegation depth limit. |
  | `sleep` | Pauses the session until a message arrives, a subagent reports, or an optional `wake_after_seconds` timer fires. A sleeping session holds no running turn. |
  | `update_scratchpad` | Updates the session's pinned working notes, which stay in the prompt every turn and are visible to you. A subagent's notes are also visible to the agent that delegated to it. |
  | `view_image` | Looks again at an image the session already saw. Offered to models that accept images. |
</Accordion>

**Tools added by agent features** appear when the agent uses skills, subagents, an advisor, teams, referenced sessions, agent file history, or memory.

<Accordion title="Every feature tool">
  | Feature | Tools |
  | - | - |
  | [Skills](/recursion/skills) | `activate_skill` loads one attached skill's full instructions. Offered when the session has at least one usable skill. |
  | [Subagents](/recursion/multi-agent) | A session that can delegate also gets `send_to_agent`, `list_agents`, `wait_for`, `interrupt_agent`, and `archive_thread` to manage its subagents. A subagent gets `message_parent` and `submit_result` to report back. |
  | Advisor | `consult_advisor` asks the roster's advisor for guidance. Offered when the roster has an advisor. |
  | [Teams](/recursion/teams) | Every member shares a task board through `post_task`, `update_task`, `add_note`, `list_tasks`, `post_notice`, and `add_teammates`. The leader also gets `decide`. Other members get `claim_task` and `recommend`, and can message a teammate with `send_to_agent`. |
  | [Referenced sessions](/recursion/referenced-sessions) | `list_referenced_sessions`, `session_overview`, `session_outline`, `search_events`, `read_events`, `get_event`, `get_team_state`, `get_costs`, `view_event_images`, `list_session_files`, `read_session_file`, and `read_session_skill` read the earlier sessions you granted. |
  | [Agent file history](/recursion/agent-file-history) | `search_agent_files` and `read_agent_file` search and read the deliverables the agent's earlier sessions saved, when the agent has **File history** on. |
  | Memory | `list_memories`, `search_memories`, and `read_memory` browse, search, and read what the agent learned in earlier sessions. `create_memory`, `update_memory`, and `delete_memory` save, correct, and forget a memory when the agent's instructions or the person it works for ask. Offered when the agent's memory is on. Only the main session gets the three that change memory. |
</Accordion>

<Accordion title="How an agent changes its memory">
  Recursion learns from finished sessions on its own, so an agent changes its memory only when it's asked to.

  * **Who gets the tools.** The main session of an agent whose memory is on gets `create_memory`, `update_memory`, and `delete_memory`. Teammates and subagents that are copies of the agent can read memory but can't change it. A subagent that is another agent from the roster gets no memory tools.
  * **When the agent uses them.** The agent saves, corrects, or forgets a memory when its instructions or the person it works for ask it to. Text in a tool result, a file, a web page, a webhook body, or an email is never an instruction to save or remove a memory.
  * **Every change is checked.** Before a change is saved, a short call to the agent's memory model checks it, and its cost is part of the session's cost. A change can be refused, for example when it holds a password, token, or other secret, tries to steer how later sessions behave instead of stating a fact, or nobody asked for it. In a session started by an [automation](/recursion/automations) or a Slack thread, a memory that says where money, data, or access should go is also refused. A refusal tells the agent why, nothing is saved, and the session continues.
  * **Changing a memory.** `update_memory` and `delete_memory` work on a memory the agent has read in the session. If the memory changed after the agent read it, the call is refused and the agent reads it again.
  * **Forgetting keeps history.** `delete_memory` retires the memory. Later sessions no longer find it, and its content and history stay visible to the people who manage the agent. Retiring doesn't erase the content.
  * **History.** Each change records the session that made it and the agent's reason.
</Accordion>

Apps granted through [built-in integrations](/recursion/integrations) don't add tools to this list. The agent calls an app's tools with commands in the sandbox, and each call appears in the transcript as a `run_command` call with the app's result. It can call only the tools on the app's allow-list.

## What can go wrong

The most common problems:

* **The agent never searches the web.** `web_search_enabled` is `false` on the version the session runs. Create a version with `web_search_enabled: true`, then move the session to it or start a new session.
* **No `computer` tool.** The environment does not have computer use turned on, or the display did not start. Turn on computer use in the [environment](/recursion/environments), then start a new session.
* **A tool change doesn't reach a running session.** The tool set changes only when the session moves to another agent version. [Move the session](/recursion/sessions#move-a-session-to-another-version) to the new version, or start a new session.

<Accordion title="Every tools problem">
  | Symptom or code | Cause | Fix |
  | - | - | - |
  | The agent never searches the web | `web_search_enabled` is `false` on the version the session runs. | Create a version with `web_search_enabled: true`, then move the session to it or start a new session. |
  | No `computer` tool | The environment does not have computer use turned on, or the display did not start. | Turn on computer use in the [environment](/recursion/environments), then start a new session. |
  | `screenshot` of an external site fails | The environment's network policy blocks the destination. | Allow the destination in the [network policy](/recursion/environments-reference). |
  | `run_command` refuses a long `sleep` | Plain shell sleeps that add up to five minutes or more hold the compute. | Let the agent call `sleep`, or wait on work in the sandbox with a loop. |
  | No `web_fetch` in a session started by an automation or Slack | The environment has a network policy that limits internet access. | Use an environment without a network policy, or have the agent reach allowed hosts from the sandbox. |
  | A tool change doesn't reach a running session | The tool set changes only when the session moves to another agent version. | [Move the session](/recursion/sessions#move-a-session-to-another-version) to the new version, or start a new session. |
  | The agent can't save or forget a memory | The session is a subagent or teammate, the agent's memory is off, the session reached its limit of memory changes, or the check refused the change. | Ask in the main session, and read the refusal in the transcript. The agent still learns from the session when it finishes. |
  | `400 invalid_request` on `disabled_integration_mcp_providers` | The list names an unsupported provider. | Use only the values in [Withhold integration tools](#withhold-integration-tools). |
  | `409 revision_conflict` | `base_agent_version_id` is no longer the latest version. | Get the agent, apply your change again, and retry. |

  The full error catalog is on [Errors](/recursion/errors).
</Accordion>

## Limits

| Limit | Value |
| - | - |
| `web_search` results per call | 10 |
| `screenshot` size | 200 to 4096 pixels wide, 200 to 8192 pixels high |
| `screenshot` wait | 30000 ms |
| `computer` display | 1280 by 800 pixels |
| Straight-line shell sleeps in one `run_command` | Under five minutes |
| Memory changes by one session | 20 in any 24 hours. At most 5 of them can be a `delete_memory` or an `update_memory` that moves the memory to a new path. A refused change can still count. |
| Content of a memory the agent saves | 16 KiB |

See [Limits](/recursion/limits) for request and session limits.

## Next steps

<CardGroup>
  <Card title="MCP servers" href="/recursion/mcp-servers">
    Add remote tools and keep their credentials in a vault.
  </Card>

  <Card title="Environments" href="/recursion/environments">
    Choose the compute, network policy, and computer use.
  </Card>

  <Card title="Deliverables and artifacts" href="/recursion/artifacts">
    Ask for deliverables and see how they're kept.
  </Card>

  <Card title="Skills" href="/recursion/skills">
    Give the agent procedures it loads when they apply.
  </Card>
</CardGroup>


# Troubleshooting
Source: https://docs.labelbox.com/recursion/troubleshooting

Match a Managed Agents symptom or error code to its cause and fix, for sign-in, keys, organizations, billing, environments, integrations, files, automations, MCP servers and credentials, and sessions.

Start with the stable field, not the message: the response's `code`, a session's `failure.code`, or a setup run's `failure_code`. Messages can change; codes don't. HTTP errors are flat objects with required `code` and `message` fields and optional `details`. Every code is listed in [Errors](/recursion/errors).

## Sign-in and the console

| Symptom | Cause | Fix |
| - | - | - |
| **Sign-in is limited to approved accounts.** | Your account isn't approved for access. | Sign in with an approved account. |
| **Verify your email address with your provider, then sign in again.** | Your identity provider hasn't verified your email address. | Verify it with the provider, then sign in again. |
| **That sign-in link is no longer valid.** | The sign-in attempt can no longer be completed. | Start again from the sign-in page. |

## API keys and authentication

| Code or symptom | Cause | Fix |
| - | - | - |
| `401 unauthorized` | The `Authorization` header is missing or malformed, or the key expired, was revoked, or is inactive. | Send `Authorization: Bearer $RECURSION_API_KEY`. Check the key's status under **API keys**. |
| `401` right after exporting a key | The key was exported in a different shell, or copied with extra characters. | Export it in the shell that runs your code, then check it starts with `rma_`. |
| `400 invalid_request` mentioning `x-organization-id` | A tenant-scoped key sent no `x-organization-id`. | Add `x-organization-id: default` or an organization id, or use an organization-scoped key. See [API keys](/recursion/api-keys). |
| `404 not_found` on every request | `x-organization-id` names an organization you can't reach. | Use `default`, or remove the header for an organization-scoped key. |
| A key stopped working without being revoked | It expired. | Check its status, then create a new key. |
| `503 service_unavailable` before anything ran | The key couldn't be checked right now. | Retry shortly. |

## Organizations and roles

| Code or symptom | Cause | Fix |
| - | - | - |
| `403 forbidden` on create, update, or delete | Your organization role is read-only. | See [Organizations and roles](/recursion/organizations-and-roles). Use an account or key with the developer or admin role. |
| `403 forbidden` when testing an environment's setup | Testing setup needs create and update permission, because it provisions compute. | See [Organizations and roles](/recursion/organizations-and-roles). Use an account or key with the developer or admin role. |
| **Test connection** is missing on an app's row, an MCP server's test says **Testing requires create access.**, or `probeMcpServer` returns `403 forbidden` | Testing a connection or an MCP server needs create permission, which the **Organization user** role doesn't have. | Use an account or key with the developer or admin role. |

## Billing

| Symptom | Cause | Fix |
| - | - | - |
| **Billing** isn't in the sidebar | Your tenant role isn't billing, admin, owner, or primary owner. | Use an account with one of those roles. |
| "The card was declined. Replace it in Stripe, then try again." | The bank declined the charge. | Click **Replace card** below the message, choose another card on Stripe, then come back and buy again. |
| Starting a session or sending a message returns `403 session_start_not_admitted` with `details.reason: "out_of_credit"` | Your tenant has used the credits it bought. New sessions and messages that start a turn are refused. Interrupts still work. | Buy credits, then start the session or send the message again. |
| A session is `idle` with `stop_reason: out_of_credit` | Your tenant ran out of credit while the session was running. It paused before its next model request. Requests already in flight for the tenant complete and are charged, so the balance can end slightly below zero. | Buy credits. It continues on its own within about a minute if it paused in the last 72 hours; otherwise send it a message. A session bound to a Slack thread continues on your next reply there. |
| A session is `idle` with `stop_reason: tenant_suspended` | Your account was suspended while the session was running. The session paused and keeps its data. | Contact support to restore the account, then send a message to continue. |
| Testing setup returns `403 setup_run_not_admitted` with `details.reason: "out_of_credit"` | Your tenant has used its credits, so no setup run was recorded. | Buy credits, then test setup again. |
| Starting work or testing setup returns `403 session_start_not_admitted` or `403 setup_run_not_admitted` with `details.reason: "tenant_suspended"` | Your account is suspended. New work is refused and running sessions pause with `stop_reason: tenant_suspended`; your data is kept. Buying credits does not restore it. | Contact support. |
| Testing setup returns `503 setup_run_admission_unavailable` | The credit check was unavailable, so no setup run was recorded. | Retry with the same `Idempotency-Key`. |

See [Billing](/recursion/billing).

## Environments and setup

| Code or symptom | Cause | Fix |
| - | - | - |
| `422 environment_not_verified` on session start | The setup script has never passed, failed, changed since it passed, or is still being tested. `details.next_action` names the call to make. | Run a setup test, wait for `verified` with `stale: false`, then start again. |
| Setup test fails with `phase: setup` | The script exited with an error or timed out. | Read `failed_line`, `failed_command`, `exit_code`, `hint`, and `stderr_tail`. Fix the script and test again. |
| Setup test fails in another phase | A temporary problem preparing compute. | Retry once before changing the script. |
| `429 setup_run_limit` | Your organization has too many setup runs in flight. | Wait for one in `details.in_flight`, or cancel it. |
| `409 setup_run_in_progress` | A setup test is already running for this environment. | Wait for `details.setup_run_id`, or cancel it. |
| `428 precondition_required` on an environment update | You changed a stored network policy, or turned on privileged access, without `expected_access`. | Read the environment and send its current access settings. |
| Package installs fail in setup or in the session | The environment has no internet access. | Allow the needed destinations, or set **Internet access** to **Enabled**, then test again. |

See [Environments](/recursion/environments).

## Integrations

| Code or symptom | Cause | Fix |
| - | - | - |
| **Add integration** is missing on **Integrations** | Your role can view integrations but not add them. | Ask someone with the organization developer or admin role. |
| An app's row stays **Pending setup** | Nobody finished signing in to the app. | Click **Connect** on the row and finish the sign-in. |
| An app's row shows **Revoked**, or the agent can't reach an app | The app no longer accepts the sign-in. | Reconnect the app on **Integrations**. See [Reconnect an app](/recursion/integrations#reconnect-an-app). |
| The agent can't use a tool in an app | The tool isn't on the app's allow-list, or the app isn't granted to the agent. | Add the tool with **Tools** on the app's row, or grant the app to the agent, then move the session to the new version or start a new session. |
| Sessions stopped using an app after a catalog update | A tool on the allow-list was reclassified or removed. | Open **Tools** on the app's row, click **Remove**, then **Save tools**. |
| **Reconnect first** when granting an app to an agent | The connection isn't active. | Connect or reconnect it on **Integrations**. |
| `400 invalid_request` on `built_in_integrations` | More than 25 entries, a non-empty `tools`, or a connection that isn't an active built-in integration with at least one tool. | Read `message` and fix the entry. See [Grant apps to an agent](/recursion/integrations#grant-apps-to-an-agent). |
| A session has an `integration_token_mint_failed` warning | Recursion couldn't get a token for the agent's grant from the connected app. The message names the repository or account and the reason, such as a repository that isn't in the GitHub App installation. | Fix what the message names: add the repository to the GitHub App installation and **Reconnect** the connection, or remove it from the grant. If GitHub was unavailable, start a new session. See [GitHub access](/recursion/github). |
| `git` or `gh` is unauthenticated in a session | Expected: GitHub access goes through the agent's GitHub tools, not the sandbox. | Grant GitHub on the agent and ask it to use its GitHub tools. See [GitHub access](/recursion/github). |
| `git` can't reach `github.com` | The environment's network policy blocks GitHub. | Allow the GitHub hosts. See [Allow specific hosts](/recursion/environments#allow-specific-hosts). |

See [Integrations](/recursion/integrations#what-can-go-wrong) for every integration error.

## Files and automations

| Symptom or code | Cause | Fix |
| - | - | - |
| `413 payload_too_large` on public REST `uploadFile` | The file part is over 30 MiB, or its multipart request exceeds the framing allowance. | Upload at most 30 MiB through the API, or use the console for a larger file. See [Files](/recursion/files). |
| `413 file_quota_exceeded` on upload | Your organization's files are at their storage quota. | Set expiries on uploads so they stop counting once they expire. See [Files](/recursion/files). |
| The agent can't find an attached file | It was attached mid-turn. | It's in place on the next tool call and visible from the next turn. |
| A deliverable isn't on the session's **Files** tab | The agent saved it outside the deliverables folder. | Ask for it by name as a deliverable. See [Deliverables and artifacts](/recursion/artifacts). |
| An automation never starts a run | It's paused, its trigger is off, or its event source is paused. | Resume the automation and the source. See [Automations](/recursion/automations). |
| A webhook delivery to a custom source returns `401 unauthorized` | The signature didn't match the source's signing secret. | Sign the exact body bytes with the source's signing secret. See [Send a custom webhook delivery](/recursion/automations#send-a-custom-webhook-delivery). |
| An automation run is `failed` | The session couldn't start. | Read the run's `error`, fix the named field or resource, and run it again. |
| An automation paused itself, or resuming it is refused | Its environment was deleted, or it names an agent that no longer exists. | Point the automation at a live agent and environment, then resume it. See [Automations](/recursion/automations). |

## MCP servers and credentials

| Code or symptom | Cause | Fix |
| - | - | - |
| MCP tools are missing from a session | The server couldn't be reached, its handshake failed, it refused the token, or no granted credential matched its URL. | Probe the server and fix the URL, server, token, or vault grant, then start a new session. A running session lists a server's tools again only when you [move it to another agent version](/recursion/sessions#move-a-session-to-another-version). See [MCP servers](/recursion/mcp-servers). |
| An environment-variable secret shows up in the transcript | The agent printed it. Any command in the sandbox can read these secrets. | Rotate the secret. Prefer an MCP credential, which never enters the sandbox. See [Security](/recursion/security). |
| Session fails with `credential_resolution_failed` | A granted credential couldn't be opened, or was deleted. | Check the session's vaults and `credential_refs`. |

## Sessions

| Code or symptom | Cause | Fix |
| - | - | - |
| Session stays `queued` | The agent is at its `max_concurrent_sessions` cap. Child sessions don't count. | Wait for a root session to finish, hard stop one you don't need, or raise the cap. See [Usage and cost](/recursion/usage-and-cost). |
| Session stays `provisioning` | Compute is being prepared. | Read the session, not just its events. After 5 minutes it fails with `sandbox_provision_timeout`; resume it or start again. |
| No events yet, but the session is `running` | The event stream is written separately and can lag the session record. | Trust `status` and `execution_state` from `getSession`. |
| Session `failed` | See `failure.code`, `failure.category`, and `failure.retryable`. | `transient`: send a follow-up to resume. `caller_error`: fix the configuration first. `internal`: report the `session_id`. See [Session failure codes](/recursion/errors#session-failure-codes). |
| `422 sandbox_provider_disabled` | The request names a runtime this deployment doesn't offer for new work. | Use the **Agent runner** runtime (`runs`). |
| Session is `idle` with `end_turn` but the work is incomplete | The agent stopped on its own. `end_turn` doesn't mean correct. | Send a follow-up message with what's missing. |
| `409 idempotency_conflict` | You reused an `Idempotency-Key` with a different request. | Use a new key for a new request. |
| Event stream returns `503 service_unavailable` | The service is at event-stream capacity. | Close streams you no longer need, then retry with backoff. |
| Event stream disconnects | Networks drop long connections. | Reconnect with `Last-Event-ID` or `after_event_id` set to the last event you received. See [Events](/recursion/events). |
| A follow-up after a hard stop starts from a clean workspace | Hard stop tears down the sandbox. | Use **Interrupt** instead when you want to keep the sandbox. |

## Rate limits and server errors

| Code | Cause | Fix |
| - | - | - |
| `429 rate_limit_exceeded` | More than 300 requests in 60 seconds against one of your allowances: all your API keys share one, and console use has its own. | `Retry-After` tells you how long to wait, not whether replay is safe. Retry only when the operation's strategy in **Endpoints** permits it and `details.retryable` is not `false`. |
| `503` | A dependency is temporarily unavailable, or a keyed request could not reserve its idempotency key. | Read `code` and `details.retryable`. Retry only when the operation's strategy in **Endpoints** permits it. `idempotency_unavailable` did not run the request, so resend the same key and request bytes. |
| `504 gateway_timeout` | The request took too long and may have run. | Retry a keyed mutation only with the same key and request bytes. Read the resource before deciding whether to repeat any other write. |
| `500 internal_error` | An unexpected failure. | Retry reads once. For a write, follow the operation's strategy in **Endpoints**. If it persists, report the request id. |

## What to include in a support request

| Field | Why |
| - | - |
| `code` | Selects the recovery path. Do not match `message`. |
| `details.field` | When present, names the request field, query parameter, or header at fault. |
| `details` | When present, holds code-specific state, measured values, limits, and next actions. |
| `details.retryable` | When present, advises whether the failure is transient. It does not override the operation's retry strategy. |
| `details.requestId`, or the `x-request-id` header | Lets support find the request. |
| `session_id` | Identifies the full session tree and its events. |
| `failure.code` and `failure.phase` | Say which stage of a session failed. |
| `setup_run_id` | Identifies a setup test and its log. |
| `recursion-organization-id` response header | Confirms which organization the request ran in. |

Never include an API key or secret value in a support request.

## Next steps

<CardGroup>
  <Card title="Errors" href="/recursion/errors">
    Every error and failure code with retry guidance.
  </Card>

  <Card title="Limits" href="/recursion/limits">
    Every limit you can hit.
  </Card>

  <Card title="Event types and states" href="/recursion/reference">
    Decode statuses, stop reasons, and results.
  </Card>

  <Card title="API conventions" href="/recursion/api">
    Retries, idempotency, and paging.
  </Card>
</CardGroup>


# Track usage and cost
Source: https://docs.labelbox.com/recursion/usage-and-cost

Read what a session cost, including its helper agents, tools, and sandbox compute, and token usage for many sessions at once.

Every session has one cost: what your credit balance is charged for its whole session tree. It covers the session and every subagent and teammate it started, and all of their model calls, provider tools such as a provider's web search, and sandbox compute. Read it in the console or as `costUsd` on the session, and use `costState` to tell a final cost from one that can still grow.

<Note>
  Any organization role that can read sessions can read their cost: user, developer, or admin. The billing role cannot see sessions. See [Organizations and roles](/recursion/organizations-and-roles) and [API keys](/recursion/api-keys). Credits are bought separately on the **Billing** screen; see [Billing](/recursion/billing).
</Note>

<Tip>
  To estimate cost before a session, open an agent in the console. Its **Estimated cost per session hour** section shows what one session hour typically costs, priced on the compute you choose. Pick a model first to see an estimate. [Pricing](/recursion/pricing) explains what that price covers.
</Tip>

| Question | Operation |
| - | - |
| What did this session cost, and is that final? | `getSession` |
| What did each run in a list cost? | `listSessions` with `root_only=true` |
| How many events and tokens did each session in a list use? | `listSessionUsage` |

## Read one session's cost

`getSession` returns `costUsd` and `costState` with the rest of the session, in both the `full` and `summary` views.

<Tabs>
  <Tab title="Console">
    The console shows one figure for the whole session tree: the session and every subagent and teammate under it.

    1. In **Sessions**, read the **Cost** column.
    2. Open the session. The session header and the **Session** tab of its details show the same **Cost**.
    3. While the figure can still change, it reads **so far**. Session costs can take up to an hour to settle.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38?view=summary' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

Some fields are omitted from this response.

```json theme={"theme":"css-variables"}
{
  "session_id": "b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38",
  "root_session_id": "b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38",
  "session_path": "/",
  "status": "completed",
  "execution_state": "idle",
  "stop_reason": "end_turn",
  "costUsd": "1.234567",
  "costState": "final"
}
```

Record `costUsd` once `costState` is `final` (see [Know when a figure is final](#know-when-a-figure-is-final)).

<Accordion title="Cost fields">
  | Field | Description |
  | - | - |
  | `costUsd` | The billed cost of the whole session tree in US dollars: model calls, tools, and sandbox compute. This is what your balance is charged. Compute that is still running is included as an estimate. |
  | `costState` | `final` or `so_far`. Present exactly when `costUsd` is. |

  A subagent or teammate session returns the cost of its whole tree, the same figure as its root. The cost is one total; it is not split by session, model call, or kind of charge. Read the root session to learn when a cost is `final`: a subagent or teammate session can still report `so_far` after its tree has settled.

  A `so_far` cost can be behind what the session has cost. When the cost can't be read again in time, `costUsd` is the last amount that was read. When running compute can't be priced right now, `costUsd` leaves it out until it can be.

  Both fields are absent when there is no cost to return yet. An absent cost is not zero; read the session again later.
</Accordion>

## Compare many sessions

Rows from `listSessions` carry `costUsd` and `costState` too. Send `root_only=true` to get one row per run. A child session repeats its tree's cost, so adding child rows counts the same cost twice.

<Tabs>
  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl 'https://api.recursion.labelbox.com/managed-agents/v1/sessions?root_only=true&limit=50' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

For token counts, `listSessionUsage` takes `session_ids`, a comma-separated list of up to 1000 ids.

<Tabs>
  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/usage?session_ids=b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38,0c6f2b9e-4d17-4a38-b5e2-9f1a3c7d8e06' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

```json theme={"theme":"css-variables"}
{
  "session_usage": [
    {
      "session_id": "b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38",
      "event_count": 214,
      "input_tokens": 182340,
      "output_tokens": 9215,
      "cache_read_tokens": 140112,
      "cache_write_tokens": 21480,
      "cost_micros": 420000
    }
  ]
}
```

<Accordion title="Usage rows">
  * `listSessionUsage` returns `session_usage`: the event count and token counts for each session's own events. Children are not included. Unknown ids and sessions with no events are left out, so compare the number of rows with the number of ids you sent.
  * `cost_micros` is the cost the session's own model events reported, in millionths of a US dollar. It is for diagnostics, not what your balance is charged: it leaves out child sessions and sandbox compute. Use `costUsd` on the session for what you are charged.
</Accordion>

## Know when a figure is final

| `costState` | Meaning |
| - | - |
| `so_far` | The cost can still grow. |
| `final` | The session has ended, its sandbox has stopped, and every charge has settled. `costUsd` won't change unless you resume the session. |

<Accordion title="Why a cost can still grow">
  1. **The session is still running.** The session or one of its subagents or teammates is working or waiting for you.
  2. **The sandbox is still running.** A sandbox can keep running after the session ends, until the environment's idle stop. Its compute is included as an estimate until it stops.
  3. **Charges are still settling.** Session costs can take up to an hour to settle.
  4. **You resumed the session.** A message that starts a new turn makes a `final` cost `so_far` again.
</Accordion>

`costUsd` is an exact decimal string, such as `"1.234567"`. Add costs with a decimal type, such as `Decimal` in Python or a decimal library in TypeScript. Floating-point sums drift.

<Accordion title="Amount format">
  * All amounts are in US dollars. No other currency is used.
  * `costUsd` has up to nine decimal places and no trailing zeros. A zero cost is `"0"`.
  * `cost_micros` in usage rows is an integer in millionths of a dollar.
</Accordion>

## What can go wrong

The most common problems:

* **The cost changed after you recorded it.** `costState` was `so_far`. Record the cost once it is `final`.
* **A list adds up to more than the runs cost.** Child rows repeat their tree's cost. Send `root_only=true`.
* **Usage `cost_micros` and `costUsd` differ.** `cost_micros` covers only the session's own model events and is for diagnostics. Use `costUsd`.

<Accordion title="Every usage and cost error">
  | Symptom or code | Cause | Fix |
  | - | - | - |
  | `costUsd` and `costState` are missing | The cost can't be read right now. | Read the session again later. Don't treat it as zero. |
  | The cost changed after you recorded it | `costState` was `so_far`: the session, its sandbox, or its charges hadn't settled, or it was resumed. | Record the cost once `costState` is `final`. See [Know when a figure is final](#know-when-a-figure-is-final). |
  | A list adds up to more than the runs cost | Child rows repeat their tree's cost. | Send `root_only=true` and add root rows only. |
  | Usage `cost_micros` and `costUsd` differ | `cost_micros` covers only the session's own model events and is for diagnostics. | Use `costUsd`. |
  | `400 invalid_request` on `session_ids` | The list is missing or has more than 1000 ids. | Split the list into batches of 1000. |
  | `listSessionUsage` returns fewer rows than ids | Unknown ids and sessions with no events are left out. | Match rows by `session_id`. |
  | `404 not_found` | The session does not exist or is outside your organization. | Check the id and your API key's organization. |
  | `429 rate_limit_exceeded` | Too many requests. | Wait for the time in `Retry-After`, then retry. |

  The full list of codes is in [Errors](/recursion/errors).
</Accordion>

## Limits

| Limit | Value |
| - | - |
| Ids in one `listSessionUsage` request | 1000 |
| Time for a session's cost to settle | Up to an hour |

Every other limit is on [Limits](/recursion/limits).

## Next steps

<CardGroup>
  <Card title="Billing" href="/recursion/billing">
    Buy the credits that session costs are charged to.
  </Card>

  <Card title="Teams" href="/recursion/teams">
    See why a team costs more than one agent.
  </Card>

  <Card title="Multi-agent" href="/recursion/multi-agent">
    Understand the session trees behind one session cost.
  </Card>

  <Card title="Limits" href="/recursion/limits">
    Check quotas and request limits.
  </Card>
</CardGroup>


# Fix a GitHub issue
Source: https://docs.labelbox.com/recursion/use-cases/fix-a-github-issue

Connect GitHub, give an agent write access to one repository, and run a session that fixes an issue and opens a pull request.

In this tutorial, an agent fixes a bug from a GitHub issue, adds a test, and opens a pull request. You then check the pull request and review it like any other. It takes about 20 minutes.

The examples use the repository `acme/web` and its issue 412; replace them with yours.

<Note>
  You need the Developer or Admin role in the organization; see [Organizations and roles](/recursion/organizations-and-roles). You also need permission to install a GitHub App on the GitHub account that owns the repository, or an existing GitHub connection in Recursion. The repository should run its tests on pull requests, for example in GitHub Actions, so you can see them pass on the agent's pull request.
</Note>

## Step 1: Connect GitHub

Connecting is an interactive authorization in GitHub, so do it in the console. It creates a connection for your organization but gives no agent access yet. Use the console or cURL.

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Integrations**, then click **Add integration**.
    2. Choose **GitHub**, then click **Continue**.
    3. On GitHub, choose the account and select the repositories the connection may reach, including `acme/web`.
    4. Back in Recursion, choose the account if asked. The connection appears under **Integration connections**.
    5. Enable `issue_read`, `get_file_contents`, `create_branch`, `push_files`, and `create_pull_request` in the connection's tool selection. New connections start with read-only tools selected.
    6. Open the connection's actions menu and click **Test connection** to confirm it can issue a token.
  </Tab>

  <Tab title="cURL">
    Connect in the console first, then list connections to get the connection id you'll use in step 3.

    ```bash theme={"theme":"css-variables"}
    curl 'https://api.recursion.labelbox.com/managed-agents/v1/integrations/connections' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

```json theme={"theme":"css-variables"}
{
  "connections": [
    {
      "connection_id": "31e6c9a4-2d75-48b0-a1f3-7c5e9d2b6a84",
      "organization_id": "org_01a08a705220724f9a2fe1bcd8638c9d",
      "provider": "github",
      "external_id": "58213377",
      "account_login": "acme",
      "account_type": "Organization",
      "resource_selection": "selected",
      "state": "active",
      "created_at": "2026-09-25T16:40:12Z",
      "updated_at": "2026-09-25T16:40:12Z"
    }
  ]
}
```

Continue when `state` is `active`.

## Step 2: Create an environment

The agent works on GitHub through its GitHub tools, which run outside the sandbox, so the environment needs no GitHub network rules. New environments block all outbound traffic, which is fine for this task.

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Environments**, then click **Create environment**.
    2. Enter a **Name**, such as `github-fixes`.
    3. Click **Create environment**.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/environments' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      -d '{
        "name": "github-fixes",
        "provider": "runs"
      }'
    ```
  </Tab>
</Tabs>

```json theme={"theme":"css-variables"}
{
  "organization_id": "org_01a08a705220724f9a2fe1bcd8638c9d",
  "environment_id": "9d3e7b52-1a4c-4f80-b6e9-2c8a5d0f7e13",
  "name": "github-fixes",
  "scope": "organization",
  "provider": "runs",
  "computer_use": false,
  "created_at": "2026-09-25T16:44:03Z",
  "updated_at": "2026-09-25T16:44:03Z"
}
```

See [Environments](/recursion/environments).

## Step 3: Create the agent with GitHub access

Give the agent the GitHub tools it needs and access to only the repository it needs. The selected tools determine its short-lived token permissions. GitHub's MCP tools run outside the sandbox; `git` and `gh` in the sandbox are not authenticated for GitHub.

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Agents**, then click **Create agent** and choose **Blank**.
    2. Enter a **Name**, choose a model, and paste the `system` value from the cURL tab into **System prompt**. Click **Create agent**.
    3. On the agent's **Configuration** tab, under **Integrations**, click **Add integration access** and select the GitHub connection.
    4. Select `issue_read`, `get_file_contents`, `create_branch`, `push_files`, and `create_pull_request` under **Tools**.
    5. Clear **All authorized repositories**, then select `acme/web`.
    6. Click **Use this access**, then click **Save new version**.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/agents' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      -d '{
        "name": "Issue fixer",
        "model": "<model-id>",
        "system": "You fix GitHub issues with your GitHub tools.\nRead the issue and the code it touches, then find the cause of the bug.\nOn a new branch named fix/issue-<number>, commit the fix and a test that fails without it.\nOpen a pull request whose description starts with \"Fixes\" and the issue URL. Never commit to main.\nEnd with the pull request URL.",
        "nativeIntegrations": [
          {
            "connectionId": "31e6c9a4-2d75-48b0-a1f3-7c5e9d2b6a84",
            "permission": "write",
            "tools": ["issue_read", "get_file_contents", "create_branch", "push_files", "create_pull_request"],
            "resources": ["acme/web"]
          }
        ]
      }'
    ```
  </Tab>
</Tabs>

```json theme={"theme":"css-variables"}
{
  "agent_id": "5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55",
  "latest_agent_version_id": "c1a94e07-2f6b-4d18-b3a5-0e7d8c6f4a21",
  "name": "Issue fixer",
  "nativeIntegrations": [
    {
      "connectionId": "31e6c9a4-2d75-48b0-a1f3-7c5e9d2b6a84",
      "permission": "write",
      "resources": ["acme/web"]
    }
  ]
}
```

You can grant GitHub through a vault instead, which is useful when several agents share one grant. See [GitHub](/recursion/github) and [Vaults](/recursion/vaults).

## Step 4: Start the session

The opening message names the issue. The system prompt already says how to work: find the cause, commit the fix and a test on a new branch, and open a pull request.

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Sessions**, then click **Launch session**.
    2. Choose the **Issue fixer** agent and the **github-fixes** environment.
    3. In **Opening message**, enter `Fix https://github.com/acme/web/issues/412 and open a pull request.`
    4. Click **Launch session**.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/sessions' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Idempotency-Key: acme-web-issue-412' \
      -H 'Content-Type: application/json' \
      -d '{
        "agent_id": "5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55",
        "environment_id": "9d3e7b52-1a4c-4f80-b6e9-2c8a5d0f7e13",
        "message": "Fix https://github.com/acme/web/issues/412 and open a pull request."
      }'
    ```
  </Tab>
</Tabs>

The API answers `202 Accepted`.

```json theme={"theme":"css-variables"}
{
  "session_id": "b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38",
  "status_path": "/managed-agents/v1/sessions/b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38"
}
```

The `Idempotency-Key` makes the start safe to retry: sending the same key again returns the same session instead of starting a second one.

## Step 5: Wait for the session to finish

Watch the session in the console, or poll it until the agent's turn ends.

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Sessions**, then open the session.
    2. Follow the transcript as the agent reads the issue and code, commits, and opens the pull request.
    3. When the agent is done, its last message gives the pull request URL.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

A finished session looks like this. Some fields are left out here.

```json theme={"theme":"css-variables"}
{
  "session_id": "b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38",
  "status": "completed",
  "execution_state": "idle",
  "stop_reason": "end_turn",
  "updated_at": "2026-09-25T17:09:51Z"
}
```

### What success means

* The session's `stop_reason` is `end_turn`, and the agent's last message gives the pull request URL.
* A pull request is open in `acme/web` from a branch other than `main`, its description starts with "Fixes" and the issue URL, and it adds a test for the bug.
* A finished session doesn't prove the work is right. Check the pull request and its checks before you merge. See [What success means](/recursion/sessions#what-success-means) for other session states.

## Step 6: Review the pull request

Open the URL from the agent's last message, or list the repository's open pull requests through the connection.

```bash theme={"theme":"css-variables"}
curl 'https://api.recursion.labelbox.com/managed-agents/v1/integrations/connections/31e6c9a4-2d75-48b0-a1f3-7c5e9d2b6a84/repositories/acme/web/pull-requests?state=open' \
  -H "Authorization: Bearer $RECURSION_API_KEY"
```

```json theme={"theme":"css-variables"}
{
  "page": 1,
  "pull_requests": [
    {
      "number": 418,
      "title": "Escape quotes in search queries",
      "head_sha": "3f9c2e7a1b4d6e8f0a2c4e6b8d0f1a3c5e7b9d2f",
      "draft": false
    }
  ]
}
```

Review and merge it the way you would any contributor's pull request.

## What can go wrong

| Symptom | Cause | Fix |
| - | - | - |
| The agent has no GitHub tools | The agent version has no GitHub access, or the session started from an older version. | Add integration access and save a new version. Then move the session to it, or start a new session. |
| The repository returns `404` | The GitHub App installation or the grant's repository list doesn't include `acme/web`. | Add the repository in GitHub and in the agent's integration access. |
| Creating a branch or pull request is refused | The connection or agent lacks `create_branch`, `push_files`, or `create_pull_request`. | Enable those tools on the GitHub connection, then select them for the agent. |
| The pull request's checks fail | The fix or its test is wrong. | Send a follow-up message with the failing check, and ask the agent to update the branch. |
| The agent stopped before opening a pull request | It hit a problem it couldn't solve. | Read its last message, then send a follow-up message with what to do next. |
| The agent committed to the wrong branch | The system prompt didn't forbid it strongly enough. | Keep "Never commit to main" in the system prompt, and protect `main` in GitHub. |

## Limits

* A grant can list up to 500 repositories.
* An agent can hold up to 25 integration connections.
* See [Limits](/recursion/limits) for session and environment limits.

## Next steps

<CardGroup>
  <Card title="GitHub" href="/recursion/github">
    Permission presets, vault grants, and how GitHub tools work.
  </Card>

  <Card title="Automations" href="/recursion/automations">
    Start this agent when an issue is labeled.
  </Card>

  <Card title="Research team tutorial" href="/recursion/use-cases/research-team">
    Split a larger task across a team.
  </Card>

  <Card title="Environments" href="/recursion/environments">
    Network policy, setup scripts, and compute.
  </Card>
</CardGroup>


# Run a research team
Source: https://docs.labelbox.com/recursion/use-cases/research-team

Run a team of agents that researches three options in parallel, reviews each finding, and delivers one recommendation.

In this tutorial, a team compares three vector databases for a semantic search service. The leader splits the work into one research task per database plus a review of each, teammates work in parallel, and the leader writes one recommendation as a named deliverable.

<Note>
  You need the Developer or Admin role in the organization; see [Organizations and roles](/recursion/organizations-and-roles). Read [Teams](/recursion/teams) for how the board works. A team runs several agents at once, so it costs more than a single agent. This tutorial caps the team at four teammates.
</Note>

## Step 1: Create the research agent

The agent needs web search, which is on by default, and a team size. The team size is `max_concurrent_threads`. No roster entries are needed, because teammates are copies of this agent.

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Agents**, then click **Create agent** and choose **Blank**.
    2. Enter a **Name**, choose a model, and paste the `system` value from the cURL tab into **System prompt**. Click **Create agent**.
    3. On the **Configuration** tab, under **Multiagent**, set **Work as a team** to **Always**.
    4. Under **Tools**, confirm **Web search** is on.
    5. Click **Save new version**.

    The console doesn't set the team size, so the team uses the default of 8. Use the API to cap it.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/agents' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      -d '{
        "name": "Research lead",
        "model": "<model-id>",
        "system": "You research technical choices for an engineering team.\nCite a source URL for every factual claim, and prefer official documentation and independent benchmarks.\nKeep working notes in /workspace/research. Save final reports as deliverables.",
        "multiagent": {
          "type": "coordinator",
          "agents": [],
          "team": {"mode": "on"},
          "limits": {"max_concurrent_threads": 4}
        }
      }'
    ```
  </Tab>
</Tabs>

```json theme={"theme":"css-variables"}
{
  "agent_id": "5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55",
  "latest_agent_version_id": "c1a94e07-2f6b-4d18-b3a5-0e7d8c6f4a21",
  "name": "Research lead",
  "web_search_enabled": true,
  "multiagent": {
    "type": "coordinator",
    "agents": [],
    "team": {"mode": "on"},
    "limits": {"max_concurrent_threads": 4}
  }
}
```

## Step 2: Create an environment

Web search and web fetch run outside the sandbox, so they work even when the environment blocks internet access. The team only needs a sandbox for its notes and the deliverable, so the default closed environment is enough.

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Environments**, then click **Create environment**.
    2. Enter a **Name**, such as `research`, and leave **Internet access** at **No access**.
    3. Click **Create environment**.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/environments' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      -d '{"name": "research", "provider": "runs"}'
    ```
  </Tab>
</Tabs>

```json theme={"theme":"css-variables"}
{
  "organization_id": "org_01a08a705220724f9a2fe1bcd8638c9d",
  "environment_id": "9d3e7b52-1a4c-4f80-b6e9-2c8a5d0f7e13",
  "name": "research",
  "scope": "organization",
  "provider": "runs",
  "computer_use": false,
  "created_at": "2026-09-25T16:50:31Z",
  "updated_at": "2026-09-25T16:50:31Z"
}
```

## Step 3: Start the team session

The `message` is the brief the leader splits into tasks. It also carries the context every agent needs and names the deliverable, so the leader knows what to save.

A good brief names the independent pieces, where each one writes, the review you want, how the result is judged, and the deliverable. See [Write a good team brief](/recursion/teams#write-a-good-team-brief).

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Sessions**, then click **Launch session**.
    2. Choose the **Research lead** agent and the **research** environment.
    3. In **Opening message**, paste the `message` value from the cURL tab.
    4. Under **Team**, confirm **Work as a team** is **Always**.
    5. Click **Launch session**.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/sessions' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Idempotency-Key: vector-db-review-2026-09-25' \
      -H 'Content-Type: application/json' \
      -d '{
        "agent_id": "5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55",
        "environment_id": "9d3e7b52-1a4c-4f80-b6e9-2c8a5d0f7e13",
        "message": "Recommend one vector database for our semantic search service: Qdrant, Weaviate, or Milvus. Research each one in its own task from official docs and at least one independent benchmark, writing notes to /workspace/research/<name>.md. Have a different teammate check each set of notes. Save the recommendation as vector-db-recommendation.md, and end with its comparison table and recommendation. Context: we run on Kubernetes, index about 40 million 768-dimension embeddings, filter every query by tenant, and have two engineers for operations.",
        "team": {"mode": "on"}
      }'
    ```
  </Tab>
</Tabs>

```json theme={"theme":"css-variables"}
{
  "session_id": "b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38",
  "status_path": "/managed-agents/v1/sessions/b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38"
}
```

## Step 4: Watch the board

The leader posts the round, and teammates join to claim it. You'll typically see three research tasks and three reviews that wait on them.

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Sessions**, then open the session.
    2. Open the **Work** tab. The **Board** shows each task, its status, and its owner.
    3. Click a task to read its body, its result, and the reviewer's notes.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38/board' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

Midway through, the board shows tasks like these:

```text theme={"theme":"css-variables"}
T1 explore done Research Qdrant for 40M tenant-filtered embeddings
T2 explore claimed Research Weaviate for 40M tenant-filtered embeddings
T3 explore done Research Milvus for 40M tenant-filtered embeddings
T4 review done Check the Qdrant notes
T5 review open Check the Weaviate notes
T6 review claimed Check the Milvus notes
```

When every task is `done`, the board holds the teammates' recommendations and then the leader's decision:

```json theme={"theme":"css-variables"}
{
  "post_id": "c5b1e8a4-2f73-4d96-a0e7-8d3c6b1f4a29",
  "kind": "decision",
  "task_ids": ["7d2e4b19-3a5c-4e81-b6f0-9c1a2d7e5b43"],
  "text": "Qdrant: payload-indexed tenant filtering without per-tenant collections, a maintained Helm chart, and the lowest operational effort for two engineers. The reviewer confirmed the RAM estimate in T1.",
  "created_by": "b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38",
  "created_at": "2026-09-25T17:26:04Z"
}
```

## Step 5: Wait for the leader to finish

After the decision, the leader writes `vector-db-recommendation.md` and ends its turn. The deliverable is kept with the root session, and the session goes `idle`.

<Tabs>
  <Tab title="Console">
    1. Stay on the session page. The transcript shows the leader's final message when it ends its turn.
    2. Open the **Files** tab to see `vector-db-recommendation.md` under **Outputs**.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl 'https://api.recursion.labelbox.com/managed-agents/v1/sessions/b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

```json theme={"theme":"css-variables"}
{
  "session_id": "b7e1c9a4-3d62-4f15-8a07-5c2e9f6d1b38",
  "status": "completed",
  "execution_state": "idle",
  "stop_reason": "end_turn"
}
```

### What success means

* The root session's `execution_state` is `idle` and its `stop_reason` is `end_turn`.
* `vector-db-recommendation.md` is kept as a deliverable of the root session.
* The board has a `decision` post, and no task is `open` or `claimed`.

`end_turn` means the leader finished, not that the recommendation is right. Read it before you act on it, and send a follow-up message if something is missing.

## Step 6: Read the recommendation

<Note>
  The brief asks the leader to end with the comparison table and recommendation, so the final message carries the result. Any files the team saved as deliverables are on the session's **Files** tab; see [Download session deliverables](/recursion/files#download-session-deliverables).
</Note>

* **In the console:** the leader's last message in the transcript holds the table and the recommendation.
* **Through the API:** page the root timeline with `listSessionEvents` and read the last `message` event whose `role` is `assistant`.
* **In a later session:** start a session that [references this one](/recursion/referenced-sessions). It can read `vector-db-recommendation.md` and, for example, turn it into a design doc or open a pull request with it.

The whole team's cost is one figure: read `costUsd` on the root session, and record it once `costState` is `final`. See [Usage and cost](/recursion/usage-and-cost).

## What can go wrong

| Symptom | Cause | Fix |
| - | - | - |
| The board read returns `404 not_found` | The session started with team mode `off`. | Start with `team.mode` `on`. |
| The leader researches everything itself | The brief didn't name separate pieces. | Name each option and its output path in the brief. |
| Two teammates wrote the same notes file | The brief didn't give each piece its own path. | Give each piece a distinct path, as with `/workspace/research/<name>.md`. |
| No sources in the notes | Web search is off for the agent. | Turn on **Web search** in the agent's **Tools** and save a new version. Then move the session to it, or start a new session. |
| The recommendation has claims without sources | The brief didn't ask for citations strongly enough. | Send a follow-up message asking for a source URL on every row. |
| `vector-db-recommendation.md` isn't on the **Files** tab | The report was saved outside the deliverables folder. | Keep "Save the recommendation as `vector-db-recommendation.md`" in the brief, or ask for it in a follow-up message. See [Deliverables and artifacts](/recursion/artifacts). |
| The session costs more than expected | Each teammate runs its own model calls. | Lower `max_concurrent_threads`, or split fewer pieces. |

## Limits

* Team size is `max_concurrent_threads`: default 8, maximum 25. This tutorial uses 4.
* A deliverable can be at most 64 MiB. See [Deliverables and artifacts](/recursion/artifacts#limits).
* See [Limits](/recursion/limits) for every product limit.

## Next steps

<CardGroup>
  <Card title="Teams" href="/recursion/teams">
    Team modes, the board, and writing briefs.
  </Card>

  <Card title="Multi-agent" href="/recursion/multi-agent">
    Delegation, rosters, and the session tree.
  </Card>

  <Card title="Deliverables and artifacts" href="/recursion/artifacts">
    What gets kept, and how to get results out.
  </Card>

  <Card title="Referenced sessions" href="/recursion/referenced-sessions">
    Let a later session read this one's deliverables.
  </Card>
</CardGroup>


# Vaults and credentials
Source: https://docs.labelbox.com/recursion/vaults

Store secrets once, grant each session exactly the credentials it needs, and know what the agent can and cannot see.

A vault is a named group of credentials. You grant vaults to an agent or a session by id, and Recursion delivers each credential to the place it is needed. Secret values are write-only: no vault or credential read, in the API or the console, returns them.

<Note>
  Creating, changing, or deleting vaults and credentials needs the organization developer or admin role. The organization user role can list vaults and read credential details, which never include secret values. See [Organizations and roles](/recursion/organizations-and-roles). For API calls, create a key under **API keys**. See [API keys](/recursion/api-keys).
</Note>

<Warning>
  Never put a secret in a system prompt, a message, a file, environment settings, or any `metadata` field. Those values are stored in session history or returned by API reads.
</Warning>

## Create a vault

Create one vault for each set of credentials that should travel together. A common pattern is one vault per project, or one vault per end user, with your own user id in `metadata`.

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Credential vaults**.
    2. Click **Create vault**.
    3. Enter a **Vault name**, then click **Create vault**.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/vaults' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Idempotency-Key: release-operations-vault' \
      -H 'Content-Type: application/json' \
      -d '{"display_name": "Release operations", "metadata": {"team": "platform"}}'
    ```
  </Tab>
</Tabs>

```json theme={"theme":"css-variables"}
{
  "vault_id": "26d4b1a8-73c9-4f60-8a15-9e2c7b5d4f31",
  "organization_id": "org_01a08a705220724f9a2fe1bcd8638c9d",
  "display_name": "Release operations",
  "metadata": { "team": "platform" },
  "credential_count": 0,
  "created_at": "2026-09-17T09:12:44Z",
  "updated_at": "2026-09-17T09:12:44Z"
}
```

`display_name` is required and can be at most 256 characters. Keep `vault_id`: you grant vaults by id. Next, add a credential to the vault.

<Accordion title="Retry a create safely">
  `createVault` accepts an optional `Idempotency-Key` header. Use one stable key when an ambiguous network failure may require a retry. The key also determines the vault id, so an unchanged vault remains recoverable after the response receipt expires; reusing it after the vault changes or is deleted returns a conflict. See [Idempotent mutations](/recursion/api#idempotent-mutations).

  Send `Idempotency-Key` as exactly one header value containing 1 to 256 visible ASCII characters. The removed body field `idempotency_key` is rejected with `400 invalid_request`.

  | You send | Result |
  | - | - |
  | The same key, exact HTTP method, escaped path, raw query, and raw body bytes while the receipt exists | The original response, with `Idempotency-Replayed: true`. No second vault is created. |
  | The same key with any different request-identity bytes while the receipt exists | `409 idempotency_conflict`. Use a new key for a new request. |
  | The same key while the first create still runs | `409 idempotency_in_progress` with `Retry-After`. Wait, then resend the same request byte for byte. |
  | The same key and create body after the response receipt expires, while the vault is unchanged | The existing vault. No second vault is created. |
  | The same key after that vault changed or was deleted | `409 conflict` on `Idempotency-Key`. Use a new key. |
  | No key, followed by an ambiguous transport failure | Do not automatically retry. The API cannot guarantee that a second unkeyed request will not create another vault. |
</Accordion>

## Add a credential

Choose the type by what the agent may see. A **Token for an MCP server** authenticates to one MCP server and is attached to its requests outside the sandbox, so the agent sees only tool results. Any command in the sandbox can read an **Environment variable**. See [How credentials reach a session](#how-credentials-reach-a-session) for the full comparison.

This example adds a token for an MCP server. The response describes the credential and never includes `secret_value`.

<Tabs>
  <Tab title="Console">
    1. Open the vault and click **Add credential**.
    2. Under **What will use this secret?**, choose **Token for an MCP server**.
    3. Enter the **Server URL** and the **API token**.
    4. Optionally enter a **Label**.
    5. Under **Acknowledgement**, confirm that you understand the credential is shared, then click **Add credential**.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    jq -n '{
      credential_type: "bearer_token",
      display_name: "Project tracker",
      mcp_server_url: "https://mcp.example.com/mcp",
      secret_value: env.TRACKER_TOKEN
    }' | curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/vaults/26d4b1a8-73c9-4f60-8a15-9e2c7b5d4f31/credentials' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      --data-binary @-
    ```

    Export `TRACKER_TOKEN` first. `jq` reads it from the environment, so the token stays out of your shell history and out of process arguments.
  </Tab>
</Tabs>

```json theme={"theme":"css-variables"}
{
  "vault_id": "26d4b1a8-73c9-4f60-8a15-9e2c7b5d4f31",
  "credential_id": "91a7c4e2-5d38-46b0-a9f1-3c6e8d2b7a54",
  "organization_id": "org_01a08a705220724f9a2fe1bcd8638c9d",
  "credential_type": "bearer_token",
  "display_name": "Project tracker",
  "mcp_server_url": "https://mcp.example.com/mcp",
  "created_at": "2026-09-17T09:20:03Z",
  "updated_at": "2026-09-17T09:20:03Z"
}
```

To confirm the token works before a session uses it, [test the server](/recursion/mcp-servers#test-a-server-before-you-use-it).

A vault never limits where the sandbox can connect. An environment-variable secret can be sent to any host the environment's [network policy](/recursion/environments-reference#network-policy) allows.

<Accordion title="Add an environment variable">
  An environment variable needs `secret_name`, the variable name the sandbox sees. The console asks you to confirm that the agent can read the value.

  ```json theme={"theme":"css-variables"}
  {
    "credential_type": "env_var",
    "display_name": "Acme API key",
    "secret_name": "ACME_API_KEY",
    "secret_value": "<value>"
  }
  ```
</Accordion>

<Accordion title="Credential fields and retries">
  | Field | Applies to | Description |
  | - | - | - |
  | `credential_type` | All | Required. `bearer_token` or `env_var`. |
  | `secret_value` | All | Required. At most 64 KiB. Encrypted at rest and never returned. |
  | `mcp_server_url` | `bearer_token` | Required. An `http` or `https` URL with no user name or password in it. It can't be changed later. |
  | `secret_name` | `env_var`, `bearer_token` | For `env_var`, required: the environment variable name. It can't be changed in the console. For `bearer_token`, optional and API only: a request header name, such as `X-Api-Key`, that carries the raw token instead of `Authorization: Bearer`. Without it, the token is sent as `Authorization: Bearer <token>`. |
  | `display_name` | All | Optional label. |
  | `metadata` | All | Your own JSON, returned as-is. Never store secrets here. |

  To retry a create after an ambiguous network failure without adding the credential twice, send one stable `Idempotency-Key` header and resend the same request. See [Idempotent mutations](/recursion/api#idempotent-mutations).
</Accordion>

## Grant credentials to a session

A session receives credentials in one of two ways:

* **Agent defaults.** Set `default_vault_ids` on the agent. Every session started without its own `vault_ids` receives those vaults. Optionally narrow them with `default_credential_refs`. See [Agents](/recursion/agents). When you [move a session to another agent version](/recursion/sessions#move-a-session-to-another-version), it gains the vaults and credentials that version's defaults add and loses the ones they remove.
* **At launch.** Send `vault_ids` and, optionally, `credential_refs` when you start the session.

<Tabs>
  <Tab title="Console">
    1. Open the agent and click **Launch session**.
    2. Under **Credentials**, check the vaults or individual credentials for this run. The agent's defaults start checked.
    3. Fill in the task, then click **Launch session**.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X POST 'https://api.recursion.labelbox.com/managed-agents/v1/sessions' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Idempotency-Key: release-status-2026-09-17' \
      -H 'Content-Type: application/json' \
      -d '{
        "agent_id": "5f0c2a1e-8b7d-4c3a-9e21-6d4f0b9a7c55",
        "environment_id": "9d3e7b52-1a4c-4f80-b6e9-2c8a5d0f7e13",
        "vault_ids": ["26d4b1a8-73c9-4f60-8a15-9e2c7b5d4f31"],
        "credential_refs": [{
          "vault_id": "26d4b1a8-73c9-4f60-8a15-9e2c7b5d4f31",
          "credential_id": "91a7c4e2-5d38-46b0-a9f1-3c6e8d2b7a54"
        }],
        "message": "Summarize the open release blockers."
      }'
    ```
  </Tab>
</Tabs>

Every vault id must exist in your organization, or the start fails with `404` on `vault_ids`. Credentials are resolved when the session's sandbox is prepared, not at the start request. See [Sessions](/recursion/sessions) for the start response.

**What success means:** each MCP server authenticated by the grant contributes its tools to the session, and each environment variable is set in the sandbox. If an MCP server can't be reached or rejects the token, the session still starts without that server's tools and records a warning in its events: `mcp_auth_rejected` for a rejected token, `mcp_discovery_failed` otherwise.

<Accordion title="What the session receives">
  | You send at launch | The session receives |
  | - | - |
  | Neither field | The agent version's default vaults and default credential refs. |
  | `vault_ids` with one or more ids | Exactly those vaults. They replace the defaults; they are not added to them. |
  | `vault_ids: []` | No vaults. |
  | `credential_refs` with `vault_ids` | Only the listed credentials, across all listed vaults. |
  | `credential_refs: []` with non-empty `vault_ids` | Rejected with `400`, because it would grant nothing. |

  `credential_refs` is an allowlist across every granted vault, not a filter inside one vault. If you list credentials from one vault only, the session receives nothing from the other vaults. List every credential the session needs.

  **Order matters.** When two granted vaults hold a credential for the same MCP server URL or the same environment variable name, the vault listed first wins. Within one vault, store only one credential per server URL or variable name. The console marks a duplicate as shadowed, and it never reaches a session.

  An agent version saved without `default_credential_refs` keeps the credentials its default vaults held when you saved it. A credential you add to one of those vaults later doesn't reach the agent's sessions until you save a new agent version that includes it. A running session gets it when you [move the session to that version](/recursion/sessions#move-a-session-to-another-version).
</Accordion>

<Accordion title="Credentials in multi-agent sessions">
  A subagent that is a copy of the same agent receives the same vaults and credential refs as its parent. A different agent from the roster receives only its own default vaults and credential refs, never the parent's. To share a credential, attach the same vault to both agents.

  Environment-variable credentials work differently, because every agent in the tree shares one sandbox:

  * They come only from the vaults granted to the root session, and they're set once, when the sandbox starts.
  * Every agent in the tree, including roster agents, can read them.
  * A roster agent's own environment-variable credentials are not added. Its session records a `delegated_env_credentials_unavailable` warning that names the missing variables. To make a variable available, grant its vault to the root session.

  See [Multi-agent](/recursion/multi-agent#what-a-child-inherits) for the full inheritance rules.
</Accordion>

## Rotate a credential

Send a new `secret_value` to store a new version of the secret. Omit it to keep the current value. You can also change `display_name`, `secret_name`, and `metadata`. You can't change `credential_type` or `mcp_server_url`; delete the credential and add a new one instead.

<Tabs>
  <Tab title="Console">
    1. Open the vault. In the credential's actions menu, click **Edit**.
    2. Enter the new value in the **Replace** field, for example **Replace API token**.
    3. Click **Save credential**.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    jq -n '{secret_value: env.NEW_TRACKER_TOKEN}' | \
      curl -X PATCH 'https://api.recursion.labelbox.com/managed-agents/v1/vaults/26d4b1a8-73c9-4f60-8a15-9e2c7b5d4f31/credentials/91a7c4e2-5d38-46b0-a9f1-3c6e8d2b7a54' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      --data-binary @-
    ```

    Export `NEW_TRACKER_TOKEN` first. If it isn't exported, `jq` sends `null`, which keeps the current secret: the request succeeds without rotating.
  </Tab>
</Tabs>

New sessions use the new value. An environment variable is set when a session's sandbox is created, so a running sandbox keeps the value it started with, including a variable that a move to another agent version removed from the session. Revoke the old secret at its provider only after running sessions that need it have finished.

## List vaults and credentials

`listVaults` returns every vault in your organization in one response. Each entry has a `credential_count`, so you don't need to open each vault. `getVault` returns one vault in the same shape as the create response. `listVaultCredentials` returns a vault's credentials, newest first, under `vault_credentials`; the list is empty when the vault is empty. `getVaultCredential` returns one credential. Neither credential read returns secret values.

<Tabs>
  <Tab title="Console">
    1. In the sidebar, click **Credential vaults**.
    2. To rename a vault, open its actions menu and click **Rename**.
    3. Open a vault. The table lists each credential with its type and label.
    4. Expand a row to see its id and, for a token for an MCP server, a connection check.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl 'https://api.recursion.labelbox.com/managed-agents/v1/vaults' \
      -H "Authorization: Bearer $RECURSION_API_KEY"

    curl 'https://api.recursion.labelbox.com/managed-agents/v1/vaults/26d4b1a8-73c9-4f60-8a15-9e2c7b5d4f31' \
      -H "Authorization: Bearer $RECURSION_API_KEY"

    curl -X PATCH 'https://api.recursion.labelbox.com/managed-agents/v1/vaults/26d4b1a8-73c9-4f60-8a15-9e2c7b5d4f31' \
      -H "Authorization: Bearer $RECURSION_API_KEY" \
      -H 'Content-Type: application/json' \
      -d '{"display_name": "Release operations (prod)"}'

    curl 'https://api.recursion.labelbox.com/managed-agents/v1/vaults/26d4b1a8-73c9-4f60-8a15-9e2c7b5d4f31/credentials' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

A vault update changes only the fields you send. `metadata` is replaced as a whole object, not merged.

## Delete a credential or a vault

Deleting a credential removes it from its vault. Deleting a vault hides the vault and every credential in it. In both cases, new sessions don't receive them, running sessions are not stopped, and there is no undo.

<Tabs>
  <Tab title="Console">
    1. To remove one credential, open the vault. In the credential's actions menu, click **Remove**, then confirm.
    2. To delete a vault, open it and click **Delete**, then confirm. The dialog warns you when agents still use the vault as a default.
  </Tab>

  <Tab title="cURL">
    ```bash theme={"theme":"css-variables"}
    curl -X DELETE 'https://api.recursion.labelbox.com/managed-agents/v1/vaults/26d4b1a8-73c9-4f60-8a15-9e2c7b5d4f31/credentials/91a7c4e2-5d38-46b0-a9f1-3c6e8d2b7a54' \
      -H "Authorization: Bearer $RECURSION_API_KEY"

    curl -X DELETE 'https://api.recursion.labelbox.com/managed-agents/v1/vaults/26d4b1a8-73c9-4f60-8a15-9e2c7b5d4f31' \
      -H "Authorization: Bearer $RECURSION_API_KEY"
    ```
  </Tab>
</Tabs>

```json theme={"theme":"css-variables"}
{ "deleted": true }
```

<Accordion title="What happens to sessions and agents that used it">
  A session that used a deleted vault keeps running without its credentials. From its next turn, the vault's MCP tokens no longer reach it: MCP servers it declared are called without them, and servers only the vault provided become unavailable. A sandbox that's already running keeps the environment variables it started with; a sandbox started later, such as after an idle session's compute is released, doesn't get them. The session can record a `session_vault_deleted` warning naming the vault.

  Through the API, an agent whose `default_vault_ids` still names a deleted vault can't start sessions with its defaults (`404` on `vault_ids`) until you save a new agent version without that vault, or launch with explicit `vault_ids`. The console's **Launch a session** drawer leaves the deleted vault out and says so.
</Accordion>

## How credentials reach a session

A session uses the vaults you send at launch, or the agent's default vaults otherwise. Each credential then goes to one of two places.

```mermaid theme={"theme":"css-variables"}
flowchart TD
  launch["Session vault_ids"] -->|"when set"| session["Session"]
  defaults["Agent default vaults"] -->|"otherwise"| session
  session --> mcp["Token for an MCP server<br>sent outside the sandbox"]
  session --> env["Environment variable<br>set in the sandbox"]
```

| Type (`credential_type`) | Console name | Used for | What the agent can see |
| - | - | - | - |
| `bearer_token` | **Token for an MCP server** | Authenticating to one MCP server, matched by its URL. | Nothing. The token is attached to MCP requests outside the sandbox. The agent sees only tool results. |
| `env_var` | **Environment variable** | A CLI or SDK in the sandbox that reads a key from the environment. | The full value. It's a plaintext environment variable in the sandbox, so any command any agent in the session tree runs can read it. See **How environment variables are masked** below. |

<Accordion title="How environment variables are masked">
  Tool output masks the current value, the value without leading or trailing whitespace, the value as JSON and shell quoting print it, and its plain base64 and URL-encoded forms as `[REDACTED:vault_credential]` before the agent or the transcript sees it. It can still appear if a command transforms it some other way, if output is cut off partway through it, if it's shorter than 8 characters, if it was rotated or deleted after the sandbox started, or in a file the agent writes. A variable that a move to another agent version removed from the session's grant stays in the running sandbox until that sandbox is replaced, and its value stays masked while it is there.
</Accordion>

A vault can also hold `integration` credentials, which grant a [native integration](/recursion/integrations#native-integrations) connection per session. [Built-in integrations](/recursion/integrations) don't use vaults: you grant those apps on the agent.

## What can go wrong

The most common problems:

* **A session's MCP tools are missing.** The token's `mcp_server_url` doesn't match the server URL on the agent, the vault wasn't granted, or `credential_refs` left the credential out. [Test the server](/recursion/mcp-servers#test-a-server-before-you-use-it) with the same vault, then check the session's grants.
* **An environment variable is missing in the sandbox.** The credential wasn't granted, or another granted vault listed earlier defines the same name. Check `vault_ids` order and `credential_refs`.
* **`404 not_found` on `vault_ids`.** A vault in the start request doesn't exist, was deleted, or belongs to another organization. Check the id with `listVaults`.
* **A secret appears in the transcript.** An `env_var` value was printed in a form the mask doesn't recognize. Rotate the secret, and prefer a token for the MCP server when the service has an MCP server.

<Accordion title="Every vault and credential error">
  | Symptom or code | Cause | Fix |
  | - | - | - |
  | `400 invalid_request` on `Idempotency-Key` | The removed body field `idempotency_key` is present, or the header is malformed or sent more than once. | Remove the body field. If you want retry protection, send exactly one header value of 1 to 256 visible ASCII characters. |
  | `400 invalid_request` on `secret_value` | The value is missing or over 64 KiB. | Send `secret_value`, at most 64 KiB. |
  | `400 invalid_request` on `mcp_server_url` | The URL is missing, isn't `http` or `https`, or contains a user name or password. | Use the server's full endpoint URL and put the token in `secret_value`. |
  | `400 invalid_request` on `secret_name` | An `env_var` credential has no variable name. | Send `secret_name`, for example `ACME_API_KEY`. |
  | `400 invalid_request` on `credential_refs` | You sent `credential_refs: []` with vaults. | Omit `credential_refs` to grant whole vaults, or list the credentials. |
  | `404 not_found` on `vault_ids` | A vault in the start request doesn't exist, was deleted, or belongs to another organization. | Check the id with `listVaults`. |
  | `409 idempotency_conflict` | The header key was reused with different method, path, query, or body bytes while its receipt exists. | Use a new key for a new request. |
  | `409 idempotency_in_progress` | The first create with this key is still running. | Wait for `Retry-After`, then resend the same request byte for byte. |
  | `409 conflict` on `Idempotency-Key` | The response receipt expired and the vault this key created was changed or deleted. | Use a new key. |
  | `503 idempotency_unavailable` | The API could not check or reserve the key. The create did not run. | Resend the same request with the same key. |
  | `429 rate_limit_exceeded` | Too many requests. | `Retry-After` tells you how long to wait. Retry only when the operation's strategy in **Endpoints** permits replay. |
  | A session's MCP tools are missing | The token's `mcp_server_url` doesn't match the server URL on the agent, the vault wasn't granted, or `credential_refs` left the credential out. | [Test the server](/recursion/mcp-servers#test-a-server-before-you-use-it) with the same vault, then check the session's grants. |
  | An environment variable is missing in the sandbox | The credential wasn't granted, or another granted vault listed earlier defines the same name. | Check `vault_ids` order and `credential_refs`. |
  | A session has a `session_vault_deleted` warning | A vault granted to the session was deleted after it started, so the session runs without its credentials. | Start a new session with a live vault. |
  | The console won't delete a vault | The vault holds a credential the console shows as read-only, such as an `integration` credential. | Delete it with `deleteVault`. |
  | `409 conflict` when you delete a vault or a credential | An event source still verifies its deliveries with that credential, or with a credential in that vault. The console names the event sources, and the response carries them in `details.dependents`. | Delete those event sources, or give them another verification credential, then delete again. An event source can't be deleted while an automation has a trigger on it: see [Receive events](/recursion/automations#receive-events). |
  | A secret appears in the transcript | It was an `env_var` credential that a command printed in a form the mask doesn't recognize, or one of the other cases in **How environment variables are masked**. | Rotate the secret. Prefer a token for the MCP server when the service has an MCP server. |
</Accordion>

For every error code, see [Errors](/recursion/errors). For limits, see [Limits](/recursion/limits).

## Next steps

<CardGroup>
  <Card title="Connect MCP servers" href="/recursion/mcp-servers">
    Add remote tools and authenticate them with a vault credential.
  </Card>

  <Card title="GitHub access" href="/recursion/github">
    Grant agents access to the repositories you choose.
  </Card>

  <Card title="Environments" href="/recursion/environments">
    Set up the sandbox and its network policy.
  </Card>

  <Card title="Security" href="/recursion/security">
    See what reaches the sandbox and how secrets are protected.
  </Card>
</CardGroup>