Workflows

A workflow in Labelbox is a customizable, multi-step pipeline for reviewing and reworking your labeled data. It allows you to create a structured quality assurance process to ensure the accuracy and consistency of your annotations. Think of a workflow as a conveyor belt. A data row moves from one task to the next in a sequential order. When a data row completes a task, its status is updated, and it moves to the next appropriate task in the workflow. Workflows are built using an interactive, node-based editor where you can define the path that your data rows will follow.

​

Building your workflow

You can customize the default workflow by adding your own tasks and logic to create a process that fits your specific needs. By default, every project comes with a simple, four-step workflow: Data rows automatically move to the next task only when the previous step is completed. As a data row moves, its status updates automatically and cannot be changed manually. For example, a data row with the To Label status must first be labeled before it can proceed to a review or rework task.

​

Create a new task

You can add custom review and rework tasks to your workflow. To do so:

1. Go to the Workflow tab and click Edit workflow.
2. Click the + button in the workflow editor.
3. Select Step → Review task or Custom rework task.
4. In the task configuration settings (Review task and Custom rework task only), you will need to provide:
   • Task name: A descriptive name for your task.
   • Instructions: Clear instructions for the users who will be working on this task.
   • Group assignment: The user group that will be responsible for this task.
   • Maximum number of labels per task queue: This setting puts a per-contributor cap on how many labels a single user can create in that specific task queue. Once an individual contributor hits that cap, they stop receiving more tasks from that queue even if tasks remain, unless the field is left empty for no limit.
   • Per-user/group task limits: This setting lets you define who the task caps apply to, at a more granular level:
     • For specific users: you can set a maximum number of tasks that each listed user can complete in that queue. Once a user hits their per-user limit, they stop receiving additional tasks from that queue, even if more tasks are available.
     • For specific groups: you can set a cumulative task limit per group for that queue (e.g., a reviewer group can do at most N tasks total across that queue). Once the group collectively reaches that limit, no one in that group will receive more tasks from that queue, even if other groups or users can.
   • Individual assignment (custom rework tasks only): For custom rework tasks, you can configure this setting to ensure that any data rows sent to rework get re-assigned to the original label creator.

​

Add filtering logic to tasks

You can use filtering logic to control which data rows are sent to which tasks. This allows you to create specialized review steps for different types of data. To add logic:

1. Go to the Workflow tab and click Edit workflow.
2. Click the + button in the workflow editor and select Step → Logic.
3. Click + Add to choose one or more filters.
4. Select All or Any to specify whether all or any of the filter conditions must be met.

The following filters are available:

​

Connect logic and tasks

Once you have created your tasks and logic, you can connect them in the workflow editor by dragging and dropping the arrows between the nodes. This allows you to visually map out the flow of your review process. For complex workflows, you can add multiple Rework and Done terminal nodes to the editor to make your workflow easier to read. This is a visual aid and does not create additional tasks.

​

Managing your workflow

Workflows are designed to be flexible, allowing you to optimize your quality assurance process as your project evolves. To modify a task or logic node, simply click on it in the editor to open its settings. You can delete a node by clicking the trash can icon. To change the connections between nodes, select the arrow connecting them and either drag it to a different node or click the trash can icon to delete it. To see a breakdown of how many data rows are in each workflow status, click the info icon next to the editor.

​

Advanced workflow actions

Here are some more advanced actions you can perform within your workflow to manage your data quality.

​

Add terminal nodes

Terminal nodes represent the default Rework and Done tasks. You can create multiple terminal nodes on the editor instead of connecting everything to a single Rework or Done node, making your workflow easier to read and visually cleaner. Adding multiple terminals does not create additional tasks and is purely for improving the visual layout.

​

Reject and rework data rows

When a label is rejected, the data row is automatically sent to the “Rework” task.

• To send a single data row to rework:
  1. Select the task in the Workflow tab.
  2. Click Start Reviewing.
  3. Click Reject on the data rows you want to send to rework.
• To send multiple data rows to rework in bulk:
  1. Go to the Data rows tab.
  2. Use the filters to find the data rows you want to send to rework.
  3. Select the data rows (up to 10,000 at a time).
  4. From the dropdown menu, select Move to task and choose Rework.

​

Use benchmarks in your workflow

At any stage of the project, if a label on a data row is set as a benchmark, the data row (and thus any current and future labels made on the data row) will move to the Done task until the benchmark status is removed from the label. If you are trying to review a data row with a benchmark label, remove the benchmark from the label and then proceed to move the data row to the appropriate task.

​

View audit logs

Labelbox automatically creates an audit log for every data row that enters a workflow. This log provides a complete history of all the events that have occurred on a data row, which is useful for investigating its review history. To view the audit log:

1. From the Workflow tab, click on a task to view the data rows within it.
2. Click on a data row to open it.
3. Click on the clock icon to view the audit log.

​

Recursion Managed Agent node

The Recursion Managed Agent node lets an AI agent you built in Recursion review data rows as a step in your workflow. When a data row reaches the node, Labelbox starts a session for your agent. The agent checks the work, gives it a score, and the data row moves on based on that score. Use it to automate quality checks that would otherwise need a human reviewer. For example, an agent can confirm labels match your ontology and send only low-scoring data rows to a person for review. The node has three outputs:

• Approved: the agent’s score meets your threshold.
• Rejected: the score is below your threshold, or the agent reports a failure.
• Timeout: the agent didn’t respond in time, so the data row moves on instead of getting stuck.

Setup has three parts. First, connect your workspace to Recursion (a one-time step). Next, add the node to a project workflow. Then, create an agent from the node. Recursion sets up the environment the agent runs in for you.

​

Before you begin

Make sure you have the following:

• Permission to manage integrations in your Labelbox workspace (typically a workspace Admin). You need this to connect to Recursion.
• A Recursion account with access to the tenant and organization where your agents will live.

You don’t need an agent or environment ahead of time. You’ll create both from the node in step 3.

​

Step 1: Connect Labelbox to Recursion

This connection lets Labelbox start sessions for your Recursion agents. You only need to set it up once per Recursion organization.

1. In Labelbox, go to Workspace settings > Integrations.
2. Find the Recursion Managed Agents card and click Connect with Recursion.
3. Sign in to Recursion if prompted.
4. Select your Recursion tenant, then the organization you want to connect.
5. Choose which agents and environments Labelbox can use. All are selected by default. Narrow the list if you only want certain agents available in Labelbox.
6. Click Allow access.

You return to the Integrations page and see a confirmation message. Your new connection appears as a row with a Connected status. To connect another organization, even one in the same tenant, repeat these steps. You can have as many connections as you need, and you’ll pick one each time you set up a node.

​

Step 2: Add the node to your workflow

1. Go to the Workflow tab and click Edit workflow.
2. Click the + button in the workflow editor and select Step → Recursion Managed Agent.
3. Connect the step before it (usually your Initial labeling task) to the node’s input.
4. Connect each of the node’s three outputs: Approved, Rejected, and Timeout.

A common setup:

• Approved goes to Done, so data rows that pass skip manual review.
• Rejected goes to Rework, so the labeler can fix the issues the agent found.
• Timeout goes to Initial review task, so a person still checks data rows the agent couldn’t finish.

​

Step 3: Create your agent

You can create the agent right from the node. Recursion builds it from a template made for this node, and creates the environment it runs in automatically. The template already knows how to score each data row and report back to Labelbox.

1. Select the node to open its settings.
2. Under the Agent to run field, click Create Recursion Agent. Recursion opens in a new tab with the Labelbox workflow node template ready. Sign in if prompted.
3. If Recursion asks you to connect to Labelbox, click to connect, then click Allow access. This lets your agent read your project, data rows, labels, and ontology. Pick the Labelbox organization that owns your project.
4. Click Continue, then Create. Recursion creates your agent and its environment.
5. Optional: In the agent’s instructions, add your own review criteria under the Task heading, such as what to check and what counts as a good label.
6. Go back to the Labelbox tab. Your new agent and environment now appear in the node’s dropdowns.

​

Step 4: Finish the node settings

With the node still selected, fill in its settings: The node’s name is sent to the agent with each session, so give it a clear name if you use more than one agent node.

​

Step 5: Save and test

1. Click Save. The workflow won’t save if an output is unconnected or the node has no Recursion connection.
2. Label and submit a data row from the labeling queue.
3. Back on the Workflow tab, the data row should show as waiting in the Recursion Managed Agent node. In Recursion, you’ll see a new session for that data row.
4. When the agent finishes, the data row moves to the step connected to Approved, Rejected, or Timeout.

​

How scoring and routing work

The agent scores each data row from 0 to 1, where 1 means the work fully meets your bar. The node compares that score to your threshold to pick an output. When the agent rejects a data row, it can also list the issues it found. Today the agent mainly reviews and scores labels. As more Labelbox tools become available to agents, they’ll be able to do more, such as adjusting labels directly.

​

Manage Recursion connections

Go to Workspace settings > Integrations to see your Recursion connections. Each row shows its status (Connected or Connection error) and which agents and environments it can use. From a row’s menu you can:

• Test connection: checks that Labelbox can still reach Recursion and lists the agents and environments it can use.
• Reconnect / change access: signs in again or changes which agents and environments are allowed.
• Disconnect: removes the connection from both Labelbox and Recursion. Any node that uses it stops working until you pick another connection.

​

Troubleshooting