Create customizable, multi-step review and rework pipelines using the node-based editor
Workflows let you customize your review queue for greater efficiency and automation. A workflow consists of tasks. Each task represents a specific step in the labeling process and is associated with a status. Tasks are fully configurable, while statuses are predefined and not configurable, allowing for both flexibility and consistency for customizing your pipeline. For example, you can create custom review task to filter and review data rows with certain annotations or those labeled by a specific labeler or group.
If you have the admin or project lead role assigned, you can use the project Workflow tab to create, review, and maintain workflows using an interactive, node-based editor. You can also set filtering logic to control which data rows are sent to which review tasks.
Default workflow in the editor
The default workflow has the following four tasks:
Initial labeling task: reserved for all data rows that have been queued for labeling
Initial review task: first review task for data rows with submitted labels
Rework task: reserved for data rows that have been rejected. All data rows rejected at any of the intermediate review tasks are automatically sent to the Rework task without manual intervention.
Done task: reserved for data rows that have either:
Among the default tasks, only the Initial review task allows you to change its name and add filtering criteria. Since the review process is iterative, workflows automatically update as data rows are labeled or rejected.
A workflow status represents the current state of a data row within the workflow. A data row can have the following statuses:
Status | Definition |
---|---|
To label | Data rows that have less than the number of required labels associated with them. In other words, these are data rows that have no labels (if they are non-consensus or non-benchmark data rows). For consensus or benchmark data rows, these would be data rows with less the number of required labels, for example, two labels created when the data row needs three labels for completion. |
In review | Data rows that are in one of the review tasks within the . |
In rework | Data rows that are in the rework task within the workflow. |
Done | Data rows that have all required labels completed and have passed through all review steps within the workflow. |
If a project does not have Workflow set up, the only statuses that will show on the data rows tab are:
A project workflow is similar to a conveyor belt, where data rows move along sequentially through the tasks like checkpoints. Here is how a data row moves through a workflow.
If a data row fits the filtering criteria for a review task, the data row will enter that review task. Then, there are two possible outcomes:
a. The data row gets an Approve review and goes on to the next task.
b. The data row gets a Reject review and gets directed to Rework (all rejected).
If a data row does not fit the criteria for a review task, it automatically gets passed to the next review task until it reaches a task that does fit the criteria.
Once a data row is modified/reworked in the Rework (all rejected) task, it goes through each task in the workflow that fits the criteria.
If a data row receives an Approve review and meets the criteria for both review tasks, the data row will go through the following tasks: Initial labeling task -> Labeler review -> Annotation review -> Done. If the data row receives a Reject review in either of the review tasks, it will move to the Rework task. Hence, a successful step-by-step flow is: Rework (all rejected) -> Labeler review -> Annotation review -> Done.
A data row moves to the next task in the workflow when the current task is completed, which may update its status. You can’t manually change a data row’s status. For example, data rows with the To Label status can’t be assigned directly to a review or rework task without first completing the initial labeling task.
You can create custom review and rework tasks in addition to the default Initial review task and Rework task in your workflow. To add a custom task, click the + button, then select Step > Review task or Custom rework task. In the task configuration settings, add the following:
You can define filtering logic that determines whether a data row enters a custom task. This logic appears as a separate node in the editor and is configured independently from the task itself, making it easier to manage complex workflows and reuse logic across tasks. To add logic, click the + button, then select Step > Logic. In the logic configuration settings:
The following filters are available:
Filter | Description |
---|---|
Labeled by | The user who labeled the data row |
Annotation | The label added to the data row |
Labeled at | The date range for when the label is added |
Consensus average | A range of average label-level consensus scores |
Feature consensus | A range of average feature-level consensus scores. Can select more than one feature schemas in the ontology. |
Dataset | The dataset where the data row comes from |
Issue category | The issue category of issues on the data row |
Batch | The batch where the data row is sent |
Metadata | The non-annotation metadata information on the data row |
Model prediction | The Foundry model predicted label of the data row |
Labeling time | The labeling time spent on the data row |
Review time | The review time spent on the data row |
Natural language | A natural language expression and confidence score range matching data rows with similar vector embeddings. |
Label feedback | The label feedback score range |
Sample probability | The likelihood of a data row being included in the task. For example, a sample probability of 40% means each data row that meets the other filters has a 40% chance of being included in the task. |
Sample probability is processed after all other filters. It’s a probability, not a guarantee, so it’s possible that none or only a few data rows are included for the task, even if many data rows meet the other filters.
After creating custom review or rework tasks and the entering logic, arrange and connect them in the workflow:
Terminal nodes represent the default Rework and Done tasks. You can create multiple terminal nodes on the editor instead of connecting everything to a single Rework or Done node, making your workflow easier to read and visually cleaner. Adding multiple terminals does not create additional tasks and is purely for improving the visual layout.
Workflows are designed to be highly configurable, giving you the flexibility to optimize your QA process.
To modify a task or logic, select the node in the workflow editor to open its settings. To delete a node, click the trash can icon on the node.
To change the connections between tasks and logic, select the flow arrow connecting two nodes. Drag it to a different node to reconnect or click the trash can icon to delete the connection.
To view the breakdown of data rows in each workflow status, click the info icon next to the editor.
The rework flow is designed to automate the flow of specifying which data rows need to be fixed. The Move to task button allows you the flexibility to maintain a structured workflow while being able to manually move data rows to another task, if necessary. Moving data rows to different tasks in a single bulk action is also covered in the audit logs.
Send a single data row to rework
To reject a single data row and send it to rework:
Bulk send data rows to rework
To bulk reject a set of data rows and send them to rework:
Labelers can rework data rows
Labelers (in addition to reviewers) can rework labels on any data rows sent to the Rework task, regardless of whether they created the label. The rework task offers the same functionality as the legacy delete and requeue workaround, which will be sunsetted.
In addition to the structured workflow, sometimes it can be helpful to conduct ad-hoc reviews to get a regular pulse check on the quality of the annotations from your project. However, all reviews must be done through workflows, so you must create another task and add it to your project’s workflow.
Even with a structured workflow, this allows you to flexibly do ad-hoc reviews and keep track of them in an audit trail.
To create an ad-hoc task:
At any stage of the project, if a label on a data row is set as a benchmark, the data row (and thus any current and future labels made on the data row) will move to the Done task until the benchmark status is removed from the label.
If you are trying to review a data row with a benchmark label, remove the benchmark from the label and then proceed to move the data row to the appropriate task.
The audit logs allow admins to know the entire set of events that happened on a data row after it was labeled. Labelbox automatically creates this audit log for all data rows that enter the workflow. The purpose of the audit log is to help you understand the complete journey of each data row, especially when you need to investigate the review history of a particular data row or set of data rows.
To view the audit log for a data row:
This will show the complete review history for a data row after it is labeled.