📘

Closed beta

Workflows is in closed beta. If you are interested in participating, please contact [email protected]

As you work with large, complex projects in Labelbox, having to review all labeled Data Rows becomes increasingly time-consuming and expensive. With Workflows, you have more granular control over how your Data Rows get reviewed. Now you can create a highly customizable, step-by-step review pipeline to drive efficiency and automation into your review process.

How it works

Every project has one highly customizable review workflow. You can access it by clicking on the Workflow tab within a project.

A workflow is made up of a series of tasks - each serving a specialized purpose in the overall review flow. For example, if you want all Data Rows that contain stop sign annotations to be reviewed, you can add a task and set filtering criteria so that only Data Rows with stop sign annotations will be admitted to this task. You can also create a task for reviewing Data Rows labeled by a particular labeler or a group of labelers. Each step in the workflow is configurable so you have the most flexibility in creating your review pipeline.

By default, there are three tasks that will appear in your workflow by default: Initial labeling task, Rework, and Done. Your project's workflow will automatically update when Data Rows are labeled or rejected.

• Initial labeling task: reserved for all Data Rows that have been labeled
• Rework task: reserved for Data Rows that have been Rejected
• Done task: reserved for Data Rows that have a) moved through their qualified tasks in the workflow or b) did not qualify for any of the tasks

Because the process for reviewing Data Rows is iterative, we designed Workflows to automatically update when Data Rows are labeled or rejected. You can set filtering criteria to control which Data Rows are sent to which review tasks. All Data Rows rejected at any of the intermediate review tasks are automatically sent to a specialized review task without any manual intervention.

Go to the Workflow tab to create, review, and maintain your review workflows.

How Data Rows move through the Workflow

A project Workflow is similar to a conveyor belt, where Data Rows move along sequentially through the tasks like checkpoints. Here is how a Data Row would move through a workflow.

1. If a Data Row fits the filtering criteria for a review task, the Data Row will enter that review task. From here, there are two possible outcomes:

   a. It gets an Approve review and goes on to the next task.

   b. It gets a Reject review and gets directed to Rework (all rejected).

2. If a Data Row does not fit the criteria for a review task, it automatically gets passed to the next review task.

3. Once Data Rows are modified/reworked in the Rework (all rejected) task, they go through the remaining tasks in the workflow again.

In the example above, if a Data Row a) does not receive an Approve review or b) does not meet the criteria for either review task, the Data Row will go through the following tasks: Initial labeling task -> Labeler review -> Annotation review -> Done.

However, if the Data Row receives a Reject review in either of the review tasks, it will move to the Rework task.
Hence, a successful flow order for would be: Rework (all rejected) -> Labeler review -> Annotation review -> Done.

Create a new task

📘

Limit 10 tasks

A workflow can have up to 10 tasks.

1. To set up a task, click on the + New task button. This will open the task configuration modal where you can specify:
   • Task name: provides a high-level description of the review task
   • Instructions: provides reviewers direction for reviewing Data Rows in that particular task
2. Next, select Filters to customize which Data Rows will qualify for the review task. These are the filters you can choose from:
   • Created by: person who labeled the Data Row
   • Annotation: the annotation present in the Data Row
   • Dataset: the dataset the Data Row belongs to
   • Labeled at: date the Data Row was labeled

Work on a task

1. To see the Data Rows in a given task, click on the task name and click View Data Rows.

2. To start working on a review task, click on the gear button and click Work on task.
3. This will open another browser where you can review assets with Approve and Decline reviews on a Data Row. Approved Data Rows will move to the next review task while declined will move into Rework.

Configure and close tasks

Workflows are designed to be highly configurable. The ability to configure any task in the workflow offers you the most flexibility to drive the efficiency of your QA process.

To modify any task, go to Edit task and modify the filter OR add a new filter.

To close a task (if you don’t need the task anymore), click on the gear icon and click Close task. You can only close a task if it has 0 Data Rows.

Reject & rework Data Rows

This Rework flow is designed to automate the flow of sending rejected Data Rows to be fixed. The Move to task button allows you the flexibility to maintain a structured workflow while being able to move Data Rows to another Task, if necessary. Bulk moves to tasks are also covered in the audit logs.

Send a single Data Row to Rework

To reject a single Data Row and send it to rework:

1. Select the task on the Workflow tab
2. Click Start Reviewing
3. Click Reject on the Data Rows you want to send to rework. All rejected Data Rows are automatically sent to Rework.

Bulk send Data Rows to Rework

To bulk reject a set of Data Rows and send them to rework:

1. Go to the Data Rows tab
2. Use the filters to specify the Task the Data Rows are in
3. Select a set of Data Rows (max 100 at a time)
4. From the dropdown, select Move to task & select Rework

Create an ad-hoc review task

In addition to the structured workflow, sometimes it can be helpful to conduct ad-hoc reviews to get a regular pulse check on the quality of the annotations from your project. However, all reviews must be done through Workflows, so you will need to create another task and add it to your project's workflow.

Even with a structured workflow, this allows you to flexibly do ad-hoc reviews and keep track of them in an audit trail.

To create an ad-hoc task:

1. Click on the + New task button.
2. Customize the filters, if any.
3. Go to the Data rows tab.
4. Select data rows in bulk and select Move to task.
5. Go back to the Workflow tab.
6. Click on the ad-hoc task you created & click Start Reviewing.

View Audit logs

The audit log allows admins to know the entire set of events that happened on a Data Row after it was labeled. Labelbox automatically creates this audit log for all Data Rows that enter the Workflow. The purpose of the audit log is to help you understand the complete journey of each Data Row, especially when you need to investigate the review history of a particular Data Row or set of Data Rows.

To view the audit log for a Data Row:

1. From the Workflow tab, click on a task. This will open another browser where you can see the Data Row.
2. Click on the clock icon.

This will show the complete review history for a Data Row after it is labeled.