Migrating to Workflows

A migration guide for Workflows, Labels tab, and Batch-based queueing.

What is Workflows?

Many AI labeling teams struggle to prioritize the right data to label and end up spending more money on data labeling than they should. In the past few months, we released two features – the Data Rows tab and Batches – to help teams better navigate and queue data for labeling.

Now, we’re excited to introduce a third feature called Workflows. Workflows give you more granular control over how your Data Rows are reviewed. You can use Workflows to create rule-based review tasks and multi-step sequences to reduce cost and increase the quality and efficiency of your labeling operations.

Watch these video to see how it works.

Here's another video that explains how Workflows, Batches, and the Data Row tab work together.

📘

Workflows rollout

We are enabling Workflows for our customers on a rolling basis. If you are interested in using Workflows, please contact [email protected] If you would like to give feedback on our Workflows feature, please fill out this form.

Old paradigm vs. new paradigm

Below are the advantages of the new Workflow + Data Rows tab + Batches paradigm.

Function

Labels tab + Review step + Dataset-based queueing

Data Rows tab + Workflow + Batches

Multi-step review

No

Up to 10 review steps per Workflow

Customizable review step

No

All 10 steps in the Workflow are customizable

Review history

No

Audit log shows all actions on a Data Row

Bulk actions

Yes

Yes

Adhoc review

Yes

Yes

Voting

Thumbs up/Thumbs down

Approve/Reject

Filters

Limited/manual

Automated quick filters by Data Row status (unlabeled, in review, in rework, etc)

Rework labels

Manually delete & requeue

All rejected Data Rows are automatically sent to “Rework” task in Workflow

Re-review

Limited

Select Data Rows and click “Move to task”

Limitations

Function

Labels tab + Review step + Dataset-based queueing

Data Rows tab + Workflow + Batches

Performance dashboard

Yes

No

Consensus

Yes

Coming soon

Current limitation for Workflows is 10K Data Rows. Coming soon: 25K Data Row limit.
Currently, only new Label imports are supported.

Upcoming changes

These changes will be implemented on a rolling basis.

When you create a new project, you’ll be asked to select a quality setting (between Benchmark or Consensus) that will determine your data upload method.

All Benchmark projects will default to Batch-based queueing, while all Consensus projects will default to Dataset-based queueing. You cannot change upload methods once you choose a quality setting.

By the end of the year, we will be disabling the following features for all customers as they are being replaced by better, more robust features. Stay tuned for more information regarding migrating your projects.

📘

Python SDK

If you're creating a project through our SDK, please refer to this section for code snippets on how to set your queuing mode.

Recommendations

When you queue your data for labeling, we recommend using Batch-based queueing instead of Dataset-based queueing. You will still be able to upload Datasets to Labelbox, however, in a few months, customers will no longer be able to directly add Dataset to a Project for labeling. All customers will be instructed to use Batches to queue data for labeling instead.

When you want to navigate or filter your labeled data, we recommend using the Data Rows tab. In a few months, we will discontinue support for the Labels tab and remove it from the platform.

When you want to set up a QA pipeline for your labeled data, we recommend using Workflows. In a few months, the Review step will be completely replaced by Workflow.

Additionally, the Project overview page now contains a quick view of the number of Data Rows per status (unlabeled, in review, in rework, etc).

📘

Benchmarks projects

Workflows is available for Benchmark projects only (Consensus not yet supported). New Benchmark projects will have an initial review task automatically set up. For existing projects, please go into the Workflow tab and create a new review task by clicking the ‘New Task’ button.

How to submit feedback

If you would like to submit feedback about Workflow, Data Rows tab, or Batches, please use this feedback form. Our product team reviews this feedback regularly.

FAQ

Why are we making this change?

Having worked with hundreds of AI teams, we recognized the need for more granular control over labeling workflows. In order to streamline and improve the creation, maintenance, and quality control of data rows, we’re introducing a new way for teams to queue and review.

How do these changes affect me?

Rather than queueing an entire dataset, we strongly encourage Batch-based queueing for more flexibility and control over your workflow. With Batches, you can:

  • Prioritize slices of data by adding batches to a project in priority
  • Manage batches & view batch history
  • Enable active learning workflows to identify the most high-impact data rows for labeling

After this change, if you’re creating a project manually within Labelbox, you’ll be asked to select a quality setting (between Benchmark or Consensus) that will determine your project’s data upload method. Benchmark projects will default to Batch-based queuing and Consensus projects will default to Dataset-based queueing. You cannot change upload methods once you choose a quality setting.

In the interim, if you’re creating a project through our SDK, you’ll need to specify if you wish to enable Batch-based queueing. You can refer to our documentation for specific code examples.

By the end of the year, we’ll be removing Dataset-based queueing in favor of Batch-based queueing. We’ll also be disabling the following features for all customers. Stay tuned for more information regarding migrating your projects.


Did this page help you?