Migrating to Workflows
A migration guide for Workflows, Labels tab, and Batch-based queueing.
What is Workflows?
Many AI labeling teams struggle to prioritize the right data to label and end up spending more money on data labeling than they should. In the past few months, we released two features – the Data Rows tab and Batches – to help teams better navigate and queue data for labeling.
Now, we’re excited to introduce a third feature called Workflows. Workflows give you more granular control over how your Data Rows are reviewed. You can use Workflows to create rule-based review tasks and multi-step sequences to reduce cost and increase the quality and efficiency of your labeling operations.
Watch these video to see how it works.
Here's another video that explains how Workflows, Batches, and the Data Row tab work together.
Workflows rollout
We are enabling Workflows for our customers on a rolling basis. If you are interested in using Workflows, please contact [email protected] If you would like to give feedback on our Workflows feature, please fill out this form.
Old paradigm vs. new paradigm
Below are the advantages of the new Workflow + Data Rows tab + Batches paradigm.
Function | Labels tab + Review step + Dataset-based queueing | Data Rows tab + Workflow + Batches |
|---|---|---|
Multi-step review | No | Up to 10 review steps per Workflow |
Customizable review step | No | All 10 steps in the Workflow are customizable |
Review history | No | Audit log shows all actions on a Data Row |
Bulk actions | Yes | Yes |
Adhoc review | Yes | Yes |
Voting | Thumbs up/Thumbs down | Approve/Reject |
Filters | Limited/manual | Automated quick filters by Data Row status (unlabeled, in review, in rework, etc) |
Rework labels | Manually delete & requeue | All rejected Data Rows are automatically sent to “Rework” task in Workflow |
Re-review | Limited | Select Data Rows and click “Move to task” |
Limitations
Function | Labels tab + Review step + Dataset-based queueing | Data Rows tab + Workflow + Batches |
|---|---|---|
Performance dashboard | Yes | No |
Consensus | Yes | Coming soon |
Current limitation for Workflows is 10K Data Rows. Coming soon: 25K Data Row limit.
Currently, only new Label imports are supported.
Upcoming changes
These changes will be implemented on a rolling basis.
When you create a new project, you’ll be asked to select a quality setting (between Benchmark or Consensus) that will determine your data upload method.
All Benchmark projects will default to Batch-based queueing, while all Consensus projects will default to Dataset-based queueing. You cannot change upload methods once you choose a quality setting.
By the end of the year, we will be disabling the following features for all customers as they are being replaced by better, more robust features. Stay tuned for more information regarding migrating your projects.
Features to be disabled | Replaced by |
|---|---|
Python SDK
If you're creating a project through our SDK, please refer to this section for code snippets on how to set your queuing mode.
Recommendations
When you queue your data for labeling, we recommend using Batch-based queueing instead of Dataset-based queueing. You will still be able to upload Datasets to Labelbox, however, in a few months, customers will no longer be able to directly add Dataset to a Project for labeling. All customers will be instructed to use Batches to queue data for labeling instead.
When you want to navigate or filter your labeled data, we recommend using the Data Rows tab. In a few months, we will discontinue support for the Labels tab and remove it from the platform.
When you want to set up a QA pipeline for your labeled data, we recommend using Workflows. In a few months, the Review step will be completely replaced by Workflow.
Additionally, the Project overview page now contains a quick view of the number of Data Rows per status (unlabeled, in review, in rework, etc).
Benchmarks projects
Workflows is available for Benchmark projects only (Consensus not yet supported). New Benchmark projects will have an initial review task automatically set up. For existing projects, please go into the Workflow tab and create a new review task by clicking the ‘New Task’ button.
How to submit feedback
If you would like to submit feedback about Workflow, Data Rows tab, or Batches, please use this feedback form. Our product team reviews this feedback regularly.
FAQ
Why are we making this change?
Having worked with hundreds of AI teams, we recognized the need for more granular control over labeling workflows. In order to streamline and improve the creation, maintenance, and quality control of data rows, we’re introducing a new way for teams to queue and review.
How do these changes affect me?
Rather than queueing an entire dataset, we strongly encourage Batch-based queueing for more flexibility and control over your workflow. With Batches, you can:
- Prioritize slices of data by adding batches to a project in priority
- Manage batches & view batch history
- Enable active learning workflows to identify the most high-impact data rows for labeling
After this change, if you’re creating a project manually within Labelbox, you’ll be asked to select a quality setting (between Benchmark or Consensus) that will determine your project’s data upload method. Benchmark projects will default to Batch-based queuing and Consensus projects will default to Dataset-based queueing. You cannot change upload methods once you choose a quality setting.
In the interim, if you’re creating a project through our SDK, you’ll need to specify if you wish to enable Batch-based queueing. You can refer to our documentation for specific code examples.
By the end of the year, we’ll be removing Dataset-based queueing in favor of Batch-based queueing. We’ll also be disabling the following features for all customers. Stay tuned for more information regarding migrating your projects.
Updated 16 days ago