Migrating to workflows
A guide for switching to Workflows, Data Rows tab, and Batch-based queueing.
What are workflows?
Many AI labeling teams struggle to prioritize the right data to label and end up spending more money on data labeling than they should. In the past few months, we released two features – the data rows tab and batches – to help teams better navigate and queue data for labeling.
Now, we’re excited to introduce a third feature called workflows. Workflows give you more granular control over how your Data Rows are reviewed. You can use workflows to create rule-based review tasks and multi-step sequences to reduce costs and increase the quality and efficiency of your labeling operations.
Watch these videos to see how it works.
Batches + data rows tab + workflows
This video explains how workflows, batches, and the data row tab work together.
Rollout plan: batches + data rows tab + workflows
New projects
Over the next few months, Labelbox will automatically configure all new projects with batches + data rows tab + workflows for new projects. These changes will happen on a rolling basis.
The table below indicates the cutoff date after which all new projects will be automatically configured with batches + data row tab + workflows.
Change | Free/EDU | Starter | Pro | Enterprise | Trial |
---|---|---|---|---|---|
New projects automatically configured with - Batches - Workflows - Data row tab - Batch-level consensus | 11/21/22 (week of) | 11/21/22 (week of) | 12/13/22 (week of) | 12/13/22 (week of) | 12/13/22 (week of) |
New projects will not have access to - Dataset-queue mode - Labels tab - Review step - Labeling parameter overrides (LPO) - Project-level consensus | 11/21/22 (week of) | 11/21/22 (week of) | 12/13/22 (week of) | 12/13/22 (week of) | 12/13/22 (week of) |
Old projects
The table below indicates the planned migration deadlines for all old projects to be automatically migrated to use Batches + Data Row tab + Workflows.
Migrating old projects
We will begin migrating old projects to the new batches + data row tab + workflow paradigm starting on 3/31/23. Soon, we will be sharing a migration schedule for each customer tier. For now, no action is required for your old projects.
Change | Free/EDU | Starter | Pro | Enterprise | Trial |
---|---|---|---|---|---|
Migration deadline for old projects | TBD | TBD | TBD | TBD | TBD |
Old projects will not have access to - Dataset-queue mode - Labels tab - Review step - Labeling parameter overrides (LPO) - Project-level consensus | 4/30/23 | 4/30/23 | 4/30/23 | 4/30/23 | 4/30/23 |
Features being replaced
For clarity, here is a table that states which features are being replaced by newer, better features.
Features to be disabled | Replaced by |
---|---|
Review step | Workflows |
Delete & requeue | Workflows |
Dataset-based queueing | Batch-based queueing |
Label activity table | Data rows tab |
LPO & project-level consensus | Consensus at the batch level |
Comparison: Old paradigm vs new paradigm
Below are the advantages of the new Workflow + Data Rows tab + Batches paradigm.
Function | Labels tab + review step + dataset-based queueing | Data rows tab + workflows + batches |
---|---|---|
Multi-step review | No | Up to 10 review steps per workflow |
Customizable review step | No | All 10 steps in the workflow are customizable |
Review history | No | Audit log shows all actions on a Data Row |
Bulk actions | Yes | Yes |
Ad-hoc review | Yes | Yes |
Voting | Thumbs up/thumbs down | Approve/reject |
Filters | Limited/manual | Automated quick filters by data row status (unlabeled, in review, in rework, etc) |
Rework labels | Manually delete & requeue | All rejected data rows are automatically sent to Rework task in Workflow |
Re-review | Limited | Select data rows and click Move to task |
Limitations
- The current limitation for workflows is 10K data rows. Coming soon: 25K data row limit.
- Currently, only new label imports are supported.
How to submit feedback
If you would like to submit feedback about workflow, data rows tab, or batches, please use this feedback form. Our product team reviews this feedback regularly.
FAQ
Why are we making this change?
Having worked with hundreds of AI teams, we recognized the need for more granular control over labeling workflows. In order to streamline and improve the creation, maintenance, and quality control of data rows, we’re introducing a new way for teams to queue and review.
How do these changes affect me?
Rather than queueing an entire dataset, we strongly encourage batch-based queueing for more flexibility and control over your workflow. With batches, you can:
- Prioritize slices of data by adding batches to a project in priority
- Manage batches & view batch history
- Enable active learning workflows to identify the most high-impact data rows for labeling
Changes for Free/EDU only (before 11/21)
Free/EDU only
Our Free and EDU customers received an early version of the batches + data row tab + workflow paradigm. These changes only apply to you if you are a free or EDU customer. And these changes outlined from here to the end of this page will be superseded by the changes outlined in the sections above.
If you’re creating a project through the UI
- You will be prompted to select a quality setting (benchmark or consensus) that will determine your project’s queueing mode.
- You cannot change the quality setting or queueing mode after a project is created.
- Benchmark projects will default to batch-based queuing.
- Consensus projects will default to dataset-based queueing.
Benchmarks projects
Workflows are available for benchmark projects only (consensus not yet supported). New benchmark projects will have an initial review task automatically set up. For existing projects, please go into the Workflow tab and create a new review task by clicking the New Task button.
If you’re creating a project through the SDK
- When creating a project, you must specify a queueing mode and quality mode (the backend will no longer infer defaults since these values cannot be changed after a project has been created). Learn more here on how to set up your queueing and quality modes.
a. Projects configured with benchmarks will default to batch queuing mode.
b. Projects configured with consensus will default to dataset-queueing mode.
c. You cannot update the quality setting or queueing mode after a project has been created.
d. Datasets cannot be attached to batch projects.
e. Batches cannot be attached to dataset projects. - All new projects will require media type upon creation. Learn how to set media type here.
- Batches have new arguments for configuring quality settings.
Queue mode / quality settings combinations
Use the table below to understand the supported combinations and default configurations available when configuring the queue mode and quality mode for your project.
Queue mode | Quality settings | Mode |
---|---|---|
None specified | None specified | Batches + Benchmark |
None specified | Consensus | Dataset + Consensus |
Dataset | None specified | Dataset + Benchmark |
Dataset | Consensus | Dataset + Consensus |
Dataset | Benchmark | Dataset + Benchmark |
Batches | None specified | Batches + Benchmark |
Batches | Consensus | ERROR (Batches + Consensus not yet supported) |
Batches | Benchmark | Batches + Benchmark |
None specified | Consensus | Dataset + Consensus |
None specified | Benchmark | Batches + Benchmark |
Python SDK
See our reference to learn how to set the queueing mode when setting up a project.
Updated 26 days ago