Rework with labels

📘

Rework in workflows

In order to provide a less involved process of re-working labels during QA, Labelbox will automatically place rejected data rows in a specialized rework task in the Workflow tab. Learn more here.

Re-working labels are one of the best ways to keep your labeling costs low while radically improving its quality. A lot of customers come to Labelbox after experiencing pure outsourcing services, and they are delighted by experiencing the ability to fully control the label lifecycle in Labelbox. Rework with labels can be perhaps best understood with an analogy: imagine, if you had to ask your software programmers to write software from scratch every time. Clearly, we don't do that. Even when software is outsourced, it is done so using git-based collaborative cloud platforms.

Below are a few common ways you can rework labeled data rows.

Process of re-queueing assets

When you delete the annotations on an asset, Labelbox automatically re-enters that asset into the labeling queue to get relabeled. You can choose to use the existing label as a template for the future labeler.

Re-queueing through Data rows tab

Follow these steps to relabel an asset:

1. Go to the Data rows tab.

2. Select the data row(s) you wish to relabel. This will enable a button on the top right to say N selected where N denotes the number of data rows that have been selected.

3. Click the highlighted button to show the dropdown and select Delete and requeue from the options. This will prompt you to confirm whether to use deleted labels as templates.

   • If you select the option of using a template and the dataset or batch is still attached to the project, the asset along with its label template will be added back to the label queue.
   • If the template option isn't selected, only the asset will be added to the label queue.

4. Click Start labeling to load the asset in the Editor and label it again.

Re-queueing large number of assets from the Data Rows tab

Labelbox makes it possible for you to delete and re-queue upto 10,000 assets in one go from the Data rows tab. Note that the system makes a distinction based on whether it is greater than 1,000 assets in one go.

1. If the number of assets is less than or equal to 1,000, the operation will be done synchronously.
2. If the number of assets exceeds 1,000, the operation will trigger an asynchronous job that you will be able to track the progress for in the Notifications tab.
   1. Each of these jobs will provide the following information - job Id, job name, user triggering the job and progress/success.

Re-queueing through Labels tab

Follow these steps to relabel an asset:

1. Go to the Labels tab.

2. Select the data row you wish to relabel and click Delete labels.

3. Check the box to use the deleted label as a template (starting point) the next time the asset is loaded into the Editor. As long as your dataset is attached to your project, the asset along with its label template will be added back to the label queue. If your dataset is not attached to your project, re-enqueued assets and label templates will not be available until the dataset is reattached to the project.

4. Click Start labeling to load the asset in the Editor and label it again.

Filter, inspect and create an issue for resolution

Driven by any hypothesis about the data, you can filter labels or data rows and create an issue on the label to be resolved.

• Using filters in data row activity
• Using Issues & Comments