Labelbox documentation

Queue system

queue-system-2.png

Labelbox uses a queue system to organize and distribute data to labelers and reviewers in your project. Each project has one Label queue and one optional Review queue.

When you add your data to a project in Labelbox, those assets are added to the project’s Label queue. Every time a Label is skipped or submitted in the labeling interface, Labelbox automatically de-queues the Label. The Label queue is rebuilt every time the project consensus settings are changed, a dataset is attached or detached from a project, a label is deleted, or you explicitly rebuild it via the GraphQL API.

When you have the Review step enabled, a percentage of your Labels are added to the Review queue. The Review queue is not customizable and there is no exclusive reservation system for distribution. Admins can view and modify reviewed Labels by clicking on the Data Row from the Activity table.

Label queue

Review queue

Contains unlabeled assets

Contains submitted and skipped labels

Queue customization supported

Queue customization not supported

Assets are reserved upon distribution to labelers

Labels are not reserved upon distribution to reviewers

View enqueued assets under Labels tab > Queue

No way to view enqueued Labels in the Labelbox UI

Automatically enabled for all projects

Must be manually enabled in the project settings

Reservation system

Labelbox distributes assets from the Label queue in batches to each active individual labeler. When a batch of assets is distributed to an active labeler, Labelbox “checks out” those assets to that labeler. Once the individual’s queue of reserved assets empties, Labelbox automatically fills it with a new batch of unlabeled assets.

5f19dc415a252.png

If a user starts labeling an asset, then is idle for more than 90 min or logs out, Labelbox will assign the asset to another active labeler. However, if the original labeler signs back in, the reservation will be refreshed. This may result in two separate Labels on the same asset.

Label queue customization

You can use the GraphQL API to make the following customizations to your Label queue. See our GraphQL docs on Labeling parameters to learn more.Labeling parameters

Number of Labels

By default, each asset in the Label queue will be labeled once. However, if you need a specific asset to be labeled more than once, you can use the GraphQL API to target an individual asset and specify the number of times it should get labeled. Labelbox will automatically re-enter that asset into the Label queue and redistribute it to active labelers until it has received the number of Labels specified.

Priority

If you need your assets to appear in the queue in a certain order, you can assign them individual priority scores. The order of non-prioritized assets in the queue is not guaranteed.

In this prioritization example, asset E will be labeled first, A will be second, and D will be third.

queue-priority-1.png

If you skip numbers when you set the prioritization, the Label queue will default to the next asset in the priority order. In this example, asset E will be labeled first, A will be second, and D will be third.

queue-priority-2.png

When you assign multiple assets the same priority number without rebuilding the queue, they will be ordered lexicographically. In this example, asset B will be labeled first, E will be second, A will be third, and D will be fourth.

queue-priority-3.png