Labeling queue

When you send a batch to a project, Labelbox will enter those data rows into the project's labeling queue. When a batch of data rows enters the queue of the Initial labeling task, Labelbox automatically distributes the data rows among each active labeler.

Labelbox reserves several data rows for that labeler — meaning no other team member can label an asset that is reserved by a teammate. The reservation system ensures no contention occurs and results in duplicate labeling. Additionally, this approach allows Labelbox to pre-load assets for a faster labeling experience.

View queued data rows

To view your assets queued for labeling, go to the data row tab in your project and click on the To label filter in the left-side panel. In this tab, you can easily explore your queued data to see what remains in the labeling queue. You can visualize the queued data using the gallery view.

To easily view the active reservations per team member, navigate to Project > Settings > Queue.

Set priority for queued data rows

You can set the priority of the data rows queued in your labeling project. To do so:

Use Annotate to open your project and then select your project.
When the project overview appears, select the Data Rows tab to display a list of the the data rows queued for your project. The current priority for each row is displayed in the Priority column.
Select the row (or rows) you want to change. When you do this, the Selection menu displays the number of selected data rows.
From the Selection menu, choose Change priority, which displays the Change data priority dialog.
Set the Data row priority field to the desired priority value and then select the Confirm button.

Priorities occur in the background. When you confirm new priorities, a notification banner acknowledges the changes and includes a link to the notification center, where you can follow progress.

The Data row priority field accepts 32-bit integer values, which means you can set priority to any integer value between -2,147,483,648 to 2,147,483,647. (For practical purposes, we recommend against using large priority values.)

The Change data row priority dialog considers lower values to be more important than higher values. That is, a priority value of 1 is considered higher priority than a value of 10. This means that negative values are considered more important than positive values. (In practice, you're free to interpret the values in any way you see fit.)

You can also control data row priorities by:

Setting the priority when you create a batch to be sent to a project.
Using the SDK to set or update the priority for individual data rows.

Set priority at batch creation

While setting up a batch to label, you can set the priority of data rows from 1 through 5. Note that a lower value of priority means it will get labeled earlier.

The above image is from the batch creation modal, where the priority for all data rows is selected to the default value of 5.

Update priority via the Python SDK

Even after data rows have been added to a project, the priority can be modified. The priority here can take all integer values. Currently, the modification of priority is supported via the SDK. See Project for instructions.

How reservations work

Once a labeler starts labeling, Labelbox will reserve three assets for the labeler.

As labelers work through their reservations, the queue will continue to replenish the reservations. This ensures that the labeler will have sufficient work available, thus avoiding unintended downtime due to latency spikes or changes made to the queued data.

Labelbox will keep reservations active for 24 hours. This timer renews every 10 minutes while a user has the asset open in their tab and is actively working. After 24 hours of idle time, the reservations will be cleared and are eligible to be served to other labelers.

An admin can clear the reservations for a user before the automated clearing using the instructions described in the following section.

Clear reservations

You can manually clear out reservations for a user by navigating to Project > Settings > Queue. Find the user in the queue for whom you wish to clear the reservations, click the vertical ellipses menu, and select Clear all reservations for this user.

🚧
Clearing reservations during active labeling
Any work performed and not yet submitted by the labeler will be lost if you clear the reservations (unless the same asset is taken by the same labeler at a later point in time, which Labelbox cannot guarantee will occur).

Real-time labeling

When you enable real-time labeling, it allows your team to label continuously. This can be helpful if you have labeling time constraints or if you need to boost your labeling team’s efficiency by enabling a continuous flow in the labeling queue.

You can enable real-time labeling in Annotate > select a project > Settings > Automation.

When all assets in the queue have been labeled, the labeler will see a button that says Wait for next asset. When a labeler clicks on Wait for next asset, Labelbox will display a real-time screen that will automatically display the next task that enters the queue.

Best practices

To avoid lost work, Labelbox recommends that labelers consult their administrator if they anticipate being away from their computer for over 24 hours. Any unsubmitted work done by the original labeler will be lost if you clear the reservations.
The timer will stop counting review and label times if our system detects 5 minutes of inactivity. When we detect user activity again, we restart the timer. This is something Labelbox does intentionally to record only active screen time on the application.
All labeling work should be limited to one browser tab, and no mobile devices should be used. Duplicate labels may be unintentionally created when a user has multiple tabs open and is actively labeling.