Labeling queue

When you send a batch to a project, Labelbox will enter those data rows into the project's labeling queue. When a batch of data rows enters the queue of the Initial labeling task, Labelbox automatically distributes the data rows among each active labeler.

Labelbox reserves several data rows for that labeler — meaning no other team member can label an asset that is reserved by a teammate. The reservation system ensures no contention occurs and results in duplicate labeling. Additionally, this approach allows Labelbox to pre-load assets for a faster labeling experience.

View queued data rows

To view your assets queued for labeling, go to the data row tab in your project and click on the To label filter in the left-side panel. In this tab, you can easily explore your queued data to see what remains in the labeling queue. You can visualize the queued data using the gallery view.

To easily view the active reservations per team member, navigate to Project > Settings > Queue.

Set priority in labeling queue

Labelbox is a flexible platform when it comes to labeling data. This is also reflected in the order in which the labeling needs to be done.

Given the iterative nature of active learning that the most advanced ML teams conduct today, it is often the case that the order in which labeling needs to occur can change dynamically. A host of reasons can contribute to this, such as:

• Prioritizing high-importance data where the model is uncertain
• Prioritizing diverse types of data rows that the team has not annotated before and hence has not been seen by the model
• Just-in-time tagging by humans in the loop when there isn't a model that has been built out

In all of these and related cases, the labeling team needs a mechanism to be able to modify the order in which data rows get labeled so the highest priority data rows, as expressed by the teams, are labeled the earliest.

Labelbox provides a couple of ways to set the priority of data rows to be labeled:

Set priority at batch creation

While setting up a batch to label, you can set the priority of data rows from 1 through 5. Note that a lower value of priority means it will get labeled earlier.

The above image is from the batch creation modal, where the priority for all data rows is selected to the default value of 5.

Update priority via the Python SDK

Even after data rows have been added to a project, the priority can be modified. The priority here can take all integer values. Currently, the modification of priority is supported via the SDK. See Project for instructions.

How reservations work

Once a labeler starts labeling, Labelbox will reserve a certain number of assets for the labeler, depending on the quality setting for the project. The number of assets reserved will be 3 for projects that do not use consensus and 10 for projects that use consensus.

As labelers work through their reservations, the queue will continue to replenish the reservations. This ensures that the labeler will have sufficient work available, thus avoiding unintended downtime due to latency spikes or changes in made to the queued data.

Labelbox will keep reservations active for 90 minutes. This timer renews every 10 minutes while a user has the asset open in their tab and is actively working. After 90 minutes of idle time, the reservations will be cleared and are eligible to be served to other labelers.

Clear reservations

You can manually clear out reservations for a user by navigating to Project > Settings > Queue. Find the user in the queue for whom you wish to clear the reservations, click the vertical ellipses menu, and select Clear all reservations for this user.

🚧

Warnings for clearing reservations

Please note that clearing all reservations for a user will clear every reservation for the user across all projects.

Additionally, any work performed and unsubmitted by the labeler will be lost if you clear the reservations (except if the same asset is taken by the same labeler, in this event work will persist).

Real-time labeling

When you enable real-time labeling, it allows your team to label continuously. This can be helpful if you have labeling time constraints or if you need to boost your labeling team’s efficiency by enabling a continuous flow in the labeling queue.

You can enable real-time labeling in Annotate > select a project > Settings > Automation.

When all assets in the queue have been labeled, the labeler will see a button that says Wait for next asset. When a labeler clicks on Wait for next asset, Labelbox will display a real-time screen that will automatically display the next task that enters the queue.

Best practices

• To avoid lost work, Labelbox recommends that labelers consult their administrator if they anticipate being away from their computer for over 90 minutes. Any unsubmitted work done by the original labeler will be lost if you clear the reservations.
• The timer will stop counting review and label times if our system detects 5 minutes of inactivity. When we detect user activity again, we restart the timer. This is something Labelbox does intentionally to record only active screen time on the application.
• All labeling work should be limited to one browser tab, and no mobile devices should be used. Duplicate labels may be unintentionally created when a user has multiple tabs open and is actively labeling.