Labelbox documentation

Review labels

This section explains how to use the review tools in Labelbox, how the review queue works, and how to configure the review step to meet your quality assurance needs.

Configure review percentage

You can adjust the percentage of labels to be reviewed by going to Settings > Quality > Review Step. This indicates the percentage of labels that will enter the review queue.

5f19dac9b63e4.png

As an admin, you can bypass the percentage restriction set in Review Step by navigating to Labels > Activity and clicking on any label to enter open review mode for all labels in the review queue. Open review mode offers the most functionality.

Queue-based review

The review queue contains the percentage of submitted or skipped assets configured in the Review step for the project. The review queue abides by the following rules:

  1. Only submitted or skipped assets can be entered into the review queue.

  2. With additional configuration, an asset in the review queue may be reviewed by more than one user, but never twice by the same user.

Members with the Reviewer role will see these assets in a special review interface. Reviewers are able to modify the annotations and submit their review by approving or rejecting the labeled assets.

Open review

Open review refers to the review interface that appears when an admin user clicks on a row in the activity list within a project. It presents a streamlined and transparent way to review, modify, copy, and re-enqueue asset into the label queue, and track progress. You can do the following things in review mode:

  • Modify a review

  • Modify a label

  • Copy a label URL

open-review-label-browser-2.png

In the Label browser panel, you will see a list of all assets in your project and whether their labels were accepted or rejected.

Review report

The Reviews section at the bottom of the Overview tab contains a simple metric for understanding the general trends in user reviews.

5f19dacb5d6b6.png

Un-reviewed: No reviews at all

Accepted: Labels where the majority upvoted

Declined: Labels where the majority downvoted

Ambiguous: Labels where upvotes and downvotes cancel each other out