Labelbox documentation

Understanding the metrics

Annotation breakdown

You can view the Annotation usage count for your Organization by navigating to Account > Usage. The Annotation breakdown chart shows annotation count by asset type (i.e. Text, Video, Images) and they are bucketed by creation date (GMT). Deleted annotations are included in the annotation usage count.

This section also contains two tables for Organization usage and Labeling breakdown. You can also get billable annotation counts programmatically via the GraphQL API.

account-section-2.png

Labels created

You can find the Labels created chart by selecting a project and looking in the Overview tab. This chart shows the number of Labels created in that project over time. You can toggle between daily, weekly, and monthly in the upper right corner of the chart.

5f19cf1c7bfa4.png

Team member performance

Go to the Performance tab to view the average metrics across all labelers or drill down into individual performance for Label time or Review time.

There are two sub-tabs — one for Labels and one for Reviews.

Average metrics across all labelers

Metric

Label sub-tab definition

Review sub-tab definition

Avg Count

Average number of labels created.

Average number of labels reviewed.

Avg Total Time

Average label time across all labels.

Average review time across all labels.

Avg Time per

Average time labeling per label.

Average time reviewing per label.

Individual performance

Metric

Labels sub-tab

Review sub-tab

Count

Number labels created.

Number of labels reviewed.

Total Time

Total time spent labeling in project.

Total time spent reviewing in project.

Time per

Average labeling time per label.

Average reviewing time per label.

Label time & review time

Labelbox keeps track of label and review time and displays them in two separate columns within the Activity table for each data row. You can find the Activity table by selecting a project and looking under the Labels tab.

Label time the creator of the label spends viewing or editing an unsubmitted label in the labeling interface. The timer starts when the image is fully loaded and stops when the user clicks "skip", "submit", or exits out of the labeling interface. To ensure idle time is not captured, the timer automatically pauses when the user is inactive on the UI for 30 seconds and resumes when the user interacts with the keyboard or mouse or refreshes the page. If the user goes back to a previous label in the queue, the timer resumes after 3 seconds and the time is added to Label time for that data row.

The Review time column indicates the total time all users who did not create the label view, edit, or review the submitted label in review mode. When an image undergoes review, the timer starts when the label loads and stops when the user moves on to the next label in the review queue.