Labelbox documentation

Understanding the metrics

Annotation breakdown

You can view the Annotation usage count for your Organization by navigating to Account > Usage. The Annotation breakdown chart shows annotation count by asset type (i.e. Text, Video, Images) and they are bucketed by creation date (GMT). Deleted annotations are included in the annotation usage count.

This section also contains two tables for Organization usage and Labeling breakdown. You can also get billable annotation counts programmatically via the GraphQL API.

updated-usage-tab.png

Labels created

You can find the Labels created chart by selecting a project and looking in the Overview tab. This chart shows the number of Labels created in that project over time. You can toggle between daily, weekly, and monthly in the upper right corner of the chart.

5f19cf1c7bfa4.png

Performance dashboard

The Performance dashboard is one of the primary tools used to manage labeling operations in a Labelbox project. It reports the throughput, efficiency, and quality of the labeling process. The throughput, efficiency, and quality analytics are reported at the overall project level and at an individual level. In addition, diagnostic analysis of the labeling operation can be performed at each level of detail.

Evaluating the performance of your data labeling operation can be broken down into three components:

  • Throughput

  • Efficiency

  • Quality

Each of these has its own view so you can better understand the overall performance of your labeling operation.

Note

All times shown in the Performance dashboard are in UTC. All data from skipped and deleted labels and reviews will be shown in the Performance dashboard.

The Throughput view provides insight into the amount of labeling work being produced. The metrics in this section help you answer questions like, How many assets were labeled in the last 30 days?, How much time is being spent reviewing labeled assets?, and What is the average amount of labeling work being produced?

Clicking on a bar element in the bar charts will show the associated labels comprising that data in the Activity Table.

performance-dashboard-v2.png

The metrics shown above are available at the project level (i.e., across all members in the project) and at the individual level. Here are the descriptions for each metric under the Throughput view.

Metric

Description

Labels

The count of Labels (labeled assets) over the selected period of time.

Reviews

The count of Reviews plotted over the selected period. A Review is created when a Thumbs Up (Approved) or Thumbs Down (Rejected) is submitted while reviewing a labeled asset. This includes when a labeled asset is being reviewed/reworked from the Label browser or the queue-based review. A Review can be created from either screen.

Labeling time

The sum of labeling time over the selected period.

Reviewing time

The sum of reviewing and editing (rework) time plotted over the selected period.

NOTE: When a user reviews and/or edits a labeled asset that they created, that time is not counted toward Reviewing time. Instead, it is counted toward Labeling time. Review time is captured when a member who did not create the labeled asset spends time viewing, editing, and/or reviewing that labeled asset in the Label browser or via queue-based review.

Total time

The sum of all labeling time, reviewing time, and reworking time plotted over the selected period.

The Efficiency view displays the time spent per unit of work (e.g., per labeled asset or per review). The metrics in this section help you answer questions like, What is the average amount of time spent labeling an asset?, and How can I reduce time spent per labeled asset?

Effieciency-subtab.png

The metrics shown above are available at the project level (i.e., across all members in the project) and at the individual level. Here are the descriptions for each metric under the Efficiency view.

Metric

Description

Avg time per label

The average time spent labeling an asset before submitting or skipping it in the Editor.

Avg time per review

The average time spent reviewing and editing (reworking) a labeled asset plotted over the selected period.

NOTE: When a user reviews and/or edits a labeled asset that they created, that time is not counted toward Reviewing time. Instead, it is counted toward Labeling time. Review time is captured when a member who did not create the labeled asset spends time viewing, editing, and/or reviewing that labeled asset in the Label browser or via queue-based review.

The Quality view helps you understand the accuracy and consistency of the labeling work being produced. The metrics in this section help you answer questions like, What is the average quality of a labeled asset?, and How can I ensure label quality is more consistent across the team?

Clicking on a bar element in the bar charts will show the associated labels comprising that data in the Activity Table.

quality-subtab.png

The metrics shown above are available at the project level (i.e., across all members in the project) and at the individual level. Here are the descriptions for each metric under the Quality view. Note that only relevant dashboard components will appear depending on whether you have Benchmarks or Consensus set up.

Metric

Description

Benchmark

The average benchmark score of labeled assets over the selected period.

Benchmark distribution

A histogram of benchmark scores (grouped by 10) for labeled assets plotted over the selected period.

Consensus

The average consensus score of labeled assets over the selected period.

Consensus distribution

A histogram of consensus scores (grouped by 10) for labeled assets plotted over the selected period.

Individual labeler performance is shown for the selected time period. The table can be sorted by all columns. You can sort columns to diagnose labeler performance issues that are impacting project metrics. One such common use case is finding individuals with the shortest or longest average time per label as this may be an indication of over-prioritizing speed over quality in the former, and vice versa in the latter; among many other underlying issues.

The table data can be exported in CSV format using the download button shown on the top right of the table. For label level detail, export data from the Export tab.

Label time & review time

Labelbox keeps track of label and review time and displays them in two separate columns within the Activity table for each data row. You can find the Activity table by selecting a project and looking under the Labels tab.

Label time the creator of the label spends viewing or editing an unsubmitted label in the labeling interface. The timer starts when the image is fully loaded and stops when the user clicks "skip", "submit", or exits out of the labeling interface. To ensure idle time is not captured, the timer automatically pauses when the user is inactive on the UI for 30 seconds and resumes when the user interacts with the keyboard or mouse or refreshes the page. If the user goes back to a previous label in the queue, the timer resumes after 3 seconds and the time is added to Label time for that data row.

The Review time column indicates the total time all users who did not create the label view, edit, or review the submitted label in review mode. When an image undergoes review, the timer starts when the label loads and stops when the user moves on to the next label in the review queue.