Performance dashboard
Information on the project performance dashboard and the included metrics.
The Performance dashboard is one of the primary tools for managing labeling operations in a Labelbox project. It reports the throughput, efficiency, and quality of the labeling process. The throughput, efficiency, and quality analytics are reported at the overall project level and at an individual level. In addition, diagnostic analysis of the labeling operation can be performed at each level of detail.
Evaluating the performance of your data labeling operation can be broken down into three components. Each of these has its own view so you can better understand the overall performance of your labeling operation.
Throughput
The Throughput view provides insight into the amount of labeling work being produced. The metrics in this section help you answer questions like, How many assets were labeled in the last 30 days?, How much time is being spent reviewing labeled assets?, and What is the average amount of labeling work being produced?
Clicking on a bar element in the bar charts will show the associated labels comprising that data in the activity table.
The metrics shown above are available at the project level (i.e., across all members of the project) and at the individual level. Here are the descriptions for each metric under the Throughput view.
| Metric | Description |
|---|---|
| Labels | The count of labeled assets over a selected period of time. |
| Reviews | The count of Reviews plotted over the selected period. A Review is created when a Thumbs Up (Approved) or Thumbs Down (Rejected) is submitted while reviewing a labeled asset. This includes when a labeled asset is being reviewed/reworked from the Label browser or the queue-based review. A Review can be created from either screen. |
| Labeling time | The sum of labeling time over the selected period. |
| Reviewing time | The sum of reviewing and editing (rework) time plotted over the selected period. |
| Total time | The sum of all labeling time, reviewing time, and reworking time plotted over the selected period. |
Timer timeout
The timer will stop counting review time and label time if our system detects 5 minutes of inactivity (such as user leaving Labelbox to go to another tab or taking a break). When we detect user activity again, we start the timer back up. This is something we do intentionally to record only active screen time on the application.
Efficiency
The Efficiency view displays the time spent per unit of work (e.g., per labeled asset or per review). The metrics in this section help you answer questions like, What is the average amount of time spent labeling an asset?, and How can I reduce time spent per labeled asset?
The metrics shown are available at the project level (i.e., across all members of the project) and at the individual level. Here are the descriptions for each metric under the Efficiency view.
| Metric | Description |
|---|---|
| Avg time per label | The average time spent labeling an asset before submitting or skipping it in the Editor. |
| Avg time per review | The average time spent reviewing and editing (reworking) a labeled asset plotted over the selected period. NOTE: When the user reviews and/or edits a labeled asset that they created, that time is not counted toward Reviewing time. Instead, it is counted toward Labeling time. Review time is captured when a member who did not create the labeled asset spends time viewing, editing, and/or reviewing that labeled asset in the Label browser or via queue-based review. |
Quality
The Quality view helps you understand the accuracy and consistency of the labeling work being produced. The metrics in this section help you answer questions like, What is the average quality of a labeled asset? and How can I ensure label quality is more consistent across the team?
Clicking on a bar element in the bar charts will show the associated labels comprising that data in the activity table.
The metrics shown are available at the project level (i.e., across all members of the project) and at the individual level. Here are the descriptions for each metric under the Quality view. Note that only relevant dashboard components will appear depending on whether you have benchmarks or consensus set up.
| Metric | Description |
|---|---|
| Benchmark | The average benchmark score of labeled assets over the selected period. |
| Benchmark distribution | A histogram of benchmark scores (grouped by 10) for labeled assets plotted over the selected period. |
| Consensus | The average consensus score of labeled assets over the selected period. |
| Consensus distribution | A histogram of consensus scores (grouped by 10) for labeled assets plotted over the selected period. |
Filtering based on users
To filter the performance dashboard to a specific group of users, you can click the filter option at the top right and a filter module will appear on the right side of the screen. From the filter module, select the users you want to be included in the Performance dashboard and click Apply.
FAQs
How does label time get measured?
Currently, the performance dashboard only takes into account labeling time when a label was skipped/submitted. Hence, if a labeler spends time browsing an asset without hitting Skip/Submit that time does not get added to the label time.
How do the label and review times and counts get populated in the graph?
The label and review counts reflect the date when the label was created and not when the label was submitted. This means that if you open a label that was created yesterday and edit it, you will observe that labeling time appears in today’s performance metrics, but instead will appear in yesterday's. This implies that the metrics for a given day can vary over time.
How does the timer work when labels are imported as ground truth?
If a label is imported as ground truth, the label time will appear as zero on the Data Rows tab and also on the performance dashboard. Any time spent looking at the label or modifying it will count as review time.
How does the timer work when labels are imported as pre-labels (MAL)?
If a label is imported as a pre-label, the label time will be considered in a manner similar to regular labeling. Hence the time spent by a labeler before hitting Skip/Submit will be added to label time.
What is the time zone that is used in the metric view?
The system uses UTC as the reference time zone to render all metrics.
Updated about 1 month ago