Model run custom metrics

Upload custom metrics

If the auto-generated metrics are not sufficient for your use case, you can upload custom metrics to your model run. This will help you even more precisely evaluate your model performance in Labelbox.

Upload custom metrics to individual prediction annotations.

To upload custom metrics to individual predictions, you can append the following list of dictionaries to the respective prediction. The custom metric fields are supported for all annotation types except raster segmentation.

# Include the following list of dictionaries in your prediction annotation built with python classes
custom_metrics = [
  { 'name': 'iou', 'value': 0.5 },
  { 'name': 'f1', 'value': 0.33 },
  { 'name': 'precision', 'value': 0.55 },
  { 'name': 'recall', 'value': 0.33 },
  { 'name': 'tagsCount', 'value': 43 },
  { 'name': 'metric_with_a_very_long_name', 'value': 0.334332 }
]

# Include the following list of dictionaries in your prediction annotation built with ndjson
'customMetrics': [
  { 'name': 'iou', 'value': 0.5 },
  { 'name': 'f1', 'value': 0.33 },
  { 'name': 'precision', 'value': 0.55 },
  { 'name': 'recall', 'value': 0.33 },
  { 'name': 'tagsCount', 'value': 43 },
  { 'name': 'metric_with_a_very_long_name', 'value': 0.334332 }
]

Scalar custom metrics

A ScalarMetric is a custom metric with a single scalar value. It can be uploaded at the following levels of granularity:

Data rows
Features
Nested features

from labelbox.data.annotation_types import (ScalarMetric,
                                            ScalarMetricAggregation,
                                            ConfusionMatrixMetric)

# custom metric on a data row 
data_row_metric = ScalarMetric(metric_name="iou_custom", value=0.5)

# custom metric on a feature
feature_metric = ScalarMetric(metric_name="iou_custom", feature_name="cat", value=0.5)

# custom metric on a nested feature
subclass_metric = ScalarMetric(metric_name="iou_custom",
                               feature_name="cat",
                               subclass_name="orange",
                               value=0.5)

Aggregation of custom metrics

This is an optional field on the ScalarMetric object, to control how custom metrics are aggergated. By default, the aggregation uses ARITHMETIC_MEAN.

Aggregations occur in the following cases:

When you provide a feature or nested-feature metric, Labelbox automatically aggregates the metric across features and nested-features on the data row.
For example, say you provide a custom metric Bounding Box Width (BBW) on the features "cat" and "dog" . The data row-level metric for BBW is the average of these two values.
When you create slices, the custom metric is aggregated across data rows of the Slice.
When you filter data inside a Model Run, the custom metric is aggregated across the filtered data rows.

"""
If the following metrics are uploaded then
in the Labelbox App, users will see:
true positives dog = 4
true positives cat = 3
true positives = 7
"""

feature_metric = ScalarMetric(metric_name="true_positives",
                              feature_name="cat",
                              value=3,
                              aggregation=ScalarMetricAggregation.SUM)

feature_metric = ScalarMetric(metric_name="true_positives",
                              feature_name="dog",
                              value=4,
                              aggregation=ScalarMetricAggregation.SUM)

Add labels to a model run

Adds data rows and labels to a model run. By adding labels, the associated data rows will also be upserted to the model run.

# upsert using label ids 
label_ids = ["<label_id_1>","<label_id_2>", ...]
model_run.upsert_labels(label_ids)

Alternatively, you can add all labels from a project to a Model run directly. This will also add all data rows from that project to the model run.

# upsert using project id
model_run.upsert_labels(project_id=<project_id>)