Model
An experiment is a container in Labelbox that houses all of the information related to the iterative development of a specific model. It contains the data rows for training, model error analysis metrics, model versioning, and versioned snapshots (model runs) of data rows, predictions, etc, associated with a model’s development.
Experiments are designed to help you track and compare all of the iterations associated with your model development.
Create a model
model = client.create_model(
name="<model_name>",
ontology_id="<ontology_id>"
)
Get a model
model = client.get_model("<model_id>")
# from labelbox import Model
models = client.get_models(where=(Model.name == "<model_name>"))
Methods
Create a model run
model.create_model_run(name="<model_run_name>"
# optionally, you can supply details of the model run config
model.create_model_run(
name="<model_run_with_config>",
config={
"learning_rate": 0.001,
"batch_size": 32
}
)
Delete a model, and its model runs
Deleting a model also deletes its model runs.
This action is permanent; it cannot be undone or rolled back.
Attributes
Get the basics
Get the model runs
# get the model runs (relationship to ModelRun objects)
model_runs = model.model_runs()
# inspect one model run
next(model_runs)
# inspect all model runs
for model_run in model_runs:
print(model_run)
Model run
A model run represents a single iteration within a model training experiment. Each model run contains a versioned snapshot of the data rows, annotations (predictions and/or ground truth), and data splits for each iteration within a model training experiment.
Model runs make it easy for you to reproduce a model training experiment using different parameters. You can also use model runs to track and compare model runs trained on different data versions.
Get all model runs inside a Model
model_runs = model.model_runs()
Create a model run
Creates a model run belonging to this model.
model_run_name = "<your_model_run_name>"
example_config = {
"learning_rate": 0.001,
"batch_size": 32,
}
model_run = model.create_model_run(name=model_run_name, config=example_config)
Get model run
model_run_id = "<your_model_run_id>"
model_run = client.get_model_run(model_run_id=model_run_id)
model_run_data = model_run.model_run_data_rows()
model_run_config = model_run.get_config())
Add data rows to a model run
Add data rows to a model run without any associated labels. You can use either data_row_id or global_key to specify the data rows.
# Using data row ids
data_row_ids = ["<data_row_id_1>", "<data_row_id_2>"]
model_run.upsert_data_rows(data_row_ids=data_row_ids)
# Using global keys
global_keys = ["<global_key1>", "<global_key_2>"]
model_run.upsert_data_rows(global_keys=global_keys)
Assign data row training, validation, and test split
Note that assign_data_rows_to_split only works on data rows or labels that are already in a model run. You can assign them to one of “TRAINING”, “VALIDATION”, “TEST” split.
client.enable_experimental=True
dataset = client.get_dataset("<Dataset_id>") # Your training dataset
# using data row ids
model_run.assign_data_rows_to_split(
data_row_ids=data_row_ids[:100],
split="TRAINING",
)
model_run.assign_data_rows_to_split(
data_row_ids=data_row_ids[100:150],
split="VALIDATION",
)
model_run.assign_data_rows_to_split(
data_row_ids=data_row_ids[150:200],
split="TEST",
)
# using global keys
model_run.assign_data_rows_to_split(
global_keys=global_keys[:100],
split="TRAINING",
)
model_run.assign_data_rows_to_split(
global_keys=global_keys[100:150],
split="VALIDATION",
)
model_run.assign_data_rows_to_split(
global_keys=global_keys[150:200],
split="TEST",
)
Add labels to a model run
Adds data rows and labels to a model run. By adding labels, the associated data rows will also be upserted to the model run.
# upsert using label ids
label_ids = ["<label_id_1>","<label_id_2>", ...]
model_run.upsert_labels(label_ids)
Alternatively, you can add all labels from a project to a Model run directly. This will also add all data rows from that project to the model run.
# upsert using project id
model_run.upsert_labels(project_id=<project_id>)
Export labels from a model run
See Export for Model Runs for more details and export JSON format.
# Set the export params to include/exclude certain fields. Make sure each of these fields are correctly grabbed
export_params= {
"attachments": True,
"metadata_fields": True,
"data_row_details": True,
}
export_task = model_run.export(params=export_params)
export_task.wait_till_done()
# Stream the export using a callback function
def json_stream_handler(output: labelbox.BufferedJsonConverterOutput):
print(output.json)
export_task.get_buffered_stream(stream_type=labelbox.StreamType.RESULT).start(stream_handler=json_stream_handler)
# Collect all exported data into a list
export_json = [data_row.json for data_row in export_task.get_buffered_stream()]
Create, modify, and delete model run config to track hyperparameters.
example_config = {
"learning_rate": 0.001,
"checkpoint_path": "/path/to/checkpoint/file",
"early_stopping": False,
"batch_size": 32,
"optimizer": {
"adam": {
"beta1": 0.899999976158,
"beta2": 0.999000012875,
"epsilon": 9.99999993923e-9
}
},
"ngpu": 1,
}
model_run_1 = model.create_model_run(name="run 1", config=example_config)
# You can also create a model with config specified, see above.
# Here is how to create a model run first and update the model config field.
model_run_2 = model.create_model_run(name="run 2")
#The update will repace the previous model run config with the new json input.
model_run_2.update_config(example_config)
Get model run config
model_run_parameters = model_run.get_config()
Delete the model run config
Delete data rows from a model run
data_row_ids = ["<data_row_id_1>","<data_row_id_2>", ...]
model_run.delete_model_run_data_rows(data_row_ids=data_row_ids)
Delete model run
