Initialize the SDK
Import the Labelbox Python SDK:API key and client
Provide a valid API key to authenticate the Labelbox client. See Labelbox API keys to learn how to generate your key.Create a multimodal chat ontology
You can create ontologies for multimodal chat projects using eitherclient.create_ontology()
or client.create_ontology_from_feature_schemas()
, with the ontology_kind
parameter set to lb.OntologyKind.ModelEvaluation
. See Supported annotation types for the annotation types you can include in a multimodal chat evaluation ontology.
Option A: create_ontology
Use create_ontology
to create an ontology and define its schema:
Option B: create_ontology_from_feature_schemas
Use create_ontology_from_feature_schemas
with feature schema IDs to create ontologies that reuse existing feature schemas instead of defining new ones. To obtain these IDs, go to the Schema tab.
Create multimodal chat evaluation projects
The two types of multimodal chat evaluation projects have different project creation methods and data row setups:-
For offline multimodal chat evaluation projects, use
create_offline_model_evaluation_project
and import data rows of existing conversations. -
For live multimodal chat evaluation projects, use
client.create_model_evaluation_project
and either:- (Recommended) Create data rows and send them to projects, like other types of projects.
- Generate empty data rows upon project creation, which can’t create data rows with attachments and metadata.
Set up offline multimodal chat evaluation projects
Useclient.create_offline_model_evaluation_project
to create offline multimodal chat evaluation projects. This method takes the same parameters as client.create_project
: a name
and an optional description
.
Set up live multimodal chat evaluation projects
Useclient.create_model_evaluation_project
to create a live multimodal chat evaluation project. This method takes the same name
and optional description
parameters as client.create_project
, with a few additional parameters specific to multimodal chat evaluation projects:
-
data_row_count
(optional): The number of data rows to generate for your project. Defaults to 100 if adataset_name
ordataset_id
is included. -
dataset_name
(optional): The name of a new dataset. Include this parameter only if you want to create a new dataset for the generated data rows. -
dataset_id
(optional): The dataset ID of an existing Labelbox dataset. Include this parameter only if you want to append generated data rows to an existing dataset.
Option A: Create and send data rows to projects
Option B: Generate empty data rows
No metadata support
Only use this option if your project doesn’t require metadata attachments or embeddings.Set up model configurations
You can create, delete, attach, and remove model configurations that evaluate your live multimodal chat responses.Create model configs
Useclient.create_model_config
to create a model configuration. This method takes the following parameters:
-
name
: The name of the model configuration. -
model_id
: The ID of the model to configure. To get this value, go to the Model tab, select your model, and copy the ID from the URL. -
inference_params
: Model configuration parameters in the JSON format. Each model has unique parameters.
First-time setup
If you’re setting up model configurations for the first time, it’s recommended to set one up using the web platform instead of the SDK to understand all the associated parameters.Attach model configurations to projects
Useproject.add_model_config to attach
or project.remove_model_config
to remove model configurations. Both methods take just a model_config
ID.
Delete model configurations
Useproject.delete_project_model_config()
or client.delete_model_config
to delete model configurations. Both methods require the model_config
ID as a parameter. You can obtain this ID in one of the following ways:
- From the response when you create a model configuration
- By accessing
project.project_model_configs
and iterating through the list of model configurations attached to your project
Mark project setup as complete
Once you’ve completed your project setup and model configuration, useproject.set_project_model_setup_complete()
to mark the setup as complete. After that, you can add more data rows to the project, but you can’t further add, modify, or delete project model configurations.
Export multimodal chat evaluation projects
Theexport()
method provides a unified way to export data from all project types, including multimodal chat evaluation projects. It retrieves model responses, annotations, and related metadata in a structured format. For detailed instructions on using export()
to export multimodal chat evaluation projects and sample export formats, see Export multimodal chat annotations.