Learn how to set up a multimodal chat evaluation project using the SDK.
client.create_ontology()
or client.create_ontology_from_feature_schemas()
, with the ontology_kind
parameter set to lb.OntologyKind.ModelEvaluation
. See Supported annotation types for the annotation types you can include in a multimodal chat evaluation ontology.
create_ontology
create_ontology
to create an ontology and define its schema:
create_ontology_from_feature_schemas
create_ontology_from_feature_schemas
with feature schema IDs to create ontologies that reuse existing feature schemas instead of defining new ones. To obtain these IDs, go to the Schema tab.
create_offline_model_evaluation_project
and import data rows of existing conversations.
client.create_model_evaluation_project
and either:
client.create_offline_model_evaluation_project
to create offline multimodal chat evaluation projects. This method takes the same parameters as client.create_project
: a name
and an optional description
.
client.create_model_evaluation_project
to create a live multimodal chat evaluation project. This method takes the same name
and optional description
parameters as client.create_project
, with a few additional parameters specific to multimodal chat evaluation projects:
data_row_count
(optional): The number of data rows to generate for your project. Defaults to 100 if a dataset_name
or dataset_id
is included.
dataset_name
(optional): The name of a new dataset. Include this parameter only if you want to create a new dataset for the generated data rows.
dataset_id
(optional): The dataset ID of an existing Labelbox dataset. Include this parameter only if you want to append generated data rows to an existing dataset.
client.create_model_config
to create a model configuration. This method takes the following parameters:
name
: The name of the model configuration.
model_id
: The ID of the model to configure. To get this value, go to the Model tab, select your model, and copy the ID from the URL.
inference_params
: Model configuration parameters in the JSON format. Each model has unique parameters.
project.add_model_config to attach
or project.remove_model_config
to remove model configurations. Both methods take just a model_config
ID.
project.delete_project_model_config()
or client.delete_model_config
to delete model configurations. Both methods require the model_config
ID as a parameter. You can obtain this ID in one of the following ways:
project.project_model_configs
and iterating through the list of model configurations attached to your projectproject.set_project_model_setup_complete()
to mark the setup as complete. After that, you can add more data rows to the project, but you can’t further add, modify, or delete project model configurations.
export()
method provides a unified way to export data from all project types, including multimodal chat evaluation projects. It retrieves model responses, annotations, and related metadata in a structured format. For detailed instructions on using export()
to export multimodal chat evaluation projects and sample export formats, see Export multimodal chat annotations.