Skip to main content
The offline multimodal chat evaluation editor allows you to evaluate generative models by importing existing conversations and adding annotations to model responses. The editor supports various data types, including text, images, videos, audio, and PDFs.

​
Set up offline multimodal chat evaluation projects

The following steps walk you through how to set up an offline multimodal chat evaluation project on the Labelbox platform. To learn how to set up an offline multimodal chat evaluation project using the SDK, see Multimodal chat evaluation projects.

​
Step 1: create a project

  1. On the Annotate projects page, click the + New project button.
  2. Select Multimodal chat, and then select Offline multimodal chat.
  3. Provide a name and an optional description for your project.

​
Step 2: add data

  1. Click the Add data button to select a conversation v2 JSON dataset or create a new dataset. Alternatively, you can import data using the SDK.

​
Step 3: Set up an ontology

Create an ontology for evaluating model response, like the following example:
The editor supports the following options: Classification tasks can apply globally to the entire conversation or individually to a message. They can also nest subclassification tasks.

​
Experimental feature

Message step reasoning is an experimental feature. Currently, you can’t import step reasoning labels using the SDK.

​
Step 4: Complete annotation tasks

Click the Start labeling button to add annotations to evaluate the responses. Complete all tasks in your workflow.