> ## Documentation Index
> Fetch the complete documentation index at: https://docs.labelbox.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Import multimodal chat annotations
> Learn how to import multimodal chat evaluation annotations and sample import formats.
## Overview
To import annotations in Labelbox, you need to create an annotations payload. In this section, we provide this payload for every supported annotation type.
### Annotation payload types
Labelbox supports two formats for the annotations payload:
* Python annotation types (recommended)
* Provides a seamless transition between third-party platforms, machine learning pipelines, and Labelbox.
* Allows you to build annotations locally with local file paths, numpy arrays, or URLs
* Easily convert Python Annotation Type format to NDJSON format to quickly import annotations to Labelbox
* Supports one-level nested classification (radio, checklist, or free-form text) under a tool or classification annotation.
* JSON
* Skips formatting annotation payload in the Labelbox Python annotation type
* Supports any levels of nested classification (radio, checklist, or free-form text) under a tool or classification annotation.
### Label import types
Labelbox supports two types of label imports:
* [Model-assisted labeling (MAL)](/docs/model-assisted-labeling)
* This workflow allows you to import computer-generated predictions (or simply annotations created outside of Labelbox) as pre-labels on an asset.
* [Ground truth](/docs/import-ground-truth)
* This workflow functionality allows you to bulk import your ground truth annotations from an external or third-party labeling system into Labelbox *Annotate*. Using the label import API to import external data is a useful way to consolidate and migrate all annotations into Labelbox as a single source of truth.
## Supported annotations
The following annotations are supported for a LLM human preference data row:
* `Tool`
* Message ranking
* Single message selection
* Multiple message selection
* `Classification`
* Radio (single-choice)
* Checklist (multi-choice)
* Free-form text
### Message and global-based annotations
Radio and free-form text annotations can be both message and global based. To make a message-based annotation global, remove the `message_id` key inside the annotation.
## Tools
### Message ranking
```python Python Annotation theme={null}
message_ranking_annotation = lb_types.MessageEvaluationTaskAnnotation(
name="Message ranking",
value=MessageRankingTask(
parent_message_id="message-0",
ranked_messages=[
OrderedMessageInfo(
message_id="message-1",
model_config_name="model-config-1",
order=1,
),
OrderedMessageInfo(
message_id="message-2",
model_config_name="model-config-2",
order=2,
),
],
),
)
```
```json NDJSON theme={null}
message_ranking_annotation_ndjson = {
"name": "model output multi ranking",
"messageEvaluationTask": {
"format": "message-ranking",
"data": {
"parentMessageId": "message-0",
"rankedMessages": [
{
"messageId": "message-1",
"modelConfigName": "model-config-1",
"order": 2,
},
{
"messageId": "message-2",
"modelConfigName": "model-config-2",
"order": 1,
},
],
},
},
}
```
### Single message selection
```python Python Annotation theme={null}
single_message_selection_annotation = lb_types.MessageEvaluationTaskAnnotation(
name="Single message selection",
value=MessageSingleSelectionTask(
message_id="message-1",
parent_message_id="message-0",
model_config_name="model-config-1",
),
)
```
```json NDJSON theme={null}
single_message_selection_annotation_ndjson = {
"name": "Single message selection",
"messageEvaluationTask": {
"format": "message-single-selection",
"data": {
"messageId": "message-1",
"parentMessageId": "message-0",
"modelConfigName": "model-config-1",
},
},
}
```
### Multiple message selection
```python Python Annotation theme={null}
multiple_message_selection_annotation = lb_types.MessageEvaluationTaskAnnotation(
name="Multi message selection",
value=MessageMultiSelectionTask(
parent_message_id="message-0",
selected_messages=[
MessageInfo(
message_id="message-1",
model_config_name="model-config-1",
),
MessageInfo(
message_id="message-2",
model_config_name="model-config-2",
),
],
),
)
```
```json NDJSON theme={null}
multiple_message_selection_annotation_ndjson = {
"name": "Multi message selection",
"messageEvaluationTask": {
"format": "message-multi-selection",
"data": {
"parentMessageId": "message-0",
"selectedMessages": [
{
"messageId": "message-1",
"modelConfigName": "model-config-1",
},
{
"messageId": "message-2",
"modelConfigName": "model-config-2",
},
],
},
},
}
```
## Classifications
### Radio
```python Python Annotation theme={null}
radio_annotation = lb_types.ClassificationAnnotation(
name="Choose the best response",
value=lb_types.Radio(answer=lb_types.ClassificationAnswer(
name="Response B")))
```
```json NDJSON theme={null}
radio_annotation_ndjson = {
"name": "radio_convo",
"answer": {
"name": "first_radio_answer"
}
}
```
### Checklist
```python Python Annotation theme={null}
checklist_annotation= lb_types.ClassificationAnnotation(
name="checklist_convo", # must match your ontology feature"s name
value=lb_types.Checklist(
answer = [
lb_types.ClassificationAnswer(
name = "first_checklist_answer"
),
lb_types.ClassificationAnswer(
name = "second_checklist_answer"
)
]
),
message_id="message-1" # Message specific annotation
)
```
```json NDJSON theme={null}
checklist_annotation_ndjson = {
"name": "checklist_convo",
"answers": [
{"name": "first_checklist_answer"},
{"name": "second_checklist_answer"}
]
}
```
### Free-form text
```python Python Annotation theme={null}
text_annotation = lb_types.ClassificationAnnotation(
name="Provide a reason for your choice",
value=lb_types.Text(answer="the answer to the text questions right here")
)
```
```json NDJSON theme={null}
text_annotation_ndjson = {
"name": "text_convo",
"answer": "the answer to the text questions right here",
}
```
## Example: Import pre-labels or ground truths
The steps to import annotations as pre-labels (machine-assisted learning) are similar to the steps to import annotations as ground truth labels, and we will describe the slight differences for each scenario.
### Before you start
The below imports are needed to use the code examples in this section.
```python Python theme={null}
import labelbox as lb
import uuid
import labelbox.types as lb_types
from labelbox.types import (
Label,
MessageEvaluationTaskAnnotation,
MessageInfo,
MessageMultiSelectionTask,
MessageRankingTask,
MessageSingleSelectionTask,
OrderedMessageInfo,
)
```
### Step 1: Import data rows
You need to import data rows to **Catalog** to attach annotations.
This example shows how to create a data row in **Catalog** by attaching it to a [dataset](/reference/dataset) .
```python Python theme={null}
mmc_asset = "https://storage.googleapis.com/labelbox-datasets/conversational_model_evaluation_sample/offline-model-chat-evaluation.json"
global_key = "offline-multimodal_chat_evaluation"
# Upload data rows
convo_data = {
"row_data": mmc_asset ,
"global_key": global_key
}
# Create a dataset
dataset = client.create_dataset(name="offline-multimodal_chat_evaluation_demo")
# Create a datarow
task = dataset.create_data_rows([convo_data])
task.wait_till_done()
print("Errors:",task.errors)
print("Failed data rows:", task.failed_data_rows)
```
```json Sample import theme={null}
{
"type": "application/vnd.labelbox.conversational.model-chat-evaluation",
"version": 2,
"actors": {
"clxfzhair00013b6vlatdodq2": {
"role": "human",
"metadata": {
"name": "User"
}
},
"clxfzhbwl00033b6vbq14ufri": {
"role": "model",
"metadata": {
"modelConfigName": "GPT 4 with temperature 0.7"
}
},
"clxfzhbwl00043b6vgajlwovx": {
"role": "model",
"metadata": {
"modelConfigName": "GPT 5"
}
}
},
"messages": {
"clxfzhair00023b6vb607bqo6": {
"actorId": "clxfzhair00013b6vlatdodq2",
"content": [
{
"type": "text",
"content": "The best restaurant in Wroclaw"
}
],
"childMessageIds": [
"clxfzi3r400053b6vm5udpdgo",
"clxfzi3r400063b6vuaeajylo"
]
},
"clxfzi3r400053b6vm5udpdgo": {
"actorId": "clxfzhbwl00033b6vbq14ufri",
"content": [
{
"type": "text",
"content": "It's impossible to definitively say which restaurant in Wroclaw is \"the best\" because it's subjective and depends on individual preferences. However, I can provide you with a list of some of the most popular and highly-rated restaurants in Wroclaw, representing a variety of cuisines and dining experiences:\n\n**Fine Dining:**\n\n* **JaDka:** Offers modern Polish cuisine with a focus on local and seasonal ingredients. \n* **Szara Ges:** Located in the heart of the city, this restaurant is known for its elegant atmosphere and refined Polish and European dishes. \n* **Młoda Polska Bistro & Pianino:** Serves traditional Polish cuisine with a modern twist in a cozy and stylish setting.\n\n**Casual Dining:**\n\n* **Bernard:** A popular spot for traditional Polish food, known for its generous portions and affordable prices. \n* **Konspira:** Offers a unique dining experience in a former safe house from the communist era, serving Polish and European cuisine. \n* **U Gruzina:** A Georgian restaurant with delicious and authentic dishes, including khachapuri (cheese-filled bread) and khinkali (dumplings).\n\n**International Cuisine:**\n\n* **Dinette:** Serves up delicious and creative breakfast and brunch dishes all day long.\n* **Pizza Pany:** A great option for Neapolitan-style pizza with fresh ingredients and a lively atmosphere. \n* **Curry House:** Offers a wide variety of authentic Indian curries and dishes. \n\n**Vegetarian/Vegan:**\n\n* **Vega:** A vegetarian restaurant with a diverse menu featuring Polish and international dishes. \n* **Zielony Szpinak:** Offers vegan versions of traditional Polish dishes as well as international options. \n\n**For a truly unique experience:**\n\n* **Wrocławskie Podwale:** Dine in a medieval cellar with vaulted ceilings and enjoy traditional Polish cuisine. \n* **Stary Młyn:** Located in a former mill, this restaurant offers a romantic atmosphere and serves Polish and European dishes. \n\nTo narrow down your search, consider what kind of cuisine you're in the mood for, your budget, and the occasion. I recommend checking online reviews and making a reservation, especially if you're dining during peak hours. \n"
}
],
"childMessageIds": ["clxfznjb800073b6v43ppx9ca"]
},
"clxfzi3r400063b6vuaeajylo": {
"actorId": "clxfzhbwl00043b6vgajlwovx",
"content": [
{
"type": "text",
"content": "There are several highly-rated restaurants in Wroclaw, but one of the best is \"Szynkarnia\". It's known for its delicious Polish cuisine, cozy atmosphere, and excellent service. Other top contenders include \"The White Dog\" known for its modern European cuisine, and \"Restauracja JaDka\" for those who want to try traditional Polish dishes."
}
],
"childMessageIds": ["clxfznjb800073b6v43ppx9ca"]
},
"clxfznjb800073b6v43ppx9ca": {
"actorId": "clxfzhair00013b6vlatdodq2",
"content": [
{
"type": "text",
"content": "Tell me more"
}
],
"childMessageIds": [
"clxfzocbm00083b6v8vczsept",
"clxfzocbm00093b6vx4ndisub"
]
},
"clxfzocbm00083b6v8vczsept": {
"actorId": "clxfzhbwl00033b6vbq14ufri",
"content": [
{
"type": "text",
"content": "Alright, tell me more about what you're looking for in a restaurant! To give you a more personalized recommendation, I need some more information. Could you tell me:\n\n* **What kind of food are you craving?** (e.g., Italian, Polish, Asian, seafood, etc.) \n* **What's your budget?** (e.g., fine dining, mid-range, budget-friendly)\n* **What's the occasion?** (e.g., romantic dinner, family meal, night out with friends)\n* **What kind of atmosphere are you looking for?** (e.g., lively, cozy, romantic, trendy)\n* **Are there any dietary restrictions or preferences I should know about?** (e.g., vegetarian, vegan, gluten-free)\n\nThe more information you can give me, the better I can tailor my recommendation to your specific needs and preferences. \n"
}
],
"childMessageIds": []
},
"clxfzocbm00093b6vx4ndisub": {
"actorId": "clxfzhbwl00043b6vgajlwovx",
"content": [
{
"type": "text",
"content": "Sure, here's more information about these restaurants:\n\n1. Szynkarnia: This is a traditional Polish restaurant located in the heart of Wroclaw. It offers a variety of Polish dishes, including pierogi (dumplings), bigos (hunter's stew), and kielbasa (sausage). The restaurant is also known for its wide selection of local beers. The interior is rustic and cozy, with wooden furniture and warm lighting.\n\n2. The White Dog: This restaurant offers a modern take on European cuisine. The menu changes seasonally to take advantage of fresh, local ingredients. The White Dog also has an extensive wine list and a stylish, contemporary interior. It's a great place for a romantic dinner or a special occasion.\n\n3. Restauracja JaDka: This restaurant is located in a beautiful, historic building in Wroclaw. It specializes in Polish cuisine, with dishes like duck with apples, beetroot soup, and a variety of pierogi. The interior is elegant and atmospheric, with exposed brick walls and vintage furniture.\n\nRemember, it's always a good idea to make a reservation in advance, as these popular spots can fill up quickly."
}
],
"childMessageIds": []
}
},
"rootMessageIds": ["clxfzhair00023b6vb607bqo6"]
}
```
### Step 2: Set up ontology
Your project ontology needs to support the classifications required by your annotations. To ensure accurate schema feature mapping, the value used as the name parameter needs to match the value of the name field in your annotation.
For example, if you provide a name `annotation_name` for your message ranking annotation, you need to name the message ranking tool as `anotations_name` when setting up your ontology. The same alignment must hold true for the other tools and classifications that you create in the ontology.
This example shows how to create an ontology containing all supported [annotation types](#supported-annotations) .
```python Python theme={null}
ontology_builder = lb.OntologyBuilder(
tools=[
lb.Tool(
tool=lb.Tool.Type.MESSAGE_SINGLE_SELECTION,
name="Single message selection",
),
lb.Tool(
tool=lb.Tool.Type.MESSAGE_MULTI_SELECTION,
name="Multi message selection",
),
lb.Tool(tool=lb.Tool.Type.MESSAGE_RANKING, name="Message ranking"),
],
classifications=[
lb.Classification(
class_type=lb.Classification.Type.RADIO,
scope=lb.Classification.Scope.GLOBAL,
name="Choose the best response",
options=[lb.Option(value="Response A"), lb.Option(value="Response B"), lb.Option(value="Tie")]
),
lb.Classification(
class_type=lb.Classification.Type.TEXT,
name="Provide a reason for your choice"
),
lb.Classification(
class_type=lb.Classification.Type.CHECKLIST,
scope=lb.Classification.Scope.INDEX,
name="checklist_convo",
options=[
lb.Option(value="first_checklist_answer"),
lb.Option(value="second_checklist_answer")
]
)
]
)
# Create ontology
ontology = client.create_ontology(
"MMC ontology",
ontology_builder.asdict(),
media_type=lb.MediaType.Conversational,
ontology_kind=lb.OntologyKind.ModelEvaluation,
)
```
### Step 3: Set up a labeling project
Use the following code to create an offline multimodal evaluation project:
```python Python theme={null}
# Create Labelbox project
project = client.create_offline_model_evaluation_project(
name="Offline MMC Import Demo",
description="", # optional
)
# Setup your ontology
project.connect_ontology(ontology) # Connect the ontology to your project
```
### Step 4: Send data rows to the project
Use the following code to send data rows to the project you just created:
```python Python theme={null}
# Create a batch to send to your project
batch = project.create_batch(
"first-batch-convo-demo", # Each batch in a project must have a unique name
global_keys=[global_key], # Paginated collection of data row objects, list of data row ids or global keys
priority=5 # priority between 1(Highest) - 5(lowest)
)
print("Batch: ", batch)
```
### Step 5: Create annotation payloads
To declare payloads, you can use Python annotation types (*preferred*) or NDJSON objects. To understand annotation payloads, see [overview](#overview).
These examples demonstrate each format and how to compose annotations into labels attached to data rows.
### Replace placeholder fields with actual values
Replace `message_id` and `model_config_name` with the actual message ID and model configuration name before appending annotations.
```python Python annotation theme={null}
label = []
label.append(
lb_types.Label(
data={"global_key" : global_key },
annotations=[
message_ranking_annotation,
single_message_selection_annotation,
multiple_message_selection_annotation,
text_annotation,
checklist_annotation,
radio_annotation,
]
)
)
```
```python NDJSON theme={null}
label_ndjson = []
for annotations in [
message_ranking_annotation_ndjson,
single_message_selection_annotation_ndjson,
multiple_message_selection_annotation_ndjson,
text_annotation_ndjson,
checklist_annotation_ndjson,
radio_annotation_ndjson,
nested_checklist_annotation_ndjson,
nested_radio_annotation_ndjson
]:
annotations.update({
"dataRow": {
"globalKey": global_key
}
})
label_ndjson.append(annotations)
```
### Step 6: Import annotation payload
For prelabeled (model-assisted labeling) scenarios, pass your payload as the value of the `predictions` parameter. For ground truths, pass the payload to the `labels` parameter.
#### Option A: import as [prelabels (model assisted labeling)](/docs/model-assisted-labeling)
This option is helpful for speeding up the initial labeling process and reducing the manual labeling workload for high-volume datasets.
```python MAL import theme={null}
# Upload MAL label for this data row in project
upload_job = lb.MALPredictionImport.create_from_objects(
client = client,
project_id = project.uid,
name="mal_job"+str(uuid.uuid4()),
predictions=label
)
print(f"Errors: {upload_job.errors}", )
print(f"Status of uploads: {upload_job.statuses}")
```
#### Option B: Import as [ground truth](/docs/import-ground-truth) labels
This option is helpful for loading high-confidence labels from another platform or previous projects that just need review rather than manual labeling effort.
```python Label import theme={null}
# Upload label for this data row in project
upload_job = lb.LabelImport.create_from_objects(
client = client,
project_id = project.uid,
name="label_import_job"+str(uuid.uuid4()),
labels=label
)
print(f"Errors: {upload_job.errors}", )
print(f"Status of uploads: {upload_job.statuses}")
```