Subscribe to release notesStatus pageCommunity
Developer guides

Import multimodal chat annotations

Learn how to import multimodal chat evaluation annotations and sample import formats.

Overview

To import annotations in Labelbox, you need to create an annotations payload. In this section, we provide this payload for every supported annotation type.

Annotation payload types

Labelbox supports two formats for the annotations payload:

  • Python annotation types (recommended)
    • Provides a seamless transition between third-party platforms, machine learning pipelines, and Labelbox.
    • Allows you to build annotations locally with local file paths, numpy arrays, or URLs
    • Easily convert Python Annotation Type format to NDJSON format to quickly import annotations to Labelbox
    • Supports one-level nested classification (radio, checklist, or free-form text) under a tool or classification annotation.
  • JSON
    • Skips formatting annotation payload in the Labelbox Python annotation type
    • Supports any levels of nested classification (radio, checklist, or free-form text) under a tool or classification annotation.

Label import types

Labelbox supports two types of label imports:

  • Model-assisted labeling (MAL)
    • This workflow allows you to import computer-generated predictions (or simply annotations created outside of Labelbox) as pre-labels on an asset.
  • Ground truth
    • This workflow functionality allows you to bulk import your ground truth annotations from an external or third-party labeling system into Labelbox Annotate. Using the label import API to import external data is a useful way to consolidate and migrate all annotations into Labelbox as a single source of truth.

Supported annotations

The following annotations are supported for a LLM human preference data row:

  • Tool
    • Message ranking
    • Single message selection
    • Multiple message selection
  • Classification
    • Radio (single-choice)
    • Checklist (multi-choice)
    • Free-form text

📘

Message and global-based annotations

Radio and free-form text annotations can be both message and global based. To make a message-based annotation global, remove the message_id key inside the annotation.

Tools

Message ranking

message_ranking_annotation = lb_types.MessageEvaluationTaskAnnotation(
    name="Message ranking",
    value=MessageRankingTask(
        parent_message_id="message-0",
        ranked_messages=[
            OrderedMessageInfo(
                message_id="message-1",
                model_config_name="model-config-1",
                order=1,
            ),
            OrderedMessageInfo(
                message_id="message-2",
                model_config_name="model-config-2",
                order=2,
            ),
        ],
    ),
)

message_ranking_annotation_ndjson = {
    "name": "model output multi ranking",
    "messageEvaluationTask": {
        "format": "message-ranking",
        "data": {
            "parentMessageId": "message-0",
            "rankedMessages": [
                {
                    "messageId": "message-1",
                    "modelConfigName": "model-config-1",
                    "order": 2,
                },
                {
                    "messageId": "message-2",
                    "modelConfigName": "model-config-2",
                    "order": 1,
                },
            ],
        },
    },
}

Single message selection

single_message_selection_annotation = lb_types.MessageEvaluationTaskAnnotation(
    name="Single message selection",
    value=MessageSingleSelectionTask(
        message_id="message-1",
        parent_message_id="message-0",
        model_config_name="model-config-1",
    ),
)

single_message_selection_annotation_ndjson = {
    "name": "Single message selection",
    "messageEvaluationTask": {
        "format": "message-single-selection",
        "data": {
            "messageId": "message-1",
            "parentMessageId": "message-0",
            "modelConfigName": "model-config-1",
        },
    },
}

Multiple message selection

multiple_message_selection_annotation = lb_types.MessageEvaluationTaskAnnotation(
    name="Multi message selection",
    value=MessageMultiSelectionTask(
        parent_message_id="message-0",
        selected_messages=[
            MessageInfo(
                message_id="message-1",
                model_config_name="model-config-1",
            ),
            MessageInfo(
                message_id="message-2",
                model_config_name="model-config-2",
            ),
        ],
    ),
)

multiple_message_selection_annotation_ndjson = {
    "name": "Multi message selection",
    "messageEvaluationTask": {
        "format": "message-multi-selection",
        "data": {
            "parentMessageId": "message-0",
            "selectedMessages": [
                {
                    "messageId": "message-1",
                    "modelConfigName": "model-config-1",
                },
                {
                    "messageId": "message-2",
                    "modelConfigName": "model-config-2",
                },
            ],
        },
    },
}

Classifications

Radio

radio_annotation = lb_types.ClassificationAnnotation(
    name="Choose the best response",
    value=lb_types.Radio(answer=lb_types.ClassificationAnswer(
        name="Response B")))

radio_annotation_ndjson = {
    "name": "radio_convo",
    "answer": {
        "name": "first_radio_answer"
    }
}

Checklist

checklist_annotation= lb_types.ClassificationAnnotation(
  name="checklist_convo", # must match your ontology feature"s name
  value=lb_types.Checklist(
      answer = [
        lb_types.ClassificationAnswer(
            name = "first_checklist_answer"
        ),
        lb_types.ClassificationAnswer(
            name = "second_checklist_answer"
        )
      ]
    ),
  message_id="message-1" # Message specific annotation
 )

checklist_annotation_ndjson = {
    "name": "checklist_convo",
    "answers": [
        {"name": "first_checklist_answer"},
        {"name": "second_checklist_answer"}
    ]
}

Free-form text

text_annotation = lb_types.ClassificationAnnotation(
    name="Provide a reason for your choice",
    value=lb_types.Text(answer="the answer to the text questions right here")
)

text_annotation_ndjson = {
    "name": "text_convo",
    "answer": "the answer to the text questions right here",
}

Example: Import pre-labels or ground truths

The steps to import annotations as pre-labels (machine-assisted learning) are similar to the steps to import annotations as ground truth labels, and we will describe the slight differences for each scenario.

Before you start

The below imports are needed to use the code examples in this section.

import labelbox as lb
import uuid
import labelbox.types as lb_types

from labelbox.types import (
    Label,
    MessageEvaluationTaskAnnotation,
    MessageInfo,
    MessageMultiSelectionTask,
    MessageRankingTask,
    MessageSingleSelectionTask,
    OrderedMessageInfo,
)

Step 1: Import data rows

You need to import data rows to Catalog to attach annotations.

This example shows how to create a data row in Catalog by attaching it to a dataset .

mmc_asset = "https://storage.googleapis.com/labelbox-datasets/conversational_model_evaluation_sample/offline-model-chat-evaluation.json"
global_key = "offline-multimodal_chat_evaluation"

# Upload data rows
convo_data = {
    "row_data": mmc_asset ,
    "global_key": global_key
}

# Create a dataset
dataset = client.create_dataset(name="offline-multimodal_chat_evaluation_demo")
# Create a datarow
task = dataset.create_data_rows([convo_data])
task.wait_till_done()
print("Errors:",task.errors)
print("Failed data rows:", task.failed_data_rows)

{
  "type": "application/vnd.labelbox.conversational.model-chat-evaluation",
  "version": 2,
  "actors": {
    "clxfzhair00013b6vlatdodq2": {
      "role": "human",
      "metadata": {
        "name": "User"
      }
    },
    "clxfzhbwl00033b6vbq14ufri": {
      "role": "model",
      "metadata": {
        "modelConfigName": "GPT 4 with temperature 0.7"
      }
    },
    "clxfzhbwl00043b6vgajlwovx": {
      "role": "model",
      "metadata": {
        "modelConfigName": "GPT 5"
      }
    }
  },
  "messages": {
    "clxfzhair00023b6vb607bqo6": {
      "actorId": "clxfzhair00013b6vlatdodq2",
      "content": [
        {
          "type": "text",
          "content": "The best restaurant in Wroclaw"
        }
      ],
      "childMessageIds": [
        "clxfzi3r400053b6vm5udpdgo",
        "clxfzi3r400063b6vuaeajylo"
      ]
    },
    "clxfzi3r400053b6vm5udpdgo": {
      "actorId": "clxfzhbwl00033b6vbq14ufri",
      "content": [
        {
          "type": "text",
          "content": "It's impossible to definitively say which restaurant in Wroclaw is \"the best\" because it's subjective and depends on individual preferences. However, I can provide you with a list of some of the most popular and highly-rated restaurants in Wroclaw, representing a variety of cuisines and dining experiences:\n\n**Fine Dining:**\n\n* **JaDka:** Offers modern Polish cuisine with a focus on local and seasonal ingredients. \n* **Szara Ges:** Located in the heart of the city, this restaurant is known for its elegant atmosphere and refined Polish and European dishes. \n* **Młoda Polska Bistro & Pianino:** Serves traditional Polish cuisine with a modern twist in a cozy and stylish setting.\n\n**Casual Dining:**\n\n* **Bernard:** A popular spot for traditional Polish food, known for its generous portions and affordable prices. \n* **Konspira:** Offers a unique dining experience in a former safe house from the communist era, serving Polish and European cuisine. \n* **U Gruzina:** A Georgian restaurant with delicious and authentic dishes, including khachapuri (cheese-filled bread) and khinkali (dumplings).\n\n**International Cuisine:**\n\n* **Dinette:** Serves up delicious and creative breakfast and brunch dishes all day long.\n* **Pizza Pany:** A great option for Neapolitan-style pizza with fresh ingredients and a lively atmosphere. \n* **Curry House:** Offers a wide variety of authentic Indian curries and dishes. \n\n**Vegetarian/Vegan:**\n\n* **Vega:** A vegetarian restaurant with a diverse menu featuring Polish and international dishes. \n* **Zielony Szpinak:** Offers vegan versions of traditional Polish dishes as well as international options. \n\n**For a truly unique experience:**\n\n* **Wrocławskie Podwale:** Dine in a medieval cellar with vaulted ceilings and enjoy traditional Polish cuisine. \n* **Stary Młyn:** Located in a former mill, this restaurant offers a romantic atmosphere and serves Polish and European dishes. \n\nTo narrow down your search, consider what kind of cuisine you're in the mood for, your budget, and the occasion. I recommend checking online reviews and making a reservation, especially if you're dining during peak hours. \n"
        }
      ],
      "childMessageIds": ["clxfznjb800073b6v43ppx9ca"]
    },
    "clxfzi3r400063b6vuaeajylo": {
      "actorId": "clxfzhbwl00043b6vgajlwovx",
      "content": [
        {
          "type": "text",
          "content": "There are several highly-rated restaurants in Wroclaw, but one of the best is \"Szynkarnia\". It's known for its delicious Polish cuisine, cozy atmosphere, and excellent service. Other top contenders include \"The White Dog\" known for its modern European cuisine, and \"Restauracja JaDka\" for those who want to try traditional Polish dishes."
        }
      ],
      "childMessageIds": ["clxfznjb800073b6v43ppx9ca"]
    },
    "clxfznjb800073b6v43ppx9ca": {
      "actorId": "clxfzhair00013b6vlatdodq2",
      "content": [
        {
          "type": "text",
          "content": "Tell me more"
        }
      ],
      "childMessageIds": [
        "clxfzocbm00083b6v8vczsept",
        "clxfzocbm00093b6vx4ndisub"
      ]
    },
    "clxfzocbm00083b6v8vczsept": {
      "actorId": "clxfzhbwl00033b6vbq14ufri",
      "content": [
        {
          "type": "text",
          "content": "Alright, tell me more about what you're looking for in a restaurant! To give you a more personalized recommendation, I need some more information.  Could you tell me:\n\n* **What kind of food are you craving?** (e.g., Italian, Polish, Asian, seafood, etc.) \n* **What's your budget?** (e.g., fine dining, mid-range, budget-friendly)\n* **What's the occasion?** (e.g., romantic dinner, family meal, night out with friends)\n* **What kind of atmosphere are you looking for?** (e.g., lively, cozy, romantic, trendy)\n* **Are there any dietary restrictions or preferences I should know about?** (e.g., vegetarian, vegan, gluten-free)\n\nThe more information you can give me, the better I can tailor my recommendation to your specific needs and preferences. \n"
        }
      ],
      "childMessageIds": []
    },
    "clxfzocbm00093b6vx4ndisub": {
      "actorId": "clxfzhbwl00043b6vgajlwovx",
      "content": [
        {
          "type": "text",
          "content": "Sure, here's more information about these restaurants:\n\n1. Szynkarnia: This is a traditional Polish restaurant located in the heart of Wroclaw. It offers a variety of Polish dishes, including pierogi (dumplings), bigos (hunter's stew), and kielbasa (sausage). The restaurant is also known for its wide selection of local beers. The interior is rustic and cozy, with wooden furniture and warm lighting.\n\n2. The White Dog: This restaurant offers a modern take on European cuisine. The menu changes seasonally to take advantage of fresh, local ingredients. The White Dog also has an extensive wine list and a stylish, contemporary interior. It's a great place for a romantic dinner or a special occasion.\n\n3. Restauracja JaDka: This restaurant is located in a beautiful, historic building in Wroclaw. It specializes in Polish cuisine, with dishes like duck with apples, beetroot soup, and a variety of pierogi. The interior is elegant and atmospheric, with exposed brick walls and vintage furniture.\n\nRemember, it's always a good idea to make a reservation in advance, as these popular spots can fill up quickly."
        }
      ],
      "childMessageIds": []
    }
  },
  "rootMessageIds": ["clxfzhair00023b6vb607bqo6"]
}

Step 2: Set up ontology

Your project ontology needs to support the classifications required by your annotations. To ensure accurate schema feature mapping, the value used as the name parameter needs to match the value of the name field in your annotation.

For example, if you provide a name annotation_name for your message ranking annotation, you need to name the message ranking tool as anotations_name when setting up your ontology. The same alignment must hold true for the other tools and classifications that you create in the ontology.

This example shows how to create an ontology containing all supported annotation types .

ontology_builder = lb.OntologyBuilder(
    tools=[
        lb.Tool(
            tool=lb.Tool.Type.MESSAGE_SINGLE_SELECTION,
            name="Single message selection",
        ),
        lb.Tool(
            tool=lb.Tool.Type.MESSAGE_MULTI_SELECTION,
            name="Multi message selection",
        ),
        lb.Tool(tool=lb.Tool.Type.MESSAGE_RANKING, name="Message ranking"),
    ],
  classifications=[
    lb.Classification(
      class_type=lb.Classification.Type.RADIO,
      scope=lb.Classification.Scope.GLOBAL,
      name="Choose the best response",
      options=[lb.Option(value="Response A"), lb.Option(value="Response B"), lb.Option(value="Tie")]
    ),
    lb.Classification(
      class_type=lb.Classification.Type.TEXT,
      name="Provide a reason for your choice"
    ),
    lb.Classification(
      class_type=lb.Classification.Type.CHECKLIST,
      scope=lb.Classification.Scope.INDEX,
      name="checklist_convo",
      options=[
        lb.Option(value="first_checklist_answer"),
        lb.Option(value="second_checklist_answer")
      ]
    )
  ]
)
# Create ontology
ontology = client.create_ontology(
    "MMC ontology",
    ontology_builder.asdict(),
    media_type=lb.MediaType.Conversational,
    ontology_kind=lb.OntologyKind.ModelEvaluation,
)

Step 3: Set up a labeling project

Use the following code to create an offline multimodal evaluation project:

# Create Labelbox project
project = client.create_offline_model_evaluation_project(
    name="Offline MMC Import Demo",
    description="<project_description>",  # optional
    media_type=lb.MediaType.Conversational,
)

# Setup your ontology
project.connect_ontology(ontology) # Connect the ontology to your project

Step 4: Send data rows to project

Use the following code to send data rows to the project you just created:

# Create a batch to send to your project
batch = project.create_batch(
  "first-batch-convo-demo", # Each batch in a project must have a unique name
  global_keys=[global_key], # Paginated collection of data row objects, list of data row ids or global keys
  priority=5 # priority between 1(Highest) - 5(lowest)
)

print("Batch: ", batch)

Step 5: Create annotation payloads

To declare payloads, you can use Python annotation types (preferred) or NDJSON objects. To understand annotation payloads, see overview.

These examples demonstrate each format and how to compose annotations into labels attached to data rows.

📘

Replace placeholder fields with actual values

Replace message_id and model_config_name with the actual message ID and model configuration name before appending annotations.

label = []
label.append(
  lb_types.Label(
    data={"global_key" : global_key },
    annotations=[
      message_ranking_annotation,
      single_message_selection_annotation,
      multiple_message_selection_annotation,
      text_annotation,
      checklist_annotation,
      radio_annotation,
    ]
  )
)

label_ndjson = []
for annotations in [
    message_ranking_annotation_ndjson,
    single_message_selection_annotation_ndjson,
    multiple_message_selection_annotation_ndjson,
    text_annotation_ndjson,
    checklist_annotation_ndjson,
    radio_annotation_ndjson,
    nested_checklist_annotation_ndjson,
    nested_radio_annotation_ndjson
    ]:
  annotations.update({
      "dataRow": {
          "globalKey": global_key
      }
  })
  label_ndjson.append(annotations)

Step 6: Import annotation payload

For prelabeled (model-assisted labeling) scenarios, pass your payload as the value of the predictions parameter. For ground truths, pass the payload to the labels parameter.

Option A: import as prelabels (model assisted labeling)

This option is helpful for speeding up the initial labeling process and reducing the manual labeling workload for high-volume datasets.

# Upload MAL label for this data row in project
upload_job = lb.MALPredictionImport.create_from_objects(
    client = client, 
    project_id = project.uid, 
    name="mal_job"+str(uuid.uuid4()), 
    predictions=label
)

print(f"Errors: {upload_job.errors}", )
print(f"Status of uploads: {upload_job.statuses}"

Option B: Import as ground truth labels

This option is helpful for loading high-confidence labels from another platform or previous projects that just need review rather than manual labeling effort.

# Upload label for this data row in project
upload_job = lb.LabelImport.create_from_objects(
    client = client, 
    project_id = project.uid, 
    name="label_import_job"+str(uuid.uuid4()),  
    labels=label
)

print(f"Errors: {upload_job.errors}", )
print(f"Status of uploads: {upload_job.statuses}")