Labeling editors
A summary of labeling editor controls, enhancements, and supported annotations.
Labelbox provides several native editor interfaces, each designed for specific use cases. Explore the links below to learn which editor best fits your use case.
- Computer vision: Image editor | Video editor | Documents editor | Geospatial editor
- Language: Text editor | Conversational text editor | Audio editor | HTML editor
- Large language models: LLM human preference editor | Prompt and response generation
- Model evaluation: Multimodal chat evaluation
Editor interface
Labelbox automatically adjusts the editor interface based on the asset type selected for the project.
However, there are some global editor settings and enhancements shared across all of the editor interfaces. These global editor settings are designed to provide an optimal labeling experience for your team through increased customization and quick accessibility.
Additionally, there are some data enrichment capabilities and supplementary tools available for use in the editor only with certain data modalities.
Features such as attachments, instructions, and the data row information panel are available in the editor regardless of the type of asset being labeled.
Attachments
Attachments can be used to provide supplementary content to any asset to help provide additional context for the labeling team. An attachment applies to an individual asset and may comprise an image, video, text, or HTML content. Markdown rendering is supported for attachments, allowing you to use to format text, add links, and include images within attachments. Multiple attachments can be linked to an singular data row.
Supported attachment types
Below are the types of attachment data that we support in the editor.
| Type | Value | Description |
|---|---|---|
IMAGE | URL of Image (PNG/JPG) (HTTPS or IAM delegated access path) | Labelers can see the attached image(s) while labeling the primary data row |
VIDEO | URL of Video (MP4) (HTTPS or IAM delegated access path) | Labelers can see the attached video(s) while labeling the primary data row |
RAW_TEXT | Text string or hyperlink (see below for cloud-hosted text files) | Labelers can see the attached text or hyperlink while labeling the primary data row If passing a hyperlink, we will show a clickable hyperlink. If you want to display text from a URL endpoint, please use the TEXT_URL type. |
TEXT_URL | URL of a text file (HTTPS or IAM delegated access path) | Labelers can see the attached text from the linked text URL while labeling the primary data row |
HTML | URL of an HTML file (HTTPS or IAM delegated access path) | Renders HTML in an iframe as an attachment. Labelers can see and interact with the attached HTML widget while labeling the primary data row. |
IMAGE_OVERLAY | URL of the image layer (PNG/JPG) (HTTPS or IAM delegated access path). | A visualization tool designed to help you view the asset to be labeled in different ways by adding image layers over the asset. |
PDF_URL | URL of the PDF file (HTTPS or IAM delegated access path) | Labelers can see the attached document while labeling the primary data row |
* The HTML attachment type can take a text or HTML file and render it in an iframe in the attachments panel. Markdown and HTML can also be used with this type to style the content.
Import attachments
When creating an attachment on a data row, you must provide the following information.
| Parameter | Required | Value |
|---|---|---|
type | Yes | See the options above. |
value | Yes | See the table above for the accepted values. |
name | No | User-determined string for reference. |
For details on how to create attachments or import data rows with attachments using the Python SDK, please view the documentation here.
JSON format
If uploading a JSON file of data rows through the SDK, use the below format to include attachments. Note how multiple attachments link to an individual data row.
[
{
"row_data": "https://lb-test-data.s3.us-west-1.amazonaws.com/image-samples/sample-image-1.jpg",
"global_key": "https://lb-test-data.s3.us-west-1.amazonaws.com/image-samples/sample-image-1.jpg",
"media_type": "IMAGE",
"metadata_fields": [{"schema_id": "cko8s9r5v0001h2dk9elqdidh", "value": "tag_string"}],
"attachments": [
{
"type": "RAW_TEXT",
"value": "IOWA, Zone 2232, June 2022 [Text string]"
},
{
"type": "RAW_TEXT",
"value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt"
},
{
"type": "TEXT_URL",
"value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt"
},
{
"type": "IMAGE",
"value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/disease_attachment.jpeg"
},
{
"type": "VIDEO",
"value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/drone_video.mp4"
},
{
"type": "HTML",
"value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/windy.html"
}
]
}
]
[
{
"row_data": "https://lb-test-data.s3.us-west-1.amazonaws.com/video-samples/sample-video-1.mp4",
"global_key": "https://lb-test-data.s3.us-west-1.amazonaws.com/video-samples/sample-video-1.mp4",
"media_type": "VIDEO",
"metadata_fields": [{"schema_id": "cko8s9r5v0001h2dk9elqdidh", "value": "tag_string"}],
"attachments": [
{
"type": "RAW_TEXT",
"value": "Some sample text"
},
{
"type": "RAW_TEXT",
"value": "Some more sample text"
}
]
}
]
[
{
"row_data": "https://lb-test-data.s3.us-west-1.amazonaws.com/text-samples/sample-text-1.txt",
"global_key": "https://lb-test-data.s3.us-west-1.amazonaws.com/text-samples/sample-text-1.txt",
"media_type": "TEXT",
"metadata_fields": [{"schema_id": "cko8s9r5v0001h2dk9elqdidh", "value": "tag_string"}],
"attachments": [
{
"type": "RAW_TEXT",
"value": "Some sample text"
},
{
"type": "RAW_TEXT",
"value": "Some more sample text"
}
]
}
]
[
{
"row_data":{
"tile_layer_url": "https://s3-us-west-1.amazonaws.com/lb-tiler-layers/mexico_city/{z}/{x}/{y}.png",
"bounds": [
[
19.405662413477728,
-99.21052827588443
],
[
19.400498983095076,
-99.20534818927473
]
],
"min_zoom": 12,
"max_zoom": 20,
"epsg": "EPSG4326",
"alternative_layers": [
{
"tile_layer_url": "https://api.mapbox.com/styles/v1/mapbox/satellite-streets-v11/tiles/{z}/{x}/{y}?access_token=pk.eyJ1IjoibWFwYm94IiwiYSI6ImNpejY4NXVycTA2emYycXBndHRqcmZ3N3gifQ.rJcFIG214AriISLbB6B5aw",
"name": "Satellite"
},
{
"tile_layer_url": "https://api.mapbox.com/styles/v1/mapbox/navigation-guidance-night-v4/tiles/{z}/{x}/{y}?access_token=pk.eyJ1IjoibWFwYm94IiwiYSI6ImNpejY4NXVycTA2emYycXBndHRqcmZ3N3gifQ.rJcFIG214AriISLbB6B5aw",
"name": "Guidance"
}
]
},
"global_key": "https://s3-us-west-1.amazonaws.com/lb-tiler-layers/mexico_city/{z}/{x}/{y}.png",
"media_type": "TMS_GEO",
"metadata_fields": [{"schema_id": "cko8s9r5v0001h2dk9elqdidh", "value": "tag_string"}],
"attachments": [
{
"type": "TEXT_URL", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt"
}
]
}
]
[
{
"row_data": {
"pdf_url": "https://lb-test-data.s3.us-west-1.amazonaws.com/document-samples/sample-document-1.pdf",
"text_layer_url": "https://lb-test-data.s3.us-west-1.amazonaws.com/document-samples/sample-document-custom-text-layer.json"
},
"global_key": "https://lb-test-data.s3.us-west-1.amazonaws.com/document-samples/sample-document-1.pdf",
"media_type": "PDF",
"metadata_fields": [{"schema_id": "cko8s9r5v0001h2dk9elqdidh", "value": "tag_string"}],
"attachments": [
{
"type": "HTML", "value": "https://www.wikipedia.org/"
}
]
},
{
"row_data": {
"pdf_url": "https://lb-test-data.s3.us-west-1.amazonaws.com/document-samples/sample-document-2.pdf",
"text_layer_url": "https://lb-test-data.s3.us-west-1.amazonaws.com/document-samples/sample-document-2-custom-text-layer.json"
},
"global_key": "https://lb-test-data.s3.us-west-1.amazonaws.com/document-samples/sample-document-2.pdf",
"media_type": "PDF",
"metadata_fields": [{"schema_id": "cko8s9r5v0001h2dk9elqdidh", "value": "tag_string"}],
"attachments": [
{
"type": "TEXT_URL", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt"
}
]
}
]
[
{
"row_data": "https://lb-test-data.s3.us-west-1.amazonaws.com/conversation-samples/sample-conversation-1.json",
"global_key": "https://lb-test-data.s3.us-west-1.amazonaws.com/conversation-samples/sample-conversation-1.json",
"media_type": "CONVERSATIONAL",
"metadata_fields": [{"schema_id": "cko8s9r5v0001h2dk9elqdidh", "value": "tag_string"}],
"attachments": [
{
"type": "TEXT_URL", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt"
}
]
}
]
[
{
"row_data": "https://lb-test-data.s3.us-west-1.amazonaws.com/audio-samples/sample-audio-1.mp3",
"global_key": "https://lb-test-data.s3.us-west-1.amazonaws.com/audio-samples/sample-audio-1.mp3",
"media_type": "AUDIO",
"metadata_fields": [{"schema_id": "cko8s9r5v0001h2dk9elqdidh", "value": "tag_string"}],
"attachments": [
{
"type": "VIDEO", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/drone_video.mp4"
}
]
},
{
"row_data": "https://lb-test-data.s3.us-west-1.amazonaws.com/audio-samples/sample-audio-2.mp3",
"global_key": "https://lb-test-data.s3.us-west-1.amazonaws.com/audio-samples/sample-audio-2.mp3",
"media_type": "AUDIO",
"metadata_fields": [{"schema_id": "cko8s9r5v0001h2dk9elqdidh", "value": "tag_string"}],
"attachments": [
{
"type": "TEXT_URL", "value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/text_attachment.txt"
}
]
}
]
Viewing attachments
When any attachments have been provided for an asset, labelers will see the Attachment button appear in the top left of the editor.
You can also use the hotkey Shift + ? to open the attachments panel.
Labeling instructions
Labeling instructions (or Annotation guidelines) are essential for any large-scale labeling operation. Whether your labeling team is in-house or outsourced, machine learning teams can leverage labeling instructions to communicate best practices with their labelers.
The following template shows one way to create labeling instructions; a PDF download is also available.
Expand to view labeling instruction template
Add labeling instructions to an ontology
Each ontology can have one labeling instructions document. You can attach instructions whenever editing an ontology by clicking the Instructions tab, and you can update the instructions at any time. For details, see Add/update instructions.
Keep in mind that ontologies can be connected to multiple projects. Updating the instructions for one project impacts all projects sharing the ontology.
If instructions vary between projects, create individual copies of your ontology for each unique project or set of instructions.
Data row information panel
Access to additional context and information at the data row level can be extremely helpful to labelers as they label data. Users can click to view a side panel that brings up data row information, providing teams with ample context and easy access to information related to the particular data row being served.
Labelers can view information such as the global key, the dataset, and when the data row was created. Based on data type, labelers can also view specific media attributes, such as an image’s width and height, the number of pages in a PDF document, the frame rate and duration of a video, and more. Additionally, labelers can also see any metadata attached to the data row.
Supplementary layers and tools
Some complementary features are only available for use in the editor when labeling specific data modalities.
Image overlay
Image overlay can be used to provide labelers with additional view options for the image being labeled. For example, if you have additional cameras capturing images of your subject matter in different formats (greyscale, thermal, etc.), you may want to provide these images as contextual layers to the primary image.
Layers are applied to individual images, and a maximum of 10 layers may be added per image. Image layers are a visualization tool designed to help you view the asset to be labeled in different ways. You may change visual settings on the image layers (e.g., transparency, brightness) and still create annotations on the data row.
Image overlay limits
To view the maximum number of images that can be attached as layers, visit our limits page.
Import image overlay layers
Image overlays are imported as attachments to a data row. You must include the following information for each image layer.
| Parameter | Required | Value |
|---|---|---|
type | Yes | IMAGE_OVERLAY |
value | Yes | URL of the image layer (PNG/JPG) (HTTPS or IAM delegated access path). |
name | No | User-determined string for reference. |
For details on how to create image layers or import data rows with image layers using the Python SDK, please view the documentation here.
JSON format
If uploading a JSON file of data rows through the SDK, use the below format to include image layers. Note how the image overlay layers are imported as attachments on an individual data row.
[
{
"row_data": "https://lb-test-data.s3.us-west-1.amazonaws.com/image-samples/sample-image-1.jpg",
"global_key": "https://lb-test-data.s3.us-west-1.amazonaws.com/image-samples/sample-image-1.jpg",
"media_type": "IMAGE",
"attachments": [
{
"type": "IMAGE_OVERLAY",
"value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/rgb.jpg",
"name": "RGB"
},
{
"type": "IMAGE_OVERLAY",
"value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/cir.jpg",
"name": "CIR"
},
{
"type": "IMAGE_OVERLAY",
"value": "https://storage.googleapis.com/labelbox-sample-datasets/Docs/weeds.jpg",
"name": "Weeds"
}
]
}
]
View image overlay layers
When layers have been provided for an image, you will see an Image overlay button appear in the top bar of the editor directly to the left of the zoom options.
Clicking on this button opens a dropdown menu where you can select an image to overlay. Alternatively, you can use the keyboard shortcut Option/Alt + Layer # to quickly switch between layers.
Smart labeling tools
Labelbox provides various AI-based tools to help you label faster. By eliminating some of your most manual work, these tools allow you to take advantage of the latest AI technologies to move fast and achieve higher throughput.
Auto-segment
This tool is embedded into the segmentation mask tool. To use auto-segment, select a segmentation mask tool, toggle on auto-segment by selecting the magic wand icon or using the hotkey R, and draw a box around an object.
Labelbox will automatically draw a segmentation mask on the object inside the box. Then, you can make edits, as usual, using the segmentation mask's pen tools. For more details, please view the documentation here.
Bounding box tracking (beta)
This tool is embedded into the bounding box tool and is only available when labeling video assets. When you draw a bounding box around an object, click Track Object to activate bounding box tracking.
Labelbox will automatically track the object for a specified number of frames. Edit the results by selecting and removing any desired keyframes. For more details, please view the documentation here.
Supported annotations
The table below indicates which top-level annotation types you can use to label each asset type.
| Annotation type | Image | Video | Text | Audio | Documents | Tiled imagery | Conversational text | HTML |
|---|---|---|---|---|---|---|---|---|
| Segmentation mask | ✓ | ✓ (use video segmentation tool) | N/A | N/A | - | - (Use polygon) | N/A | - |
| Bounding box | ✓ | ✓ | N/A | N/A | ✓ | ✓ | N/A | - |
| Polygon | ✓ | - | N/A | N/A | - | ✓ | N/A | - |
| Point | ✓ | ✓ | N/A | N/A | - | ✓ | N/A | - |
| Polyline | ✓ | ✓ | N/A | N/A | - | ✓ | N/A | - |
| Entity | N/A | N/A | ✓ | N/A | ✓ | N/A | ✓ | - |
| Relationship | ✓ | - | ✓ | N/A | ✓ | - | ✓ | - |
| Radio | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Checklist | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Free-form text | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
Updated about 2 months ago