Webhooks
A guide for managing webhooks in the UI
A webhook is an endpoint on your server that receives requests from Labelbox. The purpose of a webhook event is to notify your application whenever something interesting happens in your Labelbox account. Rather than periodically polling the Labelbox API to check for any updates in Labelbox, setting up webhooks allows your application to receive updates automatically.
For security purposes, Labelbox includes a signature in each webhook event it sends to your application endpoint. Setting up your webhook configuration to verify the signature in each webhook message allows you to confirm that the events were sent by Labelbox. See an example here.
Labelbox IP address to whitelist
Any traffic coming out of Labelbox will be coming from this IP address:
35.223.142.181. Make sure this IP address is whitelisted when you are setting up your webhooks.
Webhook events
Labelbox offers the following events either per-project organization-wide:
| Entity | Event Types | Description | Method & Payload |
|---|---|---|---|
| Label | LABEL_CREATED, LABEL_UPDATED, LABEL_DELETED | Triggers when labels have been created, updated, or deleted. | HTTP POST |
| Review | REVIEW_CREATED, REVIEW_UPDATED* | Triggers when reviews have been created or updated for a label. | HTTP POST |
| Model run training | Created (HTTP POST) | Triggers when training of a model run has been initiated Note this webhook does not yet conform to standard webhook subscriptions. It must be registered in the Model User Interface. | HTTP POST |
| Models list | Queries (HTTP GET) | Callback to retrieve a list of model types for a given model training backend | HTTP GET |
| Workflow | MOVE | Triggers when a workflow action happens, e.g. after a data row was approved or rejected during a review step, or after it was manually moved to a different queue. | HTTP POST |
* When a review is removed from a label, the REVIEW_UPDATED event will be triggered.
Manage webhooks in the UI
You can manage your webhooks from the application by going to the Webhooks tab in your Workspace settings section. Here you can create and manage your webhooks without any code.
Manage webhooks via the Python SDK
Code sample
public_url = "https://example.com/webhook-endpoint" # Where the messages will be sent
secret = b"CHANGE-ME" # Use to verify Labelbox is sending the message
client = Client()
project = client.get_project("<project-id>")
# Create a project specific webhook
topics = {topic.value for topic in Webhook.Topic} # Specify 1+ topics
webhook = Webhook.create(
client,
topics=topics,
url=public_url,
secret=secret.decode(),
project=project
)
# --- Example Flask Server Code ---
@app.route("/webhook-endpoint", methods=["POST"])
def print_webhook_info():
payload = request.data
computed_signature = hmac.new(
secret,
msg=payload,
digestmod=hashlib.sha1
).hexdigest()
if request.headers["X-Hub-Signature"] != "sha1=" + computed_signature:
print(
"Error: computed_signature does not match signature provided in the headers"
)
return "Error", 500, 200
print("=========== New Webhook Delivery ============")
print("Delivery ID: %s" % request.headers["X-Labelbox-Id"])
print("Event: %s" % request.headers["X-Labelbox-Event"])
print("Payload: %s" % json.dumps(json.loads(payload.decode("utf8")), indent=4))
return "Success"
Sample response
{'agreement': None,
'benchmarkAgreement': None,
'createdAt': '2023-02-13T18:22:46Z',
'dataRow': {'createdAt': '2023-01-25T17:54:20Z',
'deletedAt': None,
'externalId': 'multi_thread_update_4cfdcb29-be33-4e07-b586-849829385210',
'id': 'cldbyu7c7n4jp072scuqt66ll',
'labelCountInProject': 1,
'rowData': 'https://labelbox.s3-us-west-2.amazonaws.com/datasets/mapillary_traffic/images/F1xqHzDSi1qP3qSLBXVyJQ.jpg',
'updatedAt': '2023-01-25T20:42:57Z'},
'dataset': {'createdAt': '2023-01-25T17:53:58Z',
'deleted': False,
'description': '',
'id': 'cldbytq0w2uw8074hgl3hbxr2',
'name': 'Datarow Update Test - 5K',
'updatedAt': '2023-01-25T17:54:21Z'},
'deleted': False,
'id': 'cle357uex0g5307z996y28ic0',
'label': '{"objects":[{"featureId":"cle357vrp00013b6jsi6dfei0","schemaId":"cldan3u0701sc07yi6xyw6up6","color":"#ff0000","title":"bounding_box","value":"bounding_box","bbox":{"top":453,"left":951,"height":670,"width":952},"instanceURI":"https://api.labelbox.com/masks/feature/cle357vrp00013b6jsi6dfei0"}],"classifications":[],"relationships":[]}',
'project': {'createdAt': '2023-02-13T18:20:52Z',
'deleted': False,
'description': '',
'id': 'cle355i370h86070hf1v7ata1',
'name': 'Webhooks',
'updatedAt': '2023-02-13T18:22:17Z'},
'secondsToLabel': 0,
'skipped': False,
'updatedAt': '2023-02-13T18:22:46Z',
'user': {'email': '[email protected]',
'id': 'clbpgxfaehww6076tedo5ejwx'}}
{
"id": "cl4g555g81ltr07cka1a2b1wv",
"createdAt": "2022-06-16T19:41:49Z",
"updatedAt": "2022-06-16T19:58:25Z",
"secondsToLabel": 6.002,
"label": "{\"objects\":[{\"featureId\":\"cl4hg2phm00013b6j2thsko0u\",\"schemaId\":\"cl2i6hbju1o2y10a321w3cusv\",\"title\":\"grape\",\"value\":\"grape\",\"color\":\"#1CE6FF\",\"bbox\":{\"top\":298,\"left\":445,\"height\":520,\"width\":930},\"instanceURI\":\"https://api.labelbox.com/masks/feature/cl4hg2phm00013b6j2thsko0u\"}],\"classifications\":[],\"relationships\":[]}",
"agreement": null,
"benchmarkAgreement": null,
"deleted": false,
"skipped": false,
"labelCountInProject": 2,
"project": {
"id": "cl2i6gssn1o2210a3gq549fq0",
"createdAt": "2022-04-27T22:57:43Z",
"updatedAt": "2022-06-15T22:19:36Z",
"name": "grapes",
"description": "",
"deleted": false
},
"dataRow": {
"id": "cl2hzfg8g18kw0ztwbtnq0b6q",
"createdAt": "2022-04-27T19:40:43Z",
"updatedAt": "2022-04-27T20:13:51Z",
"deletedAt": null,
"externalId": "CFR_1620.jpg",
"rowData": "https://storage.labelbox.com/ckhmnux5zeutm0825bt0kj34y%2F5375dc13-6442-22d7-f523-e9d252e715ea-CFR_1620.jpg?Expires=1656619105&KeyName=labelbox-assets-key-3&Signature=iOg3tbKe4j7xWZHw1r5rQD4iJIk="
},
"dataset": {
"id": "cl2hz9qay0unl10a3f4wmcsfi",
"createdAt": "2022-04-27T19:36:16Z",
"updatedAt": "2022-04-27T19:40:43Z",
"name": "data",
"description": "",
"deleted": false
},
"user": {
"id": "ckhmnux6hi0p907898jext199",
"email": "[email protected]"
}
}
{
"id": "cl2i6hegh1x660z9i9fzz01a0",
"createdAt": "2022-04-27T22:58:35Z",
"updatedAt": "2022-06-16T20:12:41Z",
"secondsToLabel": 31.448,
"label": "{\"objects\":[{\"featureId\":\"cl2i6hhgd00013f6ghfnrcn6h\",\"schemaId\":\"cl2i6hbju1o2y10a321w3cusv\",\"title\":\"grape\",\"value\":\"grape\",\"color\":\"#1CE6FF\",\"bbox\":{\"top\":701,\"left\":402,\"height\":316,\"width\":238},\"instanceURI\":\"https://api.labelbox.com/masks/feature/cl2i6hhgd00013f6ghfnrcn6h\"},{\"featureId\":\"cl2i6hp3h00033f6gaofl29lx\",\"schemaId\":\"cl2i6hbju1o3010a30ksn2yiy\",\"title\":\"grape mask\",\"value\":\"grape_mask\",\"color\":\"#FF34FF\",\"instanceURI\":\"https://api.labelbox.com/masks/feature/cl2i6hp3h00033f6gaofl29lx\"}],\"classifications\":[],\"relationships\":[]}",
"agreement": null,
"benchmarkAgreement": null,
"deleted": true,
"skipped": false,
"labelCountInProject": 1,
"project": {
"id": "cl2i6gssn1o2210a3gq549fq0",
"createdAt": "2022-04-27T22:57:43Z",
"updatedAt": "2022-06-16T20:06:25Z",
"name": "grapes",
"description": "",
"deleted": false
},
"dataRow": {
"id": "cl2hzfg8g18kk0ztwg22ke4vz",
"createdAt": "2022-04-27T19:40:43Z",
"updatedAt": "2022-04-27T20:13:52Z",
"deletedAt": null,
"externalId": "SYH_2017-04-27_1324.jpg",
"rowData": "https://storage.labelbox.com/ckhmnux5zeutm0825bt0kj34y%2F1d883928-046f-97d5-e3f7-e6c296967c7e-SYH_2017-04-27_1324.jpg?Expires=1656619961&KeyName=labelbox-assets-key-3&Signature=0ma5Yw5qxdJE6_x7ACR7aPEHePM="
},
"dataset": {
"id": "cl2hz9qay0unl10a3f4wmcsfi",
"createdAt": "2022-04-27T19:36:16Z",
"updatedAt": "2022-04-27T19:40:43Z",
"name": "data",
"description": "",
"deleted": false
},
"user": {
"id": "ckhmnux6hi0p907898jext199",
"email": "[email protected]"
}
}
{
"id": "cl4hgj3ma26wy087b1chnayro",
"createdAt": "2022-06-16T20:11:05Z",
"updatedAt": "2022-06-16T20:11:05Z",
"score": -1,
"deleted": false,
"label": {
"id": "cl4g555g81ltr07cka1a2b1wv",
"secondsToLabel": 0,
"label": "{\"objects\":[{\"featureId\":\"cl4hg2phm00013b6j2thsko0u\",\"schemaId\":\"cl2i6hbju1o2y10a321w3cusv\",\"title\":\"grape\",\"value\":\"grape\",\"color\":\"#1CE6FF\",\"bbox\":{\"top\":298,\"left\":445,\"height\":520,\"width\":930},\"instanceURI\":\"https://api.labelbox.com/masks/feature/cl4hg2phm00013b6j2thsko0u\"}],\"classifications\":[],\"relationships\":[]}",
"agreement": null,
"benchmarkAgreement": null,
"deleted": false,
"skipped": false,
"project": {
"id": "cl2i6gssn1o2210a3gq549fq0",
"name": "grapes",
"description": ""
},
"dataRow": {
"id": "cl2hzfg8g18kw0ztwbtnq0b6q",
"externalId": "CFR_1620.jpg",
"rowData": "gs://labelbox-193903.appspot.com/ckhmnux5zeutm0825bt0kj34y/5375dc13-6442-22d7-f523-e9d252e715ea-CFR_1620.jpg"
},
"dataset": {
"id": "cl2hz9qay0unl10a3f4wmcsfi",
"name": "data",
"description": ""
},
"user": {
"id": "ckhmnux6hi0p907898jext199",
"email": "[email protected]"
}
},
"user": {
"id": "ckhmnux6hi0p907898jext199",
"email": "[email protected]"
}
}
{
"id": "cl4hghcx72nbd074r35etdcgn",
"createdAt": "2022-06-16T20:09:44Z",
"updatedAt": "2022-06-16T20:11:36Z",
"score": 1,
"deleted": true,
"label": {
"id": "cl4g5555r0ojj07fa025043sh",
"secondsToLabel": 0,
"label": "{\"objects\":[{\"featureId\":\"cl4g5563k00013f6al4spj591\",\"schemaId\":\"cl2i6hbju1o2y10a321w3cusv\",\"title\":\"grape\",\"value\":\"grape\",\"color\":\"#1CE6FF\",\"bbox\":{\"top\":471,\"left\":718,\"height\":246,\"width\":553},\"instanceURI\":\"https://api.labelbox.com/masks/feature/cl4g5563k00013f6al4spj591\",\"classifications\":[{\"featureId\":\"cl4g55ahz00043f6axelsgl1x\",\"schemaId\":\"cl4g551g50p1307ezb2otfmzs\",\"title\":\"test\",\"value\":\"test\",\"answer\":{\"featureId\":\"cl4g55ahz00033f6akoxqn6s3\",\"schemaId\":\"cl4g551g50p1407ezf6x7czou\",\"title\":\"a\",\"value\":\"a\"}}]},{\"featureId\":\"cl4g55d4900063f6aixpq44fx\",\"schemaId\":\"cl2i6hbju1o3010a30ksn2yiy\",\"title\":\"grape mask\",\"value\":\"grape_mask\",\"color\":\"#FF34FF\",\"instanceURI\":\"https://api.labelbox.com/masks/feature/cl4g55d4900063f6aixpq44fx\",\"classifications\":[{\"featureId\":\"cl4g55iqv00083f6aetgrin6j\",\"schemaId\":\"cl4g551g60p1807ez7o5w1mqd\",\"title\":\"text\",\"value\":\"text\",\"answer\":\"test\"}]},{\"featureId\":\"cl4hgh77m00013b6jyhazj23e\",\"schemaId\":\"cl2i6hbju1o2y10a321w3cusv\",\"title\":\"grape\",\"value\":\"grape\",\"color\":\"#1CE6FF\",\"bbox\":{\"top\":238,\"left\":648,\"height\":47,\"width\":568},\"instanceURI\":\"https://api.labelbox.com/masks/feature/cl4hgh77m00013b6jyhazj23e\"}],\"classifications\":[],\"relationships\":[]}",
"agreement": null,
"benchmarkAgreement": null,
"deleted": false,
"skipped": false,
"project": {
"id": "cl2i6gssn1o2210a3gq549fq0",
"name": "grapes",
"description": ""
},
"dataRow": {
"id": "cl2hzfg8g18ks0ztwbh1ohgd5",
"externalId": "CDY_2036.jpg",
"rowData": "gs://labelbox-193903.appspot.com/ckhmnux5zeutm0825bt0kj34y/10043afd-f3a7-6706-8473-9717c482dafd-CDY_2036.jpg"
},
"dataset": {
"id": "cl2hz9qay0unl10a3f4wmcsfi",
"name": "data",
"description": ""
},
"user": {
"id": "ckhmnux6hi0p907898jext199",
"email": "[email protected]"
}
},
"user": {
"id": "ckhmnux6hi0p907898jext199",
"email": "[email protected]"
}
}
{
"organizationId": "cl81imdtk0cie0yx4g39i0du8",
"projectId": "clbxofp0h05d2074475g14f9v",
"action": "APPROVE",
"dataRowIds": [
"cl9elx44f001i076k2pz84tw6"
],
"originTaskId": "92665e1f-a507-4308-bdfc-d6be3c629ec7",
"originTaskName": "My custom review step",
"destinationTaskId": null,
"destinationTaskName": "Done",
"actorId": "cl81imdtx0cif0yx4ak0ndq8a",
"timestamp": "2023-01-20T12:06:01.421Z"
}
You can find an end-to-end Python tutorial on Webhooks here.
FAQ
What is the outgoing rate for webhooks?
The rate would be determined by the configuration (the subscribed topics) and their usage (how many labels are being created)
Is there an upper limit on the size of the webhook payload?
No, because the payload is a function of the number of annotations on a label and there is no limit to the number of annotations on a data row.
Does Labelbox use any non-https connections to send webhook events?
All our webhook events are sent in https.
Is the Content-Type header from the labelbox webhook events only “application/json”?
Yes, the Content-Type will only ever be application/json.
Is the “id” nested under the “payload” field unique for each payload?
Event: LABEL_CREATED
Payload: {
"id": "cl0sjr9nb9tfi0zatagsq5zw1",
Is the order of events from a Labelbox webhook guaranteed to be the same as the order in which the label/review was created/updated/deleted?
There are no guarantees on the delivery of sequencing.
Updated 4 days ago