Projects
Data Rows tab
The Data Rows tab enables you to easily view, filter, and manage the data rows in your project.
Every project contains a Data Rows tab. Prior to any filtering, the table contains all the data rows in the project, either labeled or queued. Each data row has information displayed, including labeling and review time, the dataset the data row belongs to, if there are any open issues, and more details dependent on project settings.
The count of data rows for each given status is visible on the left panel. Clicking on any particular status will populate the table with a filtered view of data rows having that status.
You can query your data rows using any combination of the following filters:
For any applicable filter, you can customize the query to reflect both the inclusion or exclusion of a certain attribute. For example, if you are using the Batch filter, you can choose to select batches that a resulting data row either is one of or is not one of. You can improve the precision of your queries by creatively combining your search statements, as expanded on below.
The following bulk actions are currently supported:
Unauthorized to view this data row
Workflow status definitions
| Status | Definition |
|---|---|
| To label | Data rows that have less than the number of required labels associated with them. In other words, these are data rows that have no labels (if they are non-consensus or non-benchmark data rows). For consensus or benchmark data rows, these would be data rows with less the number of required labels, for example, two labels created when the data row needs three labels for completion. |
| In review | Data rows that are in one of the review tasks within the . |
| In rework | Data rows that are in the rework task within the workflow. |
| Done | Data rows that have all required labels completed and have passed through all review steps within the workflow. |
Reduced set of statuses
If a project does not have Workflow set up, the only statuses that will show on the data rows tab are:- Unlabeled when the data row has no labels associated
- Labeled when the data row has at least one label associated
Label time and review time
The Label time for a data row reflects the time spent creating the label on this data row plus any time spent editing the label by the original labeler. The Review time for a data row reflects the time spent by any other user viewing or editing the label made on this data row. For projects that use workflows for review, time spent viewing as well as modifying labels in all review and rework tasks will be added to the review time.Timing metrics with consensus and benchmarks
For projects that use consensus or benchmarks, these timing metrics will reflect a sum of the time spent labeling or reviewing on all labels created on the data row.Gallery view
If you want to preview the data rows, select the gallery view option in the top right corner of the table. This will take you to a preview mode where you can view a thumbnail of each data row.Filter data rows
You can modify the list of data rows to a desired collection by applying a filter or combination of filters. To do so, click on Create a search query to bring up the different options for filtering.| Filter | Description |
|---|---|
| Dataset | The dataset a data row belongs to |
| Metadata | Metadata fields and corresponding values assigned to the data row |
| Media attribute | Attributes of a data row’s asset, with different fields relative to each data type |
| Batch | The batch a data row belongs to |
| Data row | Information about a data row, including subqueries for Global key, Data row ID, Created at, and Last activity at |
| Find text | See find text |
| Natural language | See natural language search |
| Task | The task to which a data row currently belongs |
| Issue | Issues associated with a label |
| Label actions | Information about a label, including subqueries for ID, Labeled by, Created at, Is skipped, Is not skipped, Reworked at, Reworked by, Reviewed by, and Reviewed atNote: Bulk Move to step is not considered a review, therefore it will not show up when filtering using Reworked at, Reworked by, Reviewed by, or Reviewed at. |
| Annotation | Information on the counts of any annotation made on the labels associated with a data row |
| Benchmark agreement | Benchmark agreement metrics by Label or Feature |
| Consensus agreement | Consensus agreement metrics by Label or Feature |
Bulk actions are not considered a review or rework
Bulk Move to step actions are not considered a review or rework. Therefore, these actions will not show up when applying Reworked at, Reworked by, Reviewed by, or Reviewed at filters.Flexible querying
The search and filtering interface allows you to conduct flexible querying by combining any supported filters. Additionally, you can construct complex AND/OR conditions on attributes and their values to curate your search as desired. You can then act on these highly customized queries through the bulk actions explained below or by exporting only the data rows you care about with our export v2 capabilities.Bulk actions with data rows
You can select multiple data rows to perform operations in bulk.- Change priority: Assign a new priority ranking. See Labeling queue for more information.
- Export data: Export selected data rows. See exporting labels from a project for more information on export settings.
- Hide/Unhide from labelers: Restrict data row visibility for labelers. Hidden data rows display an Authorized only tag. Admins can click to view row details, while labelers can only copy the external ID without accessing further details.
Unauthorized to view this data rowIf a labeler clicks on a hidden row, they receive an message.
- Assign labels as benchmarks Assign labels on selected data rows as Benchmark reference labels.
Create a batch
Go to the Data Rows tab and click New Batch. That will take you to the Catalog view where you can select data rows and add them to the project. Follow the steps in Send batch from Catalog.View and manage batches
Click on Batches to view all of the existing batches that have been added to your project. When you click the menu option on a batch, you can perform the following actions:- Rename the batch to a different unique name.
- Remove queued data rows. This will remove any data rows in the batch that do not have any completed labels.
- Delete labels made on data rows in the batch. This will prompt you with the option to requeue the data rows with or without their existing label as a template.
- View history of the batch, which includes a changelog of data rows added and removed from the batch.
View and edit labels
Click on any data row in the activity table to open the Data row browser. In the browser, you can perform the following actions:- To edit the label, click the Edit button in the top right corner. From here, you can create and edit any annotations and then press Save to modify the label.
- To view all the labels for benchmark or consensus data rows, click the > icon for the desired data row in the left-side panel. You can then select any labels made on this data row and proceed with editing, as described above. Additionally, you can view the ID of each label and select a label as the “winner”.