Quiz

Quizzes are a project-level quality check for labelers and Alignerrs. Use them to confirm readiness before work starts, reinforce quality standards, or reassess contributors after guidelines change. Configure and manage quizzes from the project overview, separately from ontology schema and labeling instructions. Validating that labelers understand your instructions is one common use case; quizzes can also test edge cases, quality criteria, or other project-specific knowledge. For the contributor experience (review instructions → take quiz → results), see Labeling instructions.

​

Configure a project quiz

Save project instructions before you configure a quiz. Then:

1. Go to your project overview.
2. In the sidebar, click Quiz (or Click to configure quiz if no quiz exists yet).
3. On the Project Quiz page, create questions with AI or add them manually.
4. Configure attempt policy, pass threshold, feedback, and whether the quiz is required.
5. Click Save quiz.

Saving the quiz does not change ontology schema or instructions. Saving ontology schema or instructions does not change the quiz.

​

Automatically generate quiz questions

To generate questions from your project instructions:

1. Open the project Quiz page.
2. Under Generate with AI, choose how many questions to create (1–10; default 3).
3. Click Generate Quiz using AI.

AI generation uses the project’s instruction text and any uploaded PDF as a source. Videos and external links in instructions are not included. Instructions must be at least 50 characters (and at most 8,000 characters) for generation. You can edit generated questions or write your own to cover quality standards beyond the instruction text. You can also click Add Question using AI on an existing quiz to append one AI-generated question at a time.

​

Create quiz questions manually

1. Open the project Quiz page.
2. Click Add Questions Manually (or Add Question Manually when a quiz already exists).
3. Enter each question and expected answer, then click Save quiz.

Questions and answers can be up to 1,000 characters each.

​

Quiz settings

​

Force retakes and lockouts

After contributors have already passed, editing questions and checking Force users to retake the quiz on save requires those contributors to pass the updated quiz. You can also click Force retake now to require a retake without changing the questions—for example after a quality standard or guideline update. With One attempt only or Limited attempts, a required quiz that is failed for the last time locks the contributor out until an admin clears the failure. Use Manage lockouts on the project Quiz page to review failed answers and reset a lockout (a reason is required). Force retake does not clear final-failure lockouts. Optional quizzes never lock contributors out of work, even if attempt limits are exhausted.

​

Instruction Quiz metrics

The Instruction Quiz tab on the project Performance dashboard provides analytics for quiz performance when a project quiz is configured. Use these metrics to gauge contributor readiness, spot weak quality areas, and decide when to revise questions or force a retake.

​

How to access your quiz analytics

To view quiz analytics for your project, please follow these steps:

1. Navigate to your project.
2. Go to the Performance tab.
3. Select the Instruction Quiz tab at the top of the page.
4. Use the date range picker to filter analytics by your desired time period.

​

An overview of the metrics

The dashboard displays the following key performance indicators at the top:

​

Understanding the visual analytics

The dashboard includes two key distribution charts to help you visualize the quiz data.

• Pass attempt distribution: This chart shows how many attempts labelers need to pass the quiz and helps you identify if the quiz difficulty is appropriate. For example, if most labelers pass on the first attempt, the quiz may be too easy. If they need many attempts, it may be too difficult.
• Score distribution: This chart displays the range of scores across all attempts and shows how scores are distributed on the 1–5 scale. This helps you identify if most labelers are performing well or struggling.

​

Analyzing the question performance table

This table shows detailed metrics for each quiz question so you can identify any problematic questions. You can use this table to identify the following:

• Questions with low average scores may need clearer wording or better supporting guidelines.
• Questions requiring many attempts suggest the topic needs a better explanation or training.
• Negative improvement trends indicate labelers aren’t learning from retakes.

​

Analyzing the user performance table

This table shows individual labeler performance, so you can identify labelers who may need additional training or support.

​

Using analytics to improve your quiz

Based on the analytics data, you can take several actions to improve your quiz.

• If overall pass rates are low:
  • Clarify the quality standards or guidelines the quiz is testing.
  • Consider breaking down complex concepts into simpler explanations.
  • Add more examples that illustrate correct decisions.
• If specific questions have low scores:
  • Revise the question or expected answer.
  • Ensure the question accurately tests the intended knowledge or quality criterion.
  • Adjust the expected answer to be more flexible when appropriate.
• If labelers need many attempts to pass:
  • Simplify your quiz questions or make the criteria more explicit.
  • Add practice examples to your guidelines.
  • Consider lowering the Pass threshold (%) if appropriate.
• If improvement trends are negative:
  • Review the feedback provided by the AI scoring.
  • Ensure questions test applied understanding, not memorization.
  • Consider whether the quiz is testing the right quality concepts.