🚧

Private Beta

This feature is currently in private beta and is not enabled by default. Access is limited to a restricted group of users as we continue to refine and improve the experience. If you're interested in using this feature, please contact [email protected] to request access.

The Multi-step review feature enhances quality control by introducing multiple levels of review, each assigned to a specific group of reviewers. This allows for a structured and refined validation process, ensuring that annotations meet the highest quality standards before final approval.

With this feature, teams can define different review stages, assign specialized reviewers at each step, and systematically validate data with increasing levels of scrutiny. This is particularly useful for complex annotation tasks that require expert validation or compliance checks.

Setting up a multi-step workflow

To configure a multi-step review workflow, navigate to Quality & Workflow settings and open the Workflow tab.

From this section, you can:

• Enable automated progression based on the Sampling rate: This determines how many assets move to the next review step automatically. For example, if the sampling rate between two steps is set to 80%, then 80% of the assets that complete the previous step will be sent to the next one. Each transition follows its own configured sampling rate.
• Add additional review steps: Define multiple levels of review for a more granular quality control process.

For each step in the workflow, you must assign members who can work on it. Only users with at least the Reviewer role can be assigned to a review step.

By default, the Enforce Step Separation option is enabled. This ensures that a member cannot work on multiple steps for the same asset, which helps maintain an objective review process. For example, this is useful when requiring team members to review each other’s work without reviewing their own.

Tracking Progress

Asset statuses

Each review step includes various statuses that help you monitor the progress of assets through the workflow:

• To Label or To Review X (X being the step level): The asset is ready to be processed in the respective labeling or reviewing step.
• Labeling or Reviewing X: The asset has entered the labeling or reviewing process. This status is triggered when the first autosave occurs during the labeling or reviewing process.
• Labeled or Reviewed X: The asset has completed the step and is marked as finished for that stage.
• Re-Labeling or Re-Reviewing: The asset requires corrections or needs to be reviewed again after changes have been made.
• Skipped: The asset was skipped and discarded from the labeling workflow, meaning it will not be processed further.

📘

Send back action

When an asset is sent back from a review step (regardless of the review level), it will be reassigned to the initial labeler for correction.

Overall progress

The Overall Progress report provides a clear overview of asset distribution across different workflow stages.

The status breakdown categorizes assets into three main groups:

• To label: Assets that are waiting to be labeled, currently in the "To label" status.
• Review (all review steps): Assets that are either awaiting review (To Review X) or currently being reviewed (Reviewing X).
• Done (all steps): Assets that have reached a final status, either Labeled or Reviewed X, indicating they have successfully passed through the required steps.
• Skipped: Assets that were skipped and therefore discarded from the labeling workflow.

Handling asset progression

With the multi-step review workflow, you have flexibility in managing how assets move through the process stages.

• As a Project Admin or Manager, you can select completed assets (Labeled or Reviewed X) from the Queue page and bulk-add them to the next review step. These assets will then move to the "To Review X" status, ready to be processed by the assigned reviewers.
• As a Reviewer, if you are assigned to the next review step, you can still work on a Labeled or Reviewed X asset from the Explore or Review interface by clicking the Take for Review button. This allows you to manually pick assets for review, even if they haven't been automatically added to the next step.
• As a Labeler or Reviewer, after submitting a label or completing a review, if no one else is working on it, you can still correct it. Click the Correct Label button, and the asset will move back to the Labeling or Reviewing X state, allowing you to make adjustments.

This flexibility ensures that assets progress through the review process efficiently while allowing manual operation when needed.

📘

Access permissions

Reviewers, Managers, and Admins still have access to all assets, but if they are not assigned to the step the asset is currently in, they will only be able to view it in read-only mode.

Additionally, an asset cannot be edited simultaneously by multiple users. If one user is actively working on an asset, others will only have read-only access until the asset is released. This rule also applies when interacting with assets via the SDK, ensuring data consistency across all access points.