Log In

Reviewing labeled assets

Reviewing assets

You can review assets in two ways:

In both cases, the review interface is similar to the labeling interface and allows you to review labeled assets and then:

Manually selecting assets to be reviewed

To manually select labeled assets for review, from the project "Queue" page do one of the following:

Navigating labeled assets from the Explore view

From the Explore view, you can also review labeled assets, add issues, and answer all your labelers' open questions.

By default, all assets visible in the Explore view are sorted in the ascending order. You can change that by clicking the order icon:

For additional sorting options, click the pull-down menu next to Sort by:

For example, to check the most recent labeled asset after changing instructions, select Last labeling date. Or you can select Consensus and start by reviewing the asset with the lowest consensus score.

For details on filtering, refer to the Filtering assets through complex search queries.


You can use these keyboard shortcuts to navigate between labels and assets:

  • Jump from one asset to the other using Alt/Option + Up arrow and down arrows: you will see the latest label each time.
  • To view the full history of labels for a specific asset, click Alt/Option + Right arrow.
  • To collapse the full history of labels, click Alt/Option + Left arrow.

Reviewing labeled assets

  1. From your project "Queue" page, go to the To review tab and click Start Reviewing.
  2. If the labels were added correctly, click Review.


Be careful when reviewing assets that are used as honeypot. Clicking the Review button changes the current honeypot setting for the asset.

  1. If the labels were added incorrectly, you can make corrections, if necessary or send assets back to labeling.

Sending assets back to labeling

To send an asset back to the labeler, from the project "Queue" page, go to the Reviewed tab, select an asset and click Send back to queue.


  1. Click the Issue button.
  1. Click Send back to queue.

Selected assets will return to the Project queue To do backlog with status "Ongoing".


If an asset is sent back to queue, it gets top labeling priority. This means that the labeler will see this asset as the first one to be labeled.

Did this page help you?