You can review assets in two ways:
- You manually select the annotated assets.
- Assets are randomly selected for review from the review queue.
In both cases, the review interface is similar to the labeling interface and allows you to review labeled assets and then:
- make corrections, if necessary
- send assets back to the queue
To manually select labeled assets for review, from the project "Queue" page do one of the following:
- Click Explore, navigate through the labeled assets and then review assets.
- Go to the Labeled tab, filter specific assets, select assets for review, and then click Add to review.
By default, all assets visible in the Explore view are sorted in the ascending order. You can change that by clicking the order icon:
For additional sorting options, click the pull-down menu next to
For example, to check the most recent labeled asset after changing instructions, select
Last labeling date. Or you can select
Consensus and start by reviewing the asset with the lowest consensus score.
For details on filtering, refer to the Filtering assets through complex search queries.
You can use these keyboard shortcuts to navigate between labels and assets:
- Jump from one asset to the other using Alt/Option + Up arrow and down arrows: you will see the latest label each time.
- To view the full history of labels for a specific asset, click Alt/Option + Right arrow.
- To collapse the full history of labels, click Alt/Option + Left arrow.
- From your project "Queue" page, go to the To review tab and click Start Reviewing.
- If the labels were added correctly, click Review.
Be careful when reviewing assets that are used as honeypot. Clicking the Review button changes the current honeypot setting for the asset.
- If the labels were added incorrectly, you can make corrections, if necessary or send assets back to labeling.
To send an asset back to the labeler, from the project "Queue" page, go to the Reviewed tab, select an asset and click Send back to queue.
- Click the Issue button.
- Click Send back to queue.
Selected assets will return to the Project queue To do backlog with status "Ongoing".
If an asset is sent back to queue, it gets top labeling priority. This means that the labeler will see this asset as the first one to be labeled.
Updated about 1 month ago