Reviewing labeled assets

You can review assets in two ways:

You manually select the annotated assets.
Assets are randomly selected for review from the review queue.

In both cases, the review interface is similar to the labeling interface and allows you to review labeled assets and then:

Make corrections, if necessary
Send assets back to the queue

📘
If an asset’s status changes to “To review”, labelers are no longer allowed to make any changes to the labels in that asset: when they try to do it, a warning message is displayed and the changes are not saved. This eliminates possible quality issues.

Manually selecting assets to be reviewed

To manually select labeled assets for review:

From your project Queue page, run Filters to search for labeled assets
From the Quick actions menu, select Add to review

Reviewing labeled assets

From your project Queue page, or from the Explore view, run Filters to search for assets labeled To review.
Click to open a specific asset that you want to review.
If the labels were added correctly, click Review.

🚧
Be careful when reviewing assets that are used as honeypot. Clicking the Review button changes the current honeypot setting for the asset.

If the labels were added incorrectly, you can make corrections, if necessary or send assets back to labeling.

To check the current review score on an asset, use the Explore view or the project Analytics page. For calculation rules, refer to Calculation rules for quality metrics.

Sending assets back to labeling

To send an asset back to the labeler:

From your project Queue page, run Filters to search for specific assets
Select assets
From the Quick actions menu, click Send back to queue

Alternatively, you can use the issue button:

From your project Queue page, or from the Explore view run Filters to search for specific assets.
Open the asset.
Add an issue. Refer to Handling questions and issues.

Selected assets will return to the project Queue page with status Ongoing.

📘

The asset returns to the queue of the same labeler who added the disputed label. Kili does not impose any specific timeframe for the labeler to update the label.

If an asset is sent back to queue, it gets top labeling priority. This means that the labeler will see this asset as the first one to be labeled.

If you want to access review scores:

From the Explore viewyou can access the review score per asset. To access this metric, open and expand an asset.
To check the review score per labeler, or open the Analytics page and then go to the Quality Insights tab.

For information on how the review score is calculated, refer to Calculation rules for quality metrics.

📘
To speed up the review for images or short bits of text, you can open your Text or Image project in the thumbnail view. The thumbnail view shows you miniatures of all the assets so you can make quick decisions on many assets at once. For example, you can select five assets and then use the quick actions menu to send them back to be project queue or add them to the list of assets that must be reviewed.