Handling questions and issues

As part of the QA flow, you can create issues or address the questions asked by labelers. When all the issues and questions are addressed, you can resolve the issue.

Creating issues

When you are reviewing labeled assets, you can add issues:

• To the whole labeled asset
• To a specific annotation

Adding an issue to an asset

To add an issue to an asset:

1. Click on the Issues & Questions button.

2. Type your comment to explain the issue and then click on Add issue.

Adding an issue to a specific annotation

📘

Currently, you can add issues to these types of annotations:

• Bounding boxes
• Polygons
• Named entities

1. From the asset viewer, right-click an annotation.
2. From the context menu, select Add issue.

3. Type your comment to explain the issue and then click on Add issue.

When you're done adding issues, click on Send back. The asset will re-appear in labeler's queue and the status of the asset will change from Labeled to Ongoing.

📘

If an asset is sent back to queue, it gets top labeling priority. This means that the labeler will see this asset as the first one to be labeled.

Addressing the questions asked by labelers

📘

You can filter displayed assets to show only those that have open questions/issues linked to them. You can do it from the project Queue page or from the Explore view, using the advanced query builder.

Answering questions

Reviewers and admins can answer questions in the review. To answer a question:

1. Click on the Issues & Questions button to access open questions.

2. Type your answer and then click on Reply.

3. When you're done answering questions, click on Send back. The asset will re-appear in labeler's queue and the status of the asset will change from Labeled to Ongoing.

📘

• If a labeler added a question that is related to a specific label, reviewers see the question icon next to the label as well.
• If an asset is sent back to queue, it gets top labeling priority. This means that the labeler will see this asset as the first one to be labeled.

Resolving issues

When the labeler corrects the annotation and re-submits the asset, reviewers can resolve the issue by either of these actions:

• Clicking the tick symbol (✓) next to the issue.
• Reviewing labeled assets.