Quality management overview

Automatic workload distribution

Kili's built-in automatic workload distribution mechanism facilitates load management, allows for faster project annotation, and prevents duplicating work.
For example, if several users collaborate on an annotation project, Kili app distributes the data to be annotated to each project member so that each annotator processes unique data. The same data will not be annotated twice, unless Consensus is activated.

Labeling instructions

To help labelers and reviewers meet the labeling requirements for the project, you can set labeling instructions. Your instructions can also serve as reference, if questions arise.

Labelers who are unsure about a specific label can ask a question. Reviewers, team managers and project admin will be notified and will have the option to answer and close the question.
The number of open questions is shown on the issue button.

For information on how to handle questions asked by labelers, refer to Handling questions and issues.

Issues

To provide feedback to labelers, reviewers can add issues.
The number of open issues is shown on the issue button and is also available from the Analytics page.

Issues & Questions button

For information on how to find, add, and resolve issues, refer to Handling questions and issues.

Additional QA tools

To further increase your label quality, use the following tools:

Based on these quality management tools, you can generate specific KPIs to optimize your quality control process.

Quality KPIs

You can access quality KPIs at 2 levels:

Asset level, from the table view in the Queue page.
Labeler level, from the Analytics page.

Learn more

For more information on the review process, refer to Reviewing labeled assets.

For a list of best practices, refer to Best practices for quality workflow.