How to use consensus in your project

Setting up consensus

To set up consensus:

1. From the project "Settings" page, click on Quality management, and toggle "Consensus".
2. Set the following parameters:

• Total Asset Coverage: the percentage of the dataset that will be annotated several times. If the dataset contains 10,000 assets, and the Total asset coverage is at 10%, it means that 1,000 assets will be labeled several times. The assets to be labeled multiple times will be chosen randomly.
• Number of Labelers: this is the number of labelers that will have to annotate the same asset. It must be set to a number less than or equal to the total number of labelers. For optimal performance, it is limited to 10 labelers for standard projects.
  Contact [email protected] if you need more.

Deactivating consensus for the whole project

To deactivate consensus in an ongoing project where the parameters have been set earlier, you just need to set the Total Asset Coverage to 0 and the Number of Labelers to 1.

🚧

If you change the consensus settings in an ongoing project, only the unlabeled assets (with status TODO) will be distributed to labelers according to the new setup.

You can also set and modify consensus in Kili SDK. Refer to our recipe:

Deactivating consensus for a specific labeling job

You can deactivate consensus computation for specific labeling jobs in your project (for example, in projects where some labeling jobs have been added for workflow-related purposes). For details on how to do this, refer to Customizing the project interface through json settings.

Learn more

For an end-to-end example of how to programmatically configure consensus in your project using Kili's Python SDK, refer to our tutorial on how to set up workflows.