How to use consensus in your project
Setting up consensus
To set up consensus:
- From the project "Settings" page, click on Quality management, and toggle "Consensus".
- Set the following parameters:
- Total Asset Coverage: the percentage of the dataset that will be annotated several times. If the dataset contains 10,000 assets, and the Total asset coverage is at 10%, it means that 1,000 assets will be labeled several times. The assets to be labeled multiple times will be chosen randomly.
- Number of Labelers: this is the number of labelers that will have to annotate the same asset. It must be set to a number less than or equal to the total number of labelers. For optimal performance, it is limited to 10 labelers for standard projects.
Contact [email protected] if you need more.
Deactivating consensus
To deactivate consensus in an ongoing project where the parameters have been set earlier, you just need to set the Total Asset Coverage to 0 and the Number of Labelers to 1.
If you change the consensus settings in an ongoing project, only the unlabeled assets (with status
TODO
) will be distributed to labelers according to the new setup.
You can also set and modify consensus in Kili SDK. Refer to our recipe:
Modifying consensus
Open Recipe
Learn more
For an end-to-end example of how to programmatically configure consensus in your project using Kili's Python SDK, refer to our tutorial on how to set up workflows.
Updated about 2 months ago