To set up consensus:
- From the project "Settings" page, click on Quality management, and toggle "Consensus".
- Set the following parameters:
- Total Asset Coverage: the percentage of the dataset that will be annotated several times. If the dataset contains 10,000 assets, and the Total asset coverage is at 10%, it means that 1,000 assets will be labeled several times. The assets to be labeled multiple times will be chosen randomly.
- Number of Labelers: this is the number of labelers that will have to annotate the same asset. It must be set to a number less than or equal to the total number of labelers. For optimal performance, it is limited to 10 labelers for standard projects.
Contact [email protected] if you need more.
To deactivate consensus in an ongoing project where the parameters have been set earlier, you just need to set the Total Asset Coverage to 0 and the Number of Labelers to 1.
If you change the consensus settings in an ongoing project, only the unlabeled assets (with status
TODO) will be distributed to labelers according to the new setup.
You can also set and modify consensus in Kili SDK. Refer to our recipe:
You can deactivate consensus computation for specific labeling jobs in your project (for example, in projects where some labeling jobs have been added for workflow-related purposes). For details on how to do this, refer to Customizing the project interface through json settings.
For an end-to-end example of how to programmatically configure consensus in your project using Kili's Python SDK, refer to our tutorial on how to set up workflows.
Updated 3 months ago