To set up consensus:
- From the project "Settings" page, click on Quality management, and toggle "Consensus".
- Set the following parameters:
- Total Asset Coverage: the percentage of the dataset that will be annotated several times. If the dataset contains 10,000 assets, and the Total asset coverage is at 10%, it means that 1,000 assets will be labeled several times. The assets to be labeled multiple times will be chosen randomly.
- Number of Labelers: this is the number of labelers that will have to annotate the same asset. It must be set to a number less than or equal to the total number of labelers. For optimal performance, it is limited to 10 labelers for standard projects.
Contact [email protected] if you need more.
To deactivate consensus in an ongoing project where the parameters have been set earlier, you just need to set the Total Asset Coverage to 0 and the Number of Labelers to 1.
If you change the consensus settings in an ongoing project, only the unlabeled assets (with status
TODO) will be distributed to labelers according to the new setup.
Updated 22 days ago