Queue prioritization

Kili app manages a queue for each project. The queue can be customized and redefined throughout the project. This is helpful for advanced annotation processes for business expectations, priorities, active learning, production supervision, and real-time annotation.

First-in-First-out (FiFo) is the default queue setting. Assets are loaded based on their time stamp (creation date): assets uploaded first are distributed to be annotated first.

Two other strategies are also possible:

• Assigning assets to specific labelers
• Prioritizing assets

📘

To speed up assigning and prioritizing assets for images or short bits of text, you can open your Text or Image project in the thumbnail view. The thumbnail view shows you miniatures of all the assets so you can make quick decisions on many assets at once. For example, you can select five assets and then use the quick actions menu to give them all a specific labeling priority.

Assigning assets to specific labelers

Kili's built-in automatic workload distribution mechanism facilitates load management, allows for faster project annotation, and prevents duplicating work.
For example, if several users collaborate on an annotation project, Kili app distributes the data to be annotated to each project member so that each annotator processes unique data. The same data will not be annotated twice, unless Consensus is activated.

You can assign a specific list of assets to specific labelers.
The assignments can work for the whole queue or for a specific subset of the dataset.

📘

• To prevent two users accidentally adding labels to the same asset, Kili temporarily locks the asset that is being edited.
• Assigning assets does not impact their usual order of appearance in the labeling queue. It just ensures that some assets are specifically labeled by one (or several) labelers; for example, because of their unique ability to process such assets. Refer to Prioritizing assets.

Assigning assets to labelers from the Kili app interface

To assign assets to labelers from the Kili app interface:

1. From the Queue page, select the assets.
2. From the Quick actions menu, select Assign to.
3. Type the name of the labeler, and then click Assign.

Assigning asset to the current labeler automatically

In the Quality settings, as an Org admin, you can configure the Auto assign option.

With this option enabled, labelers can maintain an uninterrupted workflow as assets are automatically assigned to them. This ensures that labelers retain ownership of the asset until submission, preventing reassignment even if they are temporarily inactive.

Assigning assets to labelers programmatically

Refer to Kili API documentation:

• GraphQL API key: toBeLabeledBy.
• Python SDK, function: update_properties_in_assets, argument: to_be_labeled_by_array.

For examples, refer to this recipe:

Prioritizing assets

You can set specific labeling priorities at the asset level. The higher the number, the higher the priority. If two assets have the same priority, FiFo applies.

For example:

• If asset 1 has priority N_1 and asset 2 has priority N_2, and N_1 > N_2, then asset 1 has higher priority than asset 2 and will be distributed first.
• If N_1 = N_2, priority follows the default strategy (FiFo).

Prioritizing assets from the Kili app interface

To prioritize assets from the the Kili app interface:

1. From the Queue page, select the assets.
2. From the quick actions menu, select Prioritize.
3. Select one of the predefined options or type the priority manually, and then click Apply.

Prioritizing assets programmatically

You can prioritize assets with the Kili SDK, using the priorities parameter of the update_properties_in_assets method (for details, refer to the documentation).

For an end-to-end example of how to programmatically prioritize your project queue and assign specific assets to specific labelers using Kili's Python SDK, refer to our tutorial on how to set up workflows.

Additionally, here’s a dedicated recipe that helps you assign asset priorities: