This topic will guide you through a typical Kili project lifecycle. You will learn how to:
- Set up a project
- Add assets to your project
- Build project ontology using tools made available by Kili
- Add users to project and assign their roles in the project
- Properly configure quality management metrics
- Review labeled assets
- Analyze project KPIs
- Export annotations
A project is a space where the work on data is done. This is where you configure the labeling jobs, import assets, supervise the production of labels and export the completed work.
Projects are created inside an organization. Only invited users can contribute.
Regarding your workflow and operational constraints, you can adopt two different approaches to labeling in Kili:
- One-off project: Upload data > label > export labels.
- Continuous work: When your initial labeling is done, you still regularly add new assets to label and improve your model.
You can either create a project from the Kili graphical user interface or programmatically.
Follow this recipe to create a new project from the Kili Python SDK:
The recipe shows how to create an image object classification project. For more options, refer to our Python SDK reference.
For information on how to create a project from Kili CLI, refer to our Command Line Interface documentation
In Kili, an asset can be a file or a document. This could be a photograph, a satellite image, a video, a PDF, an email, etc. Refer to Supported file formats.
You can add assets located on:
For information on how to add assets located on your local servers, refer to Adding assets located on premise.
You can also add assets programmatically. For information on adding assets through Kili SDK , refer to our example recipe:
Alternatively, follow examples in our Python SDK Github repository.
For information on how to add assets through the Kili CLI, refer to our Command Line Interface documentation.
- The maximum number of assets in a project is 25,000.
If you need more, contact us at [email protected].
If you need to add asset metadata, refer to Adding asset metadata.
For additional information, refer to Asset lifecycle in Kili app.
In Kili, labeling (or annotation) jobs are labeling tasks which are associated with specific tools.
For example, each one of these can be considered a Kili labeling job:
- Classification task with a multi choice dropdown
- Object detection task with polygon tool
- Named entities recognition task
Kili jobs may contain nested subjobs, for example:
- Classification tasks with nested conditional questions
- Object detection tasks with nested transcription containing additional details
Some jobs types may be interdependent. For example, available relations in a named entities relation job will depend on how classes were defined in a named entities recognition job.
Additionally, each job can be either required or optional.
Creating projects with lots of labeling jobs is not efficient. Project complexity and potential performance issues mean that it is harder for members to complete all the jobs. Such a project is also harder to review.
It is better and more efficient to split your jobs in multiple projects, and then to concatenate your labels later, even if this operation must be done outside Kili.
Kili provides the following machine learning job types:
Availability of specific job types depends on your project asset type:
|Labeling job type||Image||Text||Video||Video (legacy)|
|Named entities recognition||✓||✓|
|Named entities relation||✓||✓|
If you need to add project metadata, refer to Adding project metadata.
Kili's built-in automatic workload distribution mechanism facilitates load management, allows for faster project annotation, and prevents duplicating work.
For example, if several users collaborate on an annotation project, Kili app distributes the data to be annotated to each project member so that each annotator processes unique data. The same data will not be annotated twice, unless Consensus is activated.
To help labelers and reviewers meet the labeling requirements for the project, you can set labeling instructions. Your instructions can also serve as reference, if questions arise.
Labelers who are unsure about a specific label can ask a question. Reviewers, team managers and project admin will be notified and will have the option to answer and close the question.
The number of open questions is shown on the issue button.
For information on how to handle questions asked by labelers, refer to Handling questions and issues.
To further increase your label quality, use the following tools:
Users, often called Members in the Kili app, are managed at two levels:
- Organization they belong to
- Projects they work for
On Kili Technology platform, a user (identified by email) can only belong to a single organization.
Inside an organization, users can be part of multiple projects, with equal or different roles.
Organization admins are admins of all the projects within an organization.
For additional information, refer to:
To manage project members, go to project Settings page and select Members.
From here you can add members of an existing organization to the project or remove them from the project.
The easiest way to add users to a project is to type their email address in the
Search user field.
To add a new user to your project:
- Click Add member.
- Type user's email address.
- Specify user's role.
- Click Submit.
To add a group of users, enter all their email addresses.
Depending on user role, users will have access to different features. Refer to the following:
For information on how to add project members using our SDK, refer to our Python SDK documentation.
You can also quickly copy the code from our recipe here:
Invitees will be informed by email that they were added to a project. If they are invited for the first time, they first have to sign up to join the organization.
Maximum number of members per project is limited to 50. If you need more, contact us at [email protected].
You can review assets in two ways:
- You manually select the annotated assets.
- Assets are randomly selected for review from the review queue.
In both cases, the review interface is similar to the labeling interface and allows you to review labeled assets and then:
- make corrections, if necessary
- send assets back to the queue
To be able to quickly find any asset, you can use Kili's advanced filtering features.
To provide feedback to labelers, reviewers can add issues.
The number of open issues is shown on the issue button and is also available from the Analytics page.
For information on how to find, add, and resolve issues, refer to Handling questions and issues.
The Analytics page provides an overview of the project progress.
By using the tabs on the left side of the screen, you can select whether you want to access:
Overview contains a snapshot of the project. You can trace what's already been done and access a quick summary of open questions and skipped assets.
Use the visual progress bar to quickly check how many assets have been annotated, reviewed, or still left to do and then click on a specific category to immediately jump to the correctly filtered Explore view.
Additionally, you have access to an overall class balance, that is information on what percentage of the total number of annotations are labels given to a specific class. For extra granularity, you can select either the whole project or drill down to specific labeling jobs.
From the Quality tab, you have access to insights regarding consensus and honeypot metrics (if you set them up in your project). Information is presented per class (with an option to delve deeper, per job) and per labeler. You also have access to an overall total agreement score for consensus and honeypot. For information on how these are calculated, refer to Calculation rules for quality metrics.
When done, you can click directly on the graph to switch to the Explore view and start your review.
Use the graphs and scores to target the most relevant assets to focus your review on. For example, if you spot a class with the lowest consensus mark, you can then check on it by using the Explore view to properly research and address this issue.
From the Performance tab, you can check how many assets are being labeled and reviewed in a daily, weekly and monthly time frames. Note that we're taking into account actions, not just assets. So if two or more project members labeled (for example as part of consensus) or reviewed the same asset, the number will be higher.
From here, you can also access detailed labeling statistics that you can later export as a
.csv file and process outside Kili.
Through performance metrics you can assess the velocity of your team after the project has been running for some time. How is it evolving per asset? Are some of the assets outliers? If so, why? Is the performance consistent with what you were expecting? How does it affect your project timeline? Once you have all this information, you can then deep dive and analyze available data per annotator to see if you can spot any discrepancies and items to focus on.
- From the project Queue page, select the assets that you want to export. Quick actions menu appears above the asset list.
- From the Quick actions menu, select
- From the "Export data" popup window, select your export parameters and then click Export.
- Wait for the notification to appear in the top-right corner of the screen.
From the "Export data" popup window, you can customize these export parameters:
For more information refer to Kili data format.
Updated about 2 months ago