This topic will guide you through a typical Kili project lifecycle. You will learn how to: 1. [Set up a project](#setting-up) 2. [Add assets to your project](#add-assets) 3. [Build project ontology using tools made available by Kili](#build-project-ontology) 4. [Add users to project and assign their roles in the project](#add-users) 5. [Properly configure quality management metrics](#qa-metrics) 6. [Review labeled assets](#review-assets) 7. [Analyze project KPIs](#analyze-kpis) 8. [Export annotations](#export-annotations)
*** ## Setting up a project A **project** is a space where the work on data is done. This is where you configure the labeling jobs, import assets, supervise the production of labels and export the completed work. Projects are created inside an **organization**. Only invited users can contribute. Regarding your workflow and operational constraints, you can adopt two different approaches to labeling in Kili: * **One-off project**: Upload data > label > export labels. * **Continuous work**: When your initial labeling is done, you still regularly add new assets to label and improve your model. You can either [create a project from the Kili graphical user interface](doc:creating-a-new-project#create-ui) or [create a project programmatically](https://python-sdk-docs.kili-technology.com/latest/sdk/tutorials/basic_project_setup/) using the [Kili Python SDK](https://python-sdk-docs.kili-technology.com/). For information on how to create a project from Kili CLI, refer to our [Command Line Interface documentation](https://python-sdk-docs.kili-technology.com/latest/cli/) Next step: [Adding assets to your project](#add-assets).
*** ## Adding assets to project In Kili, an asset can be a file or a document. This could be a photograph, a satellite image, a video, a PDF, an email, etc. For a full list, refer to [Supported file formats](doc:supported-file-formats). You can add assets located on: * [Your local workstation](doc:adding-assets-to-project#workstation) * [Remote storage (list of links)](doc:adding-assets-to-project#web) * [Remote storage (integration)](doc:adding-assets-to-project#integration) > 📘 > > * For information on how to add assets located on your local servers, refer to [Adding assets located on premise](doc:adding-assets-to-project#on-prem).* Kili does not allow for hybrid storage with assets hosted in Kili's cloud storage and remote Cloud storage. To prevent unexpected app behavior, when you use remote Cloud storage in your project, other options to upload data are disabled. You can also add assets programmatically. For information on adding assets through [Kili's Python SDK](https://python-sdk-docs.kili-technology.com/) , follow [this tutorial](https://python-sdk-docs.kili-technology.com/latest/sdk/tutorials/importing_assets_and_metadata/). For information on how to add assets through the Kili CLI, refer to our [Command Line Interface documentation](https://python-sdk-docs.kili-technology.com/latest/cli/). > 📘 > > The maximum number of assets in a project is 25,000. If you need to add asset metadata, refer to [Adding asset metadata](doc:adding-asset-metadata). For additional information, refer to [Asset lifecycle in Kili app](ref:asset-lifecycle). Next step: [Building project ontology](#build-project-ontology).
*** ## Building project ontology In Kili, labeling (or annotation) **jobs** are **labeling tasks** which are associated with **specific tools**.\ For example, each one of these can be considered a Kili labeling job: * Classification task with a multi choice dropdown * Object detection task with polygon tool * Named entities recognition task Kili jobs may contain nested subjobs, for example: * Classification tasks with nested conditional questions * Object detection tasks with nested transcription containing additional details Some jobs types may be interdependent. For example, available relations in a named entities relation job will depend on how classes were defined in a named entities recognition job. Additionally, each job can be either required or optional. > 📘 > > Creating projects with lots of labeling jobs is not efficient. Project complexity and potential performance issues mean that it is harder for members to complete all the jobs. Such a project is also harder to review.\ > It is better and more efficient to split your jobs in multiple projects, and then to concatenate your labels later, even if this operation must be done outside Kili.
Kili provides the following machine learning job types: * [Classification](doc:classification-jobs) * [Object/Entity Detection](doc:objectentity-detection-jobs) * [Object/Entity Relation](doc:objectentity-relation-jobs) * [Transcription](doc:transcription-jobs) Availability of specific job types depends on your project asset type: \ Available labeling job types per asset type | Labeling job type | Image | Text | PDF | Video | | -------------------------- | :---: | :--: | :-: | :---: | | Classification | ✓ | ✓ | ✓ | ✓ | | Object detection | ✓ | | ✓ | ✓ | | Object relation | ✓ | | ✓ | ✓ | | Named entities recognition | | ✓ | ✓ | | | Named entities relation | | ✓ | ✓ | | | Transcription | ✓ | ✓ | ✓ | ✓ | You can customize your interface through the [UI-based interface builder](doc:customizing-the-interface-in-kili-ui) or through [JSON settings](doc:customizing-the-interface-through-json-settings). If you need to add project metadata, refer to [Adding project metadata](doc:customizing-the-interface-through-json-settings#proj-metadata). Next step: [Adding project members and assigning roles](#add-users)
*** ## Adding users to project and assigning their roles in the project Users, often called Members in the Kili app, are managed at two levels: * **Organization** they belong to * **Projects** they work for On Kili Technology platform, a user (identified by email) can only belong to a single organization.\ Inside an organization, users can be part of multiple projects, with equal or different roles. > 📘 > > Organization admins are admins of all the projects within an organization. For additional information, refer to: * [User roles in organization](doc:user-roles-in-organization) * [User roles in project](doc:user-roles-in-projects) * [Managing members of an organization](doc:managing-members-of-an-organization) Depending on user role, users will have access to different features. Refer to the following: * [Projects list](doc:projects-list#permissions-per-role-project) * [Queue page](doc:queue-page#permissions-per-role-queue) * [Analytics page](doc:analytics-page#permissions-per-role-analytics) * [Settings page](doc:settings-page#permissions-per-role-settings) > 📘 > > Maximum number of members per project is limited to 50. If you need more, contact us at [support@kili-technology.com](mailto:support@kili-technology.com). For hands-on examples on how to programmatically add project members using [Kili's Python SDK](https://python-sdk-docs.kili-technology.com), refer to [this tutorial](https://python-sdk-docs.kili-technology.com/latest/sdk/tutorials/basic_project_setup/). Next step: [Configuring quality management](#qa-metrics).
*** ## Configuring quality management metrics Kili's built-in **automatic workload distribution** mechanism facilitates load management, allows for faster project annotation, and prevents duplicating work.\ For example, if several users collaborate on an annotation project, Kili app distributes the data to be annotated to each project member so that **each annotator processes unique data**. The same data will not be annotated twice, unless [Consensus](doc:consensus-overview) is activated. To help labelers and reviewers meet the labeling requirements for the project, you can [set labeling instructions](doc:adding-instructions). Your instructions can also serve as reference, if questions arise. Labelers who are unsure about a specific label can [ask a question](doc:asking-questions). Reviewers, team managers and project admin will be notified and will have the option to answer and close the question.\ The number of open questions is shown on the issue button. ![](https://files.readme.io/ee553a5-questions-notification.png "questions-notification.png") For information on how to handle questions asked by labelers, refer to [Handling questions and issues](doc:handling-questions-and-issues). To further increase your label quality, use the following tools: * [Review queue automation](doc:review-queue-automation) * [Consensus](doc:consensus-overview) * [Honeypot](doc:honeypot-overview) For hands-on examples on how to programmatically configure QA workflows in Kili using the [Kili's Python SDK](https://python-sdk-docs.kili-technology.com/), refer to [this tutorial](https://python-sdk-docs.kili-technology.com/latest/sdk/tutorials/set_up_workflows/). Next step: [Reviewing labeled assets](#review-assets).
*** ## Reviewing labeled assets You can review assets in two ways: * You [manually select the annotated assets](doc:reviewing-labeled-assets#manually). * Assets are [randomly selected](doc:review-queue-automation) for review from the review queue. In both cases, the review interface is similar to the labeling interface and allows you to [review labeled assets](doc:reviewing-labeled-assets#reviewing) and then: * make corrections, if necessary * [send assets back to the queue](doc:reviewing-labeled-assets#send-back) To be able to quickly find any asset, you can use [Kili's advanced filtering features](doc:filtering-assets-through-custom-search-queries). To provide feedback to labelers, reviewers can **add issues**.\ The number of open issues is shown on the issue button and is also available from the [Analytics page](doc:analytics-page). ![](https://files.readme.io/ebc2741-issues-and-questions-icon.png) For information on how to find, add, and resolve issues, refer to [Handling questions and issues](doc:handling-questions-and-issues). Next step: [Analyzing project KPIs](#analyze-kpis).
*** ## Analyzing project KPIs The [Analytics page](doc:analytics-page) provides an overview of the project progress. By using the tabs on the left side of the screen, you can select whether you want to access: * [A general project overview](#overview) * [Project members' performance metrics](#performance) * [Quality insights](#quality) ### Overview Overview contains a snapshot of the project. You can trace what's already been done and access a quick summary of open questions and skipped assets. Use the visual progress bar to quickly check how many assets have been annotated, reviewed, or still left to do and then click on a specific category to immediately jump to the correctly filtered **Explore** view. ![](https://files.readme.io/3a18295-analytics-overview.gif) Additionally, you have access to an overall class balance, that is information on what percentage of the total number of annotations are labels given to a specific class. For extra granularity, you can select either the whole project or drill down to specific labeling jobs. ![](https://files.readme.io/d913cfc-analytics-class-balance.gif) ### Performance From the **Performance** tab, you can check how many assets are being labeled and reviewed in a daily, weekly and monthly time frames. Note that we're taking into account **actions**, not just assets. So if two or more project members labeled (for example as part of consensus) or reviewed the same asset, the number will be higher. ![](https://files.readme.io/7f98803-analytics-productivity.gif) From here, you can also access detailed labeling statistics that you can later export as a `.csv` file and process outside Kili. ![](https://files.readme.io/e3713d3-analytics-performance-team-perf2.png) Through performance metrics you can assess the velocity of your team after the project has been running for some time. How is it evolving per asset? Are some of the assets outliers? If so, why? Is the performance consistent with what you were expecting? How does it affect your project timeline? Once you have all this information, you can then deep dive and analyze available data per annotator to see if you can spot any discrepancies and items to focus on. ### Quality Insights From the **Quality Insights** tab, you have access to insights regarding consensus and honeypot metrics (if you set them up in your project). Information is presented per class (with an option to delve deeper, per job) and per labeler. You also have access to an overall total agreement score for consensus and honeypot. For information on how these are calculated, refer to [Calculation rules for quality metrics](doc:calculation-rules-for-quality-metrics). When done, you can click directly on the graph to switch to the **Explore** view and start your review. ![](https://files.readme.io/a546fca-analytics-qa.gif) Use the graphs and scores to target the most relevant assets to focus your review on. For example, if you spot a class with the lowest consensus mark, you can then check on it by using the [Explore view](doc:explore-view) to properly research and address this issue. If neither [Consensus](doc:consensus-overview) nor [Honeypot](doc:honeypot-overview) are set in your project, the **Quality Insights** tab shows the review score per labeler. The review score calculations for each asset are made by comparing the original label (either added by a human or by a model) with its reviewed version. This metric provides additional insight into how well the team is performing. Additionally, the tab contains information on how the review score evolved over time. For information on how Kili app calculates the review score, refer to [Calculation rules for quality metrics](doc:calculation-rules-for-quality-metrics). Next step: [Exporting annotations](#export-annotations)
*** ## Exporting annotations 1. From the project [Queue page](doc:queue-page), select the assets that you want to export. Quick actions menu appears above the asset list. 2. From the Quick actions menu, select `Export`. 3. From the "Export data" popup window, [select your export parameters](#customizing-export) and then click **Export**. 4. Wait for the notification to appear in the top-right corner of the screen.

![](https://files.readme.io/10b9ca1-project-export.gif "project-export.gif") ### Customizing export parameters From the "Export data" popup window, you can customize these export parameters: * [Label format](doc:exporting-project-data#label-format) * [Scope of exported labels](doc:exporting-project-data#label-scope) * (YOLO formats only) [Splitting labels per labeling jobs](doc:exporting-project-data#job-split) {1258}

For more information refer to [Kili data format](doc:data-format). For hands-on, examples on how to programmatically export data from Kili using [Kili's Python SDK](https://python-sdk-docs.kili-technology.com/), refer to [this tutorial](https://python-sdk-docs.kili-technology.com/latest/sdk/tutorials/export_a_kili_project/). *** ## Learn more For an end-to-end example of how to set up a Kili project programmatically using Kili's Python SDK, refer to our [Basic project setup tutorial](https://python-sdk-docs.kili-technology.com/latest/sdk/tutorials/basic_project_setup/).