Definitions

Annotation

The action of creating metadata to characterize the asset and allow it to be used to train machine learning models. For example, identifying a house on a satellite image, or attaching a category to a named entity on a document.

Also:

The metadata created from the labeling work on assets. For example, a box delimiting the house on the satellite image, or the selection of textual information related to its typology of named entities.

Multiple annotations can be added on a given asset, to create a label.

Asset

A file/document. This could be a photograph, a satellite image, a video, a PDF, an email, etc.

Consensus

A quality parameter to measure the agreement between several annotations for a given job made on the same asset by different labelers. Setting a minimum consensus value to a project ensures consistency between the annotators and the best data quality for your project.

Honeypot

A quality parameter to measure the agreement between annotations made by an expert labeler and the annotations made by other labelers on a certain asset. Such a pre-annotated asset is sometimes referred to as the gold standard.

Instructions

Instructions are guidance available in the annotation interface to help labelers complete their tasks. They can be defined at the project and job levels.

Interface

The graphical user interface configured at the beginning of a project and made available to users to enable them to perform the annotation tasks.

Job

Labeling (or annotation) jobs are labeling tasks which are associated with specific tools.
For example, each one of these can be considered a Kili labeling job:

• Classification task with a multi choice dropdown
• Object detection task with polygon tool
• Named entities recognition task

Label

The combination of all annotations created on an asset. For example, all houses identified on a satellite image, or all annotated fragments of text in a document.

Labeling

The action of creating metadata to characterize the asset and allow it to be used to train machine learning models. For example, identifying a house on a satellite image, or attaching textual information to a type of named entity on a document. Also called: annotation.

Organization

An organization contains users who can create projects and easily collaborate.
The number of labeled assets at the organization level is the sum of labeled assets over all members from projects belonging to the organization. A project belongs to an organization when the author is a member of the organization.
Similarly, the number of hours at the organization level is the sum of labeling hours over all members from projects belonging to the organization.

Project

A project is the combination of:

• A dataset (collection of assets to be annotated)
• An interface adapted to the annotation tasks that we want to perform on the dataset
• Members with different roles (for example, labelers and reviewers)
• Settings regarding quality management workflow

Project user

The intersection of a project and a user. For example:

• The number of labeled assets by project user is the number of assets that were reviewed by the user or that contain a default label whose author is the user.
• The total work duration for a project user is the total time spent on each review or default label in the project by a particular user.