Using semantic segmentation

The task of image semantic segmentation consists of detecting specific regions of objects within an image.
So in image semantic segmentation, our goal is not only to identify the presence of given objects, but also to identify their positions, dimensions, and shape. This requires work at pixel level in order to classify every single pixel of the object. This can be achieved in an approximate way by drawing the shape of the object.

In Kili app, semantic segmentation can be:

• Standard semantic segmentation
• Interactive semantic segmentation

Whichever method you use, you can then refine your masks with the editing tools.

---

Standard semantic segmentation

To create a semantic segmentation annotation:

1. Select a category.
2. Hover over a specific point of an image, and then press and hold your left mouse button.
3. While holding the left mouse button, draw the shape of the object.
4. To complete the shape, move your pointer to the first point that you clicked on and release the left mouse button.

You can also create a mask with the Brush or the Lasso: see Drawing tools. To refine the shape after drawing it, see Editing masks.

---

Interactive semantic segmentation with SAM 2

Interactive segmentation uses the SAM 2 model to generate high-quality masks automatically. You can choose between two models:

• Rapid (default): get results quickly with a faster model, ideal for rapid processing.
• High-Res: for tasks requiring higher precision, this model delivers greater accuracy at a slower pace.

Getting the first generated mask may take a few seconds, but subsequent masks are computed much faster. You can interrupt the process at any time by pressing the ESC key.

You can generate a mask either by clicking on the object, or by drawing a bounding box around it.

Click mode

1. Select a category in the job viewer (right panel).
2. From the toolbar, select the interactive segmentation tool (the magic wand icon).
3. Click on the object that you want to classify. The object mask is created automatically.
4. Adjust the mask:
   • To add a region, click in the center of the region that you want to add.
   • To remove a region, press and hold Alt/Option and click in the center of the region that you want to remove.
   • To cancel and remove the mask, press the Escape key.
5. Press Enter to validate the mask and immediately start creating another one, without re-selecting the class and the tool.

In this example, the category is "Car without wheels": select the car, then press and hold Alt/Option and click on each wheel to remove it from the mask.

Bounding box mode

Drawing a bounding box lets you target the object more accurately, and is often faster than clicking.

1. Select a category in the job viewer (right panel).
2. From the toolbar, select the interactive segmentation tool.
3. Draw a bounding box around the object. The object mask is created automatically.

Editing the mask

Once created, you can refine the mask with the editing tools.

---

Draw over existing objects

The "Draw over existing object" toggle allows you to choose whether the object you are drawing will overlap with existing ones or not.

---

Drawing tools

The following tools are available in the toolbar, either to create a new mask (select a category in the job viewer, then the tool) or to modify an existing mask in edit mode.

• Brush: adjust the size with the size slider, then press and hold the left mouse button and paint over the region. Creates a new mask, or adds to the mask in edit mode.
• Lasso: press and hold the left mouse button and draw around the region. While drawing, click to lock a point; press Cmd/Ctrl+Z to go back to the previous point and resume drawing from there. To close the shape, return to the first point (the cursor changes when you hover over it). Creates a new mask, or adds to the mask in edit mode.
• Eraser (edit mode only): adjust the size with the size slider, then press and hold the left mouse button and paint over the region to remove from the mask.
• Add / Subtract (edit mode only): include or exclude parts of the shape.

---

Editing masks

To edit a semantic segmentation annotation:

1. Click the annotation to select it.
2. Press Enter (or double-click the mask) to switch to edit mode.
3. Modify the mask with the drawing tools or the node editor.
4. While editing, hold the Spacebar and drag your mouse to pan across the image, if needed.
5. Press Enter again to exit edit mode and save your changes.

Node editor

In edit mode, the mask's nodes are displayed, giving you precise control over its geometry:

• Drag nodes to reshape the mask
• Click a node to delete it
• Hover over a segment to add an intermediate node
• Press the Del key to delete the mask