Interactive AI Segmentation: Label Objects With the Smart Tool
Lisa UspenyevaDraw a box, add a few clicks and get a mask from SAM, RITM or ClickSEG. Learn the Smart Tool workflow, hotkeys and how to fix or retrain its predictions.
Interactive AI segmentation lets you draw a box around an object and get a mask from a neural network, then refine it with positive and negative clicks. The Smart Tool in the Supervisely image labeling toolbox does this with models such as Segment Anything, RITM and ClickSEG. This guide covers the workflow, hotkeys and custom models. See also AI-assisted labeling and the Smart Tool docs.
What is the Smart Tool?
The Smart Tool is an interactive segmentation tool. A neural network predicts a mask from your box and clicks, so you draw less by hand.
Which models does it use? The Smart Tool works with interactive segmentation models integrated into Supervisely, such as RITM, Segment Anything and others, from our Ecosystem. Quality, accuracy and speed depend on the chosen model.
RITM interactive segmentation SmartTool
State-of-the art object segmentation model in Labeling Interface
Serve Segment Anything in High Quality
Run HQ-SAM and then use in labeling tool
The Smart Tool is useful for tasks that involve selecting areas with complex contours or a large number or variety of objects. Where the model predictions shall be amended, a combination with such complementary tools as Brush and Pen helps. They correct imperfections in the masks created by the neural network. See also other annotation tools.
The Smart Tool can be used in many domains, including:
- Urban Planning and Infrastructure: the Smart Tool is great for marking up urban scenes and infrastructure designs. Whether you're involved in city planning or site development, it is excellent at accurately annotating structures, roads, and utilities.
- Medical: the Smart Tool facilitates the precise delineation of organs, tissues, and anomalies, thereby contributing to accurate diagnostics and comprehensive treatment planning and also helps to accurately segment medical instruments, which contributes to the development of robotic surgery.
- Agriculture: farmers may rely on AI tools to assess crop health, plan harvests efficiently, and minimize waste through targeted interventions, enhancing overall agricultural productivity. The Smart Tool enables precise annotation of fruits, vegetables, weeds, crops, pests and much more, significantly improving the quality assessment of agricultural produce.
- Automotive: The Smart Tool is really helpful for showing and dividing car damage, like dents and chips. It's also good at finding spots where the tires are worn, giving a better idea of how the vehicle equipment is overall.
- Infrastructure Inspection: In infrastructure inspection, the Smart Tool can be employed to detect damage and hazards on roads, such as potholes or obstructions. This is particularly critical for the safety of autonomous vehicles, which leverage this information to navigate safely and avoid potentially dangerous areas.
Smart Tool use cases across industries
Video Tutorial for Smart Tool in the New Labeling Toolbox
A short video tutorial on the Smart Tool in the new labeling toolbox.
Smart Tool: key features
You can try the Smart Tool for free in the Supervisely Community Edition.
How do you create a mask with the Smart Tool?
Creating a mask with the Smart Tool is simple:
-
Click anywhere in the scene to make the top point of the rectangle.
-
Then click again for the bottom point in the opposite direction.
-
The neural network will attempt to identify the main object within that rectangle.
-
To adjust the neural network's predictions, place and drag positive 🟢 and negative 🔴 points around the entire area inside the bounding boxes.
Click and put a positive point on the part of the object you want IN the final mask. For parts of the background you want to leave OUT of the final mask, put a negative point by clicking.
-
Finalize the process by pressing the
SPACEhotkey.
Pro tip: The bounding box around an object doesn’t need to be precisely close to the object’s boundaries. We recommend placing the initial bounding box with about a 10% padding from the object boundaries. This way, the model will have more context, and segmentation will be more accurate near the object boundaries.
Can the Smart Tool segment complex objects?
Regardless of the number or complexity of objects in the image, Smart Tool can manage them with its neural network algorithms. The accuracy and satisfactory performance of the algorithm are significantly influenced by your involvement in editing the mask.
Can you segment simple objects with one click?
Thanks to the accuracy provided by the Smart Tool, objects with simple shapes often require no further adjustments. Simply create bitmap masks and enjoy!
How do you edit Smart Tool predictions by hand?
Neural networks can sometimes struggle with objects they were not trained on or due to other factors. If you need to manually edit the Smart Tool predictions, you can use the Brush, Eraser and Pen tools to remove unnecessary parts or add new ones.
To add a part of the object, select the shape and draw the missing part. To delete a part of the object, press and hold the SHIFT key while repeating the previous steps. The algorithm is the same for both tools.
How do you change an object class with a right-click?
For a seamless transition between object classes, initiate a right-click on the selected item and choose the desired option from the pop-up list. Beyond class changes, this menu provides convenient options for deleting or hiding an object, further streamlining your annotation workflow.
How do you switch between models?
Neural Networks have various designs and were trained on diverse data, leading to different performances with different types of data. In some cases, it might be helpful to use different neural network models for the Smart Tool within the same dataset.
Switching between interactive segmentation models for the Smart Tool is easy. You can use the ID button in the left toolbar.
Switch between interactive object segmentation models
On the Supervisely Community instance, ClickSEG is deployed by default. You can also run any pre-trained model from an interactive segmentation category (like Segment Anything, RITM, etc.) on your own computer with a GPU, or train and deploy an interactive segmentation model on a custom training dataset.
On an Enterprise instance, administrators choose the interactive segmentation models and share them with all users. To get the best experience of the Supervisely platform check out our blog post for more details about how to train a custom interactive segmentation model (plants segmentation use case in agriculture) and deploy it on your server.
How to run any serving App
Can you train the Smart Tool on your own data?
Popular neural network models, such as Segment Anything, were trained on a large portion of data; however, these models are general-purpose and may face difficulties with specific tasks, such as segmenting agricultural images in low quality or detecting road cracks.
In Supervisely, you have the capability to train the Smart Tool specifically for your data directly within the platform, and the best part is, that no coding is required! See these posts on the topic.
-
How to Train Smart Tool for Precise Cracks Segmentation in Industrial Inspection
-
Automate manual labeling with custom interactive segmentation model for agricultural images
-
Lessons Learned From Training a Segmentation Model On Synthetic Data
Visualization Settings
Tailor your settings to suit your needs by adjusting opacity levels, selecting random colors for objects while annotating and many other useful features.
Hotkeys
Control the Smart tool 0 more efficiently with HOTKEYS.
Detect object inside BBox - CLICK LBM + CLICK LBM |
Add positive point - CLICK LBM |
Add negative point - SHIFT + CLICK LBM |
Remove feedback point - ALT + CLICK LBM |
Drag rectangle - ALT + HOLD LBM |
Drag rectangle - ALT + ARROW KEYS |
Finish & start new - SPACE |
LBM stands for Left Mouse Button
Use ID button to change model. You can deploy more interactive segmentation models in Ecosystem. |
|---|
| Deployed model: Example - Serve ClickSEG#47962. |
| Scene Navigation |
|---|
Zoom with MOUSE WHEEL. Hold RIGHT MOUSE BUTTON to move scene. |
Smart Tool vs OWL-ViT: when to use which?
The Smart Tool segments one object at a time: you give it a box and clicks, and it returns a mask. OWL-ViT is an object detector: you give it one reference box or a text prompt, and it returns boxes for similar objects on other images, without training. Use the Smart Tool for precise masks and OWL-ViT to pre-label many objects. Read the OWL-ViT guide for the detection workflow, or HQ-SAM vs SAM to pick a segmentation model.
Conclusion
The Smart Tool turns a box and a few clicks into a mask, and you can fix the result with Brush and Pen. Try it for free in the Community Edition.
Frequently asked questions
What is interactive segmentation?
Interactive segmentation means you give the model feedback and it predicts a mask. In the Smart Tool you draw a bounding box around an object, and the neural network identifies the main object inside it. You then add positive and negative points to correct the mask.
How do I create a mask with the Smart Tool?
Click once for the top point of a rectangle and once for the opposite corner. Add positive points on parts that should be in the mask and negative points (SHIFT + click) on the background, then press SPACE to finish and start a new object.
Which models does the Smart Tool use?
It can use interactive segmentation models integrated into Supervisely, such as RITM, Segment Anything and ClickSEG. The quality, accuracy and speed depend on the chosen model. You can switch models with the ID button in the left toolbar.
Can I train the Smart Tool on my own data?
Yes. You can train a custom interactive segmentation model on your training dataset directly in the platform, without coding, and deploy it for the Smart Tool. This helps with tasks where general models struggle, for example low-quality agricultural images or road cracks.
How do I fix a wrong mask?
Add or move feedback points first. For manual edits use the Brush, Eraser and Pen tools to add missing parts or remove extra ones. Hold SHIFT to delete a part instead of adding it.
Is the Smart Tool free?
You can try it for free in the Supervisely Community Edition. On a private Enterprise instance, administrators choose the interactive segmentation models and share them with all users.
