Bounding Box Annotation: How to Label Boxes for Object Detection

Bounding box annotation means drawing a rectangle around each object in an image and assigning it a class. These boxes train object detection models to find and localize objects. In Supervisely you draw them with the image labeling toolbox and can pre-label with models like OWL-ViT and YOLOv8.

What is object detection and what are bounding boxes?

Object detection is a computer vision task: identify and localize objects such as people, animals and vehicles. A detector outputs a rectangle and a class for each object. Rectangles that tightly enclose an object are bounding boxes. They are defined by coordinates that give the position and size of the object.

Image classification only says whether an object is present, with a probability. Detection also says where it is.

Object detection is used across industries:

• Autonomous vehicles: road signs, pedestrians, vehicles and obstacles.
• Agriculture: plant growth, disease and fruit condition.
• Medical diagnostics: pathologies, tumors and instruments in images or surgical video.
• Retail: product availability on shelves and customer behavior.
• Security: public places and buildings, suspicious activity on video.
• Energy: inspection and maintenance of infrastructure such as power lines.

Take a pothole monitoring system. Boxes tell you where each pothole is, how many there are and roughly how big, so repairs can be planned.

Polygon vs bounding box: which one should you use?

Use bounding boxes when the model must find objects, and polygons or masks when it must know their exact shape. Boxes cost far less to label. The difference shows in published numbers: the Cityscapes team needed more than 1.5 hours per image for fine polygon annotation and quality control, and under 7 minutes per image for coarse polygons. Drawing a single box takes seconds: a study reports 35 seconds for a high-quality box drawn the classic way on ILSVRC and about 7 seconds with extreme clicking.

	Bounding box	Polygon	Mask (brush, Smart Tool)
Shape	Rectangle, 2 corners	Any number of points	Pixels
Typical task	Object detection, counting	Instance and semantic segmentation	Segmentation, thin or irregular objects
Labeling effort	Lowest	Medium to high	Medium to high, lower with AI help
Background in the label	Yes, in the corners	Little	None
Good for	Large datasets, fast iteration	Objects with clear edges	Complex shapes
Supervisely tool	Bounding Box	Polygon	Brush, Mask Pen, Smart Tool

A practical rule: start with boxes if your detector only needs location. Switch to polygons if boxes would hold mostly background (thin, diagonal or overlapping objects) or if you need the shape for measurement or segmentation. Switching later is possible. Smart Tool and models can help turn a box into a mask, and one project can hold several shapes. For a method comparison see manual segmentation approaches.

For scale, the COCO dataset has 2.5 million labeled instances in 328k images, and the Segment Anything team collected about 1 billion masks on 11 million images using a model in the data collection loop. The lesson is the same: model help is how large labeled sets get built.

How to label bounding boxes step by step?

Manual labeling matters most where accuracy is key, for example in medicine, and for test and validation sets. It is slow on large datasets, so use every shortcut. The redesigned Supervisely image labeling toolbox has several.

Watch the 5-minute video, then follow the steps below.

What you need: a Supervisely account (the free Community Edition needs no credit card), a project with images, and a Rectangle class for each object type. Read the docs for the image labeling toolbox for the full list of tools and hotkeys.

1. Create boxes with minimal clicks. Click two corners of the object. To start the next object you do not need to finish the previous one by pressing SPACE; just set the new points.

2. Auto-select the most recent class. Move the cursor over an object of the desired class and the active class changes automatically. This helps when an image holds many classes.

3. Edit with Auto-select. Auto-select also lets you fix existing boxes, including those predicted by a neural network, without selecting each object and switching tools.

4. Use the crosshair for accurate corners. The crosshair takes the color of the selected class, which makes positioning easier.

5. Randomize figure colors. Change the color of an object with one click as often as you like. It helps to tell apart close boxes of the same class.

6. Switch classes fast. Select a labeled object and choose a new class from the Figures menu, or create one if it does not exist.

How to pre-label bounding boxes with a model?

Semi-automated and automated tools mix model predictions with human review. Semi-automated labeling fits diverse objects or single-class datasets, while full automation is the fastest and simplest and fits counting or presence checks where small box errors do not matter. In both cases keep a manual check on a sample.

OWL-ViT: boxes from a text prompt or one example

OWL-ViT (Vision Transformer for Open-World Localization) is an object detector trained on image-text pairs. In text prompt mode it detects objects described by a query. In reference image mode it finds objects similar to one you framed with a box. Read the OWL-ViT guide for details.

1. Connect your GPU. OWL-ViT is a large network, so a GPU gives much faster predictions on thousands of images. This video shows how.

2. Run the model. Launch the Serve OWL-ViT app. The NN image labeling app then applies it to the whole image or a region of interest, with configurable confidence thresholds and class selection.

   

   Supervisely App

   Serve OWL-ViT

   Class-agnostic interactive detection for auto-prelabeling

   Run in SuperviselyGitHub

   

   Supervisely App

   NN Image Labeling

   Use deployed neural network in labeling interface

   Run in SuperviselyGitHub

3. Use it in the labeling tool. Open the labeling interface, launch NN image labeling and connect to the model. Label one example with a box (or define a text prompt), apply the model to the open image, correct the predictions, and go to the next image. To update the reference, pick a new object and click Apply model to ROI.

   Annotate only one reference example and OWL-ViT model will automatically detect similar objects on other images

4. Predict on all images at once. The app below configures OWL-ViT, previews results and pre-labels the dataset.

   

   Supervisely App

   Apply OWL-ViT To Images Project

   Class-angnostic object detection model

   Run in SuperviselyGitHub
   1. Select the input project or dataset and connect to the deployed model.
   2. Choose Reference Image mode (draw a box around one instance) or Text Prompt mode (describe the objects).
   3. Set inference options and press Predictions Preview to check random images.
   4. Apply. A new project is created with the input images and the generated annotations.
   User-friendly GUI for OWL-ViT batch inference on all images in your dataset

YOLOv8: boxes from a trained detector

YOLOv8 is built to run fast, even on edge devices in robotics, and predicts boxes, classes and scores. To train your own on your labeled data and deploy it without code, see the no-code YOLOv8 tutorial.

Supervisely App

Serve YOLOv8 | v9 | v10 | v11

Deploy YOLOv8 | v9 | v10 | v11 as REST API service

Run in SuperviselyGitHub

1. Connect your GPU. See this video.

2. Deploy the model. Use Serve YOLOv8 to deploy pretrained or custom weights in a few clicks. See the video tutorial or the app readme.

3. Pre-label while you annotate. Launch NN Image Labeling, connect to the model and press Apply model to image. Correct the errors by hand.

   

   Supervisely App

   NN Image Labeling

   Use deployed neural network in labeling interface

   Run in SuperviselyGitHub
   Use a model right inside the image labeling interface to speed up the annotation process

4. Apply the model to all images. The Apply NN to Images Project app lets you set inference options, choose which model classes to keep, and preview predictions.

   

   Supervisely App

   Apply NN to Images Project

   NN Inference on images in project or dataset

   Run in SuperviselyGitHub
   1. Select the input data.
   2. Connect to the deployed YOLOv8 model.
   3. Choose the classes to keep. Other classes are ignored.
   4. Set the inference options.
   5. Press Preview to see predictions on a random image.
   6. Apply. A new project is generated with the images and the YOLOv8 annotations.
   Apply the custom model to all images in your project in a few clicks

Which methods are fastest? Manual, semi-automated or automated

Method	Human effort	Speed	Best for
Manual boxes	High	Slowest	Small sets, test and validation data, medical images
OWL-ViT with one example or a prompt	Medium	Fast	New classes with no trained model
YOLOv8 pre-labeling	Low, plus review	Fastest	Large datasets with a trained model, counting

Whatever you pick, review a sample. Pre-labeled data can have lower accuracy than manual labels, so measure on a hand-labeled validation set.

How to export bounding boxes for training?

Detection frameworks differ in label format. For example, Ultralytics YOLO expects one text file per image with one row per object: class, x center, y center, width and height, normalized to the 0-1 range. Supervisely keeps boxes in one project format and exports them through Ecosystem apps to the format you need, so you do not relabel when you change framework. For custom pipelines, use the API and SDK.

For teams, assign boxes to labelers with labeling jobs and check results with quality assurance. Supervisely can run in the cloud or on-premise, see the Enterprise page.

What next?

Boxes are the first step for most detection projects. If you need shapes, continue with polygon annotation or the Smart Tool. To label faster in general, read how to speed up image labeling.