How to Tag Multi-Camera Video Segments for Action Recognition

A video segment is a section of a video between a start and an end frame, used to recognize actions. The Mark Attributed Segments on Multi-camera Videos app in the Supervisely video labeling toolbox tags such segments on one video or on pairs of synchronized or unsynchronized camera feeds, side by side. This guide explains the modes and the steps. Multiview video is covered in the docs.

Why do you need to manage video segments?

In busy places such as airports or roads, you often need to mark specific moments and areas in video. Segments let you label just the time ranges and regions that matter, instead of whole recordings.

A segment refers to a section of a video that is demarcated by its starting and ending frames, primarily utilized for recognizing actions within the video. A tag segment acts as an identifier for these segments, applicable either to an individual video or across multiple videos in a multiview setup, with each action commencing and concluding at designated timestamps.

In multiview video analysis, one of the key challenges is ensuring the synchronization of multiple camera feeds. This synchronization affects how action recognition is conducted, which can be on a single video stream or across multiple synchronized or unsynchronized streams. The Mark Attributed Segments on Multi-camera Videos application is designed to handle a variety of use cases, thereby enabling flexible and efficient action recognition across diverse video setups.

How does the app work? Video tutorial

Check out this 3-minute video guide to learn how to use multi-camera action recognition app in Supervisely. It works with both unsynchronized and synchronized video pairs and can show spatial annotations on top of the video streams, to highlight areas on videos from stationary cameras.

Where video segment labeling is used:

• Road traffic: tag vehicles entering and exiting a road section with attributes such as time and vehicle type, and mark regions such as toll lanes to label incidents in them.

• Retail: label areas of a store and tag how customers move through them, to build datasets for foot-traffic and behavior models.

• Airports: draw regions over check-in counters, checkpoints or gates, and tag events such as unattended baggage or queue build-up in each region.

The Supervisely app for tagging and managing video segments is Mark Attributed Segments on Multi-camera Videos, created by the Supervisely Development Team.

Developer Portal

The Developer Portal is an important element of the Supervisely Ecosystem, with tutorials and documentation on automating Supervisely and integrating it with your software. You can create your own apps or modify existing ones.

What is the Mark Attributed Segments on Multi-camera Videos app?

This app is a labeling tool with a dual-panel interface to tag, manage and compare segments of paired videos side by side.

Supervisely App

Mark attributed segments on multi-camera videos

Tag segments (begin and end) with custom attributes on single or multiple videos in dual-panel view

Run in SuperviselyGitHub

Recent enhancements of the App include information widgets, improved video controls, sorting capabilities, and the advanced feature for drawing masks with regions on videos.

Single video vs pair video labeling: which mode to use?

The application operates in two modes:

Mode	Use it for
Single Video Labeling	Analyzing the same video across different times or conditions
Pair Video Labeling	Comparing two distinct video segments

Single Video Labeling for analyzing the same video across different times or conditions

Pair Video Labeling for comparing two distinct video segments

The Mark attributed segments on multi-camera videos application can show spatial annotations such as bounding boxes and masks on top of the video streams.

How to tag multi-camera video segments, step by step

Here's a guide on how to utilize the Mark attributed segments on multi-camera videos application. For deeper understanding of the application functionality, please see the full user manual available in the application description and try it in the free Community Edition.

1. How do you prepare tags?

Start your project by setting up tags. These tags can vary in type (none, text, number, one of...) and are crucial for attaching attributes to segments. They add context to each segment.

It is recommended not to change existing tags. To use the tags in the app correctly, you should prepare tags before launching the app. You can also add new tags later (don’t use existing or removed tag names). Just restart the app to use created tags.

2. How do you launch the app and select videos?

Run the application either from the Ecosystem or directly through your video project or dataset's context menu. Choose video datasets for labeling. The application's dual-panel view improves the labeling process by providing a comparative view. Navigate through the video table and select the left and right videos you aim to label by clicking on the corresponding buttons.

3. How do you create regions?

After selecting the videos, click on the DRAW REGIONS ON VIDEO button to display a mask with figures.

Algorithm for creating masks with regions on a video:

1. Define the necessary number of classes that correspond to the required region names (Ex: Region 1; Region 2).

2. Open the Supervisely video labeling toolbox.

3. Select the class with required name and draw regions on the first frame of the video.

   

4. If you've introduced new classes, a quick restart of the app might be needed to reflect these changes.

Preview and adjust your segments at any time by clicking the SHOW ALL SEGMENTS button.

4. How do you tag and manage segments?

Now it's time to add tags to your segments and manage them efficiently. Users can easily manage segments with existing tags or create new ones. Customize segments with unique identifiers, tags and timestamps for a thorough labeling process.

To start tagging, i.e. create or delete tags segments, use the START SEGMENTS TAGGING button.

To manage selected segment attributes, simply click the EDIT button on the segments table.

5. Where are the results stored?

All your segmented data is efficiently stored in JSON format in Team Files. This format captures metadata details such as video IDs, names, timestamps, and user information for easy access and collaboration among team members, learn more in app readme.

Note: All segment attributes are created and added to TagCollection object using the meta tags of current project and stored in a file after serialization to json. All segment attributes are extracted from json files and serialized into TagCollection object to manage them.

Conclusion

For labeling objects and tags on a single video, see Video Annotation 3.0. For video synced with a map and sensor charts, see Video + GPS Telemetry.

Supervisely apps can be changed or extended through the Developer Portal, so you can adapt how segments are managed to your task. For team deployments, on-premise or cloud, see Enterprise.