MBPTrack Tutorial: 3D Point Cloud Object Tracking for LiDAR & Radar
Max TeselkinUpdated Track a 3D object through point cloud frames with MBPTrack, a memory-based 3D single object tracking model, and run it in Supervisely's LiDAR toolbox.
MBPTrack is a neural network for 3D single object tracking: given a 3D bounding box in the first frame, it follows that object through the next frames of a LiDAR or radar point cloud sequence. Supervisely runs it as an app, so you can track cuboids inside the 3D point cloud labeling toolbox and speed up labeling.
This tutorial covers how 3D point cloud tracking works and how to use it in the annotation toolbox. See the point cloud episodes docs for the labeling interface.
Computer vision industry has been developing rapidly over the past few years, which has led to the emergence of new methods of visual information representation and its processing. Three dimensional (3D) representation of surrounding environment began to be widely used in such promising areas as robotics and autonomous driving.
Modern neural network architectures allow to efficiently process 3D data. One of the most demanded methods of 3D LiDAR or Radar data processing is 3D single object tracking.
This video tutorial explains all steps from connecting your computer to running the MBPTrack model and using it inside 3D point cloud annotation tool to significantly speedup and automate manual labeling.
What is 3D single object tracking?
3D single object tracking is a computer vision task dedicated to tracking of a single object in three dimensional space based on an initial 3D bounding box. This task is traditionally performed using point clouds - discrete sets of data points in 3D space. Point clouds are generated by LIDARs (light detection and ranging).
Point clouds provide valuable information about surrounding environment and target object surface. At the same time point clouds can be characterized by an increasing point sparsity with distance and inhomogeneous structure, what makes 3D single object tracking task quite non-trivial.
3D bounding box of an object in a point cloud
Why was MBPTrack a leading 3D object tracker in 2023?
Most of neural network architectures for 3D single object tracking were usually based on the Siamese paradigm, which takes search area in the current frame and the crop of target template from the previous frame as an input, and then localizes the target in the current frame using a localization network such as RPN (Region Proposal Network).
A different approach for 3D object tracking, MBPTrack, was published on arXiv in March 2023. Unlike previous existing approaches, MBPTrack uses both temporal and spatial contextual information in the 3D single object tracking task with the help a memory mechanism.
The authors report state-of-the-art results at publication on the popular 3D single object tracking benchmarks KITTI, nuScenes and Waymo Open Dataset:
KITTI dataset leaderboard
nuScenes & Waymo datasets leaderboards
What is MBPTrack and how does it work?
MBPTrack is a powerful neural network architecture for 3D single object tracking. MBPTrack uses a memory mechanism to process past information and a localization head for performing coarse-to-fine bounding box prediction. In this architecture past frames with targetness masks are used as an external memory, and a transformer-based module propagates tracked target cues from the memory to the current frame.
For accurate object localization, MBPTrack first predicts the bounding box center via Hough voting (a technique used in computer vision to detect patterns in an image that can be represented by mathematical curves). By leveraging box priors given in the first frame, reference points are being adaptively sampled around the target center that roughly cover the target of different sizes with the help of a coarse-to-fine network, named BPLocNet. Then, dense feature maps are obtained by aggregating point features into the reference points, where localization can be performed more effectively.
MBPTrack architecture
You can find more details about MBPTrack architecture in original paper.
When to use MBPTrack vs interpolation?
| MBPTrack (tracking) | 3D object interpolation | |
|---|---|---|
| You label | One cuboid in the first frame | Cuboids in the first and last frames |
| Who fills the frames | A neural network | Spline interpolation |
| Best for | Objects with harder motion | Objects with simple motion |
See the interpolation tutorial for the second option.
How to run MBPTrack in Supervisely
Follow the steps below to run the model and interact with it during annotation of point cloud episodes with 3D cuboids.
Step 1. Connect your GPU
In Supervisely it is easy to connect your own GPU to the platform and then use it to run any neural networks on it for free. To connect your computer with GPU, please watch these videos for MacOS, Ubuntu, any Unix OS or Windows.
Step 2. Run the app to deploy MBPTrack model
MBPTrack 3D Point Cloud Tracking
Deploy MBPTrack as REST API service
Run MBPTrack 3D Point Cloud Tracking app on your computer, select one of the pre-trained checkpoints and press Run button. There are checkpoints for tracking numerous different objects in point clouds such as cars, pedestrians, trucks, vans, trailers, buses, cyclists.
Go to Neural Networks page and find 3D Point Cloud tracking category.
Step 3. Use MBPTrack in Supervisely 3D Point Cloud labeling tool
Open 3D Point Cloud labeling tool and create input cuboid, do not forget to set correct cuboid direction as shown below:
After that select input cuboid for tracking, define number of frames to track and press Track button.
Conclusion
Supervisely Ecosystem provides modern ways of labeling any type of data for computer vision, including 3D point clouds. In this tutorial we learned about performing 3D single object tracking using the MBPTrack neural network architecture in Supervisely.
MBPTrack will be an excellent choice for improving speed and quality of data labeling in three dimensional space. Sign up and try to label your point clouds for free in Community Edition.
Frequently asked questions
What is 3D single object tracking?
It tracks one object in 3D space through a sequence of frames, starting from an initial 3D bounding box. It is usually done on point clouds generated by LiDAR.
What is MBPTrack?
MBPTrack is a neural network for 3D single object tracking. It keeps past frames with targetness masks as an external memory, uses a transformer-based module to pass target cues to the current frame, and predicts the box coarse-to-fine.
How well does MBPTrack perform?
The MBPTrack paper (arXiv, March 2023) reports state-of-the-art results on KITTI, nuScenes and Waymo Open Dataset at the time, and 50 FPS on a single RTX3090 GPU.
Which objects can I track in Supervisely?
The pre-trained checkpoints cover cars, pedestrians, trucks, vans, trailers, buses and cyclists.
What do I need to run MBPTrack?
A computer with a GPU connected to Supervisely, and the MBPTrack 3D Point Cloud Tracking app. Pick a checkpoint and press Run.
When should I use interpolation instead of tracking?
If the object's trajectory is simple, label the first and last frames and use 3D object interpolation. Use MBPTrack when the motion is harder to predict.
