From 3D scans to training data

Reconstruction and projected labels from our paper, Figure 1.

I co-founded Stray Robots and co-developed its computer vision and annotation tools. Kenneth Blomqvist and I contributed equally to the 2021 preprint 3D Annotation Of Arbitrary Objects In The Wild.

The idea

Labeling each image separately repeats work across views of the same object. Our tool reconstructs the scene from RGB-D observations, lets a person annotate it in 3D, and projects those annotations back into the recorded images. It does not require a CAD model of each object.

The pipeline

1. Recover camera poses using ORB-SLAM3.
2. Integrate RGB-D frames into a TSDF volume and extract a mesh with Open3D.
3. Place and resize object boxes in the 3D annotation interface.
4. Use camera geometry to generate per-frame boxes and segmentation masks.

Evaluation

The paper compares generated labels with manual annotations across 34 scenes and eight object categories, and tests a detector trained on the resulting data. It reports roughly 90% IoU agreement for segmentation and 2D boxes.

The trade-off is dependence on camera-pose and reconstruction quality. The label projection reuses that geometry across frames, so errors can propagate across many generated labels.