3D Semantic Segmentation & LiDAR-Camera Fusion

by Jacob Igo

Learning how self-driving perception actually works — building 3D semantic segmentation and sensor fusion from scratch on the Waymo Open Dataset, without the Waymo Python package.

---

Why I'm doing this

I grew up in Phoenix watching Waymo go from a novelty to something I ride whenever I get the chance, and I'm convinced autonomous vehicles are one of the more important things being built right now — safer roads, and genuinely fascinating tech. I want to work in this field, and the best way I know to understand something is to rebuild it myself.

So this is me taking raw sensor data and turning it into a labeled 3D scene, one piece at a time, learning the geometry and the gotchas along the way. It's a learning project, and I'm having a great time with it.

---

The data

Waymo Open Dataset v2.0, read live from the waymo_open_dataset_v_2_0_0 GCS bucket — no local copies, and no Waymo package. I parse the parquet files directly with PyArrow.

Each file is one 20-second driving segment with 5 LiDARs and 5 cameras per frame. LiDAR comes as range images (azimuth / inclination / range), not raw XYZ, so it needs a spherical-to-Cartesian conversion first. Labels are 23 semantic classes, and only laser 1 is labeled.

Folder	What it holds
lidar/	Range-image 3D points
lidar_segmentation/	Per-point labels (laser 1 only)
lidar_calibration/	Extrinsics + beam inclinations per laser
camera_image/	JPEG frames per timestamp
camera_calibration/	Camera intrinsics + extrinsics

---

What works so far

• Range-image decode (spherical → Cartesian)
• Extrinsic transform to a global frame
• Multi-laser fusion into one point cloud
• Segmentation labels decoded & points colored by class
• Memory-safe, timestamp-aligned data loading
• Bird's-eye and interactive 3D (Plotly) visualization
• Scene animations (matplotlib / ffmpeg)
• LiDAR → camera projection
• Fused overlay video (colored by depth)
• Creating & Training PointNet
• Creating & Training PointNet++

---

Progress notes

Longer write-ups — what I learned decoding the data, the sensor-fusion approach, and the model-training story — live in docs/progress-notes.md.

---

Project layout

File	Role
semseg.ipynb	LiDAR-only segmentation pipeline
sensor_fusion.ipynb	LiDAR-camera fusion
semseg_modeling.ipynb	PointNet training
semseg_functions.py	Shared helpers
media/	Generated videos and plots

---

Roadmap

• Now: tuning the PointNet baseline; then upgrading to PointNet++
• Next: semantic labels in the fused render (dual-coloring), then a fusion segmentation model (geometry + camera RGB)
• Eventually: run these perception pieces in a CARLA sim to evaluate and iterate

---

Running it

Python 3.10, notebook-driven, data read live from GCS (no local copies).

1. Install the dependencies (plus ffmpeg on your PATH for the animations):

pip install -r requirements.txt
sudo apt install ffmpeg   # or: brew install ffmpeg

For GPU training, install the CUDA build of PyTorch first (see the note in requirements.txt); a plain install pulls the CPU-only build.

2. Authenticate with Google Cloud — the helpers shell out to gcloud, so make sure it's installed and logged in:

gcloud auth login
gcloud auth print-access-token   # sanity check: should print a token

The token expires after an hour, so re-run the notebook's auth cell (or re-import the helper module) for long sessions. Paths to gcloud and its config are set at the top of semseg_functions.py — adjust them for your machine.

3. Launch Jupyter and run a notebook top to bottom:

jupyter lab   # then open semseg.ipynb or sensor_fusion.ipynb

• semseg.ipynb — LiDAR-only segmentation pipeline
• sensor_fusion.ipynb — LiDAR-camera fusion

Generated videos and plots land in media/.

---

References

• Waymo Open Dataset · v2.0 docs · Sun et al., Waymo Open Dataset, CVPR 2020 (arXiv:1912.04838)
• PyArrow: Parquet · PyArrow: GCS filesystem
• OpenCV pinhole model / calibration · Spherical coordinates · Homogeneous coordinates