Ego-OSCAR

Ego-OSCAR is a low-cost, open-hardware, head-mounted stereo-inertial capture system for collecting egocentric data in the wild. This repository brings together the ESP32-S3 coprocessor firmware and the Radxa host-side recorder used to capture synchronized stereo video and IMU metadata.

Repository layout

Path	Purpose
firmware/esp32_coprocessor_led_watchdog/	Seeed Studio XIAO ESP32-S3 firmware for the ICM-20948 IMU, camera VSYNC tracking, status LEDs, and host watchdog
radxa/	Radxa recorder, diagnostics, and frame-sync validation tools
docs/	Architecture and supporting project documentation

System overview

1. The ESP32-S3 samples the IMU, counts camera VSYNC pulses, and sends compact hexadecimal telemetry to the Radxa over UART (or USB serial).
2. The Radxa captures the stereo camera through V4L2 and FFmpeg while recording the corresponding IMU/frame-sync stream.
3. Each recording session contains segmented video, IMU CSV data, offsets, and metadata for downstream dataset preparation.

The defaults in the source reflect the development hardware and should be reviewed before use, especially serial ports, GPIO lines, I2C bus, video format, resolution, and frame rate.

Getting started

ESP32-S3 coprocessor

Open firmware/esp32_coprocessor_led_watchdog/esp32_coprocessor_led_watchdog.ino in the Arduino IDE, install the libraries listed in that directory's README, select the XIAO ESP32-S3 board, review USE_USB_SERIAL, and flash the device.

Radxa host

On a Debian/Ubuntu-based Radxa system:

sudo apt update
sudo apt install ffmpeg python3-libgpiod python3-serial python3-smbus rsync
python3 radxa/fpv_recorder.py

The production recorder expects a Rockchip-enabled FFmpeg providing mjpeg_rkmpp and h264_rkmpp. Prebuilt FFmpeg binaries and third-party source archives are intentionally not stored here; install or build the appropriate version for the target board.

See radxa/README.md for configuration, utilities, and output details.

Resources

• Dataset — the full dataset is available for use at fpvlabs/stereo-550 on Hugging Face
• DIY guide — step-by-step build guide, including the CAD models and the complete codebase
• Code — github.com/fpv-labs/ego-oscar
• Paper — Ego-OSCAR on arXiv
• 3D visual of the hardware — interactive model
• Diagram and spec sheet of the hardware — PDF
• Dense action labels — use Vidur: fpvlabs.ai/essays/vidur-action-label
• 3D prints — print files