SID2026

A PyTorch reimplementation of the Sony pipeline from Learning to See in the Dark (Chen et al., CVPR 2018), with an additional cross-sensor experiment using Raspberry Pi 5 and IMX708 RAW captures.

The project maps amplified short-exposure Bayer RAW data to long-exposure RGB references using a SID-style U-Net.

Features

• Sony ARW loading and preprocessing with rawpy
• black-level correction and Bayer packing
• exposure-ratio amplification
• long-exposure RGB target generation
• SID-style U-Net with PixelShuffle
• config-driven training and checkpoint resume
• L1, PSNR, and SSIM evaluation
• overlapping tiled inference for full-resolution images
• Raspberry Pi IMX708 RAW decoding and preprocessing
• Raspberry Pi 5 CPU inference benchmarking

Results

Item	Result
Sony training pairs	1,865
Sony validation pairs	234
Training steps	50,000
Final training L1	0.025244
Final validation L1	0.045908
Representative Sony full-frame PSNR	28.426 dB
Representative Sony full-frame SSIM	0.8274
Raspberry Pi 5 full-frame CPU inference	5.427 s

The Sony full-frame result used a 250× exposure ratio with overlapping tiled inference. The Raspberry Pi experiment is a cross-sensor test rather than a controlled quantitative evaluation because no matching long-exposure RAW ground truth was available.

Repository Structure

configs/             Training configurations
scripts/             Training, evaluation, inference, and inspection scripts
src/sid2026/
  data/              RAW loading, Bayer packing, indexing, and datasets
  models/            SID-style U-Net
  training/          Training utilities
  inference/         Full-frame and tiled inference
  visualization/     Preview and export helpers
tests/               Unit tests

Setup

Python 3.12 and PyTorch 2.x were used for the reported experiments.

python -m venv .venv

Windows PowerShell:

.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
pip install -r requirements.txt

Install a CUDA-enabled PyTorch build separately when training on an NVIDIA GPU.

Dataset

The SID dataset is not included in this repository. Place the Sony subset under:

data/
  sid/
    Sony_train_list.txt
    Sony_val_list.txt
    Sony_test_list.txt
    Sony/
      short/
      long/

Raspberry Pi captures can be stored under:

data/
  pi/
    scene_001/
      raw.npy
      capture_settings.json

Training

python scripts/train_sony.py --config configs/<config>.yaml

Resume from a checkpoint:

python scripts/train_sony.py \
  --config configs/<config>.yaml \
  --resume outputs/<run>/checkpoints/last.pt \
  --steps 50000 \
  --output-dir outputs/<run>

Training outputs include resolved configuration files, metric logs, checkpoints, and preview images.

Evaluation

Evaluate a checkpoint on the Sony validation split:

python scripts/evaluate_sony.py \
  --checkpoint outputs/<run>/checkpoints/last.pt \
  --list-file data/sid/Sony_val_list.txt \
  --crop-size 512 \
  --mixed-precision \
  --output-dir outputs/evaluation

Sony Full-Frame Inference

python scripts/infer_full_frame.py \
  --checkpoint outputs/<run>/checkpoints/last.pt \
  --short data/sid/Sony/short/00001_00_0.04s.ARW \
  --long data/sid/Sony/long/00001_00_10s.ARW \
  --tile-size 512 \
  --overlap 64 \
  --mixed-precision \
  --output-dir outputs/sony_full_frame

The script writes the restored image, amplified RAW preview, optional target comparison, metrics, and run metadata.

Raspberry Pi Cross-Sensor Inference

The tested Pi captures were stored as byte buffers reported by the capture pipeline as SBGGR16. They are decoded into 16-bit Bayer images and packed in BGGR order before being converted to the model-facing [R, G1, B, G2] layout.

python scripts/infer_pi_raw.py \
  --checkpoint outputs/<run>/checkpoints/last.pt \
  --raw data/pi/scene_001/raw.npy \
  --metadata data/pi/scene_001/capture_settings.json \
  --bayer-pattern BGGR \
  --tile-size 512 \
  --overlap 64 \
  --output-dir outputs/pi_inference

Benchmark CPU inference on Raspberry Pi 5:

python scripts/benchmark_pi_runtime.py \
  --checkpoint outputs/<run>/checkpoints/last.pt \
  --raw data/pi/scene_001/raw.npy \
  --metadata data/pi/scene_001/capture_settings.json \
  --bayer-pattern BGGR \
  --device cpu \
  --torch-threads 4 \
  --warmup 1 \
  --runs 5 \
  --output-dir outputs/pi_benchmark

Limitations and Future Work

• collect controlled short- and long-exposure IMX708 RAW pairs
• evaluate sensor-specific fine-tuning
• improve white-balance and color consistency
• test ONNX export, quantization, pruning, and model distillation
• develop a smaller model for embedded or video-rate inference

Citation

@inproceedings{chen2018learning,
  title={Learning to See in the Dark},
  author={Chen, Chen and Chen, Qifeng and Xu, Jia and Koltun, Vladlen},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  year={2018}
}