This repository tests whether early-exit inference produces real deployment savings on a RubikPi 3 with a Qualcomm QCS6490. It compares a full ResNet-18, a calibrated two-exit ResNet-18 built with earlyon, a static MobileNetV3-Small and an INT8 HTP (NPU) deployment — on identical evaluation data with documented timing boundaries, live-stream behaviour, and thermal telemetry.

In this setup, early exit lost. Runtime staging overhead outweighed its estimated compute reduction on CPU. MobileNetV3-Small was the strongest CPU option, while a full INT8 ResNet-18 on the HTP provided the best accurate accelerated result.

These results apply to the tested models, dataset, board image, runtime and benchmark methodology; they are not a universal claim about early-exit inference or Qualcomm hardware.

Result

Early exit saved estimated backbone computation but increased measured CPU latency.

Full ResNet-18 33.6 ms · early exit 48.9 ms · static MobileNetV3-Small 7.7 ms · INT8 ResNet-18 on HTP 3.3 ms (p50). The static and accelerator-backed alternatives were the stronger deployment choices in this experiment.

Configuration	Backend	Precision	Accuracy	p50 model latency¹	Processed throughput²	Observation
Full ResNet-18	CPU (ONNX Runtime)	FP32	95.0%	33.6 ms	28.7 fps	reference
Early-exit ResNet-18	CPU (staged ORT ×3)	FP32	93.8%	48.9 ms	21.5 fps	slower than full
MobileNetV3-Small	CPU (ONNX Runtime)	FP32	94.0%	7.7 ms	116 fps	best CPU option
Full ResNet-18	HTP (QNN delegate)	INT8	95.0%	3.3 ms	~180 fps	fastest accurate option

¹ One synchronous batch-1 inference invocation; includes host routing for the staged row; excludes preprocessing and one-time session/delegate setup. ² Frames completed per second of the sequential benchmark loop including ~2.2 ms/frame preprocessing — which is why the HTP row reads 180 fps rather than 1 / 3.3 ms ≈ 305 fps. Execution is synchronous; nothing overlaps. CPU rows: median of 3 × 500-frame runs, alternating order, 60 s cooldowns. HTP row: single 500-frame run. Accuracy is measured on the CIFAR-10 test split; live room frames are out of distribution and never counted as accuracy. INT8 MobileNetV3-Small is absent because post-training quantization collapsed it to 17.5% top-1 — a negative result kept visible.

Why early exit lost here

• Only ~10% of estimated backbone compute was avoided at the calibrated operating point (11.8% of frames exit after layer2, 22.2% after layer3, 66% run the full depth).
• The exit heads add computation to every frame, exited or not.
• Three separate ONNX Runtime stage calls replace one fused graph, losing cross-graph optimization.
• Host-side routing adds synchronization and control overhead between stages.
• On live room scenes (out of distribution) the calibrated heads correctly refuse to fire — 89% of frames ran full depth — so the overhead is paid with almost no savings exactly when inputs drift.
• The static MobileNetV3-Small executes as one small optimized graph.
• The full INT8 HTP model is already fast enough (3.3 ms) that extra graph boundaries have no headroom to pay for themselves.

The routing itself works as designed — exit heads reach 88% / 98.7% accuracy on the frames they claim, and the board runner matches earlyon's reference routing decisions exactly (100% agreement on 200 samples, asserted at export). The economics, not the mechanism, are what failed here.

System architecture

Text version of the diagram

host laptop: UVC webcam → MJPEG encode (720p, ~30 fps)
     │  USB · adb reverse · TCP
     ▼
RubikPi 3 (QCS6490) — everything below runs on-device
  MJPEG decode → preprocess (224², CIFAR normalize)
     → bounded queue (size 2, drop-oldest)
     → model runner ─┬─ full ResNet-18        · CPU FP32 · one graph
                     ├─ early-exit ResNet-18  · CPU FP32 · 3 staged graphs + host routing
                     ├─ MobileNetV3-Small     · CPU FP32 · one graph
                     └─ full ResNet-18 INT8   · Hexagon HTP · TFLite + QNN delegate
  telemetry (2 s): latency · exit index · CPU% · RSS · thermal zones by name

Reproduce

# host: environment + models (fixed seed, ~13 min on an RTX 4050)
python3 -m venv .venv && .venv/bin/pip install -e ".[host,dev]"
.venv/bin/python scripts/prepare_models.py --device cuda
.venv/bin/python scripts/export_models.py     # asserts parity vs earlyon

# board: CPU benchmark (500 frames × 3 models × 3 repeats)
scripts/deploy_board.sh
adb shell "cd /opt/rec && python3 scripts/run_benchmark.py \
  --models models --frames models/eval_frames.npz \
  --out results/cpu_benchmark.json --repeats 3"

# charts + summary from committed raw data
.venv/bin/python scripts/summarize_results.py \
  --cpu-results benchmarks/raw/cpu_benchmark.json \
  --routing models/board/routing.json
.venv/bin/python scripts/make_charts.py

Board setup: offline wheel install (no internet on the board)

Download aarch64/cp310 wheels on the host, push, install offline:

pip download onnxruntime==1.18.1 --no-deps \
  --platform manylinux_2_28_aarch64 --python-version 3.10 \
  --only-binary=:all: -d wheels/
pip download coloredlogs flatbuffers packaging "protobuf<5" sympy \
  humanfriendly "mpmath<1.4" --no-deps --platform any \
  --python-version 3.10 --only-binary=:all: -d wheels/
adb shell mkdir -p /opt/rec/wheels && adb push wheels/. /opt/rec/wheels/
adb shell "pip3 install --no-index --find-links /opt/rec/wheels onnxruntime"

tflite_runtime==2.13.0 (for the HTP path) installs the same way. Board inventory: scripts/inventory_board.sh writes a sanitized benchmarks/environment.json.

Live MJPEG pipeline (webcam over adb)

# host: serve the webcam as MJPEG on localhost and bridge it to the board
gst-launch-1.0 -q v4l2src device=/dev/video0 \
  ! image/jpeg,width=1280,height=720,framerate=30/1 \
  ! multipartmux ! tcpserversink host=127.0.0.1 port=8081 sync=false &
adb reverse tcp:8081 tcp:8081

# board: camera check, then a 60 s live run
adb shell "cd /opt/rec && python3 scripts/check_camera.py --net-port 8081 --frames 90"
adb shell "cd /opt/rec && python3 scripts/run_live_camera.py \
  --models models --runner early_exit --duration 60 \
  --net-port 8081 --out results/live.json"

With a supported CSI camera (IMX219/477/708) the same scripts run without --net-port.

HTP (NPU) benchmark

# host: INT8 TFLite conversion (tensorflow venv; calibrated on train images)
.venv-tf/bin/python scripts/convert_tflite.py
adb push models/tflite_work/full_int8/full_integer_quant.tflite /opt/rec/models/full_int8.tflite

# board: same frames, delegate on ("htp") and off ("none")
adb shell "cd /opt/rec && python3 scripts/run_htp_benchmark.py \
  --models models --model full --file full_int8.tflite \
  --frames models/eval_frames.npz --out results/htp_full.json --backend htp"

HTP proof — how NPU execution was verified

Successful inference alone was not treated as proof of accelerator execution. The claim rests on a chain (benchmarks/accelerator_proof.json):

1. The FP16 path was attempted first — the HTP backend rejected it ("SocModel doesn't support FP16"), and float models silently fell back to CPU while appearing to run with the delegate. Detected, never reported as acceleration.
2. An INT8 model was prepared (post-training quantization, real calibration images).
3. The QNN TFLite delegate loaded and claimed the graph.
4. Delegate/backend behaviour confirmed the HTP configuration path is honored (an invalid precision request fails hard).
5. The identical .tflite file ran without the delegate as a control.
6. Result: 23× execution difference (3.3 ms vs 75.9 ms p50) on the same file, same frames, same harness.
7. Accuracy parity held (95.0% delegated vs 94.8% CPU-INT8 vs 95.0% FP32).

Model conversion time and delegate setup are excluded from all latency numbers. A full staged HTP early-exit runtime was not built: measured stage-one latency (2.03 ms alone) against the 3.3 ms fused result left insufficient headroom to justify extra graph boundaries, intermediate-tensor handling and host routing within this study's scope — no measured staged-HTP latency is claimed.

Live-stream and thermal behaviour

60-second runs per model against the ~30 fps MJPEG input, all inference on the board (CPU FP32):

Model (live)	Processed fps	Capture→result p50 / p95	Stale frames dropped
MobileNetV3-Small	30.1 (input rate)	11.0 / 13.6 ms	3 of 1807
Full ResNet-18	24.9	87.4 / 113.6 ms	310 of 1806
Early-exit ResNet-18	19.3	97.7 / 124.1 ms	616 of 1774

Capture continued at ~30 fps in every run; processed fps counts frames that completed inference, and the drop-oldest queue (size 2) intentionally discarded the surplus as stale. Capture→result latency runs from frame arrival on the board to the classification result, including queue wait.

Ten-minute run — early-exit ResNet-18, staged ONNX Runtime CPU FP32, queue size 2, drop-oldest: capture held ~30 fps (18,046 frames); 10,499 processed (17.5 fps); 7,547 stale frames intentionally discarded; capture→result p99 137 ms. The run completed without a crash or sustained memory growth (RSS ~180→198 MB). The cpuss0-thermal zone, sampled every 2 s, rose from ~79 °C and then held ~86–88 °C (max 88.4 °C). CPU frequency throttling was not independently verified.

Exit behaviour on in-domain test frames vs the estimated compute saving:

Camera path used in this study

The live input is an MJPEG stream from a UVC webcam, transported to the board over USB ADB. Decode, preprocessing, inference, routing, queueing and telemetry all run on the RubikPi. This is a RubikPi inference benchmark and makes no zero-copy claim. A CSI (qtiqmmfsrc) capture path is also implemented in src/rubikpi_earlyon_camera/camera.py; the published numbers use the MJPEG input (details in docs/limitations.md).

Repository layout

src/rubikpi_earlyon_camera/   preprocessing, runners, queue, telemetry, schemas
scripts/                      train/calibrate, export+parity, benchmarks, charts
benchmarks/                   environment, raw per-frame data, summary, HTP proof
docs/                         methodology, task selection, accelerator path, limitations
assets/                       generated charts and diagrams
tests/                        23 CPU-safe unit tests (hardware tests marked)

Methodology and evidence

• docs/methodology.md — one timing core, fairness controls, calibration hygiene, statistics.
• docs/task-selection.md — why CIFAR-10 at 224².
• docs/accelerator-path.md — runtime selection, INT8-only finding, proof protocol, quantization failure.
• docs/limitations.md — read this before quoting any number.
• benchmarks/ — raw per-frame data, folded summary, sanitized environment, trtcheck reports.

No model weights or datasets are committed; every artifact regenerates deterministically (seed 42) and SHA-256 hashes of the published artifacts are committed:

Artifact	Source / licence	Hashes	Regenerate
CIFAR-10	Krizhevsky 2009, research use; fetched by torchvision	md5 in torchvision	scripts/prepare_models.py
ImageNet-pretrained inits	torchvision weights (BSD-3)	—	scripts/prepare_models.py
Trained checkpoints	this project, MIT	models/prepare_report.json	scripts/prepare_models.py
ONNX graphs	this project, MIT	models/board/export_report.json	scripts/export_models.py
TFLite FP32 + INT8	this project, MIT (onnx2tf)	models/board/tflite_report.json	scripts/convert_tflite.py

No Qualcomm SDK binaries are committed; the QNN delegate and runtime ship with the Qualcomm Linux 1.3 board image.

Limitations

Upsampled CIFAR-10 frames are not photographic camera frames; live room scenes are out of distribution and used for systems behaviour only; one board, one ambient environment; exit heads trained for three epochs; FLOPs fractions are analytic estimates; frequency throttling was not measured. Full list: docs/limitations.md.

Relationship to earlyon and trtcheck

Project	Role
earlyon	trains, calibrates and benchmarks early-exit models (v0.3.0 used here; routing semantics reproduced with 100% agreement)
trtcheck	independently validates the exported ONNX graphs (v1.1.0; its TensorRT verdicts are not presented as QNN compatibility)
this repository	tests those ideas on real Qualcomm edge hardware and reports what actually happened

Licence

MIT — see LICENSE.