Harden USB firmware-flash: md5-verify settle/retry, pacing, end-to-end on-device validation

[millsmillsymills]

Context

Follow-up from #98 (multi-firmware support). The firmware-flash core works — the chunked storage_write transfers data correctly (md5 verified on-device for files up to 1.58 MB) — but the full end-to-end official↔Momentum flash does not yet complete reliably over USB. This issue tracks the robustness hardening needed to make a complete flash reliable, plus the on-device end-to-end validation.

No firmware update was ever executed during validation; every attempt aborted safely before the reboot-into-updater step.

Findings from hardware validation (Momentum mntm-012, USB)

1. Fixed in Multi-firmware support + firmware-flash tooling #98 — storage_write timeout floor was 3 s, which failed a 117 KB write whose data had actually transferred (md5 matched). Now flush-aware (max(15, len/40000)).
2. storage_md5sum returns None immediately after a large write. The device is still flushing to SD; the verify reads None and the installer treats it as a mismatch and aborts. The same 1.58 MB file md5-matched once the device had settled. Evidence:
   • 768 KB write → md5 matched.
   • 1.0 MB and 1.58 MB writes → write_ok=True but immediate storage_md5sum returned None; a later read of the same file returned the correct hash.
3. RPC session can wedge after aggressive back-to-back large writes — start_rpc_session negotiation fails, get_device_info returns empty, SD reads unavailable. Recovered only by a physical power-cycle. Suggests the post-write/inter-command sequence needs settle/drain and possibly host-side pacing.

Proposed work

• In the installer's per-file verify, add a short settle delay + bounded storage_md5sum retry (treat None as "not ready, retry" rather than "mismatch"). Distinguish a real digest mismatch from an unreadable digest.
• Investigate host-side pacing / inter-chunk drain in (or around) storage_write for multi-MB files to avoid the session wedge; consider a brief settle between files in install_bundle.
• Consider a session-health check / re-sync (drain + re-start_rpc_session) between large operations, and surface a clear recovery message (power-cycle / DFU) if the session wedges mid-flash.
• On-device end-to-end validation: real flipperzero_firmware_install official→capture official golden fixtures→back to Momentum (Task 12/13 from the plan), then the integration+usb flash round-trip test.
• Reorganize golden fixtures to tests/golden/fixtures/{momentum,official}/ and parametrize the golden suite (depends on the official capture above).
• Clean up leftover /ext/update/diag and /ext/update/mcp-update folders left on the test device's SD card.

Notes

• The per-file md5 verify means a failed/corrupted push aborts before any reboot, so this is a reliability problem, not a bricking risk.
• Diagnosis scripts used during validation were ad-hoc (/tmp), not committed.

[millsmillsymills]

Update: the md5-verify settle/retry item is now implemented in #98 (commit f8e522b) — storage_md5sum returning None after a large write is now treated as 'not ready, retry' rather than a mismatch, with unit tests (_verify_md5). This was the actual blocker for the false-mismatch abort.

Remaining items here are the ones that genuinely require the (power-cycled) test device and can't be unit-verified:

• Host-side pacing / inter-chunk drain investigation for the RPC session-wedge after aggressive multi-MB writes.
• Session-health check / re-sync between large operations.
• On-device end-to-end flipperzero_firmware_install (official → capture official golden fixtures → back to Momentum).
• Reorganize golden fixtures to tests/golden/fixtures/{momentum,official}/ + parametrize.
• integration+usb flash round-trip test.
• Clean up leftover /ext/update/diag and /ext/update/mcp-update folders on the test device's SD card.

[millsmillsymills]

AFK status — remaining work is hardware-gated

Audited the proposed-work checklist against phase1. The non-hardware code slices have already landed:

• md5 settle/retry — installer._verify_md5 treats a None digest as "still flushing, retry" (5 attempts, 2s settle) and only fails on a definitive non-None mismatch. Done.
• storage_write timeout floor — flush-aware max(15.0, len/40000) at protobuf_rpc.py:1117. Done (Multi-firmware support + firmware-flash tooling #98).

Everything still open requires a real device and cannot be done AFK:

• Host-side pacing / inter-chunk drain + inter-file settle — needs on-device repro of the session wedge to know what to pace; adding a blind settle is speculative and unverifiable without hardware.
• Session-health check / re-sync (drain + re-start_rpc_session) between large ops — same; depends on reproducing the wedge.
• On-device end-to-end validation (official→Momentum round-trip).
• Official golden capture + reorganize fixtures to tests/golden/fixtures/{momentum,official}/ and parametrize — blocked on the capture above (fixtures are currently flat).
• Clean leftover /ext/update/diag + /ext/update/mcp-update from the test SD card.

Handing back: this needs a Flipper on USB to proceed.

[millsmillsymills]

PR #105 addresses the session-health check + inter-file pacing items: install_bundle now probes the RPC session with a ping after each per-file md5 verify and fails closed (before reboot) with a power-cycle/DFU recovery message when it wedges. The remaining checkboxes (on-device round-trip validation, host-side drain tuning against a live wedge repro, official golden capture + fixture reorg, SD cleanup) still need a real Flipper on USB.

[millsmillsymills]

On-device validation (Momentum mntm-012, f7, USB, /ext 12.4 GB free)

Ran a non-flashing diagnosis (large scratch writes to /ext/update/diag, md5 verify, ping probe, then cleanup).

Wedge reproduced + characterized

write	write_ok	md5	ping after
768 KB	True (9.7 s)	matched, readable	healthy
1.0 MB	True (12.0 s)	matched (needed a retry — first storage_md5sum poll was unreadable)	wedged

• The wedge triggers earlier than reported — 1.0 MB here vs 1.58 MB before.
• The md5 settle/retry was exercised (the 1.0 MB digest was unreadable on the first poll, readable on the next), confirming _verify_md5's None-means-retry handling is load-bearing.
• PR fix(firmware): pace + recover multi-MB flash writes that wedge the RPC session #105's post-write ping probe correctly detected the wedge.

Recovery is software, not a power-cycle — corrects an assumption

The issue says the wedge "recovered only by a physical power-cycle." Not so: a transport teardown + reconnect (new RPC session) fully recovered the device — ping ok, 60-key get_device_info, real /ext/update listing. No physical power-cycle needed. PR #105's recovery message was updated to point at reconnect/retry accordingly.

Note: while wedged, storage_list returned [] and storage_delete returned True — garbage/unreliable responses (matches finding #3). So never trust ops issued on a wedged session; re-sync first.

SD cleanup — done

Removed /ext/update/mcp-update and /ext/update/diag (the latter also held leftover ad-hoc a.bin/b.bin/c.bin/big.bin from earlier validation). /ext/update now contains only legitimate f7-update-* bundles.

Checklist status

• md5 settle/retry (_verify_md5) — verified on-device.
• Session-health check + clear recovery message — PR fix(firmware): pace + recover multi-MB flash writes that wedge the RPC session #105 (ping probe, fail-closed, reconnect hint).
• Clean up leftover /ext/update/{diag,mcp-update} — done.
• Host-side pacing / inter-chunk drain tuning — wedge cause still not root-caused; PR fix(firmware): pace + recover multi-MB flash writes that wedge the RPC session #105 adds a fixed inter-file settle but the per-chunk drain inside storage_write for multi-MB files is unverified.
• Re-sync and resume mid-flash (vs the current fail-closed) — now known to be achievable via reconnect; not yet implemented.
• On-device end-to-end official↔Momentum round-trip + official golden capture + fixture reorg — pending the real flash (to be done next, with confirmation).

[millsmillsymills]

Live multi-MB write investigation — two wedge severities found

Drove the firmware-push path on-device (Momentum mntm-012, USB) without flashing. Key result: the session wedge has two severities, and that reframes the remaining work.

1. Soft wedge (reconnect-recoverable) — handled in PR #105

• A ~1.0–1.2 MB storage_write transfers its data correctly but wedges the session; storage_md5sum then reads None or garbage (so a single mismatch can't be trusted).
• A transport reconnect clears it; the already-written data verifies on the fresh session. Re-writing the file just re-wedges, so recovery must be verify-after-reconnect, not rewrite.
• PR fix(firmware): pace + recover multi-MB flash writes that wedge the RPC session #105 implements this (resync hook + _check_md5 match/mismatch/unreadable + verify-after-reconnect). Validated live for writes up to ~1.2 MB.

2. Hard wedge (needs physical power-cycle) — open, possibly unfixable host-side

• Aggressive/repeated multi-MB writes + reconnect churn drove the device into a state where start_rpc_session negotiation itself fails (3 attempts, then 5s backoff). storage_write returns False, every digest reads None, get_connection_health reports connected=False.
• A transport reconnect does not recover this — it needs a physical power-cycle. This matches the original finding Type hardening: ConnectionHealth TypedDict, frozen ServerContext, config wifi validator #3 ("recovered only by a physical power-cycle").
• Real firmware images (~1.5 MB+) are squarely in the size range that triggers this, so an end-to-end USB flash of a full image is not yet reliable. Whether this is fixable host-side (pacing/drain at the chunk level) or is a device/USB-stack limit is still open.

Updated checklist

• md5 settle/retry — verified on-device.
• Session-health detection + clear recovery message — PR fix(firmware): pace + recover multi-MB flash writes that wedge the RPC session #105.
• Soft-wedge re-sync and resume (verify-after-reconnect) — PR fix(firmware): pace + recover multi-MB flash writes that wedge the RPC session #105, live-validated ≤~1.2 MB.
• SD cleanup (/ext/update/{diag,mcp-update}) — done earlier.
• Hard wedge at multi-MB: root-cause the start_rpc_session negotiation failure after large writes; determine if host-side chunk pacing/drain avoids it or if it's a device limit. This is the blocker for an end-to-end USB flash.
• On-device official↔Momentum round-trip + official golden capture + fixture reorg — blocked on the hard-wedge above.

Note: a diagnostic run left the device in the hard-wedged state; it needs a power-cycle to be usable again.

[millsmillsymills]

Root cause of the multi-MB wedge: unpaced fragment sending

Reviewed _storage_write_internal (protobuf_rpc.py). The write loop sends every has_next fragment back-to-back and only reads a response on the final fragment. The Flipper reassembles fragments and acks only the last one, so there is no per-fragment backpressure.

Consequence: an unpaced multi-MB write pushes USB CDC at ~1.5 MB/s while the device drains to SD at ~80 KB/s (the rate already noted in the storage_write timeout comment) — a ~19× overrun. Past ~1 MB the device's RPC input path overruns and wedges mid-write; the leftover bytes then corrupt the next start_rpc_session negotiation, which is exactly the hard wedge (negotiation fails, only a power-cycle recovers). _WRITE_CHUNK_SIZE is 1024 B, so a 1.5 MB image is ~1500 fragments fired with zero pacing — matches the observed threshold.

Fix (PR #105)

Pace each non-final fragment send to the device's sustainable throughput (_WRITE_THROUGHPUT_BYTES_S = 80_000), applying host-side backpressure so the unacked fragment stream cannot overrun the device. The existing per-byte storage_write timeout (len/40000) already budgets ~2× the paced send time, so no spurious timeouts. Unit tests assert one pacing sleep per non-final fragment and none for single-frame writes.

This is the host-side mitigation the "host-side pacing / inter-chunk drain" checkbox called for, derived from the actual code path rather than guessed.

Needs on-device re-validation (device currently power-cycle-pending)

• Confirm a paced ~1.5–2 MB write no longer soft- or hard-wedges the session.
• Then the end-to-end official↔Momentum flash round-trip + official golden capture.

If pacing alone closes the wedge, the resync-and-resume recovery in PR #105 becomes a backstop rather than the primary path.

[millsmillsymills]

Real flash attempt: Momentum mntm-012 → Official (latest, f7)

Ran the full production path (download+verify → install_bundle with paced writes + resync-resume → system_update). Result splits cleanly into a solved half and a new, narrower half.

✅ Multi-MB transfer + verify is solved (the core #99 problem)

The complete official 6-file bundle pushed and md5-verified on the first pass, zero reconnects needed:

file	size
firmware.dfu	768 KB
radio.bin	117 KB
resources.ths	1.58 MB
splash.bin	528 B
update.fuf	1.57 KB
updater.bin	116 KB

Pacing (PR #105) prevented both the soft and the hard wedge — resources.ths at 1.58 MB, which reliably wedged before, transferred clean. No power-cycle, no negotiation failure.

⛔ New blocker: device-side system_update rejects with UnspecifiedError

After the push, system_update("/ext/update/mcp-update/update.fuf") returned UpdateResponse.UnspecifiedError. install_bundle failed closed before the reboot — the device never rebooted and stayed on Momentum mntm-012, fully functional. The fail-closed + typed-error surfacing (codes.py / #103) worked exactly as intended.

Diagnosis so far:

• The staged update.fuf on-device is byte-identical to the downloaded bundle's manifest. Target 7 matches the device. All 6 referenced files are present and md5-verified.
• So the bundle is correctly staged; UnspecifiedError is the device's own update_operation_prepare rejecting it — a generic code (not target/manifest/integrity/IntFull-specific). The Flipper writes the actual reason to its internal serial log, which the protobuf RPC does not expose.

Open questions for this new blocker (likely its own issue):

• Is UnspecifiedError specific to applying a stock Official bundle while running Momentum (a Momentum updater guard)? A Momentum→Momentum reflash would isolate this.
• Otherwise needs the device's UART/CLI log output captured during system_update to get the real reason.

Updated checklist

• md5 settle/retry; soft-wedge resync-resume; hard-wedge prevention via fragment pacing — all validated live, incl. the full official bundle.
• SD cleanup of prior leftovers.
• system_update UnspecifiedError on the staged official bundle — new device-side blocker; needs serial logs or a same-flavor reflash to isolate. Blocks the official↔Momentum round-trip + official golden capture.

Staged bundle left at /ext/update/mcp-update as a repro artifact.

[millsmillsymills]

Round-trip update: Official succeeds, Momentum blocked by an 11 MB resource file

✅ Momentum mntm-012 → Official 1.4.3 — full success (validated live)

The complete flash worked end-to-end: paced+resync push of all 6 files, then update + reboot + apply. Device came up as Official 1.4.3 (origin_fork=Official, flipperdevices git). The RPC system_update UnspecifiedError was a transient post-large-write settle issue — _trigger_update (settle + reconnect + retry on UnspecifiedError) now handles it; confirmed the bundle itself is valid via the CLI update install path.

⛔ Official → Momentum — blocked by resources.tar.gz (11.2 MB)

The Momentum bundle's resource blob is 11.2 MB (vs Official's 1.58 MB resources.ths). An 11 MB single storage_write wedges mid-transfer (not after), so the data never fully lands; _push_file's reconnect-then-rewrite never converges (each 11 MB rewrite re-wedges) and it fails closed after the retry budget. Device stayed safely on Official — no system_update, no reboot.

This is a distinct, harder limit from the ≤2 MB wedge that PR #105 solves:

• Pacing (80 KB/s) + resync-resume reliably handles writes up to ~2 MB (all Official bundle files, incl. 1.58 MB).
• A single ~11 MB write exceeds it — the wedge lands mid-transfer, so verify-after-reconnect can't recover without a full rewrite, which re-wedges.

Likely needs qFlipper-class flow control: proper per-fragment backpressure / draining of the device's input, or splitting the large resource into bounded appends (Flipper WriteRequest has no append/offset, so this needs a different storage approach). Its own investigation, separate from PR #105.

Net checklist

• md5 settle/retry; soft-wedge resync-resume; fragment pacing; system_update settle/retry — all live-validated; Momentum→Official flash works.
• SD cleanup.
• Large single-file writes (~11 MB Momentum resources) wedge mid-transfer — the remaining transfer-reliability gap; blocks Official→Momentum via the tool. qFlipper can restore Momentum meanwhile.
• Official golden capture + fixture reorg — still pending (now unblocked for the Official side; the device is currently on Official).

[millsmillsymills]

Official golden fixtures captured + dual-firmware reorg — PR #106

While the device is on Official 1.4.3, captured the official hardware golden set and reorganized fixtures into per-flavor dirs:

• tests/golden/fixtures/{momentum,official}/ hold the firmware-specific hardware fixtures (cli_exec device_info, fs_push, mode_switch, lock_contention); the firmware-independent offline fixtures stay flat.
• The recorder classifies the connected device's flavor and writes into the right dir; the replay/drift tests are parametrized over both flavors.

Checklist update:

• Reorganize golden fixtures to tests/golden/fixtures/{momentum,official}/ and parametrize — done (PR test(golden): dual-firmware golden fixtures (Momentum + Official) #106).
• Official capture — done (1.4.3).

Remaining open item on #99: the ~11 MB single-file write (Momentum resources.tar.gz) that wedges mid-transfer and blocks Official→Momentum via the tool. Separate, deeper transfer-reliability investigation.