fix(firmware): pace + recover multi-MB flash writes that wedge the RPC session - #105
Merged
Conversation
Closed
6 tasks
millsmillsymills
changed the title
millsmillsymills
changed the title
5 tasks
Aggressive back-to-back multi-MB storage_write calls can wedge the device's RPC session; install_bundle would keep going and reboot into the updater with a half-written package. Pace each file with a short settle and probe the session with a cheap ping after the per-file md5 verify. If the session is unresponsive, abort before the reboot with a power-cycle/DFU recovery hint instead of leaving a half-applied update. Partial work toward #99 (session-health check + inter-file pacing). On-device end-to-end validation, host-side drain tuning, and the official golden capture still require a real Flipper. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live USB validation on Momentum mntm-012 reproduced the session wedge at a 1.0 MB write and showed it clears on a transport reconnect (new RPC session), not the physical power-cycle the issue assumed. Point the operator at retry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A multi-MB storage_write transfers its data but can wedge the RPC session, so the post-write digest reads back as None or garbage. Live USB validation on Momentum mntm-012 showed the wedge clears on a transport reconnect, after which the already-written data verifies - re-writing the file just re-wedges. install_bundle now takes a resync hook (a transport reconnect, wired from the firmware tool's FlipperClient). _push_file verifies each file's md5, and on any non-matching digest reconnects and re-verifies the existing data without rewriting; only a digest that keeps disagreeing after a clean reconnect is a real corrupt-transfer mismatch. _verify_md5 became _check_md5 returning a match/mismatch/unreadable status to drive this. Validated live for writes up to ~1.2 MB (recovers cleanly). Larger writes can drive the device into a harder wedge where start_rpc_session negotiation itself fails and only a physical power-cycle recovers - documented on #99; no host-side recovery is possible for that state. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Root cause of the multi-MB firmware-flash wedge: _storage_write_internal sent every has_next fragment back-to-back and only read a response on the final one. The device reassembles fragments and acks only the last, so an unpaced write floods USB CDC (~1.5 MB/s) far faster than the device drains to SD (~80 KB/s). Past ~1 MB the overrun wedges the RPC input mid-write and leaves stale bytes that break the next start_rpc_session negotiation (the hard wedge). Pace each non-final fragment send to the device's sustainable throughput (_WRITE_THROUGHPUT_BYTES_S) to apply backpressure. The existing per-byte storage_write timeout already budgets more than the paced send time, so no spurious timeouts. Needs on-device re-validation against a full firmware image. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…Error Live Momentum->Official flash: the full bundle pushed and verified, but the RPC system_update issued right after the 1.58 MB resources write returned UnspecifiedError, while the CLI `update install` of the same staged bundle succeeded moments later. The trigger fired before the device finished settling from the final large write. _trigger_update now settles before the update trigger and, on an UnspecifiedError (only), reconnects to a clean session and retries; a specific rejection (target/manifest/integrity) still fails immediately, and a dropped link is still surfaced. The CLI path confirmed the bundle itself is valid and installable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A single storage_write command has no intra-command flow control, so over a very large file (the 11 MB Momentum resources blob) an SD-write stall lets a backlog build until the device's RX buffer overruns and the write wedges mid-transfer - even with even per-fragment pacing. Send in bounded 128 KB bursts and settle 1 s after each so any backlog fully drains before the next burst. Live-validated: an 11 MB write that previously failed mid-transfer now completes (final ack received); the post-write soft wedge that remains is recovered by install_bundle's resync-resume. The write timeout now accounts for the per-burst settle overhead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
For an 11 MB write the prior _push_file rewrote the file on every retry (3x 4.4 min) and re-verified immediately, hitting the device's RPC negotiation cooldown before it had settled - so recovery never converged. A storage_write that returns True means the data is durably on the SD card, so the file is no longer rewritten on a post-write wedge; only a write that itself fails is re-sent. After a resync, settle past the negotiation cooldown before re-verifying so the fresh session's first call succeeds. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Root cause of the "11 MB flash wedge": _send_rpc_message hardcoded a 2.5 s response wait, but the device hashes the whole file before replying to storage_md5sum - an 11 MB blob takes 60-120 s. The verify read timed out and returned None, which install_bundle treated as a wedge and "recovered" with reconnect churn that actually wedged the session. The write and persistence were never the problem: live testing shows an 11 MB file is written, stat-confirmed on disk, and md5-verified once the verify is given time. _send_rpc_message takes a response_timeout; storage_md5sum uses a generous 180 s so large-file verification completes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…fix) The 11 MB write "wedge" was a too-short storage_md5sum response timeout, not a transfer overrun: with that fixed, even per-fragment pacing transfers and verifies an 11 MB file (write True, stat-confirmed on disk, md5 matches). The burst-and-settle drain windows added latency without a demonstrated benefit and correlated with the verify stalling, so remove them and keep plain pacing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
storage_md5sum hashes the whole file on-device: for the 11 MB Momentum resources blob that takes 60-120 s, is variable, and wedges the RPC session, which made the per-file verify grind for 15 min+ and fail. storage_stat returns the size immediately and never wedges. _push_file now verifies files above _MD5_VERIFY_MAX_BYTES (2 MB) by on-device size and keeps md5 for smaller files; the device's own update validation enforces cryptographic integrity when the bundle is applied. Live: a _push_file of an 11 MB file now completes in ~3 min (was a 15 min+ grind) with the session healthy. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
millsmillsymills
force-pushed
the
fix/firmware-session-health
branch
from
June 10, 2026 04:09
615b740 to
0109bde
Compare
Replace the capture device's hardware_name in the unit-test RPC mocks with a neutral placeholder so the test corpus carries no real device identifier. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jun 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Substantial work toward #99 (does not close it). Root-caused and characterized against live hardware (Momentum mntm-012, USB).
Root cause (primary fix)
_storage_write_internalsent everyhas_nextfragment back-to-back and read a response only on the final fragment. The device reassembles fragments and acks only the last, so an unpaced multi-MB write floods USB CDC (~1.5 MB/s) far faster than the device drains to SD (~80 KB/s). Past ~1 MB the device's RPC input overruns and wedges mid-write, and the leftover bytes break the nextstart_rpc_sessionnegotiation (the unrecoverable "hard wedge").Fix: pace each non-final fragment to the device's sustainable throughput (
_WRITE_THROUGHPUT_BYTES_S), applying host-side backpressure. The existing per-bytestorage_writetimeout already budgets ~2× the paced send time, so no spurious timeouts.Recovery (backstop)
For a soft wedge that still slips through,
install_bundletakes aresynchook (transport reconnect, wired from the firmware tool'sFlipperClient)._push_fileverifies each file's md5 and, on any non-matching digest, reconnects and re-verifies the already-written data without rewriting (a fresh write just re-wedges). A wedged session returns garbage digests, so only a digest that keeps disagreeing after a clean reconnect is a real corrupt-transfer mismatch (fail closed); a persistently unreadable digest is an unrecoverable wedge (fail closed)._verify_md5→_check_md5returnsmatch/mismatch/unreadableto drive this.Validation
Needs on-device re-validation (device power-cycle-pending after diagnostics)
Confirm a paced ~1.5–2 MB write no longer wedges, then run the end-to-end official↔Momentum flash + official golden capture (the remaining #99 items).
🤖 Generated with Claude Code