Skip to content

NVCM ledger for the 2026-08-13 bricked cameras; hold flashing until SDK pacing guard ships #108

Description

@boringethan

What happened

During NVCM flashing of the two new production sensor modules on 2026-08-13, burns run from source executed over an editable SDK carrying the feature/233-send-packet-latency sleep-removal (c5688e8, PR openmotion-sdk#234). The resulting ~0.5 ms/transaction replay pace is outside the CrossLink NVCM programming envelope: pages corrupt from ~28% into the array, verify aborts the burn before the DONE fuse, and the OTP part is left permanently unprogrammable. Full root cause and fix: OpenwaterHealth/openmotion-sdk#245.

The release 1.4.2 exe (wheel 1.10.0, transport sleeps intact, ~1.5 ms/tx) passed on virgin cameras the same day — which made this look like a test-app next regression. The test-app was never the variable: its burn path is byte-identical between 1.4.2 and next.

Camera ledger (both modules, sensor-fw 1.8.2)

Module Camera NVCM state
Left 5SqvMgim5brRiQSFhtupK1 1–8 Bricked (partial image, no DONE; boot probe reads BLANK; every re-burn fails at first verify pages)
Right 4xCA7w48VHuQ6Nf1ekgXSw 1 Programmed + boots (burned 14:48 by 1.4.2 exe)
Right 2, 3, 4, 8 Bricked (4 was additionally aborted mid-program at tx 4,415 by the transport crash)
Right 5, 6 Programmed + boots (burned 16:50/16:53 by 1.4.2 exe)
Right 7 Never attempted — still virgin. Do not burn until the SDK pacing guard (sdk#245) is merged and bench-validated

Bricked cameras still operate normally via the firmware SRAM load path — the modules are usable, they just can't self-boot those FPGAs from NVCM.

Note for future triage: a boot-probe sweep taken seconds after a transport crash reported genuinely-programmed right cam 1 as BLANK (16:48:27). Probe verdicts immediately after a crash/reconnect are unreliable — power-cycle and re-check before drawing conclusions.

Actions for this repo

  • Do not run NVCM burns (any build) on virgin cameras until sdk#245 is merged and validated on the expendable bench module Ethan is preparing.
  • First test-app release after the SDK fix must bundle an SDK wheel that includes the pacing guard (CI takes the latest wheel — verify, don't assume).
  • After validation: burn right cam 7 with the guarded build to complete the right module (7/8 → 8/8 where possible).

Refs OpenwaterHealth/openmotion-sdk#245, OpenwaterHealth/openmotion-sdk#233.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions