Skip to content

deploy.py dies silently (exit 127) after flashing when stdout is piped — post-flash comeback + --power-cycle-cmd recovery never runs #121

Description

@boringethan

Symptom

On two consecutive 1.8.2 bare-metal deploys (left then right, Windows 11, invoked from Git Bash with stdout piped/captured, STM32_Programmer_CLI v2.20.0 path), scripts/deploy.py:

  • flashed successfully (File download completeRUNNING Program ... Start operation achieved successfully),
  • then terminated immediately with exit code 127,
  • printed none of its own stdout — not even the early ASCII lines ([*] Requesting DFU mode…, [*] Waiting for DFU device…, [*] Flashing … via STM32_Programmer_CLI…), and no Python traceback,
  • and never ran the post-flash _wait_for_sensor_comeback() / --power-cycle-cmd recovery — the sensor sat in the usual post-DFU jump-to-app hang until a manual Shelly power cycle.

So the documented hands-free contract ("pass --power-cycle-cmd and the deploy finishes itself") silently does not hold when the script is driven non-interactively (agents, CI, | tee). Flash integrity is fine; only the recovery/verification tail is lost.

Evidence / analysis

  1. Process dies without interpreter shutdown. All of deploy.py's stdout is block-buffered when stdout is a pipe; none of it was ever flushed, and no traceback appeared on stderr. A Python-level exception would have flushed the buffer and printed a traceback — so the process was killed hard (native crash / os._exit-like), with bash reporting 127. Timing on both runs: the instant STM32_Programmer_CLI -g 0x08000000 completed (device resets off the USB bus).
  2. omotion background threads were still noisy at DFU-entry: ValueError: Console controller not connected and RIGHT-COMM: USB read error (errno=32); exiting read loop: [Errno 32] Pipe error reached stderr during _enter_dfu_sensor() (its finally: interface.stop() did run). Suspect a native-layer (libusb) callback/thread outliving stop() and faulting when the device disappears during the flasher's reset — reproducibly, on both runs.
  3. Separate latent bug, same environment: every ✅/⚠️/❌ print in deploy.py raises UnicodeEncodeError when stdout is a pipe on Windows (cp1252). Repro: python -c "print('\u2705')" | cat. This did not cause the observed deaths (they precede any emoji print) but will break the recovery path even after the crash is fixed.
  4. _wait_for_sensor_comeback('left', timeout=8) run standalone (no flasher involved) works fine with piped stdout — the failure needs the DFU/flash sequence.

Impact

Bench automation must treat every deploy as "flash then manually power-cycle and verify". Defeats the up-front recovery logic too (that path did work — it's pre-flash).

Suggestions

  • Force UTF-8 + line-buffered/flushed output: sys.stdout.reconfigure(encoding='utf-8', errors='replace', line_buffering=True) at startup (or drop emoji), so progress lines survive regardless of how the script dies.
  • Print + flush a marker before/after _run(flasher) to bracket the crash window.
  • Investigate omotion interface teardown: ensure reader threads/native handles are fully joined/closed before invoking the flasher (or run DFU-entry in a child process so the flashing/recovery parent never holds USB state).
  • Consider subprocess.run isolation for the flasher with explicit capture, and make the comeback/power-cycle tail robust to it.

Observed during the 1.8.2 release deployment (both bench sensors ended up on 1.8.2 after manual power cycles; no data loss).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions