COMM TX busy-wait (TX_TIMEOUT 500 ms) stalls the main loop and kills a camera

[boringethan]

Problem

A stalled COMM endpoint busy-waits the firmware main loop for up to 500 ms,
which starves the camera DMA re-arm and kills a camera.

Core/Src/uart_comms.c:186:

while (!tx_flag) {
    if ((get_timestamp_ms() - start_time) >= TX_TIMEOUT) {   /* TX_TIMEOUT 500, uart_comms.c:37 */
        printf("COMM USB TX Timeout\r\n");
        ...
    }
}

The camera re-arm (start_data_reception) runs inside send_data()
(Core/Src/camera_manager.c), which the main loop calls. The frame period is
25 ms. A 500 ms spin is twenty missed deadlines, and one missed re-arm is
terminal: Overrun -> failed to setup receive for Camera N channel ->
Camera N has stopped posting data -> rail off for 10000 ms.

Because the queue-full message itself must be transmitted over COMM, the
firmware's own diagnostic is what triggers the spin. Reporting one dropped
frame costs half a second of main-loop time.

Evidence (bench, stock 1.8.2-rc.4, production flags)

Host-side outage injected into the SDK read loops, 40 s scans, mask 0xC3:

condition	queue-full	camera lost
HISTO endpoint stalled only (9 trials, 150/300/500 ms)	yes, up to 32	0/9
COMM endpoint stalled only (9 trials)	none	0/9
both stalled (9 trials)	yes	9/9
both stalled, DEBUG_FLAG_USB_PRINTF off (6 trials)	yes	0/6
both stalled, DEBUG_FLAG_USB_PRINTF on (6 trials, interleaved)	yes	6/6

Neither endpoint alone does it — it is an interaction, which is exactly the
signature of "the queue-full message is the blocker". The lost camera is index
0 (firmware "Camera 1") on both modules, matching the 2026-07-24 field event.

Archive corroboration across 168 logs: 1621 COMM USB TX Timeout in 9
files, and a cascade in 20260320_131254.log whose inter-record gaps are
1.031/1.044/1.057 s then 0.577/0.588 s — 2x and 1x TX_TIMEOUT. The main loop
was iterating at 1-2 Hz against a 40 Hz frame rate.

Field match

open-motion-20260724_143144.log, camera temp 47.2 C (not thermal):

45.506  HISTO enqueue fail: queue full (count=4 size=4 enq=7 deq=3)
45.522  [CMP] USB FAIL: status=3          <- same event; enqueue returns USBD_FAIL
45.565  Error in USART2: Overrun error / failed to setup receive for Camera 1 channel
45.603  Camera 1 has stopped posting data
45.613  Camera 1: rail off for 10000 ms

Cost: 742 of 1954 frames on that channel, terminal dark frame missing, so the
final interval's corrected data is unrecoverable.

Fix

Do not block the main loop to emit a diagnostic:

• make the COMM send non-blocking (queue and drain, as the logging ring already
  does), or
• reduce TX_TIMEOUT to something compatible with a 25 ms frame deadline, and
• consider dropping the queue-full printf entirely — [DIAG] overruns c1-c8
  and OW_CMD_DIAG_STATS already carry that information.

Not the cause (ruled out on the bench)

• Thermal — 47.2 C, latch is >110 C.
• Firmware version — 1.8.2-rc.4 and 1.8.2-dev.1 equivalent within ~10%.
• NVIC priority inversion (USB at 0 above camera RX at 4/6) — real and worth
  fixing separately, but 15/15 overruns persisted on a priority-corrected build.
• DEBUG_FLAG_SEND_DEFER — governs one regime; never set in any archive log.
• Host sink stall alone (openmotion-sdk#116) — 0 camera losses in 28 trials
  across 60-300 ms.

[boringethan]

PR up: #117 — hardware-validated with stock rc.4 on the other module as an in-run control (fixed 0/0, stock 7 overruns / 6 cameras lost, same 5 scans).

[boringethan]

Board triage 2026-08-19: moved back to Backlog to clear the review queue. The fix exists and is complete: PR #117 (branch feature/115-comm-tx-nonblocking), closed unmerged — reopen #117 when this is picked back up rather than redoing the work. It was written as a stack with #118 (issue #96) and #119 (issue #116); plan to review the trio together.