[Bug] Terminal output can permanently stall before xterm when a coalescer flush wake-up is lost

[zopenb]

Operating system

Windows

Netcatty version

1.1.71 (also reproduced with a diagnostic build based on current main)

How did you install Netcatty?

GitHub Release (.exe)

Affected area

SSH connection / terminal

Can you reproduce it?

Often (>50%) during longer TUI sessions

Steps to reproduce

1. Connect to a Linux host over SSH in a large terminal (the captured session was 266x68).
2. Run a full-screen TUI that uses DEC 2026 synchronized output and emits frequent repaint blocks. OpenCode 1.18.4 is a reliable reproducer, but no OpenCode plugin is involved.
3. Start a streaming operation and let the Netcatty window become unfocused, occluded, or minimized while output continues.
4. Return to the terminal. The visible TUI can remain permanently frozen even though the SSH session and remote process are still alive.

Expected behavior

Every renderer ingress byte should either reach xterm in order or remain attached to a scheduled drain. Losing browser animation frames, changing pane visibility, or applying flow control must not leave pending terminal output with no wake-up path.

Actual behavior

A live capture caught the permanent stall before xterm:

• The renderer continued logging renderer-receive events. The final burst contained many 12-34 KB TUI chunks.
• There were no corresponding renderer-write-done events after the stall began.
• The xterm private write buffer was empty (_writeBuffer.length=0, _pendingData=0, _isSyncWriting=false), synchronized-output mode was false, and the sync render handler was not buffering.
• Renderer CPU was nearly idle, rather than busy parsing or painting.
• The remote TUI remained alive and busy, while both PTY queues were empty (TIOCINQ=0, TIOCOUTQ=0). SSH/TCP was healthy.
• Kitty keyboard handling was disabled at runtime (useKitty=false).
• Bringing the window to the foreground caused newly requested animation frames to run, but did not drain the already-stranded terminal batch.

This places the loss between renderer ingress and term.write(), in Netcatty's write coalescer / visibility scheduling path rather than SSH, the remote PTY, xterm parsing, or Kitty keyboard handling.

Two scheduling edges appear capable of stranding pending data:

1. createWriteCoalescer relies on requestAnimationFrame for alternate-screen TUI batches without a wall-clock deadline. Chromium/Electron may suppress rAF while a window is occluded.
2. The visible idle flush is implemented as a debounce: every incoming chunk clears and rearms the timer. A sustained stream can postpone the only timer-based drain indefinitely. If flow control then pauses ingress, there may be no later push to create a new wake-up edge.
3. If a scheduled callback is rejected by shouldFlushScheduledFrame(), its scheduled state is cleared before returning while pending bytes remain. A later visibility transition needs an explicit rearm/flush path.

Suggested direction

A correctness fix should preserve all bytes, ordering, and ingress ACK accounting:

• Give rAF scheduling a bounded wall-clock fallback.
• Treat the idle flush timer as a maximum-latency deadline, not a reset-on-every-chunk debounce.
• If the flush gate rejects a scheduled callback, retain or rearm a drain instead of leaving pending bytes unscheduled.
• Force-drain the coalescer and serial write queue on focus/visibility resume.
• Add regression tests for a missing rAF callback, a gate transition while a frame is pending, and continuous sub-deadline chunks.

A local diagnostic build that bypassed the extra coalescer for SSH completed 217/217 writes with zero outstanding batches while the window was minimized. That confirms the transport and xterm path can keep up, but bypassing the normal SSH pipeline is too broad for an upstream fix because it skips timestamps, prompt formatting, output-pressure slicing, and other terminal behavior.

Related work

• fix(terminal): supersede stale synchronized TUI frames #2442 only supersedes queued, cursor-home full DEC 2026 snapshots. It intentionally does not touch incremental repaint blocks and does not fix this lost wake-up.
• [Bug] Slow terminal output during apt upgrade (output pipeline bottlenecks) #2272 covers general output-pipeline throughput and flow-control overhead. This report is narrower: a permanent scheduling stall with data already delivered to the renderer.

Logs / screenshots

Representative terminal performance sequence at the stall:

renderer-receive ingressBytes=34564
renderer-receive ingressBytes=22111
renderer-receive ingressBytes=29620
renderer-receive ingressBytes=24925
renderer-receive ingressBytes=28726
... no renderer-write-done after this point ...

Targeted tests for the proposed scheduling guards pass locally, including a simulated rAF callback that never arrives.

Before submitting

• I searched existing issues and did not find a duplicate
• I removed passwords, private keys, host addresses, and other secrets from this report

[netcatty-bot]

Thanks for the detailed report and the capture that pins this between renderer ingress and the terminal write path.

We can reproduce the failure mode in the current write batching / visibility scheduling: animation-frame drains for full-screen TUI output, the visible idle flush that keeps getting postponed while chunks keep arriving, and a flush gate that can clear a scheduled drain while bytes are still waiting. OpenCode is a solid reproducer here because of its continuous synchronized repaints; this is not an OpenCode integration bug.

We will prepare a focused fix so pending output always keeps a wake-up (bounded fallback if frames are suppressed, idle flush as a max wait rather than endless debounce, re-arm when a gated flush is skipped, and drain again when the window comes back). Regression coverage for a missing frame callback and continuous sub-deadline chunks will be part of that change.

[netcatty-bot]

A draft fix is available at #2470. Waiting on Codex review / remaining checks.

[s-celles]

Independent reproduction on a different platform and configuration, which I think narrows this usefully: the stall does not need occlusion, minimisation, or SSH.

Environment

• macOS 15 (arm64), Netcatty 1.1.72 and a dev build from main
• Local shell, no SSH, no remote host
• Terminal 172x51, window visible and focused throughout
• Reproducer: TryIt.jl, a full-screen Julia TUI on Tachikoma. It uses DEC 2026 synchronized output and repaints every cell's background colour each frame (~175 KB/frame at this size). Any animated-background TUI at a similar rate should do.

The visibility state during the stall

Captured live from the renderer while frozen:

paneVisible: true   pageHidden: false   docFocused: true
coalescerPending: 0 Ko   hasPendingWrites: true
flowPending: 1016 Ko     flowPaused: false

So requestAnimationFrame was not being suppressed, and the flush gate was open. That rules out hypothesis (1) as a necessary condition, at least for this reproduction.

Hypothesis (2) is sufficient on its own

scheduleVisibleTerminalWriteIdleFlush (terminalSessionAttachment.ts:320) clears and re-arms its timer on every incoming chunk, with VISIBLE_WRITE_IDLE_FLUSH_MS = 24.

Measured arrival rate during the stall, instrumented at writeSessionData:

 sec | KB/s | chunks/s | KB/chunk
-----+------+----------+---------
   7 | 1310 |       69 |     19.0
   8 |  428 |       22 |     19.4
     (nothing afterwards)

69 chunks/s is one every ~14 ms, shorter than the 24 ms debounce. The timer is therefore re-armed before it can ever fire, so the only timer-based drain never runs — with the window visible, focused, and rAF healthy. The threshold is simply "sustained arrivals closer together than the debounce", which any high-frame-rate TUI crosses.

Ingress ACK accounting leaks permanently

The suggested direction mentions preserving ingress ACK accounting. It is worth stating that this is not only a constraint to respect — it is where the failure becomes permanent rather than transient.

I wrapped OutputFlowController.received / .written and tallied by call site:

received : 2.25 MB   (single site: flow.received in writeSessionData)
written  : 1.25 MB   (the two flow.written sites in writeSessionDataImmediate)
LEAK     : 1023 KB

Neither drop-acknowledging path appears in the tally: not the write-queue drop handler (terminalSessionAttachment.ts:366, which does call controller.written), nor acknowledgeDroppedTerminalDisplayBytes (:389, which acks the main process but not the renderer-side controller). The stranded bytes simply never reach any accounted exit.

The consequence chain, which matches the symptoms exactly:

1. stranded bytes are counted by received() and never by written();
2. flowPending climbs monotonically and cannot come back down;
3. at FLOW_HIGH_WATER_MARK the PTY is paused, and written() can never bring it under FLOW_LOW_WATER_MARK to resume;
4. the TUI's own writes then block on a full PTY;
5. its render loop stalls, so it stops reading stdin — the application's keyboard goes dead, which is what users notice first.

Step 5 is worth calling out: Tachikoma's loop documents this exact failure mode ("the app animates but ignores every key"), and I suspect other TUIs behave the same way. A user reports "frozen and unresponsive", but the terminal emulator is idle and the remote process is alive.

What I was able to rule out

For whoever picks this up, these were eliminated by measurement rather than reading:

• xterm.js write buffer — empty during the stall (_pendingData: 0, _writeBuffer.length: 0, _callbacks: 0), matching your capture.
• Exceptions — none. A persistent collector on error / unhandledrejection / console.error caught nothing across several stalls.
• terminalWriteQueue — driven in isolation through 4 scenarios (multi-step merged items, yieldAfter, interleaved flushTerminalWriteQueueBypassingTimers, drain-between-bursts), asserting that every enqueued write eventually runs. All pass, so the queue does not drop continuations.
• Renderer CPU — 99.5% idle over a 6 s profile, consistent with your capture.

A suggestion beyond the scheduling fixes

PR #2470 addresses the wake-up paths, which should fix the stall itself. Separately, it may be worth making the leak non-permanent: when the whole pipeline is provably drained — coalescer empty, write queue empty, xterm write buffer empty — the flow controller's backlog must be zero by construction. Reconciling it at that point would close this entire family of freezes, including any future wake-up path that gets missed, rather than one instance of it.

---

Happy to test #2470 against this reproducer if that would help — it is deterministic and does not need SSH or a remote host.

Two disclosures for transparency: the dev build I instrumented also carried two unrelated local patches of mine (a fix in filterSyncBlockClears and an @xterm/addon-webgl glyph-cache change); neither touches the write scheduling path, and the original symptoms appeared on a stock 1.1.70 install before any of them. This investigation was carried out with AI assistance.

[binaricat]

Fixed and merged in #2470.

We reproduced the permanent stall with a deterministic Node/xterm harness: the previous main branch could strand the first terminal batch after a visibility-gate change, while the merged fix drains it correctly. The fix also covers continuous sub-deadline output and prevents a completed frame from lending its old safety deadline to the next frame.

Validation completed:

• 125 focused terminal tests passed
• lint, production build, and all final PR checks passed
• two independent Linux SSH stress runs completed with more than 9.5 MB per host, byte-for-byte ingress/ack accounting, zero pending output, and clean exit
• latest Codex and independent reviews were clean

Thanks @s-celles for the focused/local-shell reproduction and flow-accounting measurements. Once a build containing merge commit e3b2c6a is available, please retest OpenCode/TryIt.jl and let us know if any stall remains.

[s-celles]

I also find some issues upstream with xterm.js that lead to a nearly fixed animation and unresponsive keypress situation when using TryIt.jl (made with Tachikoma.jl)

This problem occurs in Netcatty (which rely on xterm.js) probably because of 2 layers of problems : on Netcatty side (but #2470 could help at least partially) and on xterm.js side.

I will try to improve that but now, with my fixes I'm getting 24 fps which is quite nice compared to previous situation.

On the other side I have no issue with Kitty which doesn't rely on xterm.js