Buffer QuestDB writes through an outage and replay them

[ichnograph]

Follow-up to #17. Reconnecting stopped the permanent wedge, but every reading taken while QuestDB was down was still dropped. This holds them and replays them.

Where the data was actually going

The obvious design (buffer rows once we know we are disconnected) does not catch the loss. Rows handed to the ILP client sit in its internal buffer until the next flush, and tcpLineSender.Flush discards that buffer when the write fails. The sink only discovers the outage at flush time, by which point the rows that flush was carrying are gone.

So the buffer holds every row until a flush confirms it, and clears on a successful flush. In the normal case that is one flush interval of rows.

Overflow

QuestDB.MaxBufferBytes caps it, 4 MiB by default. Sizing comes from a sizingSender that implements LineSender and measures the ILP encoding of a row; the client does not expose the encoded length. Estimates round up, so the cap is not exceeded.

Past the cap the oldest rows are evicted and the store call starts returning errors, which the service's consecutive-error escalation turns into a non-zero exit after five. The pod crash-loops for as long as QuestDB is down rather than growing until the kernel kills it. MaxBufferBytes: 0 restores the v1.5.3 drop-on-disconnect behaviour.

Health: degraded is not stuck

This needed a health-server change, otherwise the fix defeats itself: liveness flips after 90s of unhealthiness and the restart throws away the buffer that is holding the data.

healthserver.Degrader is an optional Checker extension. A checker that is failing but recovering in place reports Degraded() true and the liveness threshold skips it. Readiness is untouched and goes red on the first failure, so the outage is still visible and alertable.

State	/readyz	/healthz	Result
Connected	200	200	normal
Disconnected, buffer has room	503	200	pod stays up, readings held
Disconnected, buffer full	503	503	restart loop, readings dropped

Known limit

ILP over TCP has no server acknowledgement. The last flush before a socket error is reported as successful even though QuestDB never stored those rows, so they cannot be replayed. The buffer covers everything from the first reported failure onwards. Closing that gap properly means moving to the HTTP transport, which gets a real server response; that is a bigger change and a separate decision.

Tests

• End-to-end over a real TCP listener: write and flush through a server-side close, then assert the held rows arrive on the second accepted connection. This is what surfaced the flush-buffer gap above; the first version of the fix passed the unit tests and failed this one.
• Overflow evicts oldest, keeps newest, returns an error, and stops shielding liveness.
• Replay failure keeps the rows it never handed over.
• Buffering disabled keeps the old drop semantics.
• Liveness ignores a degraded checker indefinitely, and trips once it stops being degraded.
• qdb 95.5%, healthserver 96.6%, total 83.7%. Lint clean, race clean.