KafkaTable latches status == "failed" permanently on any exception escaping the reader loop (_on_reader_done sets _failure and nothing clears it or re-spawns the task), with no restart().
PR #34 removed the two realistic transient triggers:
- a raising
on_set/on_delete apply-observer callback (now logged + counted via stats.callback_errors, non-fatal);
- a transient
IllegalStateError from the catch-up position() under a metadata blip (now tolerated — see ADR 0003).
But a genuinely non-retriable death still freezes the table for its lifetime — e.g. TopicAuthorizationFailedError on a mid-stream ACL change, which the design already documents as terminal-by-design. There is no self-recovery for that or any other unexpected reader death.
Proposal
Add a supervised reader that can clear _failure and re-spawn the reader task (bounded backoff), or an explicit restart() the caller can invoke.
Design questions
- Which errors are retriable (auto-recover) vs terminal (stay
failed)?
- Automatic supervision vs caller-driven
restart() (or both)?
- Backoff policy and how
status/stats reflect a recovering vs permanently-failed reader.
Context: follow-up to #33 (whose original "dies on broker disconnect" mechanism was refuted — disconnects are retried and survived) and #34.
KafkaTablelatchesstatus == "failed"permanently on any exception escaping the reader loop (_on_reader_donesets_failureand nothing clears it or re-spawns the task), with norestart().PR #34 removed the two realistic transient triggers:
on_set/on_deleteapply-observer callback (now logged + counted viastats.callback_errors, non-fatal);IllegalStateErrorfrom the catch-upposition()under a metadata blip (now tolerated — see ADR 0003).But a genuinely non-retriable death still freezes the table for its lifetime — e.g.
TopicAuthorizationFailedErroron a mid-stream ACL change, which the design already documents as terminal-by-design. There is no self-recovery for that or any other unexpected reader death.Proposal
Add a supervised reader that can clear
_failureand re-spawn the reader task (bounded backoff), or an explicitrestart()the caller can invoke.Design questions
failed)?restart()(or both)?status/statsreflect a recovering vs permanently-failed reader.Context: follow-up to #33 (whose original "dies on broker disconnect" mechanism was refuted — disconnects are retried and survived) and #34.