Consumer-session start-from improvments, table improvements, test improvements - #358
Merged
Merged
Conversation
Start-from / cross-topic merge: - Replace decline/NACK at the merge's memory cap with per-stream consumer pause/resume (count and byte watermarks). Nothing is handed back to the broker mid-skip, so a stream can no longer overtake itself and the cap nack-storm is gone. Broker redeliveries are deduplicated by append-position watermark and never double-spend the skip budget. - Finish the skip on the budget's last claim (no N+1st head), release the session-wide ordering lock once the merge settles, use a monotonic stall clock granting a fresh window on resume, and route cross-target drop acks through the origin listener. - Latest-n: anchor non-contributing topics at their inspected tail so concurrent appends survive; 30s wall-clock resolution budget; refuse read-compacted targets and chunked topics; re-verify anchors against retention before any seek; UI/server cap parity (10M), pinned on both sides. - Refuse counted skips over overlapping enabled targets; keep and document latest-n's per-view duplicate contract. - Degradation is user-visible: StartFromProgress carries degraded + abandoned_streams, the first degraded frame bypasses the reporting interval, and the session shows a sticky best-effort banner. - Merge internals: O(log K) head heaps and incremental counters, per-target receiver-queue budget, 1000-topic admission guard, one shared maintenance thread for stall sweeps. Tables / UX: - Draggable column reorder, persisted per table in localStorage, for the shared Table and the consumer-session message table (header and rows follow one keyed order; sticky columns stay pinned). - Shared Table: pinFirst for summary rows, compact size variant, and a fix for TableVirtuoso mounting before its scroll parent existed (tables rendered blank until a window resize). - Topic Positions: rebuilt on the shared Table with an All-topics aggregate row, Behind / entries read / entries left columns, consumption-first default order, wider topic column, first tab and selected by default; polling is gated by the open panel + tab, and the capture toggle is removed - opening the tab is the request. - Start From defaults to Earliest message (Latest on non-persistent topics, which retain nothing). Tests: full server (747), jest (453) and e2e suites green, including the new CsFlowControlSpec - consumer.pause under an armed listener and the give-up/degradation pipeline against a real broker - plus column-reorder and refusal-path coverage.
visortelle
changed the title
visortelle
force-pushed
the
consumer-session-improvements-and-tests
branch
from
August 10, 2026 04:50
8b12d08 to
908d38a
Compare
Guaranteed no longer promises live ordering it cannot keep. It now replays
everything recorded up to Play (of what retention still holds) in strict
selected-timestamp order, auto-pauses at that boundary with a caught-up
banner, and extends the boundary on Resume. Ordering violations across a
pause seam are delivered loudly flagged - a row marker plus a session
counter - never silently.
Why: for live traffic, {no stall, no ordering exception, no coordination} is
pick-two. Pulsar has no cross-partition watermark, so a strict live mode
either stalls on an idle partition (observed: an empty table) or admits a
skew window. This drops the impossible promise instead of quietly
compromising it; live following is Best effort's job, one click away through
the existing SetDeliveryOrder RPC.
- Default delivery order is now Best effort.
- Latest x Guaranteed is disabled in the picker - replaying history from
"now" replays nothing. Saved combinations still load exactly as written.
- Approximate position modes renamed, and "% of time" now resolves ONE cutoff
across the whole selection instead of a range per logical topic, so a
multi-topic session's percentage refers to one comparable history.
- UseLatestTopicSchema finally uses the topic's latest registered schema: a
payload from a Schema.BYTES producer on a schema-registered topic rendered
as an escaped JSON string. An inferred schema that fails to decode keeps
the raw-string answer rather than turning a readable payload into an error.
- Fixes 36 branch-review findings: delivery/lifecycle races, limiter and
backpressure bounds, admission permits, cookie SameSite matching, and the
zombie-session reconnect flood (short-grace janitor rule for disconnected,
unwatched sessions).
- New e2e coverage: delivery modes, merge order, chunking, pause loops,
retention isolation, schema values, wide topologies, plus a per-config
instance lane (DekafInstance) with 10 configuration specs.
- Docs: new start-from and mode comparison page; index and tutorial aligned
to the replay contract.
Two follow-ups are recorded in the plan doc rather than fixed here: the
live-edge resume corner (disclosed, with a one-click escape) and the table
auto-refresh interval being floored at 2s by the data layer.
Verified: server 967/0, jest 663/0 (50 suites), tsc clean, e2e 333/0 with one
designed self-cancel (TOP-8 requires topic policies disabled).
visortelle
force-pushed
the
consumer-session-improvements-and-tests
branch
from
August 10, 2026 04:51
908d38a to
b2c1733
Compare
… modes, caught-up panel docked - Guaranteed is the default delivery order (third and final move of this default; absent/legacy/unrecognized configs all resolve to it). The e2e generic lanes pin Best effort explicitly; CS-MO-0 pins the default. - Proto field 61 renamed replay_seam_violation -> delivered_out_of_order (same tag): Best effort now flags its late emissions too, at the point the verdict already exists. Per-row "!" marker plus a toolbar warning badge with cause-naming tooltips replace the always-on mode chip. - The caught-up panel is docked bottom-right by the session itself, not a toast: react-toastify's animation-mediated removal made every dismiss/re-announce hand-off a race (swallowed panel in CS-DM-R2, twin panels in CS-DM-R3B); rendered conditionally, at most one can exist. Both actions primary: "Load new messages up to now" and "Switch to Best effort and follow live". - Latest x Guaranteed selectable again: Play answers an instant caught-up and the panel is where the user decides. - Config UX: help circles describe every option and tag the default, display limit defaults to 1,000,000, tools pane open by default at 340px, Topic Positions empty states use NothingToShow, index column resizable, "Topic FQN copied". - e2e: CS-DM-D6G holds its churn window through the product's own 500 msg/s delivery limit (the pipeline outgrew block-size timing); README documents the toast-lifecycle trap. Full suite 333/333.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Consumer session control improvements
Delivery ordering modes
Introduced delivery ordering modes when for consuming messages from multiple topics.
Implemented topic positions view
This view is useful when you run a consumer session on large topics, so you could understand where you are at this moment.
Other changes