Consumer-session start-from improvments, table improvements, test improvements

.github/workflows/ci.yml

@@ -22,6 +22,8 @@ jobs:
       PULSAR_ADMIN_PORT: 28080
       PULSAR_BROKER_PORT: 26650
       DEKAF_PORT: 28090
+      # Isolate the Library's data dir per run so items don't accumulate and skew counts.
+      DEKAF_FRESH_DATA: 1
     steps:
       - uses: actions/checkout@v4
         with:

.gitignore

@@ -118,3 +118,10 @@ result
 
 # Docker
 **/slim.*.json
+
+# macOS
+.DS_Store
+
+# Local design assets (~24MB of PNG/GIF concepts) - not part of the product build.
+# Anchored to the repo root so it does not also swallow any nested `design/` dir elsewhere.
+/design/

AGENTS.md

@@ -9,7 +9,7 @@ Dekaf is an open-source UI for Apache Pulsar. It's a single deployable binary th
 - A **Scala 3 / ZIO** backend exposing a gRPC API (`server/`)
 - An embedded **Envoy proxy** that translates browser gRPC-Web ↔ native gRPC
 
-The UI and server communicate over **Protobuf / gRPC-Web**. Proto definitions live in `proto/` and are the source of truth for the API contract — generated code is committed into `ui/grpc-web/` and `server/src/main/scala/pb/`.
+The UI and server communicate over **Protobuf / gRPC-Web**. Proto definitions live in `proto/` and are the source of truth for the API contract — generated code lands in `ui/grpc-web/` and `server/src/main/scala/pb/`, which are gitignored and regenerated by `cd proto && make build`.
 
 ## Development environment
 
@@ -69,7 +69,7 @@ Dekaf stores saved sessions and other user artifacts as "managed items" on disk
 
 ## Conventions & notes
 - The backend is intentionally **straightforward Scala** — avoid heavy FP / type-level acrobatics (per `CONTRIBUTING.md`).
-- After changing any `.proto`, you **must** run `cd proto && make build` and rebuild both sides; the generated code is committed.
+- After changing any `.proto`, you **must** run `cd proto && make build` and rebuild both sides. The generated output is gitignored, so a clean checkout has no `pb/` or `grpc-web/` until you run it.
 - Generated directories (`ui/grpc-web/`, `server/src/main/scala/pb/`) should not be hand-edited.
 - `demoapp/` is a sample producer app used by the quick-start docker-compose to populate demo data.
 - `desktop/` contains an Electron wrapper; `helm/` and `deployment/` are for k8s; `docker/` holds image builds and the quick-start compose file.

docs/configuration-reference.md

@@ -76,8 +76,8 @@ Also set the appropriate cookie settings.
 
 |Field                           |Description                                                                                                                                                                                                                                                                                                                                                               |
 |---                                |---                                                                                                                                                                                                                                                                                                                                                                       |
-|cookieSecure                       | `true` or `false`. Set it to `true` if you use the `https` protocol.                                                                                                                                                                                                                                                                                                                                                      |
-|cookieSameSite               | `true` or `false`. Set it to `true` if you use the `https` protocol.                                                                                                                                                                                                                                                                                      |
+|cookieSecure                       | `true` or `false`. Set it to `true` if you use the `https` protocol. Adds the `Secure` attribute, so the browser only sends the cookie over HTTPS.                                                                                                                                                                                                                                                                                                                                                      |
+|cookieSameSite               | `lax`, `strict` or `none` (case-insensitive). Controls the cookie's `SameSite` attribute, which tells the browser whether to send the cookie on cross-site requests - the built-in CSRF protection. Leave it unset to use the browser default. `none` additionally requires `cookieSecure: true`, because browsers reject `SameSite=None` on a non-`Secure` cookie; if you set it without `cookieSecure`, the attribute is omitted and a warning is logged. An unrecognised value is also omitted with a warning.                                                                                                                                                                                                                                     |
 
 ### Default Pulsar Auth
 

docs/consume/consumer-session-tutorial.md

@@ -6,7 +6,7 @@
 
 - Navigate to a Pulsar topic you're interested in
 - Click the **Consume** button
-- By default, **Consumer Session** starts from the **Latest Message** in the topic. If no producers produce new messages in realtime to the selected topic, switch the **Start From** field to **Earliest Message**
+- By default, **Consumer Session** starts from the **Earliest Message** in the topic, so pressing play shows the data that is already there. On a **non-persistent** topic, which keeps no history at all, it starts from the **Latest Message** instead and the history-based positions are unavailable.
 - Click the **▶** button to start the consumer session.
 
 ![initial consumer session](./img/start-consumer-session.png)
@@ -67,7 +67,7 @@ All the loaded messages data will be erased on the UI. If you want to save the c
 The **Search in Loaded Messages** feature that we used above, may be not enough for precise search and search in Pulsar topics with a lot of messages for the following reasons:
 
 - In case you're searching for a small subset of messages in comparison with all messages in the topic, the network bandwidth may easily became a bottleneck.
-- The amount of messages loaded to your browser simultaneously is limited by RAM available for a single browser tab. By default, the message count limit is `10,000` messages, but you can adjust it in consumer session's **advanced settings** depending on the average message size.
+- The amount of messages loaded to your browser simultaneously is limited by RAM available for a single browser tab. By default, the message count limit is `10,000` messages, but you can adjust it with **Limit num. display messages** in the consumer session settings depending on the average message size.
 
 The **Message Filter** feature solve both mentioned problems. They work on server-side, therefore processed messages that don't pass message filters, aren't being loaded to your browser.
 
@@ -226,7 +226,7 @@ Each consumed message represents a row in the table.
 
 Table has the following columns:
 
-- **#**: Index number of the message in the current view. It is determined by the order in which the message was consumed and delivered to the UI. The order of messages may differ from the **Publish Time** order in case of consuming a partitioning topic or multiple non-partitioned topics.
+- **#**: Index number of the message in the current view. It is determined by the order in which the message was consumed and delivered to the UI. Across a partitioned topic or several topics that depends on the session's delivery order: **Guaranteed** and **Best effort** (the default) merge by the selected timestamp - Guaranteed adds no disorder of its own, while under **Best effort** a message arriving after the reorder window is numbered out of order. **Fastest** delivers each topic and partition independently, so its numbering follows arrival rather than **Publish Time**.
 - **Publish Time**: The timestamp of when the message is published. The timestamp is automatically applied by the producer.
 - **Key**: The key of the message. Messages are optionally tagged with keys, which is useful for features like topic compaction and key-shared subscriptions.
 - **Value**: Value of the message serialized as JSON. It may be inefficient to observe values this way because some values may not fit the column width. To fix that, you can map specific value field to a table column by using **Projections**.

docs/consume/index.md

@@ -6,6 +6,22 @@ Dekaf allows you to explore Apache Pulsar topic data by using **consumer session
 
 - Browse live stream data
 - Consume multiple topics at once
+- Choose how Dekaf combines multiple topics or partitions. Pulsar itself does not guarantee order
+  across topics.
+  - **Guaranteed** (the default) is an exact replay: it delivers everything recorded up to Play (of what
+    retention still holds) in strict selected-timestamp order, then auto-pauses with a caught-up
+    banner, whose "Load new messages up to now" extends the replay to the present; an ordering violation across a pause seam
+    (producer clock skew) is delivered loudly flagged - a row marker plus a session counter -
+    never silently. Multi-stream sessions require persistent topics.
+  - **Best effort** merges by the selected timestamp within ~0.75 s. Late messages can
+    appear out of order; none are dropped.
+  - **Fastest** delivers each topic or partition independently, with no reordering delay.
+  - See the [comparison of start positions and modes](./modes-comparison.md) for how every
+    start-from mode, delivery order and consumption mode combine.
+- **Order by** uses Pulsar timestamps: publish time is added automatically by the producer, broker
+  publish time is broker entry metadata recorded when a message arrives, and event time is an
+  optional timestamp set by the application. Missing broker publish time or event time falls back
+  to publish time and is reported.
 - Filter messages using user-friendly basic, or advanced JavaScript filters
 - Search for specific value in loaded messages
 - Map specific message field to the search results table column
@@ -20,8 +36,33 @@ Dekaf allows you to explore Apache Pulsar topic data by using **consumer session
 
 The amount of messages that can be processed during the consumer session is unlimited, but the amount of messages that are loaded and displayed in the user-faced UI at once has limits.
 
-By default this limit equals to `10,000` messages. Depending on the average message size, you can configure this number for specific topic.
+By default this limit equals to `1,000,000` messages. Depending on the average message size in the consumed topics, you can lower it per consumer session - the help beside the toggle says when to.
 
 Operations that user can perform on the loaded data (e.g. sorting, export, or search in found) are limited to these messages.
 
 We're looking for the best way to get rid of this limitation.
+
+### Pausing loses nothing only on persistent topics
+
+Pausing closes the session's intake and hands back the messages it had already prefetched, so on a
+persistent topic the broker redelivers them and the session picks up where it stopped. A
+non-persistent topic stores nothing, so there is nothing for the broker to redeliver: whatever is
+published while the session is paused - including the messages it had already prefetched and handed
+back - is gone, and resuming shows only what is published from then on. A paused session says so on
+screen whenever it reads such topics.
+
+### A session affects retention while it exists
+
+Dekaf browses through its own non-durable subscription, so the broker holds a live cursor at the
+position the session has read up to, and a topic's data cannot be reclaimed ahead of that cursor.
+An open session therefore delays ledger deletion and storage reclamation on the topics it reads
+whenever its cursor sits behind data the broker would otherwise be free to drop - and it keeps
+doing so for as long as the session stays open. The window is longest where you would expect: a
+session that is paused - including one auto-paused at its **Guaranteed** replay boundary - or that
+started from an old position holds its cursor where it is until it moves on.
+
+Other consumers are not affected: every other subscription keeps its own cursor and backlog, and no
+topic or subscription policy is changed. Stopping or deleting the session removes its subscription,
+and the cursor with it. A session nobody is watching any more - a closed tab whose cleanup never
+reached the server - is stopped automatically after 1 hour without a live stream, which is the
+longest an abandoned session can hold a cursor.

docs/consume/modes-comparison.md

@@ -0,0 +1,72 @@
+# Start positions and modes - comparison
+
+A consumer session has three independent choices:
+
+1. **Start from** - where in the topic the session begins.
+2. **Delivery order** - how messages from several topics or partitions are combined into one stream.
+3. **Consumption mode** (per target) - whether the session reads the raw log or the compacted view.
+
+This page compares the options and what they are compatible with.
+
+## Start from
+
+| Mode | What you get | Exact? | Speed | Notes |
+|---|---|---|---|---|
+| Earliest message | Everything the topic still retains | Exact | Instant | |
+| Latest message | Only messages published after Play | Exact | Instant | The only mode available on non-persistent topics |
+| Skip first n messages | Everything except the globally-oldest n (by publish time, across all partitions) | Exact - counts real messages, not storage entries | Proportional to n; a progress bar appears for very large skips | For jumping deep into a huge topic, a time-based or percentage mode is instant instead |
+| Latest n messages | Exactly the globally-newest n (by publish time, across all partitions) | Exact | Fast at any topic size (walks only the tail) | |
+| Message with specific ID | From one exact message | Exact | Instant | |
+| Specific time | From a moment in time | To the millisecond | Instant at any topic size | |
+| Relative time ago | Same, phrased as "2 hours ago" | To the millisecond | Instant | Optional rounding to the unit start |
+| Approximate position (% of data) | Roughly this far through the stored data | Approximate - counted in storage entries, so batching skews the message percentage | Instant at any topic size - it asks the broker a fixed number of questions however much the topic holds | Each topic and partition is positioned independently |
+| Approximate position (% of time) | Roughly this far between the oldest and newest publish times | Approximate - producer clocks set the timestamps | Instant at any topic size | One shared cutoff instant for the whole session, so multi-topic sessions align |
+
+## Compatibility: start from x what you are reading
+
+| Start from | Non-persistent topic | Read-compacted target | Topic with chunked messages | Partitioned topic |
+|---|---|---|---|---|
+| Earliest message | disabled - nothing is retained | yes | yes | yes |
+| Latest message | **yes - the only mode** | yes | yes | yes |
+| Skip first n | disabled | yes - counts what is actually delivered | yes - a chunked message counts as one message | yes - the skipped n are global, not per partition |
+| Latest n | disabled | **refused** - it counts the raw log, which compaction rewrites | **refused** - storage entries are not messages there; the error names the alternatives | yes - the n are global, not n per partition |
+| Message ID | disabled | yes | yes | yes |
+| Specific / Relative time | disabled | yes | yes | yes |
+| % of data | disabled | **refused** - it measures the raw log | yes, but chunk entries skew the percentage | yes - each partition positioned independently |
+| % of time | disabled | **refused** - its range comes from the raw log | yes | yes - one shared cutoff across all partitions |
+
+A refused combination fails with a message naming the reason and a working alternative - it never
+silently returns a different result than asked for.
+
+## Delivery order (when a session merges several topics or partitions)
+
+| | Fastest | Best effort | Guaranteed (the default) |
+|---|---|---|---|
+| Order claim | None - messages appear as they arrive | Sorted by the selected timestamp, waiting up to ~0.75 s for stragglers | An exact replay of everything recorded up to Play (of what retention still holds), strictly sorted; it auto-pauses when caught up, and "Load new messages up to now" extends the replay to the present |
+| A partition goes silent | No effect | After the grace, delivery continues; a straggler arriving later is shown late and counted | Silence itself does not matter - the replay delivers each stream's recorded range and auto-pauses at the boundary; a silent stream holds delivery only while part of its recorded range is still undelivered |
+| A silent partition stays silent | No effect | Delivery simply continues without it; only while a counted start position (Skip first n) is still resolving does a ~30 s give-up set the stream aside and mark the start position approximate | If a stream's recorded range can no longer be delivered - trimmed away by retention, or the start position seeked past its end - no caught-up fires: the session discloses the wait and offers a one-click switch to Best effort. Otherwise the caught-up pause stands, and "Load new messages up to now" replays whatever was recorded since |
+| Late/out-of-order messages | Shown as they come | Delivered and counted, never dropped | Cannot occur within a replay chunk; across a pause seam a producer-clock reversal is delivered flagged (row marker + counter), never silently |
+| Non-persistent topics | yes | yes | Single stream only - nothing is retained, so Play announces caught-up instantly; merging non-persistent topics is refused |
+| Best for | Raw throughput, single topics | Everyday browsing and live following | Forensic reading of recorded history where order matters more than immediacy |
+
+**Order by** applies to Best effort and Guaranteed: publish time (producer clock), broker publish
+time (broker clock, entry metadata), or event time (application-set). A message missing the chosen
+timestamp falls back to publish time, and the session reports how often that happened.
+
+**Latest message x Guaranteed** replays nothing - a replay of recorded history starting at "now"
+has nothing behind it. The combination is selectable; Play simply answers caught-up immediately,
+and the caught-up panel offers the two ways on (load what has arrived since, or switch to Best
+effort and follow live).
+
+## Consumption mode (per target)
+
+| | Regular | Read compacted |
+|---|---|---|
+| What you read | The raw log - every retained message | The newest message per key up to the compaction horizon, then the raw tail |
+| Use it for | Everything by default | Key-value style topics where only the latest value per key matters |
+| Start-from limits | None | Latest n and both percentage modes are refused (they measure the raw log); Earliest, Latest, Skip first n, Message ID and the time modes work |
+
+## See also
+
+- [Consumer sessions overview](./index.md)
+- [Consumer session tutorial](./consumer-session-tutorial.md)