Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
108 changes: 61 additions & 47 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,10 @@
<a href="#license"><img src="https://img.shields.io/badge/license-MIT-green" alt="License"></a>
</p>

<img src="assets/demo.gif" alt="cratis diagnose reporting one failed partition, the exception that caused it, then the workbench opening on the same event store" width="900">
<img src="assets/demo.gif" alt="cratis diagnose reporting a healthy store, then the raw events on the log, then the read model those events produced" width="900">

<sub>One command for the verdict, one for the exception behind it, then the live view.<br>
The reactor could not reach its mail server; the CLI names the observer, the partition and the event.</sub>
<sub>Is it healthy, what happened, and what state did that produce —<br>
the three questions, in the order you actually ask them.</sub>
</div>

---
Expand All @@ -43,14 +43,14 @@ the server's auth, and you are on a box you reached over SSH.
```
❯ cratis chronicle diagnose

── Chronicle Diagnostics 10:15:16 ─────────────────────────────────────────────
── Chronicle Diagnostics 14:18:21 ─────────────────────────────────────────────
server: chronicle://chronicle-dev-client:***@localhost:35100/
event store: Bookshop / Default

✓ Connection connected
✓ Server version 16.7.0
✓ Event stores 2 stores: System, Bookshop
✓ Observers 6 active 3 disconnected
✓ Observers 9 active
✗ Failed partitions 1 need attention → cratis chronicle failed-partitions list
✓ Recommendations none
✓ Event sequence tail: 22
Expand Down Expand Up @@ -161,13 +161,14 @@ Measured on one store of 23 events and 9 observers:

| | `events get` | `observers list` | `event-types list` |
|---|---|---|---|
| `-o plain` | 2,133 B | 794 B | 561 B |
| `-o json-compact` | 8,250 B | 1,701 B | 1,656 B |
| `-o json` | 9,966 B | 2,110 B | 2,505 B |
| `-o plain` | 1,615 B | 777 B | 561 B |
| `-o json-compact` | 7,329 B | 1,684 B | 1,656 B |
| `-o json` | 8,981 B | 2,093 B | 2,505 B |
| `-q` | 59 B | 403 B | 310 B |

`plain` is tab-separated and about 4.5× smaller than `json` here — the gap widens with row
count, because JSON repeats every field name on every row. `-q` exists to be piped:
`plain` is tab-separated and 2.7× to 5.6× smaller than `json` on this store — widest on
`events get`, where JSON repeats four field names across every one of 23 rows. `-q` exists to
be piped:

```bash
cratis chronicle observers list -q | xargs -I {} cratis chronicle observers replay {} -y
Expand All @@ -179,57 +180,70 @@ This is the table you will spend the most time in, and two of its columns are ea
misread:

```
Id Type State Quarantined Next# LastHandled# Subscribed
Bookshop.Members Reducer Disconnected False 3 2 False
Bookshop.Books Reducer Disconnected False 11 10 False
Bookshop.OverdueNotices Reactor Disconnected False 22 21 False
Bookshop.BorrowedBooks Projection Active False 19 18 False
Bookshop.OverdueBooks Projection Active False 23 22 False
Id Type State Quarantined Next# LastHandled# Subscribed
Bookshop.Members Reducer Active False 23 2 False
Bookshop.Books Reducer Active False 23 10 False
Bookshop.BorrowedBooks Projection Active False 23 18 False
Bookshop.OverdueBooks Projection Active False 23 22 False
Bookshop.OverdueNotices Reactor Active False 23 22 False
```

| Column | What it means |
|---|---|
| `LastHandled#` | the last event this observer finished processing |
| `Next#` | the next sequence number it will look at |
| `Next#` | the next sequence number this observer will look at |
| `LastHandled#` | the last event it actually processed |
| `State` | `Active`, `Replaying`, `Suspended`, `Disconnected`, `Quarantined` or `Unknown` |
| `Subscribed` | whether a client is currently attached to it |

> [!NOTE]
> **A `Next#` behind the log tail is not by itself a problem.** An observer only advances over
> events it subscribes to. Above, the tail is 22 and `BorrowedBooks` sits at 19 — events 19
> through 22 are reservations and overdue markings, none of which it observes. It is fully
> caught up. This is why `diagnose` reports failed partitions rather than sequence lag: lag is
> ambiguous, a failed partition is not.

`Disconnected` means no client is attached — usually the application is not running.
It is the normal state for a store whose application is stopped, and it is not an error.
> **`LastHandled#` lagging the tail is normal, and is not the same thing as being behind.**
> Every observer above has `Next#` 23 against a tail of 22 — all of them are caught up. But
> `Members` last handled sequence 2, because no member has registered since; nothing between 3
> and 22 was addressed to it. The two columns answer different questions: `Next#` is how far it
> has read, `LastHandled#` is the last thing it cared about.
>
> This is why `diagnose` reports failed partitions rather than sequence lag. Lag is ambiguous.
> A failed partition is not.

`Disconnected` means no client is attached — usually the application is not running. It is the
normal state for a store whose application is stopped, and it is not an error.

## Following a failure to the event that caused it

A partition is one event source's slice of an observer. When processing throws, Chronicle
stops **that partition** and leaves the rest of the observer running, so one bad entity does
not halt everything. `failed-partitions show` prints what happened, per attempt:

<div align="center">
<img src="assets/triage.gif" alt="diagnose flagging one failed partition in an otherwise healthy store, listing it, then printing the exception that stopped it" width="860">
</div>

<sub>Nine observers running, one partition stopped. The verdict names the next command, the
list names the observer and the partition, and the partition turns out to be a book —
`978-0131177055` — whose overdue notice could not be sent.</sub>

The partition is the ISBN because that is the event source id this application uses. Whatever
your entities are keyed by is what you will see here, which is what makes the failure
addressable rather than merely reported:

```
FailedPartition: 3447a2eb-5dc8-41d5-ab80-30948951dd44
FailedPartition: caadc869-1251-41d0-9063-6947eaf74043
Observer: Bookshop.OverdueNotices
Partition: 00000014-1111-4222-8333-444444444444
Attempts: 4
Partition: 978-0131177055
Attempts: 5

--- Attempt at 2026-07-27T23:46:06.9840000+00:00 (Seq# 22) ---
--- Attempt at 2026-07-28T12:17:56.6680000+00:00 (Seq# 22) ---
Exception has been thrown by the target of an invocation.
smtp.bookshop.local: connection refused
...
--- Attempt at 2026-07-27T23:46:21.4580000+00:00 (Seq# 22) ---
```

Four attempts, and the gaps between them widen — 2 seconds, then 4, then 8. Chronicle backs
off and keeps retrying on its own, so a partition that failed on something transient
recovers without you. `retry-partition` exists for the other case: you have fixed the cause
and do not want to wait for the next attempt.
The gaps between attempts widen — 2 seconds, then 4, then 8. Chronicle backs off and keeps
retrying on its own, so a partition that failed on something transient recovers without you.
`retry-partition` exists for the other case: you have fixed the cause and do not want to wait
for the next attempt.

```bash
cratis chronicle observers retry-partition Bookshop.OverdueNotices 00000014-…-444444444444 -y
cratis chronicle observers retry-partition Bookshop.OverdueNotices 978-0131177055 -y
```

> [!WARNING]
Expand All @@ -249,12 +263,13 @@ selected observer, `T` retries a failed partition, `S` and `U` stop and resume j
`Ctrl+P` is the part worth knowing about. It searches **every kind of artifact at once**:

<div align="center">
<img src="assets/palette.gif" alt="typing one word into the workbench command palette and matching an observer, an event type, a projection, a read model and a failure at the same time" width="860">
<img src="assets/workbench.gif" alt="filtering the workbench observers view down to one application, then searching every artifact kind at once from the command palette" width="860">
</div>

<sub>One word, six matches, five kinds — the reactor and the projection's observer, the event
type they both read, the projection declaration, the read model it writes, and the failure the
reactor left behind. Picking one jumps to its view with the filter already applied.</sub>
<sub>`F` narrows the view to one application. `Ctrl+P` then matches a single word across five
kinds at once — the reactor, the projection's observer, the event type they both read, the
projection declaration and the read model it writes. Picking one jumps to its view with the
filter already applied.</sub>

That breadth is the reason it is a palette and not a search box. "Overdue" is not a name you
look up in one list; it is a thread running through five of them, and following it is what
Expand Down Expand Up @@ -331,11 +346,11 @@ can work out that a stuck observer means `failed-partitions show` without being
It is not a static word list:

<div align="center">
<img src="assets/completions.gif" alt="pressing tab after cratis chronicle observers replay and getting the nine observer ids registered on the live server" width="860">
<img src="assets/completions.gif" alt="pressing tab after cratis chronicle read-models instances and getting the read model names registered on the live server" width="860">
</div>

<sub>Completing an observer id shells back into the CLI, which connects and returns what that
server has registered right now. Typing narrows to the two that match.</sub>
<sub>Completing a read model name shells back into the CLI, which connects and returns what that
server has registered right now — then the completed command runs against it.</sub>

Observers, event stores, event types, projections, read models, jobs, recommendations,
subscriptions, applications and users all complete this way — and context names, which come
Expand Down Expand Up @@ -487,9 +502,8 @@ observes one thing, so a failure reads as a sentence.
They are scripted, not screen-captured, and re-render from a clean checkout:

```bash
dotnet build -c Release
assets/demo-store/reset.sh # a throwaway Chronicle server with a story seeded into it
vhs assets/demo.tape # or palette, completions
assets/record.sh # every clip: sets up the store, waits for it, renders
assets/record.sh workbench # or just one
```

[`assets/RECORDING.md`](assets/RECORDING.md) covers how they were made, what the fixture
Expand Down
80 changes: 49 additions & 31 deletions assets/RECORDING.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,8 @@ Everything in this directory that ends in `.tape` is a script. Nothing was perfo
nothing was screen-captured, and any of it re-renders from a clean checkout:

```bash
dotnet build -c Release
assets/demo-store/reset.sh
vhs assets/demo.tape # or palette, completions
assets/record.sh # all four
assets/record.sh workbench # or one
```

That is the point. A hand-recorded screencast is a one-off you cannot fix a typo in; a tape is
Expand Down Expand Up @@ -34,20 +33,26 @@ customer system — so [`demo-store/`](demo-store/) builds a third option: a thr
in Docker with a small bookshop seeded into it.

```bash
assets/demo-store/reset.sh # fresh server, seed, exit
assets/demo-store/reset.sh drip 40 # append a trickle against a running one
assets/demo-store/reset.sh # fresh server, seed, exit
assets/demo-store/reset.sh serve 90 # stay connected and healthy
assets/demo-store/reset.sh serve-failing 90 # stay connected with one partition failing
```

It registers eight books, three members, two projections, two reducers and one reactor, then
appends a story: borrowings, two returns, two reservations, two books going overdue. The
reactor that sends overdue notices **throws for exactly one book**, which is what leaves a
failed partition behind for the CLI to find.

Two properties matter:
Three properties matter:

- **Deterministic identifiers.** Event source ids are generated from a fixed pattern rather
than `Guid.NewGuid()`, so `00000014-1111-4222-8333-444444444444` is the same book on every
re-render and the tapes can name it directly.
- **Domain identifiers, not GUIDs.** A book is keyed by its ISBN and a member by a handle, so
`978-0131177055` is the same book on every re-render, the tapes can name it directly, and
every column the CLI prints stays readable. The first version of this fixture used
`Guid.NewGuid()` and the read model table came out as two columns of wrapped hex.
- **A connected client.** Three of the four clips run against `serve`, so observers report
Active. A client that has exited leaves everything `Disconnected`, which looks like a broken
system when it is only a stopped process — the first cut of this set had three red
`Disconnected` rows in the hero for no reason other than that the seeder had finished.
- **The failure survives.** Chronicle retries a failed partition on its own with a widening
backoff, and it *succeeds* the moment the client reconnects without the fault. So the seeder
exits after seeding and the partition stays failed. An earlier version of the hero tape ran a
Expand Down Expand Up @@ -151,8 +156,8 @@ Two consequences worth knowing before editing a tape:
## The settings, and what each is for

All of them live in [`_style.tape`](_style.tape), which every other tape pulls in with
`Source assets/_style.tape`. Three recordings that drift apart in font size or theme read as
three screencasts; one shared file makes them read as one set.
`Source assets/_style.tape`. Four recordings that drift apart in font size or theme read as
four screencasts; one shared file makes them read as one set.

| Setting | Why |
|---|---|
Expand All @@ -179,40 +184,53 @@ are not written the same way — and `completions.tape` has to be zsh.

The most useful discipline here was editorial, not technical.

**One hero, then short single-purpose clips.** The top of the README gets the full arc — the
verdict, the exception behind it, the live view — at about 35 seconds. Everything after is
15-20 seconds and shows exactly one thing, sitting next to the prose that explains it.
**Show the tool working, not the system failing.** The first version of this set was built
entirely around a broken store: the hero opened on red ✗ marks and a four-deep stack trace,
and three observers showed `Disconnected` because the seeder had exited. It read as "this is
for when everything is on fire" rather than "this is what the tool does". Three of the four
clips now run against a healthy, connected system, and exactly one is about a failure.

**One hero, then short single-purpose clips.** The top of the README gets the three questions
you actually open the tool to answer — is it healthy, what happened, what state did that
produce — at about 29 seconds. Everything after is 15-25 seconds and shows one thing, sitting
next to the prose that explains it.

**A GIF has to show something text cannot.** That is the whole test.

- ✅ `demo.gif` — `diagnose` printing a verdict and naming the next command, then a full-screen
dashboard painting itself over the same terminal and narrowing live as a filter is typed.
- ✅ `palette.gif` — one word matching an observer, an event type, a projection, a read model
and a failure *at the same time*. A screenshot shows six rows; it cannot show them arriving
as the query resolves.
- ✅ `demo.gif` — the verdict, then the raw log, then the state those events produced. Three
commands, one question each, in the order they get asked.
- ✅ `workbench.gif` — a full-screen dashboard narrowing live as a filter is typed, then one
word matching five kinds of artifact *at the same time*. A screenshot shows five rows; it
cannot show them arriving as the query resolves.
- ✅ `completions.gif` — a tab press that makes a network call. There is no way to convey that
the list came from the server rather than from the script, except by watching it happen.
- ❌ **The failed-partition trail on its own.** Four attempts of the same stack trace is a wall
of output. It is in the hero, where it lasts seven seconds and carries the narrative, and the
README quotes it as text where it is searchable and loads instantly.
- ✅ `triage.gif` — one failed partition in a healthy store, and the trail from the verdict to
the exception. Kept short and kept to one clip: four attempts of the same stack trace is a
wall of output, and the README quotes the interesting part as text where it is searchable.
- ❌ **The output formats.** The same command four ways is a table with byte counts, not a
recording.

**Watch the total weight.** Three GIFs, 1.1 MB.
**Watch the total weight.** Four GIFs, under 1 MB.

## Nothing destructive is typed on camera

## Recording something destructive
An earlier cut of `completions.tape` used `cratis chronicle observers replay ` to demonstrate
completion, because an observer id is a good thing to complete. That command reprocesses an
observer from sequence zero. It never ran — the tape stopped short of `Enter` and ended on
`Ctrl+C` — but it meant a single stray keystroke in a tape stood between a demo and a replay.

`completions.tape` types `cratis chronicle observers replay ` on camera — a command that
reprocesses an observer from sequence zero. It exists in the clip because an observer id is the
best demonstration of completion hitting the server, and it is safe because **the tape never
sends `Enter`**. It ends on `Ctrl+C`, which abandons the line.
It now completes `cratis chronicle read-models instances ` instead, which is read-only, and the
clip is better for it: the completion list has no `$system` entries in it, so there is no
backslash-escaped noise on screen either.

Audit before rendering:
Audit before rendering, and again whenever a tape changes:

```bash
awk '/^Show/{s=1} s' assets/completions.tape | grep -n "Enter"
grep -n "replay\|retry\|remove\|delete" assets/*.tape
```

That should return nothing but comment lines. Then verify the server afterwards, as above.
That should return nothing. If a clip ever does need a destructive command, keep it off `Enter`
and verify the server before and after — capture the tail sequence number, the failed-partition
count and every observer's position, and diff them.

No README asset justifies a tape typo replaying somebody's observer.
Binary file modified assets/completions.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading