Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/clients/acceptance/connection-kv-stream.md
Original file line number Diff line number Diff line change
Expand Up @@ -382,7 +382,7 @@ reject or surface the broker rejection for partial and cross-realm selectors.
**When:** Client sends `Append(session_id, expected_offset=99999, payload="event2")`
**Then:**

- Server returns error (status=1) with message indicating concurrency conflict (e.g. containing "conflict")
- Server returns error (status=2) with numeric domain code 2001, independently of message wording
- No new record is appended
- Clients MUST send expected_offset on every Append; servers MUST enforce it

Expand Down
50 changes: 33 additions & 17 deletions docs/clients/spec/notice-stream.md
Original file line number Diff line number Diff line change
Expand Up @@ -405,6 +405,18 @@ Stream is not a queue, a broker-managed consumer group system, an exactly-once c
| 608 | UNSUBSCRIBE | Client → Server |
| 609 | NOTIFY | Server → Client (delivery) |

#### Error envelope generation 2

All Stream request errors except READ use `[u8 status=2][u32 BE domain_code][string message]`.
READ retains `[u8 status=1][u32 BE domain_code][string message]`.
Success and NOTIFY layouts are unchanged. Codes are stable; messages are diagnostic.
Unclassified backend failures use `2012`, never an inferred concurrency conflict.

Updated SDKs also accept legacy status-1 non-READ errors (`[string message]`),
preserving the absence of a domain code. They MUST NOT infer codes from wording.
Upgrade all five SDKs before the broker; old decoders cannot interpret status 2.
See [migration guidance](../../operations/migration-guide.md).

#### BEGIN Request

```
Expand All @@ -418,8 +430,9 @@ Response (status=0):
[u64 BE] session_id
[u32 BE] data_len
[bytes] data (broker-defined opaque bytes; clients MUST treat as opaque and MAY ignore)
Response (status=1):
[u8] 1
Response (status=2):
[u8] 2
[u32 BE] error_code
[u32 BE] error_len
[bytes] error_msg
```
Expand All @@ -443,13 +456,14 @@ Response (status=0):
[u8] 0
[u32 BE] data_len
[bytes] data
Response (status=1):
[u8] 1
Response (status=2):
[u8] 2
[u32 BE] error_code
[u32 BE] error_len
[bytes] error_msg
```

**expected_offset (OCC):** Clients MUST send `expected_offset` on every APPEND. It is the client's view of the stream's next write offset for that route (0 for a new stream). Servers MUST enforce it: if `expected_offset` does not match the server's next offset for that route, the server MUST reject the append with status=1 and an error message (e.g. containing "conflict"). This provides optimistic concurrency control; clients that receive a conflict should re-read the stream and retry with the correct offset.
**expected_offset (OCC):** Clients MUST send `expected_offset` on every APPEND. It is the client's view of the stream's next write offset for that route (0 for a new stream). Servers MUST enforce it: if `expected_offset` does not match the server's next offset for that route, the server MUST reject the append with status=2 and domain code `2001`. COMMIT MUST also return `2001` when a competing commit invalidates a staged batch. Clients MUST classify conflicts by code, independently of message wording. The application decides whether to reload and execute a new command; SDKs MUST NOT automatically retry APPEND or COMMIT.

**Optional discriminator:** Clients MAY include an immutable discriminator string on APPEND. The broker stores it as a replay sidecar and uses it only for filtered reads. Clients that do not need filtered replay SHOULD omit it.

Expand All @@ -473,8 +487,9 @@ Response (status=0):
[u8] 0
[u32 BE] data_len
[bytes] data
Response (status=1):
[u8] 1
Response (status=2):
[u8] 2
[u32 BE] error_code
[u32 BE] error_len
[bytes] error_msg
```
Expand All @@ -490,8 +505,9 @@ Response (status=1):
[u64 BE] session_id
Response (status=0):
[u8] 0
Response (status=1):
[u8] 1
Response (status=2):
[u8] 2
[u32 BE] error_code
[u32 BE] error_len
[bytes] error_msg
```
Expand Down Expand Up @@ -759,12 +775,11 @@ class StreamSession:

#### Error Codes (2xxx)

Stream uses operation-specific error envelopes. `READ` errors are
`[status=1][u32 error_code][string message]` and preserve the numeric 2xxx
code. Every other Stream operation uses the plain
`[status=1][string message]` envelope. Clients must select the decoder from the
request message type; they must not consume the first four bytes of a
non-`READ` message as an error code.
Stream READ retains `[status=1][u32 BE error_code][string message]`.
Every other Stream operation uses `[status=2][u32 BE error_code][string message]`.
For legacy brokers only, non-READ status 1 means `[string message]` with no code.
Select the decoder using both status and request message type; never infer a
code from message wording.

- 2001 = ERR_CONCURRENCY_CONFLICT (expected_offset mismatch)
- 2002 = ERR_SESSION_ALREADY_ACTIVE
Expand Down Expand Up @@ -840,8 +855,9 @@ Unsubscribe from stream change notifications.
[bytes] route_pattern
Response (status=0):
[u8] 0
Response (status=1):
[u8] 1
Response (status=2):
[u8] 2
[u32 BE] error_code
[u32 BE] error_len
[bytes] error_msg
```
Expand Down
19 changes: 19 additions & 0 deletions docs/development/format-compatibility.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,25 @@

This page defines compatibility expectations for serialized data, protocol payloads, and storage formats.

## Stream error envelope generation 2

Stream BEGIN, APPEND, COMMIT, ROLLBACK, LAST, GET_METADATA, SUBSCRIBE, and
UNSUBSCRIBE errors now use `[u8 2][u32 BE domain_code][u32 BE message_length][UTF-8 message]`.
READ retains its coded status-1 envelope. Success and notification layouts and
persisted data are unchanged by this error-envelope change.

Deploy updated .NET, TypeScript, Go, Python, and Rust SDKs before the broker.
Updated clients decode both generations; legacy non-READ status-1 errors have
no structured code. Old SDKs cannot decode generation 2, so the broker upgrade
must wait until all consumers have migrated. Roll back the broker first while
keeping the dual-generation clients. No capability negotiation is performed.

Concurrency conflicts carry `2001`; unclassified backend errors carry `2012`.
Preserve unknown codes and original exceptions. Never classify message wording
or automatically retry a failed append/commit. Applications own command retries.
The release checklist must record exact broker and SDK versions and requalify
Portia's stale-append and pending-batch assertions before release.

## Compatibility Rules

1. Backward-incompatible wire changes require explicit release notes and migration guidance.
Expand Down
11 changes: 11 additions & 0 deletions docs/development/release-policy.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,3 +16,14 @@ Fitz release work is focused on explicit change communication and operator safet
4. Breaking wire releases include matching supported-client changes and a mixed-version prohibition when no negotiation exists.

Use [../operations/release-checklist.md](../operations/release-checklist.md) before final publish.

## Stream error envelope generation 2 release gate

For issue #238, record the released .NET, TypeScript, Go, Python, and Rust SDK
versions that decode status 2 before releasing the broker change. Verify legacy
status-1 decoding, APPEND and COMMIT code `2001` over a real broker, unrelated
wording with `2001`, misleading wording with another code, and backend `2012`.
Requalify both linked Portia assertions using the exact broker and client
artifacts, including original failure and pending-batch preservation through
cleanup failures. Follow the client-first upgrade and broker-first rollback in
[migration guidance](../operations/migration-guide.md).
76 changes: 76 additions & 0 deletions docs/development/stream-error-238-evidence.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
# Stream error codes: issue #238 qualification

Local qualification on 2026-09-06. These results describe uncommitted working
changes on `fix/stream-error-codes-238`, based on broker
`6f6e7c3ca71d8a9615e7f795cfa01329b3e23f95`. No package or broker release was
published by this qualification.

## Contract

Non-READ Stream errors use status 2, followed by a big-endian u32 code and a
length-prefixed UTF-8 message. READ retains its status-1 coded envelope.
Both generations are decoded by the coordinated .NET, TypeScript, Go, Python,
and Rust SDK changes. Legacy uncoded errors remain unclassified.

APPEND and COMMIT expose 2001 for OCC; backend failures use 2012. Explicit
codes survive unrelated wording and misleading conflict wording. Ingress
backpressure/timeout responses use the same Stream envelope and retain their
existing numeric codes. Subscription errors preserve 2010/2011.

## Broker validation

- `cargo fmt --all -- --check`: passed.
- `cargo test --locked --workspace`: 2,192 passed, zero failed, one existing
ignored benchmark-artifact acceptance test.
- Strict workspace Clippy with all targets/features: passed.
- Stream transport suite: 93 passed across TCP and WebSocket.
- Wire-contract suite: five passed, including a real store commit conflict.

A single resource permits only one active network append session. The
commit-time regression therefore stages two actors against one real store,
commits the first, and verifies the second store commit fails and encodes 2001.
It also verifies that the losing actor retains its active session for explicit
cleanup. This does not claim that two network clients can stage simultaneously.
Stale APPEND is exercised over both TCP and WebSocket.

## SDK validation

- .NET: 304 unit/Stream tests passed before adding three legacy-envelope cases;
all 13 dedicated envelope cases passed afterward. The full run passed 330
tests but failed the reconnect conformance aggregate: its fixture restarts
Docker Compose while the qualification endpoint is a separately started
local binary. Full restart qualification remains a release gate.
- TypeScript: format, lint, build, 616 unit tests, 44 Stream integration tests,
361 full integration tests, and 18 conformance tests passed.
- Go: `go test ./...` passed.
- Python: Ruff format/lint and 156 unit tests passed; integration reported six
passed and two pre-existing prerequisite skips.
- Rust: all-target/all-feature tests reported 100 passed and three existing
ignored broker tests; formatting, strict Clippy, and test-policy validation
passed.

## Portia packed-consumer qualification

An isolated copy of Portia `a3913d539108f975636d014bc74c9f5f69900c99` used locally
packed `Cntryl.Fitz` and `Cntryl.Fitz.Abstractions` version `0.1.2-issue238`.
NuGet source mapping selected those packages from a private local folder;
the original Portia checkout was unchanged. The endpoint was the newly built
broker binary at `ws://127.0.0.1:4190/ws`, with memory storage.

- `ShouldThrowConcurrencyExceptionWhenAppendingWithStaleExpectedVersion`:
passed (one case).
- `CompetingRaisedEventsStillConflictAndKeepPendingChanges` plus
`FitzPersistenceFailureTests`: passed (10 cases).

The failure tests preserve the original exception as the translated conflict's
inner exception, retain the pending batch, and preserve the original append or
commit failure through rollback/disposal failures. Misleading text with another
code and uncoded text are not translated. No automatic command retry was added.

## Release boundary

These local artifacts are not released versions. Publish and record all five
SDK versions before releasing the broker; then repeat the Portia assertions
against the exact released SDK packages and broker image digest. Follow the
client-first upgrade and broker-first rollback in the migration guide. Issue
#238 must remain open until coordinated release qualification is recorded.
19 changes: 19 additions & 0 deletions docs/operations/migration-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,25 @@

This guide covers safe upgrades between Fitz releases.

## Stream error envelope generation 2

Stream BEGIN, APPEND, COMMIT, ROLLBACK, LAST, GET_METADATA, SUBSCRIBE, and
UNSUBSCRIBE errors now use `[u8 2][u32 BE domain_code][u32 BE message_length][UTF-8 message]`.
READ retains its coded status-1 envelope. Success and notification layouts and
persisted data are unchanged by this error-envelope change.

Deploy updated .NET, TypeScript, Go, Python, and Rust SDKs before the broker.
Updated clients decode both generations; legacy non-READ status-1 errors have
no structured code. Old SDKs cannot decode generation 2, so the broker upgrade
must wait until all consumers have migrated. Roll back the broker first while
keeping the dual-generation clients. No capability negotiation is performed.

Concurrency conflicts carry `2001`; unclassified backend errors carry `2012`.
Preserve unknown codes and original exceptions. Never classify message wording
or automatically retry a failed append/commit. Applications own command retries.
The release checklist must record exact broker and SDK versions and requalify
Portia's stale-append and pending-batch assertions before release.

## Rust embedding API: write policies and delivery errors

`KvMessage::Begin::write_options` now takes `fitz::domains::WritePolicy` so the
Expand Down
11 changes: 11 additions & 0 deletions docs/operations/release-checklist.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,3 +27,14 @@ Use this checklist before approving a Fitz release.
1. Engineering sign-off.
2. Operations sign-off.
3. Security sign-off for auth or policy changes.

## Stream error envelope generation 2 release gate

For issue #238, record the released .NET, TypeScript, Go, Python, and Rust SDK
versions that decode status 2 before releasing the broker change. Verify legacy
status-1 decoding, APPEND and COMMIT code `2001` over a real broker, unrelated
wording with `2001`, misleading wording with another code, and backend `2012`.
Requalify both linked Portia assertions using the exact broker and client
artifacts, including original failure and pending-batch preservation through
cleanup failures. Follow the client-first upgrade and broker-first rollback in
[migration guidance](../operations/migration-guide.md).
12 changes: 11 additions & 1 deletion src/api/runtime_ingress/domain_frame_dispatcher.rs
Original file line number Diff line number Diff line change
Expand Up @@ -287,7 +287,17 @@ impl DomainFrameDispatcher<'_> {
domain_code: u16,
message: &'static str,
) -> Result<(), IngressDecision> {
let payload = Self::encode_domain_error_body(domain_code, message);
let payload = if (600..=608).contains(&msg_type.0) {
crate::protocol::stream_codec::encode_error_response_into(
&mut crate::protocol::payload_codec::PayloadEncoder::new(),
msg_type.0,
domain_code,
message,
)
.into()
} else {
Self::encode_domain_error_body(domain_code, message)
};
let response_ctx = crate::protocol::frame_context::FrameContext::new(
session_id,
channel_id,
Expand Down
14 changes: 12 additions & 2 deletions src/api/runtime_ingress/tests/domain_backpressure.rs
Original file line number Diff line number Diff line change
Expand Up @@ -406,7 +406,12 @@ fn should_not_close_session_when_a_domain_command_times_out() {
);
// Error body: [u8 flag][u32 code][string message].
let body = &frames[0].payload;
assert_eq!(body[0], 1, "{} should send an error body", case.domain);
assert_eq!(
body[0],
if case.domain == "stream" { 2 } else { 1 },
"{} should send an error body",
case.domain
);
let code = u32::from_be_bytes([body[1], body[2], body[3], body[4]]);
// A timed-out command was already enqueued and may still run, so the
// code must not be one `REQ-PROTO-012` classifies as retryable. Those
Expand Down Expand Up @@ -496,7 +501,12 @@ fn should_answer_sustained_mailbox_backpressure_without_killing_the_session() {
// one `REQ-PROTO-012` classifies as retryable. A fatal code here makes
// a compliant client give up on a request it could safely re-send.
let body = &frames[0].payload;
assert_eq!(body[0], 1, "{} should send an error body", case.domain);
assert_eq!(
body[0],
if case.domain == "stream" { 2 } else { 1 },
"{} should send an error body",
case.domain
);
let code = u32::from_be_bytes([body[1], body[2], body[3], body[4]]);
assert!(
DOCUMENTED_RETRYABLE_CODES.contains(&code),
Expand Down
3 changes: 2 additions & 1 deletion src/domains/stream/sink/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -323,9 +323,10 @@ fn decode_stream_error_message(payload: &[u8]) -> Result<String, String> {
return Ok(message);
}
let mut decoder = crate::dispatch::protocol::payload_codec::PayloadDecoder::new(payload);
if decoder.get_u8()? != 1 {
if decoder.get_u8()? != 2 {
return Err("stream response is not an error".to_string());
}
decoder.get_u32()?;
decoder.get_string()
}

Expand Down
Loading