From a3b843add78600484dfaa9ef227c7a0e6da37db2 Mon Sep 17 00:00:00 2001 From: nsaspy <104283403+lost-rob0t@users.noreply.github.com> Date: Thu, 13 Aug 2026 01:50:24 -0400 Subject: [PATCH] design StarRouter stabilization and StarIntel bridge --- docs/design/001-current-state-code-audit.org | 296 ++++++++++++++++ .../002-delivery-contract-and-protocol.org | 270 +++++++++++++++ .../003-starintel-bridge-architecture.org | 316 ++++++++++++++++++ .../004-durable-outbox-and-recovery.org | 303 +++++++++++++++++ .../005-testing-chaos-and-benchmark-plan.org | 309 +++++++++++++++++ ...bservability-and-unattended-operations.org | 309 +++++++++++++++++ .../007-performance-and-backpressure.org | 288 ++++++++++++++++ ...8-security-compatibility-and-migration.org | 313 +++++++++++++++++ docs/design/index.org | 142 ++++++++ 9 files changed, 2546 insertions(+) create mode 100644 docs/design/001-current-state-code-audit.org create mode 100644 docs/design/002-delivery-contract-and-protocol.org create mode 100644 docs/design/003-starintel-bridge-architecture.org create mode 100644 docs/design/004-durable-outbox-and-recovery.org create mode 100644 docs/design/005-testing-chaos-and-benchmark-plan.org create mode 100644 docs/design/006-observability-and-unattended-operations.org create mode 100644 docs/design/007-performance-and-backpressure.org create mode 100644 docs/design/008-security-compatibility-and-migration.org create mode 100644 docs/design/index.org diff --git a/docs/design/001-current-state-code-audit.org b/docs/design/001-current-state-code-audit.org new file mode 100644 index 0000000..95ffc2a --- /dev/null +++ b/docs/design/001-current-state-code-audit.org @@ -0,0 +1,296 @@ +#+title: StarRouter 001 - Current-State Code Audit +#+options: toc:3 + +* Scope + +This audit covers the current StarRouter broker, client library, protocol definitions, tests, build files, and operational defaults. It is intentionally implementation-oriented: each finding names the affected code path, expected failure mode, severity, and stabilization requirement. + +The codebase is small enough that the complete active routing surface can be treated as one system rather than as isolated modules. + +* Executive finding + +StarRouter has a useful minimal architecture -- ZeroMQ ROUTER/DEALER for command traffic and PUB/SUB for fanout -- but the current implementation cannot yet provide a reliable durable delivery contract. + +There are several source-verifiable correctness bugs in the active message path, and the existing test files are examples/long-running clients rather than automated tests. The correct next move is not a rewrite. Stabilize the protocol boundary, build a deterministic harness, add durability/backpressure semantics, then add the StarIntel bridge behind those contracts. + +* P0 correctness findings + +** P0-1: client messages are processed twice + +File: =src/starRouterpkg/server.nim= +Procedure: =handleMessage= +Status: *verified from code* + +For the =SC01= path, =handleMessage= dispatches =msg.typ= inside a =try/finally= block and then performs a second =case msg.typ= immediately afterward. + +Consequences include: + +- =newDocument= may be published twice; +- actor registration may run twice; +- target routing may run twice; +- ACK frames are emitted in the =finally= block and again in the second dispatch; +- a queued/stale ACK can be consumed by a later client request because replies are not correlated by message id. + +Requirement: + +- exactly one dispatcher invocation per accepted frame set; +- exactly one terminal protocol reply per request; +- test exact publish count and exact reply count for every event type. + +** P0-2: =Client.newMessage= drops the requested event type + +File: =src/starRouterpkg/client.nim= +Procedure: =newMessage= +Status: *verified from code* + +The procedure accepts =eventType= but does not assign it to the returned =Message.typ=. This can serialize an unintended default enum value instead of the event selected by the caller. + +Requirement: + +- assign every required field explicitly; +- remove unused parameters or make them authoritative; +- add constructor round-trip tests for every =EventType=. + +** P0-3: target dispatch mutates one message and publishes another + +File: =src/starRouterpkg/server.nim= +Procedure: =handleTarget= +Status: *verified from code* + +The procedure copies =msg= into =newMsg=, selects an actor, changes =newMsg.topic= and =newMsg.typ=, then calls =publishClientMessage(msg)= rather than publishing =newMsg=. + +The intended targeted routing is therefore not represented in the published frames. + +Requirement: + +- fix only after a target-routing conformance test exists; +- make the original message immutable in routing code where practical; +- test selected actor id and emitted topic. + +** P0-4: =emit= is not a reliable request operation + +File: =src/starRouterpkg/client.nim= +Procedure: =emit= +Status: *verified from code* + +The procedure exposes =tries= but does not use it. It performs one send, waits indefinitely for one reply, and does not inspect whether the reply is ACK or NACK. There is no request id in the reply, so a duplicate/stale ACK can satisfy a later request. + +Requirement: + +- bounded request deadline; +- correlated reply containing message id / request tracker; +- explicit ACK/NACK/error parsing; +- retry policy that preserves the original message id; +- retry only when the operation is safe under at-least-once semantics. + +** P0-5: current PUB/SUB path is not a durable bridge transport + +Files: =server.nim=, =client.nim=, =tests/testSUb.nim= +Status: *verified design limitation; runtime loss tests required* + +The test subscriber itself notes that a subscriber can miss the publisher's first messages. ZeroMQ PUB/SUB also has finite high-water marks and does not provide application-level durable replay. + +Requirement: + +- never define StarIntel durable ingest correctness as "the bridge subscribed to PUB"; +- retain PUB/SUB for transient fanout; +- create a durable bridge acceptance path/outbox. + +* P1 reliability and routing findings + +** P1-1: actor-name parsing breaks hyphenated actor names + +File: =server.nim= +Procedure: =bumpActor(router, id)= +Status: *verified from code* + +Actor ids are generated as =actorName-ULID=. The server recovers the actor name with =id.split("-")[0]=. For an actor name such as =github-actor=, this produces =github= rather than =github-actor=. + +Requirement: never derive semantic actor names by splitting a composite id. Keep actor name as explicit state or map actor id directly to registration metadata. + +** P1-2: liveness initialization ignores configured =maxLives= + +File: =server.nim= +Procedure: =newActor= +Status: *verified from code* + +=newActor= hardcodes =liveness = 5= while the router has =maxLives=. + +Requirement: one authoritative liveness policy. + +** P1-3: liveness damage is loop-frequency dependent + +File: =server.nim= +Procedures: =hurtActors=, =run= +Status: *verified from code; timing impact requires runtime validation* + +Once an actor is considered stale, =hurtActors= can decrement a life on every broker loop iteration. Under heavy traffic that loop can run rapidly, so actor eviction speed may depend on unrelated message rate rather than heartbeat intervals. + +Requirement: + +- represent liveness as an expiry/deadline rather than repeated damage; +- use a monotonic clock for elapsed-time decisions; +- remove a client only when an explicit deadline expires, or count missed heartbeat epochs exactly once. + +** P1-4: KeyError recovery mutates registry state + +File: =server.nim= +Procedure: =handleMessage= +Status: *verified from code* + +A =KeyError= during dispatch is caught and handled by calling =registerActor(msg)= regardless of why the lookup failed. This can turn an unrelated malformed/unknown state into an actor registration and hides the root cause. + +Requirement: classify errors. Missing actor, malformed message, unsupported type, and internal registry corruption need distinct outcomes. + +** P1-5: malformed framing uses an assertion + +File: =server.nim= +Procedure: =handleMessage= +Status: *verified from code; process impact requires runtime validation* + +The delimiter frame is checked with =doAssert empty.len == 0=. A network peer should not be able to turn a protocol error into an assertion path. + +Requirement: reject malformed envelopes with a bounded protocol error, count the violation, and continue serving healthy peers. + +** P1-6: public interface defaults contradict safety documentation + +Files: =README.org=, =src/starRouter.nim= +Status: *verified from code/docs* + +README says not to expose StarRouter to the web. The CLI entry point defaults both endpoints to =tcp://*=. + +Requirement: loopback by default. Explicit opt-in is required for non-loopback binding. + +** P1-7: no explicit ZeroMQ backpressure/security socket policy + +Files: =server.nim=, =client.nim= +Status: *verified absence* + +The code does not configure application-specific HWM, send/receive timeouts, linger, immediate connection behavior, ROUTER mandatory delivery, authentication, or maximum accepted payload. + +Requirement: define each option by role rather than inherit library defaults accidentally. + +* P2 client/inbox findings + +** Inbox capacity is not initialized + +=newInbox(n)= constructs a deque with capacity but does not assign =Inbox.size=n=. =isFull= compares =size < len(documents)=, which is inconsistent with its name and does not enforce a bound. + +Requirement: make capacity explicit and implement a selected overload policy: block, reject, spill, or drop only for explicitly transient data. + +** Quiet =runStringInbox= heartbeat path is suspect + +The timeout branch does not mirror =runInbox= heartbeat behavior. Test that a completely idle string inbox maintains registration for multiple expiry windows. + +** Poll timeout units are inconsistent + +The router multiplies =timeout * 1000= before =poll= while client paths pass =timeout= directly even though the field is otherwise interpreted as seconds. Normalize time units in type/name/API: =timeoutMs= or duration type. + +** Filter/callback lifecycle is implicit + +Inbox loops call =filter= and =callback= without establishing defaults or validating registration. + +Requirement: fail construction/starting the loop with a clear configuration error rather than nil-callback runtime failure. + +** Registration messages lack normal envelope fields + +=register= initializes topic/source/type/data but not id/time. The transport should define which envelope fields are mandatory for control messages and enforce them uniformly. + +** Wall clock is used for liveness + +=utils.unix= is appropriate for externally meaningful timestamps but not ideal for process-local timeout measurement because wall time can jump. + +Requirement: carry both event timestamp and monotonic deadline internally. + +* Protocol design debt + +Current =Message[T]= contains =id, data, source, typ, time, topic=. Missing fields needed for robust distributed operation include: + +- protocol version; +- correlation/request id; +- causation/root id; +- payload content type / encoding; +- attempt number; +- TTL/deadline; +- optional trace context; +- schema/document dtype distinct from transport topic; +- extension metadata with forward-compatible unknown-field handling. + +Do not add every field to legacy SC01 frames immediately. Define a v0.4 canonical envelope internally, then provide SC01 decode/encode adapters. + +* Testing audit + +Current files under =tests/= + +- =test1.nim=: long-running integration/example client; no assertions; +- =testDocsPerSec.nim=: progress counter/infinite run, not a bounded benchmark; +- =testSUb.nim=: subscriber example/infinite run; +- =config.nims=: only source path configuration. + +The files import =unittest= in places but do not constitute an automated regression suite. + +There is no current test proving: + +- one request -> one dispatch; +- one request -> one ACK/NACK; +- constructor correctness; +- reconnect/retry behavior; +- actor liveness state transitions; +- target routing; +- malformed-frame isolation; +- subscriber overload behavior; +- graceful shutdown; +- bridge replay after outage; +- sustained throughput or latency percentiles. + +* CI/build/packaging audit + +** GitHub Actions + +The current workflow is a Docker build/publish workflow. It does not provide a dedicated Nim unit/integration test, lint/static check, protocol conformance, Nix check, sanitizer, or benchmark gate. + +** Nix + +=flake.nix= supplies an x86_64-linux development shell with Nim/Nimble and ZeroMQ library path. It does not define a package or =checks= outputs. + +Requirement: =nix flake check= must become the canonical reproducible test gate. + +** Docker + +The Dockerfile uses the mutable =nimlang/nim:onbuild= image and installs =libzmq5=. Reproducibility and exact Nim/package versions are therefore not fully controlled by the repository's Nix lock. + +Recommendation: build the OCI artifact from the Nix package after the package/check contract exists. + +** Dependency metadata + +=starRouter.nimble= pins =zmq= but leaves several dependencies unpinned and does not list optional paths consistently. README documents that =nimble install= errors while still acquiring dependencies. This is not a release-grade dependency contract. + +* Stabilization order + +Do not start with performance tuning. Correctness comes first. + +1. Build deterministic protocol and actor-registry tests around current behavior. +2. Fix duplicate dispatch/ACK behavior. +3. Fix constructors and target routing. +4. Add request correlation, deadlines, structured errors, and safe framing validation. +5. Add bounded queues/backpressure and explicit socket options. +6. Add bridge durable outbox + Rabbit confirms. +7. Add reconnect/replay/chaos tests. +8. Measure throughput and optimize only from profiles/benchmarks. +9. Harden network exposure/authentication. +10. Run soak tests before unattended deployment. + +* Release blocker definition + +StarRouter should not be labeled stable until all of the following are mechanically enforced: + +- zero known P0 correctness defects; +- deterministic tests for each event type; +- no silent durable-message loss in declared failure scenarios; +- bounded memory under overload; +- crash/restart recovery demonstrated; +- documented at-least-once semantics and dedup/idempotency keys; +- health/readiness/metrics available; +- loopback-safe defaults; +- reproducible build/test gate; +- multi-hour then multi-day soak tests with zero unexplained loss/duplication beyond specified replay behavior. diff --git a/docs/design/002-delivery-contract-and-protocol.org b/docs/design/002-delivery-contract-and-protocol.org new file mode 100644 index 0000000..ac151d5 --- /dev/null +++ b/docs/design/002-delivery-contract-and-protocol.org @@ -0,0 +1,270 @@ +#+title: StarRouter 002 - Delivery Contract and Protocol +#+options: toc:3 + +* Goal + +Make delivery behavior explicit enough that clients, the broker, and bridges can be independently tested against the same contract. + +The current SC01 protocol remains a compatibility protocol. A new internal canonical envelope and correlated request/reply contract should be introduced without forcing a flag-day migration. + +* Delivery semantics + +** Durable messages + +For messages declared durable, StarRouter v0.4 targets: + +- acceptance only after a durable local record exists; +- eventual forwarding while the destination is unavailable; +- at-least-once delivery after crashes/retries; +- stable message id across retries; +- no silent discard under overload; +- bounded local storage with an explicit rejection state when the bound is reached; +- deterministic quarantine for permanently invalid messages. + +The system does *not* promise exactly once. + +** Transient messages + +Transient fanout may retain the current PUB/SUB performance model. Loss is allowed only when the producer has explicitly selected transient semantics. + +Transport choice must therefore follow semantic class rather than treating all messages alike. + +* Canonical envelope + +Internally normalize SC01 and future protocols into one envelope before routing: + +#+begin_src text +Envelope + protocolVersion string + messageId string + correlationId string? + causationId string? + source string + eventType EventType + topic string + documentType string? + createdAtUnixMs int64 + deadlineUnixMs int64? + attempt uint32 + durable bool + contentType string + payload bytes + extensions map +#+end_src + +Rules: + +1. =messageId= is generated by the producer once and is immutable across retry/replay. +2. =correlationId= identifies a request/reply exchange. For normal producer submissions it can equal =messageId=. +3. =causationId= points to the message that caused this message when relevant. +4. =createdAtUnixMs= is event metadata, not a liveness clock. +5. =deadlineUnixMs= is optional and causes an expired message to become terminal rather than retried forever. +6. =documentType= is separate from =topic=. A routing topic and a StarIntel document dtype are not the same concept. +7. Unknown extension keys must survive decode/re-encode. + +* SC01 compatibility framing + +Current SC01 producer request frames are conceptually: + +#+begin_example +[empty] +[SC01] +[source] +[id] +[time] +[type] +[topic] +[payload] +#+end_example + +The ROUTER socket prepends the transport peer identity. + +Compatibility decoder requirements: + +- require exact minimum frame count; +- validate header before body parsing; +- enforce frame/payload size limits; +- validate integer ranges without assertions; +- reject unknown event types with protocol error; +- normalize timestamps; +- preserve original message id; +- never create registry state as an incidental parser error recovery path. + +* Correlated replies + +The current reply is effectively only an ACK ordinal. Replace/extend it with a response envelope: + +#+begin_src text +Reply + protocolVersion + correlationId + status ACK | NACK | ERROR | BUSY + code stable machine-readable code + retryable bool + acceptedAtUnixMs + detail optional bounded text +#+end_src + +Compatibility option: + +- legacy SC01 clients may continue receiving the one-frame ACK during a migration period; +- v0.4-capable clients negotiate/advertise correlated replies; +- never send both legacy and v0.4 terminal replies to the same negotiated request. + +Suggested error codes: + +- =BAD_FRAME= +- =BAD_HEADER= +- =BAD_EVENT_TYPE= +- =BAD_PAYLOAD= +- =MESSAGE_TOO_LARGE= +- =UNKNOWN_ACTOR= +- =UNSUPPORTED_OPERATION= +- =OVERLOADED= +- =OUTBOX_FULL= +- =DESTINATION_UNAVAILABLE= +- =INTERNAL_ERROR= + +* Acceptance states + +A durable producer submission moves through: + +#+begin_example +RECEIVED + -> VALIDATED + -> DURABLY_ACCEPTED + -> FORWARD_PENDING + -> FORWARDED_CONFIRMED + -> RETIRED +#+end_example + +Terminal alternate states: + +#+begin_example +REJECTED_INVALID +REJECTED_OVERLOAD +EXPIRED +QUARANTINED +#+end_example + +Only =DURABLY_ACCEPTED= is sufficient for the producer-facing durable ACK. + +If the compatibility mode cannot make a durable record, it must not claim durable acceptance. + +* Retry rules + +Retry is based on stable identity, not creation of a new logical message. + +- transport timeout before acceptance reply: retry same =messageId=; +- duplicate request with already accepted =messageId=: return the prior acceptance result, do not append a second logical outbox row; +- destination publish not confirmed: retry same outbox record; +- validation error: never retry automatically; +- unsupported event mapping: quarantine/reject, do not spin; +- expired deadline: terminal. + +Use exponential backoff with jitter for destination failures and a configured maximum delay. Do not use a small fixed retry loop that hammers an unavailable service. + +* Idempotency + +The broker/bridge should maintain an idempotency record for durable submissions keyed by =messageId= for at least the replay horizon. + +The downstream StarIntel document itself should also use deterministic =_id= values whenever the producer can derive them. Bridge-level deduplication reduces duplicates; downstream idempotency is still required because a publish can be accepted by RabbitMQ while its confirmation is lost before the bridge records success. + +* Ordering + +No global ordering guarantee is proposed. + +If a workload requires order, scope it explicitly, for example: + +- same document id; +- same partition key; +- same source actor. + +Then route that key through a single ordered lane. Do not serialize unrelated traffic globally. + +Updates to the same StarIntel document are especially sensitive because CouchDB revisions can conflict. The bridge should preserve source ordering metadata but StarIntel update handling must remain conflict-safe. + +* Heartbeat contract + +Heartbeats are transport health, not application delivery acknowledgements. + +Target behavior: + +- peer advertises heartbeat interval; +- each peer tracks an expiry deadline; +- receipt of any valid traffic can count as evidence of liveness if desired; +- expiry is evaluated by elapsed time, not by decrementing a life counter on every busy loop iteration; +- liveness uses a monotonic process clock; +- heartbeat events are never forwarded into StarIntel document ingest. + +The ZeroMQ Guide's Paranoid Pirate/Majordomo patterns are useful references for heartbeat expiry and worker liveness, but StarRouter should keep its protocol smaller than Majordomo unless it actually needs the full service-broker contract. + +* Socket requirements + +Document explicit values/config for: + +- =ZMQ_SNDHWM= and =ZMQ_RCVHWM=; +- =ZMQ_SNDTIMEO= and receive deadline behavior; +- =ZMQ_LINGER= on shutdown; +- =ZMQ_IMMEDIATE= for clients where queued sends to incomplete connections are undesirable; +- =ZMQ_ROUTER_MANDATORY= when targeted sends must fail instead of silently disappearing; +- heartbeat options where library/binding support is suitable; +- reconnect interval/backoff; +- maximum message/frame size at the application layer. + +Do not depend on defaults as an undocumented reliability policy. + +* Event type contract + +Current enum values are retained for compatibility: + +| Value | Event | v0.4 disposition | +|-------+-------+------------------| +| 0 | heartbeat | transport control | +| 1 | ack | transport control | +| 2 | nack | transport control | +| 3 | newDocument | durable-capable data | +| 4 | deleteDocument | data/control; downstream contract required | +| 5 | getDocument | request/reply; not a one-way bridge event | +| 6 | updateDocument | durable-capable data | +| 7 | register | transport/service registry control | +| 8 | target | service-routing control; mapping must be explicit | + +Do not renumber existing values. New values require protocol-version documentation and compatibility tests. + +* Broker-to-broker header + +=SR01= exists but is unimplemented. Do not build the StarIntel bridge by quietly assigning new semantics to =SR01=. + +If broker federation is later implemented, give it a separate ADR/design with loop prevention, hop budgets, identity, replay semantics, and authentication. + +* XRAP relationship + +The repository TODO already points to ZeroMQ RFC 40 XRAP. + +XRAP provides useful ideas: + +- request tracking; +- explicit request/reply resources; +- pipelining; +- structured status/error behavior; +- protocol signatures. + +Recommendation: borrow the tracker/correlation and explicit error concepts rather than immediately replacing StarRouter with full XRAP. Full XRAP adoption should be justified by concrete resource/RPC use cases such as =getDocument=, broker administration, or registry inspection. + +Reference: https://rfc.zeromq.org/spec/40/ + +* Conformance tests required + +For each SC01 request: + +- decoder accepts one valid fixture; +- decoder rejects each malformed frame independently; +- one logical dispatch occurs; +- one terminal response occurs; +- correlation is correct in v0.4 mode; +- retry preserves id; +- duplicate accepted id returns previous acceptance result; +- payload and topic are byte/character exact after round trip; +- unknown extensions survive internal normalize/encode paths; +- unsupported event never mutates actor registry. diff --git a/docs/design/003-starintel-bridge-architecture.org b/docs/design/003-starintel-bridge-architecture.org new file mode 100644 index 0000000..3b88ceb --- /dev/null +++ b/docs/design/003-starintel-bridge-architecture.org @@ -0,0 +1,316 @@ +#+title: StarRouter 003 - StarIntel Bridge Architecture +#+options: toc:3 + +* Objective + +Design a bridge process that accepts StarRouter messages and forwards compatible durable data into StarIntel Server without requiring StarRouter clients to speak AMQP and without making StarIntel Server understand the legacy ZeroMQ wire protocol. + +The bridge is an anti-corruption layer between two messaging models: + +- StarRouter: legacy ZeroMQ request + PUB/SUB broker protocol; +- StarIntel Server: RabbitMQ topic exchange for distributed document traffic, CouchDB for durable documents, Sento for local actors. + +* Primary architectural choice + +Use RabbitMQ as the bridge's StarIntel-facing interface. + +Do not make the HTTP API the primary high-volume bridge path. + +Reasons: + +1. RabbitMQ is already StarIntel's documented distributed actor/document boundary. +2. StarIntel's HTTP =POST /new/document/:dtype= only proves publication, not persistence. +3. The HTTP bulk route publishes elements sequentially and can partially accept a batch. +4. The HTTP producer is serialized through one pinned agent. +5. Rabbit routing keys directly express insert/update semantics. +6. A bridge can use persistent AMQP messages, mandatory routing, and publisher confirms as its own reliable publishing boundary. + +HTTP remains useful for administrative probes, document verification, or a fallback adapter in tests. + +* Target data path + +#+begin_example +SC01 client + | + v +StarRouter compatibility ingress + | + | canonical Envelope + v +Durable acceptance / outbox + | + +---------------------> transient PUB compatibility fanout + | + v +Bridge dispatcher + | + +-- validate JSON + +-- classify EventType + +-- derive/check dtype + +-- preserve ids/provenance + +-- select routing key + v +Confirmed Rabbit publisher + | + v +exchange: documents + | + +-- documents.ingest. + +-- documents.updated. + v +StarIntel consumers + | + v +CouchDB / documents.new. +#+end_example + +* Deployment shape + +Recommended initial shape: *separate bridge process*. + +Benefits: + +- crash isolation from the broker; +- independent restart and upgrade; +- explicit queues/outbox ownership; +- no AMQP dependency in the StarRouter protocol/client library; +- bridge can be disabled without changing ordinary PUB/SUB clients; +- simpler migration path if StarRouter is later replaced or embedded elsewhere. + +After the design is proven, the durable acceptance/outbox hook may live in the broker while the AMQP publisher remains a supervised bridge worker. + +* Bridge components + +** ZeroMQ ingress adapter + +Responsibilities: + +- consume only the designated reliable bridge stream, not arbitrary PUB traffic; +- decode current SC01-compatible envelope or a dedicated internal durable stream; +- validate frame bounds; +- normalize to canonical =Envelope=; +- never acknowledge durable acceptance until the outbox owns the message. + +The very first prototype may subscribe to PUB for compatibility experiments, but that mode must be labeled =best-effort= and may not satisfy the durable contract. + +** Validator / classifier + +Responsibilities: + +- parse payload as JSON when StarIntel mapping requires JSON; +- enforce object body where required; +- extract body =dtype=; +- compare transport/topic hints with body dtype; +- validate required fields for supported mappings; +- reject/quarantine unsupported message types deterministically. + +Do not silently mutate a conflicting dtype to make a message pass. + +** Durable outbox + +Responsibilities: + +- unique record by message id; +- store canonical envelope and original payload; +- track publish destination/routing key; +- record attempt count and next retry time; +- record last error; +- atomically move accepted -> pending/confirmed/quarantined states; +- recover pending records after restart. + +See =004-durable-outbox-and-recovery.org=. + +** Rabbit publisher worker(s) + +Responsibilities: + +- maintain long-lived AMQP connection/channel; +- use publisher confirms; +- publish important messages as persistent; +- use =mandatory= when the bridge expects the routing key to reach a queue; +- correlate confirms with outbox records; +- reconnect and replay unconfirmed records; +- bound in-flight confirm window; +- expose latency and failure metrics. + +** Verifier / optional persistence observer + +Publisher confirmation proves RabbitMQ accepted the publish according to its configured routing/durability semantics; it does not prove the StarIntel ingest consumer persisted the document. + +For workloads that require proof of persistence, add an optional asynchronous verifier: + +- observe =documents.new.= and match deterministic =_id= / bridge metadata; or +- query =GET /document/:id= after a bounded delay. + +This is a higher assurance mode, not required for the baseline forwarding ACK. + +* Event mapping + +| StarRouter event | Value | StarIntel mapping | Initial bridge status | +|------------------+-------+-------------------+-----------------------| +| heartbeat | 0 | none | never forward | +| ack | 1 | none | never forward | +| nack | 2 | none | never forward | +| newDocument | 3 | =documents.ingest.= | supported | +| deleteDocument | 4 | no documented Rabbit delete contract | reject/quarantine | +| getDocument | 5 | request/reply, possible HTTP GET | unsupported in one-way bridge | +| updateDocument | 6 | =documents.updated.= | supported after tests | +| register | 7 | none | never forward | +| target | 8 | semantic ambiguity; see below | conditional/future | + +* newDocument mapping + +Input requirements: + +- payload parses as JSON object; +- payload contains non-empty string =dtype=; +- message id exists; +- configured maximum payload is not exceeded. + +Output: + +#+begin_example +exchange = documents +routing key = documents.ingest. +body = original/normalized StarIntel document JSON +persistent = true +mandatory = true +message-id = StarRouter message id +#+end_example + +Do not rely solely on StarRouter =topic= as dtype. StarIntel's ingest consumer requires a body dtype. If topic is configured as a dtype hint and it differs from body dtype, classify as =DTYPE_MISMATCH= and quarantine/reject. + +Bridge metadata may be added under a namespaced extension only if it is compatible with the active StarIntel document spec, for example: + +#+begin_src json +{ + "extensions": { + "star_router_bridge": { + "message_id": "01...", + "source": "actor-01...", + "received_at": 0 + } + } +} +#+end_src + +Do not overwrite existing provenance. + +* updateDocument mapping + +Input requirements: + +- payload object has non-empty =dtype=; +- update contains an =_id= suitable for StarIntel update handling; +- document semantics permit partial/full update. + +Output routing key: + +#+begin_example +documents.updated. +#+end_example + +The StarIntel update consumer refetches revisions and retries conflicts. The bridge must not invent CouchDB =_rev= values. + +* target mapping + +StarRouter =target= currently means "route this message to one registered actor". StarIntel's target document is a durable document with =dtype=target= that is ingested, persisted, then routed to an external actor. + +These are not automatically equivalent. + +Initial rule: + +- do not reinterpret every StarRouter =target= event as a StarIntel target document; +- support a StarIntel target only when the payload itself is an explicit valid target document, including =dtype: target= and actor semantics required by StarIntel; +- otherwise return =UNSUPPORTED_OPERATION= or quarantine according to producer mode. + +This avoids accidental task amplification or routing to the wrong actor. + +* delete mapping + +StarIntel currently documents HTTP deletion but notes that deletion does not emit a Rabbit deletion event. There is no corresponding durable distributed deletion contract to map StarRouter =deleteDocument= onto safely. + +Therefore v1 bridge must reject/quarantine =deleteDocument= instead of guessing. + +A later StarIntel ADR should define deletion as a first-class event with authorization, tombstone semantics, replay behavior, and audit history. + +* getDocument mapping + +Do not send =getDocument= through Rabbit document ingest. + +Future choices: + +1. a bridge RPC adapter performs StarIntel HTTP =GET /document/:id= and replies using correlated StarRouter replies; +2. define an XRAP resource endpoint for document retrieval; +3. expose a dedicated StarIntel RPC service. + +Keep this out of the first durable-forwarding slice. + +* Backpressure relationship + +The bridge must be allowed to slow/reject durable producers when its durable storage limit is reached. + +Never "solve" overload by: + +- dropping durable messages; +- unbounded in-memory buffering; +- acknowledging before durable ownership; +- spawning unlimited publish futures; +- converting every failure into immediate retry. + +A durable outbox is a finite resource. Its high-water mark becomes an explicit admission-control boundary. + +* Failure matrix + +| Failure | Required behavior | +|---------+-------------------| +| StarIntel/Rabbit down before local acceptance | durable request not ACKed | +| Rabbit down after local acceptance | retain outbox, retry later | +| bridge crash after local acceptance | recover pending row after restart | +| Rabbit accepts but confirm lost | replay possible; downstream idempotency required | +| unroutable AMQP mandatory publish | mark retry/error; alert; do not retire | +| invalid JSON | reject without retry | +| missing dtype | reject without retry | +| unsupported StarRouter event | reject/quarantine without hot loop | +| outbox disk full | stop accepting durable messages; readiness false | +| transient PUB subscriber slow | apply transient drop/block policy separately | + +* StarIntel compatibility requirements + +Before enabling bridge production traffic, assert against the live StarIntel version: + +- =documents= topic exchange exists and is durable; +- =documents.ingest.#= is bound to the ingest queue; +- updates binding is active; +- body =dtype= is required and accepted; +- selected document schema version is compatible; +- target semantics if enabled are verified; +- Rabbit credentials have publish permission only to required exchanges/routing keys where practical. + +* Implementation language + +The bridge can be implemented in Nim to reuse StarRouter protocol types and keep the adapter close to the source protocol. The AMQP client choice must be evaluated for: + +- publisher confirms; +- mandatory returns; +- connection recovery; +- heartbeat support; +- TLS; +- maintained status; +- Nix packaging; +- benchmark behavior. + +Do not pick a library solely because it can call =basic.publish=. + +If the available Nim AMQP clients cannot meet the contract cleanly, a small Common Lisp bridge using StarIntel's ecosystem is acceptable, but it must still implement confirms rather than merely reuse the current unconfirmed producer abstraction. + +* Acceptance criteria for bridge v1 + +- 100% of supported accepted messages survive bridge process restart; +- Rabbit outage of configured test duration causes no accepted-message loss; +- duplicate replay does not create duplicate logical StarIntel effects in deterministic-id fixtures; +- invalid/unsupported messages cannot block healthy queue progress; +- mandatory unroutable publish is visible and not retired; +- memory remains bounded at configured in-flight/outbox limits; +- bridge can run unattended under soak test with automatic reconnection; +- metrics identify accepted, pending, confirmed, retried, rejected, quarantined, and oldest-pending age. diff --git a/docs/design/004-durable-outbox-and-recovery.org b/docs/design/004-durable-outbox-and-recovery.org new file mode 100644 index 0000000..c55fe49 --- /dev/null +++ b/docs/design/004-durable-outbox-and-recovery.org @@ -0,0 +1,303 @@ +#+title: StarRouter 004 - Durable Outbox and Recovery +#+options: toc:3 + +* Purpose + +Define the durability mechanism that lets StarRouter acknowledge a durable message locally while StarIntel/RabbitMQ is slow, unavailable, restarting, or temporarily unroutable. + +The outbox is not a cache. It is the ownership record for accepted durable work until the destination confirms acceptance. + +* Reliability model + +The bridge follows this transfer-of-ownership chain: + +#+begin_example +producer owns message + | + | durable append succeeds + v +StarRouter/bridge outbox owns message + | + | Rabbit publisher confirm succeeds + v +RabbitMQ owns message + | + | StarIntel consumer ACKs after required processing + v +StarIntel durable flow owns effect +#+end_example + +At every transition, acknowledgement means the next component has accepted responsibility. + +RabbitMQ's reliability guidance makes the same distinction: writing bytes to a socket is not proof that a broker safely accepted a publish. Publisher confirms are required for reliable publishing, and unconfirmed messages must be considered candidates for retransmission. + +Reference: https://www.rabbitmq.com/docs/reliability + +* Storage choice + +For the first implementation, prefer a small embedded transactional store with WAL behavior and a mature crash-recovery story. + +SQLite is a reasonable default if a maintained Nim binding is available in Nix and the benchmark shows sufficient throughput. The important design properties are: + +- atomic transaction for dedupe check + append; +- unique index on message id; +- ordered scan by next-attempt/deadline; +- WAL/crash recovery; +- bounded file growth policy; +- observable database integrity checks; +- simple operational backup/inspection. + +Do not invent an ad-hoc append file unless benchmarks prove an embedded database cannot meet the target. A custom log requires checksums, torn-write recovery, compaction, indexing, concurrency semantics, and migration tooling that otherwise come for free. + +The storage implementation is replaceable behind an =OutboxStore= interface. + +* Logical schema + +#+begin_src sql +CREATE TABLE outbox_message ( + message_id TEXT PRIMARY KEY, + protocol_version TEXT NOT NULL, + source TEXT NOT NULL, + event_type INTEGER NOT NULL, + topic TEXT NOT NULL, + document_type TEXT, + payload BLOB NOT NULL, + content_type TEXT NOT NULL, + state TEXT NOT NULL, + routing_key TEXT, + accepted_at_ms INTEGER NOT NULL, + next_attempt_ms INTEGER NOT NULL, + attempt_count INTEGER NOT NULL DEFAULT 0, + deadline_ms INTEGER, + last_error_code TEXT, + last_error_detail TEXT, + confirmed_at_ms INTEGER, + retired_at_ms INTEGER +); + +CREATE INDEX outbox_pending_idx +ON outbox_message(state, next_attempt_ms, accepted_at_ms); +#+end_src + +A real migration may add a separate attempt/event table rather than overwrite all error history. The minimal row above is enough for correctness. + +* States + +#+begin_example +ACCEPTED + -> PUBLISHING + -> CONFIRMED + -> RETIRED + +ACCEPTED/PUBLISHING + -> RETRY_WAIT + -> PUBLISHING + +ACCEPTED/RETRY_WAIT + -> EXPIRED + +any non-retired + -> QUARANTINED +#+end_example + +=CONFIRMED= means Rabbit publisher confirmation was correlated to the message. =RETIRED= means the configured replay retention period has elapsed and the row is eligible for compaction/deletion. + +Never remove the dedupe identity immediately on confirmation. A producer can retry late after losing its local ACK. + +* Atomic acceptance + +Producer durable request algorithm: + +1. decode and validate envelope enough to identify message id and size; +2. begin transaction; +3. look up =message_id=; +4. if an existing compatible row exists, return its prior acceptance state; +5. if an existing row has the same id but materially different immutable content, return =ID_COLLISION= and alert; +6. insert accepted row; +7. commit transaction and required durability barrier; +8. only then emit producer ACK for durable acceptance. + +This makes retries idempotent at the first durable boundary. + +* Publisher algorithm + +Worker loop: + +#+begin_example +claim bounded batch of due rows + | + v +publish each with persistent delivery + mandatory routing + | + v +track delivery sequence -> message_id + | + +-- basic.return --> mark unroutable / retry or quarantine + | + +-- basic.nack ----> retry + | + +-- confirm ack ---> CONFIRMED + | + +-- connection lost before confirm --> retry unknown set +#+end_example + +Use asynchronous/batched confirms rather than synchronous confirm-per-message once correctness is established. RabbitMQ documents that batching/asynchronous confirms improve throughput and that confirms can arrive asynchronously. + +Reference: https://www.rabbitmq.com/docs/confirms + +* Persistent AMQP properties + +For durable document forwarding: + +- persistent delivery mode; +- =message_id= property = StarRouter message id; +- content type = =application/json=; +- type/property = dtype if useful and compatible; +- timestamp as metadata; +- =mandatory=true= when at least one destination queue must exist. + +A publisher confirm by itself does not prove that a message was routed if an exchange accepts an unroutable message. Mandatory publishing plus handling =basic.return= closes that gap for the bridge's expected topology. + +* Connection failure + +On AMQP connection/channel failure: + +- stop claiming new rows until a bounded in-flight set is reconciled; +- treat all in-flight but unconfirmed rows as uncertain and make them retryable; +- reconnect with exponential backoff + jitter; +- redeclare/assert expected exchange if the deployment contract allows; +- resume due rows; +- do not create new logical message ids. + +Duplicates are allowed by the contract and must be safe downstream. + +* Retry policy + +Classify failures: + +** Retryable + +- TCP/AMQP connection failure; +- channel reset not caused by invalid publish parameters; +- broker unavailable; +- publish confirm timeout; +- transient resource pressure; +- temporary no-quorum state if deployment uses quorum queues. + +** Terminal / quarantine + +- invalid JSON; +- missing/invalid dtype; +- unsupported event type; +- message exceeds configured maximum; +- ID collision with different immutable content; +- routing key generation failure; +- schema incompatibility known to be permanent. + +** Operator-action required + +- mandatory unroutable publish due to missing StarIntel binding; +- credentials/permission failure; +- outbox corruption/integrity error; +- disk exhaustion. + +Do not retry terminal errors forever. + +* Backoff + +Use capped exponential backoff with jitter, for example conceptual values: + +#+begin_example +base = 250 ms +max = 30 s +next = random(0, min(max, base * 2^attempt)) +#+end_example + +Exact defaults should be benchmarked. Operator-action failures can use a slower retry cadence while keeping readiness false and alerting. + +* Disk bounds + +Define both: + +- maximum outbox bytes; +- maximum pending message count. + +Recommended admission states: + +| Utilization | Behavior | +|-------------+----------| +| <70% | normal | +| 70-85% | warning metric/event | +| 85-95% | shed optional/transient work; slow durable intake | +| >95% | reject new durable acceptance with =OUTBOX_FULL=; readiness false | + +Percentages are starting defaults, not universal constants. Make them configuration values. + +Never ACK a durable message and then discard it to recover disk space. + +* Retention and compaction + +Keep confirmed message identities for a configurable dedupe retention window. Retain quarantined rows longer or until operator resolution. + +Compaction must be incremental and must not stall the broker event loop. If SQLite is selected, schedule checkpoint/maintenance policies and measure their latency effect under sustained writes. + +* Startup recovery + +At bridge startup: + +1. open store; +2. run lightweight integrity/schema check; +3. migrate schema transactionally if needed; +4. convert stale =PUBLISHING= rows from the previous process into retryable state; +5. expose readiness=false until store and Rabbit connection are initialized; +6. start publisher; +7. begin accepting durable traffic only when local storage can commit safely. + +Rabbit need not be healthy before accepting traffic if local outbox capacity is available and the configured operating mode explicitly permits offline buffering. + +* Shutdown + +Graceful shutdown sequence: + +1. stop accepting new durable messages; +2. allow in-progress local acceptance transactions to finish; +3. stop claiming new outbox rows; +4. wait a bounded period for in-flight publisher confirms; +5. mark remaining uncertain rows retryable; +6. checkpoint/close store; +7. close AMQP/ZeroMQ sockets with bounded linger. + +SIGTERM must not depend on an infinite wait for Rabbit or clients. + +* Optional higher assurance: persistence observation + +A publisher confirm transfers responsibility to RabbitMQ, not directly to CouchDB. + +For critical workloads, the bridge can hold a separate =PERSISTENCE_PENDING= record until it observes the resulting =documents.new.= event or verifies the document by id through StarIntel HTTP. + +Do not make this mandatory in v1 because it couples the bridge to post-ingest semantics and increases latency/state. Add it as an assurance mode with independent metrics. + +* Crash test invariants + +Automated tests must kill the process at each boundary: + +- before transaction begin; +- after row insert before commit; +- immediately after commit before producer ACK; +- after publish before Rabbit confirm; +- after confirm before local state update; +- after local =CONFIRMED= update before retirement. + +After restart, assert: + +- no accepted durable message disappears; +- uncertain publishes can replay; +- duplicate replay retains the same message id; +- terminal invalid rows do not re-enter hot retry; +- pending queue eventually drains after Rabbit recovers. + +* Rabbit topology note + +If StarIntel later uses quorum queues for critical ingest, publisher confirms become even more meaningful because RabbitMQ confirms a persistent quorum-queue message after quorum acceptance according to its documented semantics. Queue-type changes belong to StarIntel operations/design, not to this bridge PR. + +Reference: https://www.rabbitmq.com/docs/quorum-queues diff --git a/docs/design/005-testing-chaos-and-benchmark-plan.org b/docs/design/005-testing-chaos-and-benchmark-plan.org new file mode 100644 index 0000000..380842b --- /dev/null +++ b/docs/design/005-testing-chaos-and-benchmark-plan.org @@ -0,0 +1,309 @@ +#+title: StarRouter 005 - Testing, Chaos, and Benchmark Plan +#+options: toc:3 + +* Goal + +Turn StarRouter from example-driven validation into a deterministic system whose correctness, loss behavior, restart behavior, and performance are measured automatically. + +Testing is the first implementation slice because several current defects are source-verifiable but the repository has no reliable regression harness to prove fixes stay fixed. + +* Test pyramid + +** Pure unit tests + +No sockets, no sleeps, no external broker. + +Cover: + +- =Message= constructors set every field correctly; +- event enum encode/decode; +- canonical envelope normalization; +- SC01 frame parsing; +- malformed frame rejection; +- actor registry operations; +- round-robin selection; +- liveness deadline transitions with injected clock; +- retry/backoff calculations with deterministic RNG seed; +- dtype/routing-key mapping; +- idempotency collision behavior; +- outbox state machine. + +Time and randomness must be injectable. Unit tests should never wait ten real seconds for a heartbeat timeout. + +** In-process/component tests + +Run actual ZeroMQ sockets on ephemeral local ports with bounded test deadlines. + +Assertions: + +- request -> one dispatch; +- request -> one terminal reply; +- no duplicate publish for =newDocument=; +- target output uses selected actor id; +- hyphenated actor names remain valid; +- registration/heartbeat/expiry; +- malformed peer cannot kill broker; +- slow/absent peer returns bounded error; +- reconnect after router restart; +- HWM/overload policy behaves as documented. + +Every loop/test must terminate or fail by a global deadline. + +** Bridge integration tests + +Bring up RabbitMQ plus a minimal fake/real StarIntel ingest topology under Nix/containers. + +Cases: + +- =newDocument= routes to =documents.ingest.=; +- =updateDocument= routes correctly; +- missing dtype rejected; +- conflicting topic/dtype rejected; +- unsupported event quarantined; +- mandatory unroutable publish not retired; +- publisher confirm retires expected outbox row; +- connection failure causes replay; +- duplicate message id does not append second logical row. + +** StarIntel end-to-end tests + +Use a disposable StarIntel Server stack and database. + +Flow: + +1. create deterministic document fixture; +2. submit through StarRouter durable API; +3. wait for bounded acceptance; +4. verify document persisted in StarIntel; +5. repeat same message id; +6. verify idempotent logical result; +7. perform update fixture and verify final revision/content. + +This is the proof that the bridge works with actual server behavior, not only a mocked exchange. + +* Regression tests for current P0 defects + +Add these tests before fixes: + +** Duplicate dispatch + +Count calls to the publish seam after one =newDocument= request. Current code should reproduce a count greater than one; fixed code must equal exactly one. + +** Duplicate ACK + +Read all available reply frames for one request within a bounded window. Fixed code must return exactly one terminal reply. + +** Constructor event type + +Construct each =EventType= through =Client.newMessage=. Assert =message.typ == requestedType=. + +** Target mutation + +Register two test actors, submit a target, inspect emitted topic. It must equal one selected actor id and the emitted event type must match the target-dispatch contract. + +** Hyphenated actor heartbeat + +Register actor =github-actor=, heartbeat using its generated id, assert lookup does not truncate name. + +* Protocol property/fuzz tests + +Generate malformed frame sets: + +- zero frames; +- missing delimiter; +- wrong header; +- missing body frame; +- extra frames; +- huge frame; +- non-numeric time/type; +- enum below/above range; +- invalid UTF-8 where strings are required; +- empty source/id/topic; +- payload at limit and limit+1. + +Invariants: + +- broker process remains alive; +- parser never reads beyond available frame set; +- one bad peer cannot mutate unrelated actor state; +- error is bounded and machine-readable; +- no durable acceptance occurs for invalid envelope. + +Use a coverage-guided fuzz target if Nim/libFuzzer integration is practical under Nix. Otherwise start with deterministic generated cases and seeded random mutation. + +* Chaos/failure testing + +Chaos must operate against disposable local infrastructure, never production. + +** Process failures + +Kill -9 the bridge at each outbox/publish transition documented in 004. + +** Rabbit outage + +- stop Rabbit before submission; +- stop after local acceptance; +- stop while a confirm batch is in flight; +- restart with same state; +- restart with topology temporarily missing. + +Assert no locally accepted durable record disappears. + +** StarRouter restart + +Restart router while clients are active. Verify clients either reconnect or fail with a bounded error according to contract. No request may hang forever. + +** Network impairment + +Use a test proxy/netem where available to inject: + +- latency; +- jitter; +- packet loss; +- connection reset; +- half-open behavior; +- short partitions. + +Focus on state recovery, not merely throughput degradation. + +** Disk pressure + +Fill the outbox volume toward configured thresholds. Verify warning, admission throttling, readiness failure, and hard rejection occur without deleting accepted rows. + +** Poison message + +Insert invalid/unsupported message among valid messages. Verify it is quarantined and valid messages continue. + +* Soak tests + +Soak stages: + +1. 30 minutes development smoke; +2. 4 hours pre-merge extended test; +3. 24 hours release candidate; +4. 72 hours unattended stability gate before claiming production readiness. + +During soak record: + +- sent/accepted/confirmed/persisted counts; +- duplicates by message id; +- unexplained losses; +- memory RSS; +- file descriptors; +- outbox bytes/pending age; +- reconnect count; +- CPU; +- p50/p95/p99/p99.9 latency; +- throughput; +- event-loop lag if measurable. + +A soak that ends with counts that cannot be reconciled is a failure even if the process stayed alive. + +* Benchmark methodology + +The current =testDocsPerSec.nim= is not a sufficient benchmark because it runs indefinitely and does not define sent count, elapsed interval, latency distribution, correctness reconciliation, payload shapes, or resource use. + +Replace/retain it as an example and add a real benchmark executable. + +** Workload matrix + +Payload sizes: + +- 256 B +- 1 KiB +- 4 KiB +- 16 KiB +- 64 KiB +- configured maximum + +Concurrency: + +- 1 producer +- 4 producers +- 16 producers +- 64 producers where machine capacity permits + +Modes: + +- transient PUB fanout; +- durable local acceptance only; +- durable + Rabbit confirms; +- durable + full StarIntel persistence observation. + +** Measurements + +- offered messages/s; +- accepted messages/s; +- confirmed messages/s; +- persisted messages/s when applicable; +- latency p50/p95/p99/p99.9/max; +- CPU seconds per million messages; +- peak RSS; +- outbox write amplification/size; +- network bytes; +- retry/duplicate rates. + +Throughput without loss reconciliation is not a valid result. + +* Performance regression gates + +Do not choose hard numeric targets until a baseline is measured on named hardware. + +CI can still enforce relative gates: + +- no >10% throughput regression against stable baseline unless explicitly approved; +- no >20% p99 latency regression under same fixture; +- zero correctness regression; +- zero unbounded memory trend in bounded load test. + +Store benchmark metadata: CPU model, cores, OS/Nix revision, Nim version, libzmq version, Rabbit version, payload mix, commit SHA. + +* Nix/CI gate + +Target =nix flake check= should run: + +- compile; +- unit tests; +- component tests with bounded local sockets; +- formatting/static checks selected for Nim; +- protocol fixture tests; +- fast bridge integration tests if Rabbit service can be provisioned reproducibly. + +Long soak/chaos tests can run as scheduled/manual CI jobs rather than block every tiny PR. + +PR required checks should include at minimum: + +- nix build/check; +- unit; +- protocol/component; +- integration; +- Docker/OCI build only after the above pass. + +* Deterministic test rules + +- no fixed TCP ports when an ephemeral allocation works; +- no infinite loops in tests; +- every future/socket wait has a deadline; +- no test depends on execution order; +- unique temporary data directories/databases; +- seeded randomness printed on failure; +- cleanup via defer/finalizer even on assertion failure; +- retries in product code are observable, not hidden by retrying the test itself. + +* Release correctness ledger + +For a run with N producer messages, calculate: + +#+begin_example +N_sent +N_rejected_before_acceptance +N_durably_accepted +N_confirmed_to_rabbit +N_verified_persisted +N_quarantined +N_expired +N_pending +N_duplicate_replays +#+end_example + +The ledger must balance according to the selected assurance mode. If a message is neither accounted for nor explicitly still pending, the test fails. diff --git a/docs/design/006-observability-and-unattended-operations.org b/docs/design/006-observability-and-unattended-operations.org new file mode 100644 index 0000000..47ffd41 --- /dev/null +++ b/docs/design/006-observability-and-unattended-operations.org @@ -0,0 +1,309 @@ +#+title: StarRouter 006 - Observability and Unattended Operations +#+options: toc:3 + +* Objective + +StarRouter and the bridge should be able to run without an operator staring at a terminal. Unattended operation requires enough state exposure to answer four questions quickly: + +1. Is the process alive? +2. Is it able to accept new work safely? +3. Is accepted work making forward progress? +4. If not, where is it stuck and is data at risk? + +Logging alone is not sufficient. + +* Health model + +Expose distinct states rather than one boolean. + +** Liveness + +Liveness answers: should the process be restarted? + +Healthy when: + +- event loop/supervisor is running; +- internal fatal-error flag is clear; +- outbox store can still be accessed by the process. + +Rabbit/StarIntel outage should not necessarily make liveness fail if the bridge can safely buffer work. + +** Readiness + +Readiness answers: should new durable work be sent here? + +Readiness false when any configured hard condition holds, including: + +- outbox cannot commit; +- outbox above admission cutoff; +- schema migration failed; +- required local socket cannot bind; +- bridge is shutting down; +- destination topology/credentials are known permanently invalid; +- corruption/invariant violation detected. + +Rabbit being temporarily down can be either ready or not-ready depending on configured offline-buffer policy and remaining outbox capacity. Make this explicit. + +** Degraded + +A process can be live and ready while degraded, for example: + +- Rabbit reconnecting but outbox has ample capacity; +- retry rate above baseline; +- pending age increasing; +- one optional verifier unavailable; +- transient PUB subscribers dropping. + +Expose degraded reasons as structured codes. + +* Metrics + +Minimum broker metrics: + +#+begin_example +starrouter_requests_total{event_type,result} +starrouter_dispatch_total{event_type} +starrouter_replies_total{status} +starrouter_protocol_errors_total{code} +starrouter_registered_actors{actor_name} +starrouter_actor_expirations_total{actor_name} +starrouter_pub_messages_total{topic} +starrouter_pub_dropped_total{reason} +starrouter_inbox_depth{actor} +starrouter_event_loop_lag_seconds +#+end_example + +Minimum bridge metrics: + +#+begin_example +starrouter_bridge_accepted_total{event_type,dtype} +starrouter_bridge_rejected_total{reason} +starrouter_bridge_quarantined_total{reason} +starrouter_bridge_outbox_pending +starrouter_bridge_outbox_bytes +starrouter_bridge_oldest_pending_seconds +starrouter_bridge_publish_attempts_total{result} +starrouter_bridge_publish_confirmed_total +starrouter_bridge_publish_retries_total{reason} +starrouter_bridge_unroutable_total{routing_key} +starrouter_bridge_inflight_confirms +starrouter_bridge_rabbit_connected +starrouter_bridge_rabbit_reconnects_total +starrouter_bridge_accept_latency_seconds +starrouter_bridge_confirm_latency_seconds +starrouter_bridge_end_to_end_latency_seconds +#+end_example + +Control label cardinality. Do not use message id, source URL, arbitrary topic, error detail, or document id as unbounded metric labels. + +* Structured events/logs + +Each operational log/event should be machine parseable and carry bounded fields such as: + +#+begin_src json +{ + "time": "...", + "level": "warn", + "component": "bridge.publisher", + "event": "publish_retry", + "message_id": "01...", + "routing_key": "documents.ingest.person", + "attempt": 4, + "error_code": "RABBIT_CONNECTION_LOST", + "retry_in_ms": 3200 +} +#+end_src + +Message id is appropriate in logs/traces even though it is not appropriate as a metric label. + +Never log whole payloads by default. StarIntel documents can contain sensitive data. Provide an explicit debug mode with size limits/redaction and make the default metadata-only. + +* Trace/correlation + +Carry =message_id= and =correlation_id= through: + +- producer submission; +- broker acceptance; +- outbox row; +- Rabbit message properties/headers; +- StarIntel bridge logs. + +If OpenTelemetry is adopted, follow its messaging semantic conventions where practical, but keep local metric names stable behind an adapter because the messaging semantic convention is still evolving. + +Reference: https://opentelemetry.io/docs/specs/semconv/messaging/ + +* Status endpoint / command + +Provide a cheap local administrative surface, ideally loopback/Unix-socket bound, that returns: + +- version + git revision; +- uptime; +- protocol version(s); +- readiness/liveness/degraded state; +- actor counts; +- message counters; +- outbox pending/quarantined/bytes/oldest age; +- Rabbit connected + last successful confirm time; +- current retry delay; +- configured capacity thresholds without secrets. + +Do not require querying every outbox row to render status. + +Possible forms: + +- local HTTP =/health=, =/ready=, =/metrics=; +- or a small XRAP/admin resource later. + +The first implementation should pick the simplest surface that monitoring can scrape. + +* Supervision + +The process should be organized into supervised components rather than one giant async loop whose exception policy is "print stack trace and continue". + +Conceptual actors/tasks: + +#+begin_example +Supervisor + +-- ZmqIngress + +-- Registry/Liveness + +-- OutboxWriter + +-- RabbitPublisher + +-- RetryScheduler + +-- Metrics/Health + +-- OptionalPersistenceVerifier +#+end_example + +A component failure policy should be explicit: + +- restart component locally when state is recoverable; +- restart whole process when shared-state invariants may be compromised; +- fail fast if durable store cannot be trusted; +- never silently continue after an invariant violation that can lose accepted data. + +* Restart policy + +Under systemd/container orchestration: + +- restart on unexpected exit; +- bounded restart backoff to avoid hot crash loops; +- SIGTERM invokes graceful shutdown; +- startup readiness stays false until local durable state is valid; +- watchdog integration can be added after event-loop health is measurable. + +The process itself should reconnect to ordinary network outages. The supervisor should not need to restart the whole process every time RabbitMQ blips. + +* Alerts + +High-value alerts: + +** Page / urgent + +- accepted-message reconciliation mismatch; +- durable outbox integrity failure; +- outbox >95% / admission stopped; +- oldest pending exceeds critical threshold; +- mandatory unroutable messages sustained; +- repeated authentication/authorization failures; +- process crash loop; +- message loss detected in verifier/soak ledger. + +** Ticket / warning + +- retry rate elevated; +- outbox >70-85%; +- reconnect frequency abnormal; +- p99 confirmation latency regression; +- actor churn/expiry spike; +- quarantine count increasing; +- disk growth above forecast. + +Avoid paging just because one transient retry happened. + +* SLO candidates + +Do not lock targets until baseline measurements exist. Define the shape now: + +- durable acceptance availability; +- confirmed-forwarding success rate; +- p99 acceptance latency; +- p99 Rabbit confirmation latency; +- maximum oldest-pending age under healthy destination; +- zero unexplained accepted-message loss; +- bounded duplicate rate under injected failures; +- uptime between planned restarts. + +Correctness SLO is absolute: no unexplained loss of messages that were acknowledged as durably accepted. + +* Runbooks + +Create operator runbooks before unattended deployment for: + +1. Rabbit unavailable; +2. outbox filling; +3. mandatory unroutable messages; +4. poison/quarantined messages; +5. outbox integrity failure; +6. StarIntel ingest stopped; +7. disk full; +8. high memory/CPU; +9. repeated actor expiry; +10. upgrade/rollback. + +Each runbook should state: + +- symptoms/metrics; +- safe inspection commands; +- whether new intake should be stopped; +- whether replay is safe; +- data-loss risk; +- recovery steps; +- verification after recovery. + +* Quarantine inspection + +Operators need a bounded command/API to: + +- count quarantined messages by reason; +- inspect metadata for one message id; +- export payload intentionally for debugging; +- retry after correcting a recoverable configuration issue; +- retire/delete only with explicit operator action and audit record. + +Do not make =retry all= the default remediation. + +* Upgrade behavior + +Before replacing a running bridge: + +- readiness false; +- stop new intake; +- drain/mark in-flight confirms; +- leave unconfirmed rows durable; +- start new version against same/migrated store; +- verify old pending rows are recognized; +- readiness true after compatibility check. + +Schema migrations require forward/backward compatibility or a documented rollback boundary. + +* Logging cleanup in current source + +The current broker has unconditional heartbeat =echo= calls and a log filename environment variable named =FEDIWATCH_LOG=. Stabilization should: + +- remove unconditional stdout heartbeat spam; +- use StarRouter-specific configuration names; +- define log level independently from flush policy; +- ensure errors include component/event codes; +- keep default logs payload-free. + +* Unattended release gate + +Before labeling unattended mode supported: + +- 72-hour soak completes; +- no unexplained count mismatch; +- process automatically recovers from at least three Rabbit restarts; +- disk/outbox threshold behavior demonstrated; +- SIGTERM/restart replay demonstrated; +- monitoring can identify a deliberately injected poison message; +- no unbounded RSS/file-descriptor trend; +- runbooks tested against the disposable stack. diff --git a/docs/design/007-performance-and-backpressure.org b/docs/design/007-performance-and-backpressure.org new file mode 100644 index 0000000..af9acaf --- /dev/null +++ b/docs/design/007-performance-and-backpressure.org @@ -0,0 +1,288 @@ +#+title: StarRouter 007 - Performance and Backpressure +#+options: toc:3 + +* Objective + +Increase throughput only while preserving the declared delivery semantics and bounded-resource behavior. + +StarRouter's useful performance property is that ZeroMQ can move small messages cheaply. That advantage disappears if the system responds to load by duplicating work, silently dropping durable data, exhausting memory, or blocking one slow callback behind an unbounded queue. + +The optimization order is: + +1. correctness; +2. boundedness/backpressure; +3. measurement; +4. batching/concurrency; +5. serialization/copy optimization; +6. lower-level tuning only when profiles justify it. + +* Separate performance classes + +Do not benchmark all traffic as one number. + +** Transient fanout + +Use cases: + +- live notifications; +- telemetry-like events where loss is acceptable; +- compatibility subscribers that can catch up from another source. + +Optimization target: very high throughput / low latency. + +Allowed overload behavior: configured loss or disconnect for slow subscribers, with metrics. + +** Durable forwarding + +Use cases: + +- StarIntel new/update documents; +- evidence/results that must survive destination outage. + +Optimization target: sustained confirmed throughput with bounded latency and zero unexplained accepted-message loss. + +Allowed overload behavior: backpressure/reject-before-acceptance when durable capacity is full. + +Never compare transient PUB numbers directly to confirmed durable forwarding and call the latter a regression without noting the semantics. + +* Current backpressure gaps + +The current code does not establish an application-level policy for: + +- PUB/SUB HWM; +- DEALER/ROUTER HWM; +- inbox capacity; +- broker ingress rate; +- maximum payload; +- per-actor queue depth; +- durable spool capacity; +- in-flight AMQP confirms. + +=Inbox.size= is not initialized by =newInbox= and the queue is not enforced as a bound. + +These must become explicit configuration and metrics. + +* ZeroMQ high-water marks + +libzmq maintains queues per connection and uses high-water marks to limit outstanding messages. Socket behavior differs by type: PUB can drop when subscribers cannot keep up, while other socket types can block or return errors according to options and state. + +Reference: https://zeromq.org/socket-api/ +Reference: https://libzmq.readthedocs.io/en/latest/zmq_setsockopt.html + +Design rules: + +- choose HWM per socket role; +- document whether reaching it blocks, rejects, or drops; +- expose drop/rejection counters; +- do not set a huge HWM as a substitute for admission control; +- test HWM behavior with a deliberately stalled consumer. + +* Bounded actor inboxes + +Each actor/inbox needs: + +#+begin_example +capacity +current_depth +high_water_threshold +full_policy +processing_latency +oldest_message_age +#+end_example + +Full policies: + +- =BLOCK=: upstream stops until room exists; +- =REJECT=: producer receives retryable busy/overloaded result before durable acceptance; +- =SPILL=: durable work goes to an outbox/disk queue; +- =DROP=: permitted only for explicitly transient traffic. + +Default durable policy should be spill/backpressure, never silent drop. + +* Fairness + +A single hot topic/actor must not starve every other actor. + +Potential scheduler choices: + +- bounded per-topic queues + round robin; +- deficit/weighted round robin if priorities are later required; +- partition durable forwarding by routing key/document class; +- preserve an emergency/control lane for heartbeat/admin traffic. + +The current =ActorManager.nextActor= reconstructs a sequence from table values. Benchmark registry size before optimizing, but a stable explicit vector/ring of registered actor ids will provide clearer deterministic round-robin behavior and avoid allocation in the hot path. + +* Serialization + +Current payload handling mixes JSON serialization paths and has an optional =jsony= compile switch. + +Before choosing a faster serializer: + +1. establish payload fixture set; +2. measure encode/decode CPU and allocation separately from socket IO; +3. assert output semantic equivalence; +4. measure end-to-end throughput impact. + +Potential optimizations: + +- keep payload bytes opaque in broker hot path when the broker does not need to inspect JSON; +- parse JSON only in bridge classifier where dtype mapping requires it; +- avoid parse -> object -> serialize cycles unless normalization is required; +- reuse buffers where Nim/library ownership rules make it safe; +- keep a maximum payload size to prevent pathological allocation. + +Do not switch wire format before compatibility and profile data justify it. + +* Durable outbox write path + +Optimize the acceptance transaction as a short append/index operation. + +Principles: + +- one writer actor/task initially; +- prepared statements; +- WAL mode if SQLite selected; +- group commits/batching only after durability behavior is understood; +- no network call inside the acceptance transaction; +- publisher reads pending records independently. + +Measure: + +- fsync/commit latency; +- accepted msgs/s; +- database file growth; +- checkpoint stalls; +- recovery time with large pending queue. + +* Rabbit publisher confirms + +Synchronous "publish one, wait for one confirm" is easy to reason about but leaves throughput on the floor. + +After correctness tests pass, use a bounded asynchronous confirm window: + +#+begin_example +max_inflight = configurable + +while inflight < max_inflight and due outbox rows exist: + publish row + remember delivery_seq -> message_id + +on ack/nack/return: + update outbox + free inflight slot +#+end_example + +RabbitMQ recommends asynchronous confirms or batching for throughput because persistent-message confirms can have disk/replication latency. + +Reference: https://www.rabbitmq.com/docs/confirms + +Tune =max_inflight= from benchmark curves. Too small underutilizes Rabbit; too large increases duplicate replay set and memory after connection loss. + +* Batch boundaries + +Possible batches: + +- outbox claim/read batch; +- database state-update batch; +- Rabbit in-flight confirm window. + +Do not invent a StarIntel HTTP bulk batch for the primary path; the bridge is Rabbit-native. + +Preserve one logical AMQP message per StarIntel document unless StarIntel adds a documented batch message contract. + +* Parallelism + +Use concurrency where ordering is not required. + +Potential actor/task model: + +#+begin_example +Ingress actor + -> Outbox writer actor + -> Publisher supervisor + +-- publish lane 0 + +-- publish lane 1 + +-- ... +#+end_example + +Start with one publisher channel. Add lanes only if benchmarks show the confirmed publisher is the bottleneck and downstream routing/order constraints permit it. + +Partition consistently (for example by document id hash) if same-document ordering matters. + +* Slow subscribers + +PUB/SUB compatibility must treat slow subscribers as independent from durable bridge health. + +A stalled transient subscriber must not force durable ingress to run out of memory. + +Expose per-socket/aggregate drop information where the binding permits. If exact subscriber drop attribution is not available with PUB, document that limitation rather than infer it. + +* Payload limits + +Set a default maximum serialized payload based on observed StarIntel documents, with configuration override and an absolute safety cap. + +Reject oversize requests before copying/parsing repeatedly. + +Benchmark at: + +- normal p50 payload; +- p95/p99 observed payload; +- configured maximum; +- maximum+1 rejection. + +* Latency budgets + +Measure phases independently: + +#+begin_example +producer -> broker receive +broker validation -> outbox commit +outbox wait -> AMQP publish +publish -> confirm +confirm -> optional persistence observation +#+end_example + +A single end-to-end number cannot identify whether the bottleneck is disk, ZeroMQ, Rabbit, StarIntel ingest, or CouchDB. + +* Overload tests + +For every bounded queue, deliberately exceed capacity. + +Assert: + +- RSS reaches a plateau rather than growing without bound; +- durable requests receive =OVERLOADED/OUTBOX_FULL= before acceptance when capacity is exhausted; +- accepted rows remain intact; +- transient loss is counted; +- recovery drains backlog without manual restart; +- high-priority control/heartbeat path remains responsive if designed that way. + +* Performance tuning candidates after baseline + +Likely candidates, in order of evidence: + +- remove duplicate dispatch and unnecessary serialization first; +- opaque payload forwarding in router; +- allocation-free/stable actor rotation; +- batching outbox reads/state updates; +- async Rabbit confirm window; +- prepared SQLite operations/checkpoint tuning; +- jsony or alternative JSON only if profiling says JSON dominates; +- multiple publisher lanes; +- CPU affinity/advanced ZeroMQ tuning only as a late optimization. + +* Benchmark result format + +Every performance report should include a correctness footer: + +#+begin_example +sent: 10,000,000 +rejected: 0 +accepted: 10,000,000 +confirmed: 10,000,000 +pending at finish: 0 +unexplained lost: 0 +duplicate replays: 0 # or explained injected-failure count +#+end_example + +A million messages per second with unexplained loss is not a successful benchmark. diff --git a/docs/design/008-security-compatibility-and-migration.org b/docs/design/008-security-compatibility-and-migration.org new file mode 100644 index 0000000..e02bdb0 --- /dev/null +++ b/docs/design/008-security-compatibility-and-migration.org @@ -0,0 +1,313 @@ +#+title: StarRouter 008 - Security, Compatibility, and Migration +#+options: toc:3 + +* Objective + +Stabilize StarRouter without breaking existing clients, while replacing unsafe defaults and making the StarIntel bridge deployable as an explicit, reviewable boundary. + +* Security baseline + +The current README says StarRouter must not be exposed to the web. The executable entry point nevertheless defaults to wildcard TCP binds. + +Target defaults: + +#+begin_example +pubAddress = tcp://127.0.0.1:6000 +apiAddress = tcp://127.0.0.1:6001 +#+end_example + +Non-loopback binding requires explicit configuration. + +Do not market this change as sufficient authentication. Network binding is only one layer. + +* Threat model + +Assets: + +- durable StarIntel documents; +- bridge outbox contents; +- Rabbit credentials; +- actor registry/control plane; +- service availability; +- potentially sensitive StarIntel payloads. + +Untrusted actions to defend against: + +- arbitrary remote publish; +- malformed multipart frames; +- giant payload/resource exhaustion; +- spoofed actor ids; +- fake heartbeat/registration; +- target-routing abuse; +- replay/id collision attacks; +- unauthorized Rabbit publishing; +- payload leakage through logs; +- connection floods. + +* ZeroMQ transport hardening + +Evaluate CURVE/ZAP for authenticated/encrypted non-loopback deployments supported by the chosen Nim/libzmq stack. + +Regardless of transport authentication: + +- max frame/message size; +- connection/socket resource bounds; +- protocol version/header validation; +- structured rejection instead of assertions; +- rate/admission limits where exposed beyond trusted localhost; +- explicit =ROUTER_MANDATORY= behavior for targeted sends; +- safe linger/timeouts; +- no sensitive payload logging by default. + +A reverse proxy does not help a raw ZeroMQ TCP port in the same way it helps HTTP. If remote access is needed, use an authenticated network boundary or ZeroMQ-native security rather than exposing =tcp://*= casually. + +* Rabbit credentials + +Bridge credentials should follow least privilege. + +Prefer a dedicated Rabbit user/vhost policy that can: + +- connect to the intended vhost; +- publish to the =documents= exchange using required routing keys; +- optionally consume only the post-insert verification keys if verifier mode is enabled; +- not administer unrelated queues/exchanges. + +Do not reuse broad StarIntel operator credentials in the bridge configuration. + +Secrets must come from environment/secret files/runtime secret management, never from committed design/config examples with real values. + +* StarIntel HTTP boundary + +The reviewed StarIntel HTTP docs currently warn that the API has no built-in authentication/authorization and wildcard CORS. This is another reason not to expose HTTP as the bridge's high-volume network boundary. + +If the bridge uses HTTP for optional persistence verification, keep it on a trusted local/internal network and follow StarIntel's own deployment threat model. + +* Payload handling + +Treat StarIntel documents as potentially sensitive. + +Rules: + +- metadata-only default logs; +- outbox file permissions restricted to service user; +- backup/export requires explicit operator action; +- quarantine inspection does not dump payload by default; +- metrics never contain raw payload/document id/user content as labels; +- diagnostic payload samples are bounded/redacted. + +Encryption at rest for the outbox can be evaluated if the host threat model requires it; do not invent custom cryptography. + +* Compatibility strategy + +Do not replace SC01 in one change. + +Phases: + +** Phase A - characterize current protocol + +- golden SC01 fixtures; +- exact event numeric values; +- exact multipart order; +- client/server compatibility tests; +- document known bugs separately from intended behavior. + +** Phase B - internal canonical envelope + +Decode SC01 into the new internal envelope, route internally, encode legacy responses exactly as required. + +No client migration required yet. + +** Phase C - correlated protocol capability + +Add v0.4 correlated replies and durable acceptance semantics behind explicit negotiation/configuration. + +Legacy clients remain best-effort unless they opt into/are wrapped by the durable path. + +** Phase D - durable bridge + +Enable StarIntel bridge only for selected producers/topics/datasets first. + +** Phase E - deprecate ambiguous legacy behavior + +After instrumentation shows migration coverage, deprecate: + +- uncorrelated durable ACKs; +- target semantics that cannot be validated; +- unsafe wildcard defaults; +- unbounded inbox behavior. + +* Protocol negotiation + +Possible simple mechanism: + +- retain =SC01= for legacy; +- introduce a new header such as =SC02= only after its frame contract is written and tested; or +- add a capability registration request while preserving header. + +Do not invent =SC02= in implementation until the ADR decides it. The key requirement is that a peer cannot accidentally receive a reply format it does not understand. + +=SR01= remains reserved for broker-to-broker behavior and must not be reused as the bridge header. + +* Migration slices + +** Slice 1: executable test harness + +Deliverables: + +- =nix flake check= package/test baseline; +- deterministic unit tests; +- local ZeroMQ component harness; +- regressions for duplicate dispatch/ACK, constructor type, target routing, actor names; +- no behavior cleanup hidden inside the harness PR except required test seams. + +Exit gate: current behavior is reproducibly measured and known defects fail targeted tests. + +** Slice 2: broker correctness + +Fix: + +- duplicate dispatch; +- duplicate replies; +- constructor event type; +- target publishes wrong object; +- actor identity mapping; +- assertion-based malformed framing; +- liveness deadline logic. + +Exit gate: protocol/component tests green, no P0 known defects. + +** Slice 3: bounded client/broker behavior + +Implement: + +- deadlines/time units; +- inbox capacity/full policy; +- socket HWM/linger/immediate/mandatory choices; +- clean shutdown/reconnect; +- loopback defaults; +- structured status/error codes. + +Exit gate: overload and restart tests green; memory bounded. + +** Slice 4: canonical envelope + correlated replies + +Implement internal normalization and idempotent request tracking while retaining SC01 adapter. + +Exit gate: retry/stale ACK tests prove no cross-request acknowledgement confusion. + +** Slice 5: durable outbox + +Implement storage interface + selected embedded backend, acceptance states, recovery, disk bounds. + +Exit gate: kill-at-every-boundary tests show no accepted row loss. + +** Slice 6: Rabbit confirmed publisher + +Implement StarIntel routing mapping, publisher confirms, mandatory returns, persistent messages, reconnect/replay. + +Exit gate: Rabbit restart/topology failure tests reconcile all accepted messages. + +** Slice 7: StarIntel E2E + +Run disposable StarIntel stack, verify document/update persistence and duplicate behavior. + +Exit gate: full ledger balances. + +** Slice 8: observability/unattended + +Metrics, health/readiness, structured logs, service unit/container restart policy, runbooks. + +Exit gate: 24-hour then 72-hour soak with injected destination restarts. + +** Slice 9: performance pass + +Profile and optimize from measured baselines. Add async confirms/batching only with correctness tests unchanged. + +Exit gate: performance targets established and no correctness regression. + +* Rollout modes + +Use explicit bridge modes: + +#+begin_example +OFF +SHADOW +CANARY +ACTIVE +#+end_example + +=OFF=: no bridge forwarding. + +=SHADOW=: observe/classify traffic and record mapping decisions/metrics, but do not publish durable StarIntel effects. Avoid storing sensitive payload unnecessarily. + +=CANARY=: forward only configured datasets/sources/dtypes. + +=ACTIVE=: forward all supported configured traffic. + +This gives an operational rollback lever independent of code rollback. + +* Shadow reconciliation + +Before ACTIVE: + +- count what would be supported/rejected/quarantined; +- find topic/body dtype mismatches; +- find missing dtype rates; +- characterize payload sizes; +- list actual event types in use; +- estimate outbox capacity from observed traffic + outage budget. + +Do not discover these distributions for the first time after enabling durable forwarding. + +* Rollback + +Rollback must not delete pending outbox work. + +If a new bridge version is unhealthy: + +1. stop new intake / set bridge OFF or readiness false; +2. shut down cleanly; +3. preserve durable store snapshot; +4. start previous compatible version only if its schema can read the store; +5. otherwise restore software while keeping store for forward recovery; +6. reconcile counts before declaring recovery complete. + +Database migrations need a compatibility table in each implementation PR. + +* Versioning + +Version independently: + +- application release; +- wire protocol; +- outbox schema; +- StarIntel mapping profile. + +Expose all four in status output so a mixed deployment can be diagnosed. + +* Documentation updates required with implementation + +Before stable release update: + +- README installation; remove "nimble fails but gets deps" workflow; +- protocol frame reference; +- config reference with safe defaults; +- delivery semantics; +- StarIntel bridge mapping; +- runbooks; +- benchmark method/results; +- upgrade/rollback guide; +- security deployment notes. + +* Non-goals for the first stabilization program + +- full broker federation; +- exactly-once marketing claim; +- automatic semantics for StarRouter =deleteDocument=; +- automatic reinterpretation of all =target= messages; +- full XRAP replacement; +- premature protobuf/wire-format replacement; +- global total ordering; +- bypassing StarIntel's documented Rabbit ingest flow to write CouchDB directly. + +Those can be separate designs after the broker has a reliable measured core. diff --git a/docs/design/index.org b/docs/design/index.org new file mode 100644 index 0000000..7dcd662 --- /dev/null +++ b/docs/design/index.org @@ -0,0 +1,142 @@ +#+title: StarRouter Design and Stabilization Index +#+options: toc:2 + +* Purpose + +This directory is the design baseline for stabilizing StarRouter before new production integrations are added. + +The immediate integration target is a bridge from the legacy StarRouter ZeroMQ protocol into StarIntel Server's durable RabbitMQ document flow. The bridge is deliberately designed as a separate process and reliability boundary so legacy wire behavior can be preserved while correctness, durability, backpressure, observability, and recovery are upgraded independently. + +This is a design pass, not an assertion that the current broker is production ready. The current README explicitly describes StarRouter as pre-alpha. The source audit found correctness and delivery issues that must be fixed or contained before unattended operation. + +* Design documents + +1. [[file:001-current-state-code-audit.org][001 - Current-state code audit]] + - source-level audit of broker, client, protocol, tests, build, and operations + - verified defects separated from runtime hypotheses + - stabilization priorities + +2. [[file:002-delivery-contract-and-protocol.org][002 - Delivery contract and protocol]] + - explicit delivery semantics + - canonical envelope + - ACK/NACK correlation + - compatibility rules for SC01/SR01 + +3. [[file:003-starintel-bridge-architecture.org][003 - StarIntel bridge architecture]] + - StarRouter to StarIntel mapping + - RabbitMQ-native bridge path + - event-type compatibility table + - bridge actor/process decomposition + +4. [[file:004-durable-outbox-and-recovery.org][004 - Durable outbox and recovery]] + - durable acceptance boundary + - retry/replay/deduplication + - publisher confirms + - crash and outage recovery + +5. [[file:005-testing-chaos-and-benchmark-plan.org][005 - Testing, chaos, and benchmark plan]] + - deterministic unit/integration tests + - protocol conformance tests + - crash/restart/network fault tests + - benchmark methodology and release gates + +6. [[file:006-observability-and-unattended-operations.org][006 - Observability and unattended operations]] + - metrics, structured events, health/readiness + - supervision and restart policy + - runbooks and SLOs + +7. [[file:007-performance-and-backpressure.org][007 - Performance and backpressure]] + - bounded queues and high-water marks + - batching, confirms, latency/throughput tradeoffs + - overload behavior + +8. [[file:008-security-compatibility-and-migration.org][008 - Security, compatibility, and migration]] + - network exposure and authentication + - staged compatibility plan + - migration slices and rollout gates + +* Architecture decision summary + +The recommended target is: + +#+begin_example +StarRouter clients + | + | SC01 compatibility protocol + v ++-----------------+ +| StarRouter | +| validated input | ++--------+--------+ + | + | durable accepted envelope + v ++---------------------------+ +| bridge / durable outbox | +| normalize + dedupe + retry | ++-------------+-------------+ + | + | AMQP persistent publish + | publisher confirms + mandatory routing + v ++---------------------------+ +| RabbitMQ documents topic | ++-------------+-------------+ + | + | documents.ingest. + | documents.updated. + v ++---------------------------+ +| StarIntel Server | +| ingest/update consumers | ++-------------+-------------+ + | + v + CouchDB +#+end_example + +The current PUB socket remains useful for transient fanout and compatibility, but it must not be the only path used for data whose loss would violate correctness. A PUB/SUB subscriber can miss messages during startup and can drop under high-water-mark pressure. Durable forwarding therefore requires a separate acceptance and replay mechanism. + +* Delivery objective + +The intended end-to-end contract is *at-least-once durable forwarding with idempotent effects*. + +Exactly-once delivery is not claimed. A crash can occur after RabbitMQ accepts a message but before the bridge records the confirmation. The bridge may therefore replay that message. StarIntel actors already document an at-least-once processing model and require idempotency, so bridge behavior should align with that model rather than hide it. + +* Source basis + +Repository sources reviewed on 2026-08-13: + +- =src/starRouterpkg/server.nim= +- =src/starRouterpkg/client.nim= +- =src/starRouterpkg/proto.nim= +- =src/starRouterpkg/utils.nim= +- =src/starRouter.nim= +- =tests/= +- =README.org= +- =todo.org= +- =flake.nix= +- =Dockerfile= +- =.github/workflows/docker-publish.yml= +- StarIntel Server =docs/messaging.org= +- StarIntel Server =docs/architecture.org= +- StarIntel Server =docs/http-api-docs.org= +- StarIntel Server =source/producers/producers.lisp= + +External primary references: + +- ZeroMQ RFC 40 / XRAP: https://rfc.zeromq.org/spec/40/ +- ZeroMQ Guide, reliable request-reply patterns: https://zguide.zeromq.org/docs/chapter4/ +- libzmq socket options: https://libzmq.readthedocs.io/en/latest/zmq_setsockopt.html +- RabbitMQ reliability: https://www.rabbitmq.com/docs/reliability +- RabbitMQ confirms: https://www.rabbitmq.com/docs/confirms +- OpenTelemetry messaging semantic conventions: https://opentelemetry.io/docs/specs/semconv/messaging/ + +* Audit status + +The source and repository behavior were audited through the repository contents and project documentation. The current repository does not provide a deterministic automated test suite or a CI test gate that can establish runtime behavior. Runtime claims in these documents are therefore marked as either: + +- *verified from code*: directly implied by current source; or +- *runtime validation required*: plausible operational consequence that must be reproduced by the new harness before being considered closed. + +The first implementation slice should build that harness before changing protocol semantics.