Repository navigation
Persist OTLP traces and show per-Run collector delivery - #1
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Collector outages previously lost remote traces: finished Runs fired an untracked HTTP export and only logged its failure. Finished traces now enter a flushed, atomic file outbox with one background sender, persisted per-batch checkpoints and startup reconciliation from local spans. Runs await local publication rather than collector responses. Settled payloads compact to acknowledgments, duplicate enqueue calls collapse, and deleted Agents leave scheduling before queue-file cleanup.
Each Run's timeline now shows delivery independently from task execution: export off, waiting for completion, pending, paused, delivered, partial, rejected, uncertain, missing recoverable record or unavailable spans. The safe API object includes span/batch/failed-attempt/warning counts and an explicit unpersisted-acknowledgment flag, excluding collector destinations, headers and raw response text. Background refreshes use a compact authenticated endpoint; known queue records need no span-file reads. A zero-key replay demo lets a completed Run change from pending to delivered after its loopback collector recovers, and replay no longer incorrectly requests model credentials.
The exporter retries connection faults and HTTP 429/502/503/504 with backoff, jitter and Retry-After. Populated partial success and other 4xx/5xx are terminal and never resent. Responses are bounded; a known response whose checkpoint cannot be saved retries disk only. Queued content remains pinned to its original collector and credentials remain configuration-only.
Validation:
3bfa196passed all four Node 22/24 jobs in push CI and PR CI, including all 225 tests, typecheck, build and startup/replay/OTLP recovery smoke. Publication checks wait for disk, aggregate and per-Run checkpoint completion without dropping the counter or no-resend assertions.npm run checkpassed at final feature source5552e1e: typecheck, all 225 tests (3 contract, 172 server, 50 web), production build and startup/replay/recovery smoke checks.c6ff89cis committed.At source
5552e1e, the fresh 40-Run-per-arm benchmark recorded replay HTTP p95 219.998 ms with export disabled and 203.759 ms while the collector always returned 503; all measured Runs completed. Lower replay latency is host variation, not an acceleration claim. Local 100-span persistence p95 was 3.471 ms versus 12.458 ms with the flushed outbox. This measures replay HTTP completion and local persistence, not model execution or browser polling. Historical3ab5f15observations remain in separately named source-pinned files; fresh raw measurements and normalized hashes are committed.This remains a single-process outbox. A crash between receiver acceptance and local checkpoint can produce an ambiguous replay; OTLP does not guarantee exactly-once ingestion. Initial migration backfills historical final local traces once and may duplicate previous remote export. Disk exhaustion and sudden power loss are not guaranteed-delivery cases; pending data needs disk monitoring. No model, paid service, cloud deployment or merge was used.