Skip to content

Persist OTLP traces and show per-Run collector delivery - #1

Merged
LUOaini1213 merged 15 commits into
mainfrom
codex/reliable-otlp-export-20261009
Oct 9, 2026
Merged

LUOaini1213 merged 15 commits into
mainfrom
codex/reliable-otlp-export-20261009

Conversation

@LUOaini1213

@LUOaini1213 LUOaini1213 commented Oct 9, 2026 •

Copy link
Copy Markdown
Owner

Collector outages previously lost remote traces: finished Runs fired an untracked HTTP export and only logged its failure. Finished traces now enter a flushed, atomic file outbox with one background sender, persisted per-batch checkpoints and startup reconciliation from local spans. Runs await local publication rather than collector responses. Settled payloads compact to acknowledgments, duplicate enqueue calls collapse, and deleted Agents leave scheduling before queue-file cleanup.

Each Run's timeline now shows delivery independently from task execution: export off, waiting for completion, pending, paused, delivered, partial, rejected, uncertain, missing recoverable record or unavailable spans. The safe API object includes span/batch/failed-attempt/warning counts and an explicit unpersisted-acknowledgment flag, excluding collector destinations, headers and raw response text. Background refreshes use a compact authenticated endpoint; known queue records need no span-file reads. A zero-key replay demo lets a completed Run change from pending to delivered after its loopback collector recovers, and replay no longer incorrectly requests model credentials.

The exporter retries connection faults and HTTP 429/502/503/504 with backoff, jitter and Retry-After. Populated partial success and other 4xx/5xx are terminal and never resent. Responses are bounded; a known response whose checkpoint cannot be saved retries disk only. Queued content remains pinned to its original collector and credentials remain configuration-only.

Validation:

  • Final head 3bfa196 passed all four Node 22/24 jobs in push CI and PR CI, including all 225 tests, typecheck, build and startup/replay/OTLP recovery smoke. Publication checks wait for disk, aggregate and per-Run checkpoint completion without dropping the counter or no-resend assertions.
  • Local npm run check passed at final feature source 5552e1e: typecheck, all 225 tests (3 contract, 172 server, 50 web), production build and startup/replay/recovery smoke checks.
  • Actual loopback HTTP tests cover partial success, warnings, retryable/nonretryable replies, timeout, batching, checkpoint disk faults, redaction, destination changes and deletion. Malformed inner payloads cannot block healthy Runs; failed unlink cancels every deleted Run; deletion/shutdown during awaited reads issue no new HTTP request.
  • Trace and compact delivery APIs preserve counters after restart, exclude sensitive fields, validate Run IDs and enforce authentication. Compact refresh is independent of span reads. React rendering and polling regressions cover all status explanations, stale-response cancellation, transient API recovery, terminal stop and initially missing observations.
  • Built server processes were forcibly killed and restarted after 503 and timeout faults: pending payloads replayed byte-identically; acknowledged Runs were not resent after another restart. The smoke waits for disk and live status publication separately, preserving 0-pending/2-delivered assertions and checking per-Run accepted spans with no pending checkpoint. Source-pinned recovery evidence at c6ff89c is committed.
  • Actual browser QA confirmed pending-to-delivered without reload or another task, partial 11 accepted/1 rejected, rejected 12 spans and uncertain 12 spans while Run execution stayed completed. Final compact-API browser QA passed at desktop 1280x900 and mobile 390x844 without horizontal overflow or console errors; the zero-key replay banner was checked after rebuilding.

At source 5552e1e, the fresh 40-Run-per-arm benchmark recorded replay HTTP p95 219.998 ms with export disabled and 203.759 ms while the collector always returned 503; all measured Runs completed. Lower replay latency is host variation, not an acceleration claim. Local 100-span persistence p95 was 3.471 ms versus 12.458 ms with the flushed outbox. This measures replay HTTP completion and local persistence, not model execution or browser polling. Historical 3ab5f15 observations remain in separately named source-pinned files; fresh raw measurements and normalized hashes are committed.

This remains a single-process outbox. A crash between receiver acceptance and local checkpoint can produce an ambiguous replay; OTLP does not guarantee exactly-once ingestion. Initial migration backfills historical final local traces once and may duplicate previous remote export. Disk exhaustion and sudden power loss are not guaranteed-delivery cases; pending data needs disk monitoring. No model, paid service, cloud deployment or merge was used.

@LUOaini1213 LUOaini1213 changed the title Persist OTLP export and recover pending traces after restart Persist OTLP traces and show per-Run collector delivery Oct 9, 2026
@LUOaini1213
LUOaini1213 marked this pull request as ready for review October 9, 2026 12:33
@LUOaini1213
LUOaini1213 merged commit 331c931 into main Oct 9, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant