Report node errors to Sentry, stamped with the build they came from - #380
Merged
Conversation
ferrosa had a telemetry layer already, gated on FERROSA_TELEMETRY_ENABLED, but no path for an error to leave the machine. A node that logs a failure to a console nobody is attached to has not reported it. Errors become Sentry events when the installer configures FERROSA_SENTRY_DSN. WARN becomes a breadcrumb rather than an event: it is context for the error that follows, and as an event it would bury it. Everything below WARN stays in the log — a storage engine at info level would exhaust the quota in minutes and hide every error underneath the noise. The client starts BEFORE the subscriber. The layer is inert without it, and the errors most worth having from a database are the ones raised while it is still coming up. The guard is bound in main so it lives until the process exits; dropping it stops sending, which loses exactly the shutdown errors. The disabled-telemetry branch moves from the fmt() builder to registry composition so the Sentry layer can sit beside the writer. Filter and writer are unchanged. Events carry `ferrosa@version+sha`. build.rs prefers FERROSA_BUILD_SHA from CI, falls back to git, and says `unknown` last — a stale SHA would be believed. A dirty tree is marked. The DSN is not in this repository. ferrosa is PUBLIC: a committed DSN can be used by anyone to write into the project and cannot be rotated without a release. An empty value — what an unset variable looks like from a shell — is absent rather than broken, and plain http is refused; it would put diagnostics on the wire in the clear. sentry 0.49, not 0.34: the older release pins rustls 0.22, whose rustls-webpki 0.102 carries four RUSTSEC advisories. One rustls-webpki in the lock file, and it is the patched 0.103. rustls rather than native-tls because CI builds this for x86_64-unknown-linux-musl. 7 tests in sentry_reporting. clippy --all-targets -D warnings and fmt --check pass. The workspace suite: 1786 passed with 0 failures under the exact CI invocation. A plain --all-features run also surfaces real_typed_edges_paged_scan_delivers_every_distinct_row, which fails loud without FERROSA_TEST_TYPED_EDGES_DIR by design and which CI skips explicitly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ferrosa's half of the per-repo diagnostics work.
What was missing
There was a telemetry layer already, gated on
FERROSA_TELEMETRY_ENABLED, but no path for an error to leave the machine. A node that logs a failure to a console nobody is attached to has not reported it.What this adds
Errors become Sentry events when the installer configures
FERROSA_SENTRY_DSN.WARNbecomes a breadcrumb, not an event — context for the error that follows. Everything belowWARNstays in the log: a storage engine at info level would exhaust the quota in minutes and hide every error underneath the noise.The client starts before the subscriber. The layer is inert without it, and the errors most worth having from a database are the ones raised while it is still coming up. The guard is bound in
mainso it outlives the process; dropping it stops sending, which loses exactly the shutdown errors.The disabled-telemetry branch moves from the
fmt()builder to registry composition so the Sentry layer can sit beside the writer. Filter and writer unchanged.Build identity
Events carry
ferrosa@version+sha.build.rsprefersFERROSA_BUILD_SHAfrom CI, falls back togit rev-parse, and saysunknownlast — a stale SHA would be believed. A dirty tree is marked.The release workflow should set
FERROSA_BUILD_SHAso tagged builds carry the commit CI checked out.The DSN is not in this repository
ferrosa is PUBLIC. A DSN is not a secret — it's a write-only ingest key meant to be embedded — but one committed to a public repo can be used by anyone to write into the project, and rotating it then needs a release. Empty (what an unset shell variable looks like) is absent, not broken; plain
httpis refused, since it would put diagnostics on the wire in the clear.Dependency choice, deliberately
sentry 0.49, not0.34. The older release pinsrustls 0.22, whoserustls-webpki 0.102carries four RUSTSEC advisories —cargo denycaught exactly this on the ferrosa-memory branch. Onerustls-webpkiin the lock now, the patched0.103.rustlsrather thannative-tlsbecause CI builds this forx86_64-unknown-linux-musl.Adding a diagnostics dependency that ships known vulnerabilities would be a poor trade for observability.
Verification
sentry_reportingcargo clippy --all-targets -- -D warningsandcargo fmt --checkpassOne note on the suite: a plain
--all-featuresrun also surfacesreal_typed_edges_paged_scan_delivers_every_distinct_row, which fails loud withoutFERROSA_TEST_TYPED_EDGES_DIRby design. CI skips it explicitly, and so does the command above. I checked main is green before concluding that.The
__eh_framelinker warning on debug builds is pre-existing on main — I verified by stashing — andRUSTFLAGSisn't set in CI, so build warnings aren't errors there; clippy's-D warningsis the gate and it passes.