Skip to content

Report node errors to Sentry, stamped with the build they came from - #380

Merged
bkearns merged 1 commit into
mainfrom
feat/telemetry-sentry
Aug 29, 2026
Merged

Report node errors to Sentry, stamped with the build they came from#380
bkearns merged 1 commit into
mainfrom
feat/telemetry-sentry

Conversation

@bkearns

@bkearns bkearns commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

ferrosa's half of the per-repo diagnostics work.

What was missing

There was a telemetry layer already, gated on FERROSA_TELEMETRY_ENABLED, but no path for an error to leave the machine. A node that logs a failure to a console nobody is attached to has not reported it.

What this adds

Errors become Sentry events when the installer configures FERROSA_SENTRY_DSN. WARN becomes a breadcrumb, not an event — context for the error that follows. Everything below WARN stays in the log: a storage engine at info level would exhaust the quota in minutes and hide every error underneath the noise.

The client starts before the subscriber. The layer is inert without it, and the errors most worth having from a database are the ones raised while it is still coming up. The guard is bound in main so it outlives the process; dropping it stops sending, which loses exactly the shutdown errors.

The disabled-telemetry branch moves from the fmt() builder to registry composition so the Sentry layer can sit beside the writer. Filter and writer unchanged.

Build identity

Events carry ferrosa@version+sha. build.rs prefers FERROSA_BUILD_SHA from CI, falls back to git rev-parse, and says unknown last — a stale SHA would be believed. A dirty tree is marked.

The release workflow should set FERROSA_BUILD_SHA so tagged builds carry the commit CI checked out.

The DSN is not in this repository

ferrosa is PUBLIC. A DSN is not a secret — it's a write-only ingest key meant to be embedded — but one committed to a public repo can be used by anyone to write into the project, and rotating it then needs a release. Empty (what an unset shell variable looks like) is absent, not broken; plain http is refused, since it would put diagnostics on the wire in the clear.

Dependency choice, deliberately

sentry 0.49, not 0.34. The older release pins rustls 0.22, whose rustls-webpki 0.102 carries four RUSTSEC advisoriescargo deny caught exactly this on the ferrosa-memory branch. One rustls-webpki in the lock now, the patched 0.103. rustls rather than native-tls because CI builds this for x86_64-unknown-linux-musl.

Adding a diagnostics dependency that ships known vulnerabilities would be a poor trade for observability.

Verification

  • 7 tests in sentry_reporting
  • cargo clippy --all-targets -- -D warnings and cargo fmt --check pass
  • 1786 passed, 0 failures under the exact CI invocation

One note on the suite: a plain --all-features run also surfaces real_typed_edges_paged_scan_delivers_every_distinct_row, which fails loud without FERROSA_TEST_TYPED_EDGES_DIR by design. CI skips it explicitly, and so does the command above. I checked main is green before concluding that.

The __eh_frame linker warning on debug builds is pre-existing on main — I verified by stashing — and RUSTFLAGS isn't set in CI, so build warnings aren't errors there; clippy's -D warnings is the gate and it passes.

ferrosa had a telemetry layer already, gated on FERROSA_TELEMETRY_ENABLED, but
no path for an error to leave the machine. A node that logs a failure to a
console nobody is attached to has not reported it.

Errors become Sentry events when the installer configures FERROSA_SENTRY_DSN.
WARN becomes a breadcrumb rather than an event: it is context for the error
that follows, and as an event it would bury it. Everything below WARN stays in
the log — a storage engine at info level would exhaust the quota in minutes and
hide every error underneath the noise.

The client starts BEFORE the subscriber. The layer is inert without it, and the
errors most worth having from a database are the ones raised while it is still
coming up. The guard is bound in main so it lives until the process exits;
dropping it stops sending, which loses exactly the shutdown errors.

The disabled-telemetry branch moves from the fmt() builder to registry
composition so the Sentry layer can sit beside the writer. Filter and writer
are unchanged.

Events carry `ferrosa@version+sha`. build.rs prefers FERROSA_BUILD_SHA from CI,
falls back to git, and says `unknown` last — a stale SHA would be believed. A
dirty tree is marked.

The DSN is not in this repository. ferrosa is PUBLIC: a committed DSN can be
used by anyone to write into the project and cannot be rotated without a
release. An empty value — what an unset variable looks like from a shell — is
absent rather than broken, and plain http is refused; it would put diagnostics
on the wire in the clear.

sentry 0.49, not 0.34: the older release pins rustls 0.22, whose rustls-webpki
0.102 carries four RUSTSEC advisories. One rustls-webpki in the lock file, and
it is the patched 0.103. rustls rather than native-tls because CI builds this
for x86_64-unknown-linux-musl.

7 tests in sentry_reporting. clippy --all-targets -D warnings and fmt --check
pass. The workspace suite: 1786 passed with 0 failures under the exact CI
invocation. A plain --all-features run also surfaces
real_typed_edges_paged_scan_delivers_every_distinct_row, which fails loud
without FERROSA_TEST_TYPED_EDGES_DIR by design and which CI skips explicitly.
@bkearns
bkearns added this pull request to the merge queue Aug 29, 2026
Merged via the queue into main with commit 8026fdf Aug 29, 2026
18 checks passed
@bkearns
bkearns deleted the feat/telemetry-sentry branch August 29, 2026 02:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant