Repository navigation
Push metrics over OTLP/HTTP instead of Prometheus /metrics (MARSOHS-1593) - #5
Open
ElanHasson wants to merge 1 commit into
Open
ElanHasson wants to merge 1 commit into
ElanHasson wants to merge 1 commit into
Conversation
ElanHasson
marked this pull request as ready for review
September 21, 2026 14:09
Collaborator
Author
|
We need to ensure ALL telemetry is covered. |
MARSOHS-1593. On App Platform nothing can scrape a specific instance, so
the Prometheus pull endpoint was one of the reasons the worker was pinned
to one replica. Both `tester serve` and `tester run` now push metrics over
OTLP/HTTP, configured entirely by the standard OTEL_* environment variables
(endpoint, bearer header, gzip, service name, export interval), the same
path the OHP cluster's otel-gateway uses to reach DigitalOcean's
observability edge. With no endpoint set, export is disabled and logged.
New package telemetry:
- Setup installs the SDK MeterProvider with an otlpmetrichttp exporter
behind a periodic reader; resource = defaults (service.name,
service.instance.id=hostname) overridden by OTEL_SERVICE_NAME /
OTEL_RESOURCE_ATTRIBUTES; returns a shutdown that flushes on exit.
- Instruments, all under scope github.com/digitalocean/tester:
server: tester.test.duration (legacy Prometheus buckets kept),
tester.test.last_run, tester.run.queue_wait, tester.run.duration,
tester.run.claims{result}, tester.run.scheduled{package},
tester.run.reset{package,result}, tester.runs.pending{package,state}
(observed from the DB, zero-filled for every configured package).
runner: tester.runner.polls{result} (heartbeat), tester.runner.run.duration.
- bind() re-creates instruments on the installed provider so they are
direct rather than delegating, and lets tests use a manual reader.
Removed: http/metrics.go, the /metrics route, and the
prometheus/client_golang dependency.
Tests: Setup disabled/enabled (fake OTLP endpoint receives a POST with the
configured Authorization header), resource precedence, and a manual-reader
test asserting every series, unit, bucket boundaries and the zero-filled
pending gauge.
Co-authored-by: Cursor <cursoragent@cursor.com>
ElanHasson
force-pushed
the
otlp-metrics
branch
from
October 9, 2026 16:37
9ed5f72 to
6ec002e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Targets
main. Supersedes #4, which GitHub auto-closed when its base branch was deleted after #3 merged. Jira: MARSOHS-1593. Design:docs/superpowers/specs/2026-09-20-fork-bootstrap-scheduler-otlp-design.md.On App Platform nothing can scrape a specific instance, so the Prometheus
/metricsendpoint was one of the two reasonsworkeris pinned at one replica (the other is #3). Bothtester serveandtester runnow push over OTLP/HTTP, the same path the OHP cluster's otel-gateway uses to reach the DigitalOcean observability edge.Configuration
Entirely via standard
OTEL_*env vars; export is off (and logged once) when no endpoint is set. The App Platform spec will carry:New package
telemetrySetup(ctx, defaultServiceName)installs an SDKMeterProvider(otlpmetrichttp + periodic reader). Resource defaults (service.name,service.instance.id=hostname) are overridden byOTEL_SERVICE_NAME/OTEL_RESOURCE_ATTRIBUTES. Returns a shutdown that flushes on exit; both commands call it.github.com/digitalocean/tester):tester.test.duration(s)name,statetester.test.last_run(unix s)name,statetester.run.queue_wait(s)packagetester.run.duration(s)package,resulttester.run.claimsresult=claimed/empty/errortester.run.scheduledpackagetester.run.resetpackage,result=reset/failedtester.runs.pendingpackage,state=queued/runningtester.runner.pollsresulttester.runner.run.duration(s)package,resulthttp/metrics.go, the/metricsroute andprometheus/client_golangare removed. Dashboards ontester_tb_run_duration_smove to the edge's rendering oftester.test.duration.Verified
go test -race ./...green.telemetrytests: disabled path, enabled path against a fake OTLP endpoint (asserts theAuthorizationheader arrives), resource precedence, and a manual-reader test asserting every series, unit, bucket bounds and the zero-filled pending gauge.serveexported 7 metrics asservice.name=tester, two runners exportedtester.runner.*astester-runner, all gzip with the bearer header.Follow-ups (not in this PR)
OTEL_*env vars towebandworker, then raiseworkerinstance_count(MARSOHS-1594).tester.runs.pending{state=queued}andtester.run.queue_wait(MARSOHS-1479).Made with Cursor