Skip to content

Commit c2d22e1

Browse files
committed
feat(transit-ops): persistent provider health, staleness alerts, OTEL metrics, admin UI
Wire up the observability + automation surface for the self-hosted Transitous pipeline + transit provider chain. Six pieces land together because they share data: a Redis-backed provider-health tracker, per-feed staleness alerting, OpenTelemetry counters, admin UI, plus two small infrastructure refactors that fall out of the cleanup. Persistent provider health - Replace the in-memory `integrations/transit/health.ts` counter with a Redis-backed service at apps/api/src/services/provider-health/. State per provider: success/failure totals, sliding window of last 100 outcomes, lastFailureAt + lastFailureReason, EMA latency (alpha 0.2). Auto-disable when window failure rate > 50% over at least 10 samples; 5-min cooldown. - All state writes go through a single Lua script for atomicity. Keys `provider:health:<id>` with 30-day TTL. - IntegrationContext gains optional providerHealth and metricsRecorder handles. apps/api wires the singletons; integrations/transit /orchestrator.ts wraps every provider call in `timed(health, metrics, providerId, method, fn)`. - New endpoints on /api/data-manager: - GET /providers — sorted list with full state. - GET /providers/:id — single state or 404. - POST /providers/:id/reset — admin-session only. - /api/integrations/transit/health is now 410 Gone with a pointer to the new endpoint. Per-feed staleness alerts - Add `consecutive_failures` column to data_manager.feed_state (migration 0005_feed_state_consecutive_failures.sql). Pipeline stages bump it on validation failure, reset on success. - New services/data-manager/src/jobs/transitous/staleness-alerts.ts with detectStaleFeeds + emitFeedAlerts. Triggers: - stale: last_fetched_at older than 48h. - consecutive-failures: ≥ 3 in a row. - Always logs a structured warn line. Optional GitHub Issue sink controlled by TRANSITOUS_ALERT_GH_TOKEN + TRANSITOUS_ALERT_GH_REPO; dedupes by exact title against open issues. - New TRANSITOUS_STALENESS_CHECK_CRON (default `0 4 * * *`, daily after the 03:00 sync) + post-sync immediate invocation as a safety net. OpenTelemetry metrics - Add @opentelemetry/{api,sdk-metrics,exporter-prometheus} to apps/api. New apps/api/src/services/metrics/index.ts initialises a singleton MeterProvider with two instruments: - transit_provider_calls_total (counter) - transit_provider_call_duration_ms (histogram) Labels: provider_id, method, outcome (ok/empty/error/skipped). - Mount GET /api/internal/metrics returning Prometheus text format. No auth — restrict via firewall / docker network. - New Grafana dashboard at infra/docker/dashboards/transit-providers .json with three panels (call rate, p95 latency, error rate by provider). Admin UI - New apps/web/src/app/admin/transit/ page assembled from: - LockfileCard (Transitous ref + lockedAt + lockedBy, display-only). - CurrentJobCard (running state + "Run sync now" + "Restart MOTIS" buttons; live polling at 5s when in-flight). - RecentJobsTable (last 20 jobs + drawer with stage breakdown). - FeedsBreakdownChart (pure-CSS bars by region + status). - FeedsTable (paginated feed_state with region + status filters). - ProviderHealthTable (per-row Reset). - New apps/web/src/lib/admin/transitHooks.ts (read + mutation hooks). Sidebar nav entry added in AdminSidebar. - All English strings; admin shell has no i18n integration today. Sweep cleanup - Delete the deprecated useProviders hook + resolveProvider / providerAttribution / providerLabel helpers + the /api/integrations/transit/providers route + getTransitProviderAttribution orchestrator function + the now-empty attribution.test.ts. All callers were removed by the earlier attribution-canonicalisation pass; the residue is unused. - Remove the duplicated ProviderAttribution interface from packages/core/src/constants/transit.ts + its barrel export. The framework's @openmapx/integration-framework version is now the single source. Zero external imports survived the cleanup. - Extract services/motis-feed-proxy/lib/build.ts into a new @openmapx/motis-feed-proxy-config workspace package. Cross-service relative import in services/data-manager replaced by a typed workspace dep; services/data-manager/tsconfig narrows rootDir back to "." (was ".." to accommodate the relative import) and tsconfig.build.json emits to `dist/index.js` directly. Dockerfile CMD + start script updated to match. - services/motis-feed-proxy/ now holds only service.json — the nginx config rendering lives in the new shared package. - Strip plan references from code/doc comments (no §, no "plan §" citations). Comments now describe behaviour, not the plan they came from. Tests - 191 files, 2390 passed | 6 skipped. New suites: provider-health (10 unit tests covering window cap, auto-disable, cooldown expiry, reason truncation, EMA convergence), data-manager-providers route (8 cases for auth + 404 + reset), staleness-alerts (14 cases incl. GitHub sink dedup), metrics module (6 cases incl. Prometheus text endpoint), admin transit page + FeedsTable + ProviderHealthTable (10 cases).
1 parent 0fce7e3 commit c2d22e1

64 files changed

Lines changed: 7303 additions & 792 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

apps/api/package.json

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,9 @@
3232
"@openmapx/db-schema": "workspace:^",
3333
"@openmapx/integration-framework": "workspace:^",
3434
"@openmapx/place-ids": "workspace:^",
35+
"@opentelemetry/api": "^1.9.1",
36+
"@opentelemetry/exporter-prometheus": "^0.218.0",
37+
"@opentelemetry/sdk-metrics": "^2.7.1",
3538
"better-auth": "^1.6.9",
3639
"better-auth-harmony": "^1.3.2",
3740
"cached-hafas-client": "^5.1.9",
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
ALTER TABLE "data_manager"."feed_state" ADD COLUMN "consecutive_failures" integer DEFAULT 0 NOT NULL;

0 commit comments

Comments
 (0)