Commit c2d22e1
committed
feat(transit-ops): persistent provider health, staleness alerts, OTEL metrics, admin UI
Wire up the observability + automation surface for the self-hosted
Transitous pipeline + transit provider chain. Six pieces land
together because they share data: a Redis-backed provider-health
tracker, per-feed staleness alerting, OpenTelemetry counters,
admin UI, plus two small infrastructure refactors that fall out of
the cleanup.
Persistent provider health
- Replace the in-memory `integrations/transit/health.ts` counter with
a Redis-backed service at apps/api/src/services/provider-health/.
State per provider: success/failure totals, sliding window of last
100 outcomes, lastFailureAt + lastFailureReason, EMA latency
(alpha 0.2). Auto-disable when window failure rate > 50% over at
least 10 samples; 5-min cooldown.
- All state writes go through a single Lua script for atomicity. Keys
`provider:health:<id>` with 30-day TTL.
- IntegrationContext gains optional providerHealth and metricsRecorder
handles. apps/api wires the singletons; integrations/transit
/orchestrator.ts wraps every provider call in `timed(health,
metrics, providerId, method, fn)`.
- New endpoints on /api/data-manager:
- GET /providers — sorted list with full state.
- GET /providers/:id — single state or 404.
- POST /providers/:id/reset — admin-session only.
- /api/integrations/transit/health is now 410 Gone with a pointer
to the new endpoint.
Per-feed staleness alerts
- Add `consecutive_failures` column to data_manager.feed_state
(migration 0005_feed_state_consecutive_failures.sql). Pipeline
stages bump it on validation failure, reset on success.
- New services/data-manager/src/jobs/transitous/staleness-alerts.ts
with detectStaleFeeds + emitFeedAlerts. Triggers:
- stale: last_fetched_at older than 48h.
- consecutive-failures: ≥ 3 in a row.
- Always logs a structured warn line. Optional GitHub Issue sink
controlled by TRANSITOUS_ALERT_GH_TOKEN + TRANSITOUS_ALERT_GH_REPO;
dedupes by exact title against open issues.
- New TRANSITOUS_STALENESS_CHECK_CRON (default `0 4 * * *`, daily
after the 03:00 sync) + post-sync immediate invocation as a
safety net.
OpenTelemetry metrics
- Add @opentelemetry/{api,sdk-metrics,exporter-prometheus} to
apps/api. New apps/api/src/services/metrics/index.ts initialises
a singleton MeterProvider with two instruments:
- transit_provider_calls_total (counter)
- transit_provider_call_duration_ms (histogram)
Labels: provider_id, method, outcome (ok/empty/error/skipped).
- Mount GET /api/internal/metrics returning Prometheus text format.
No auth — restrict via firewall / docker network.
- New Grafana dashboard at infra/docker/dashboards/transit-providers
.json with three panels (call rate, p95 latency, error rate by
provider).
Admin UI
- New apps/web/src/app/admin/transit/ page assembled from:
- LockfileCard (Transitous ref + lockedAt + lockedBy, display-only).
- CurrentJobCard (running state + "Run sync now" + "Restart
MOTIS" buttons; live polling at 5s when in-flight).
- RecentJobsTable (last 20 jobs + drawer with stage breakdown).
- FeedsBreakdownChart (pure-CSS bars by region + status).
- FeedsTable (paginated feed_state with region + status filters).
- ProviderHealthTable (per-row Reset).
- New apps/web/src/lib/admin/transitHooks.ts (read + mutation
hooks). Sidebar nav entry added in AdminSidebar.
- All English strings; admin shell has no i18n integration today.
Sweep cleanup
- Delete the deprecated useProviders hook + resolveProvider /
providerAttribution / providerLabel helpers + the
/api/integrations/transit/providers route + getTransitProviderAttribution
orchestrator function + the now-empty attribution.test.ts. All
callers were removed by the earlier attribution-canonicalisation
pass; the residue is unused.
- Remove the duplicated ProviderAttribution interface from
packages/core/src/constants/transit.ts + its barrel export.
The framework's @openmapx/integration-framework version is now
the single source. Zero external imports survived the cleanup.
- Extract services/motis-feed-proxy/lib/build.ts into a new
@openmapx/motis-feed-proxy-config workspace package. Cross-service
relative import in services/data-manager replaced by a typed
workspace dep; services/data-manager/tsconfig narrows rootDir back
to "." (was ".." to accommodate the relative import) and
tsconfig.build.json emits to `dist/index.js` directly. Dockerfile
CMD + start script updated to match.
- services/motis-feed-proxy/ now holds only service.json — the
nginx config rendering lives in the new shared package.
- Strip plan references from code/doc comments (no §, no "plan §"
citations). Comments now describe behaviour, not the plan they
came from.
Tests
- 191 files, 2390 passed | 6 skipped. New suites: provider-health
(10 unit tests covering window cap, auto-disable, cooldown
expiry, reason truncation, EMA convergence), data-manager-providers
route (8 cases for auth + 404 + reset), staleness-alerts (14
cases incl. GitHub sink dedup), metrics module (6 cases incl.
Prometheus text endpoint), admin transit page + FeedsTable +
ProviderHealthTable (10 cases).1 parent 0fce7e3 commit c2d22e1
64 files changed
Lines changed: 7303 additions & 792 deletions
File tree
- apps
- api
- src
- db/migrations
- meta
- routes
- __tests__
- services
- metrics
- __tests__
- provider-health
- __tests__
- web/src
- app/admin/transit
- __tests__
- components/admin
- lib/admin
- docs
- infra/docker/dashboards
- integrations
- transit-gtfs-local
- transit
- __tests__
- packages
- core/src
- api
- constants
- hooks/transit
- db-schema/src
- integration-framework/src
- contracts
- motis-feed-proxy-config
- __tests__
- src
- services/data-manager
- __tests__/transitous
- src
- jobs/transitous
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
35 | 38 | | |
36 | 39 | | |
37 | 40 | | |
| |||
Lines changed: 1 addition & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
0 commit comments