Skip to content

feat: extend edge routing telemetry with resume and outcome labels - #591

Open
Angela (Angelawork) wants to merge 5 commits into
agent-substrate:mainfrom
Angelawork:feature/router-telemetry
Open

feat: extend edge routing telemetry with resume and outcome labels#591
Angela (Angelawork) wants to merge 5 commits into
agent-substrate:mainfrom
Angelawork:feature/router-telemetry

Conversation

@Angelawork

@Angelawork Angela (Angelawork) commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Fixes #564 (Part 1)

  • Tests pass
  • Appropriate changes to documentation are included in the PR

Description

This PR implements the first part of #564 by extending metric atenet.router.route.duration with singleflight activation state (ate.router.resume) and outcome status labels (ate.router.outcome). All telemetry attribute keys and label value sets are centralized in internal/ateattr to ensure consistency across components.

Key Changes:

  • Added resumed boolean to ResumeActorResponse in ateapi.proto indicating whether a cold activation workflow occurred (!state.WasRunning).
  • Updated workflow_resume_test.go to assert resumed == true on cold activations and resumed == false for already-running actors.
  • Extended classifyOutcome(err) in Envoy ext_proc to classify errors into metric labels: ok, cancelled, timeout, no_capacity, lock_conflict, not_found, unavailable, rate_limited, and resume_error.
  • Added explicit handling for codes.Unavailable ("unavailable") and codes.ResourceExhausted / StatusCode_TooManyRequests ("rate_limited").
  • Consolidated metric label names on atenet.router.route.duration using standard keys from internal/ateattr:
    • ate.template.namespace (TemplateNamespaceKey)
    • ate.template.name (TemplateNameKey)
    • ate.router.outcome (RouterOutcomeKey)
    • ate.router.resume (RouterResumeKey)
  • Updated manifests/ate-install/kind/kustomization.yaml to export atenet-router OTLP metrics to opentelemetry-collector.

Testing

  • go test -buildvcs=false ./cmd/ateapi/internal/controlapi/...
  • go test -buildvcs=false ./cmd/atenet/internal/router/...
  • make test

E2E Test

  • ./hack/create-kind-cluster.sh
  • ./hack/install-ate-kind.sh --deploy-ate-system --deploy-demo-counter
  • ./hack/run-e2e.sh ./internal/e2e/suites/metrics/...

@Angelawork
Angela (Angelawork) force-pushed the feature/router-telemetry branch 2 times, most recently from 4c0eb73 to 67c0e93 Compare July 29, 2026 18:02
Comment thread cmd/atenet/internal/router/extproc.go
Comment thread cmd/atenet/internal/router/extproc.go
Comment thread cmd/atenet/internal/router/extproc.go Outdated
Comment thread cmd/atenet/internal/router/resumer.go Outdated
Comment thread cmd/atenet/internal/router/resumer.go
Comment thread cmd/atenet/internal/router/resumer.go Outdated
Comment thread cmd/atenet/internal/router/resumer.go
@JeffLuoo

Copy link
Copy Markdown
Collaborator

Overall it LGTM, but since this change touches ResumeActor function, and also this is the first PR I review, I would like to have it also reviewed by folks who are more experienced on this project.

Julian Gutierrez Oschmann (@juli4n) Max Smythe (@maxsmythe) Zoe Zhao (@zoez7)

Comment thread cmd/atenet/internal/router/extproc.go Outdated
Comment thread cmd/atenet/internal/router/extproc.go Outdated
Comment thread cmd/atenet/internal/router/extproc.go
Comment thread cmd/ateapi/internal/controlapi/workflow_resume_test.go Outdated
@Angelawork
Angela (Angelawork) force-pushed the feature/router-telemetry branch 2 times, most recently from ff4839b to c955d56 Compare July 30, 2026 15:27

@JeffLuoo Jeff Luo (JeffLuoo) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please rebase

@Angelawork

Copy link
Copy Markdown
Collaborator Author

Please rebase

Done!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature request: More system metrics for debuggability (Activation SLI, crash accounting, and capacity signals)

3 participants