Skip to content

feat(orchestration): add default 5m telemetry lookback policy - #697

Open
dirtyren wants to merge 4 commits into
mezmo:nightlyfrom
dirtyren:feat/sre-telemetry-lookback-policy
Open

dirtyren wants to merge 4 commits into
mezmo:nightlyfrom
dirtyren:feat/sre-telemetry-lookback-policy

Conversation

@dirtyren

@dirtyren dirtyren commented Sep 14, 2026

Copy link
Copy Markdown

Summary

When Aura operates as an SRE orchestrator or incident responder, queries to metrics, logs, and telemetry systems should focus on the most recent operational state by default. Without an explicit lookback window, agents or specialist workers can either issue unbounded queries or fail to focus on recent anomalies.

This PR establishes a default 5-minute lookback window policy across Aura's quickstart configuration, SRE orchestration integration tests, and example incident response agent profiles:

  • Coordinator & Delegation Rules: Instructs workers to inspect telemetry (metrics, logs, traces, alerts) from the last 5 minutes (now-5m to now) by default.
  • Specialist Workers (incident-responder, metrics-analyst, log-analyst): Added default 5m time window guidance to preambles.
  • User Override Preservation: Explicitly respects user requests when a different or longer time window is requested (e.g. "past 2 hours", "since yesterday").
  • Focused Test Coverage: Added dedicated test coverage inspecting telemetry query arguments for both the default 5-minute lookback window and explicit user overrides.

Changes

  • quickstart.toml: Added default 5m lookback policy to coordinator delegation rules/principles and worker preambles (incident-responder, metrics-analyst, log-analyst).
  • configs/integration-sre-orchestration.toml: Added 5m lookback guidance to coordinator routing and prometheus-analyst preamble.
  • examples/complete/kubernetes-sre-orchestrated-multi-model.toml: Added 5m lookback guidance to coordinator routing and prometheus-analyst.
  • examples/complete/kubernetes-sre.toml: Added 5m telemetry lookback step to cluster troubleshooting workflow.
  • examples/complete/incident-response-mezmo.toml & examples/complete/incident-response-datadog.toml: Added default 5m telemetry lookback guideline to triage workflows.
  • crates/aura-web-server/tests/sre_orchestration_test.rs:
    • Added helper functions get_tool_calls and extract_telemetry_query to inspect tool invocation arguments.
    • Enhanced test_sre_workers_use_domain_tools to inspect arguments when telemetry queries execute.
    • Added test_sre_telemetry_query_defaults_to_five_minute_window to assert that telemetry query arguments default to a 5-minute lookback interval (5m).
    • Added test_sre_telemetry_query_respects_explicit_override_window to assert that user-specified overrides (e.g. 1 hour) are respected in query arguments and not overwritten by the 5-minute default.
  • crates/aura/src/orchestration/config.rs: Added unit tests verifying preamble builders preserve telemetry lookback policies.
  • crates/aura-config/src/config_test.rs: Added unit tests verifying shipped configurations preserve lookback guidance.

Verification

  • Validated that all modified and existing TOML configurations parse cleanly.
  • Added unit tests for preamble builders and shipped configuration validation.
  • Added SRE integration test assertions for default lookback arguments and explicit override preservation.

@dirtyren
dirtyren requested a review from a team September 14, 2026 14:21
@github-actions

github-actions Bot commented Sep 14, 2026

Copy link
Copy Markdown

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@greptile-apps

greptile-apps Bot commented Sep 14, 2026

Copy link
Copy Markdown

RetriggerConfidence Score: 4/5

The PR is not yet safe to merge because the five-minute cutoff can hide older active incidents or alerts.

Findings

  1. P1 Lookback Hides Active Incidents
  2. P2 Lookback Behavior Lacks Coverage

Summary

This PR adds a default five-minute telemetry lookback policy to shipped SRE and incident-response configurations and adds tests intended to cover the default and user-specified overrides.

  • Adds lookback guidance to coordinator and specialist-worker prompts.
  • Preserves explicitly requested longer time windows.
  • Adds configuration, preamble, and orchestration integration tests.
  • The earlier active-incident concern remains unresolved, and the attempted integration-test fix remains incomplete.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart LR
    U[User telemetry request] --> C[Coordinator]
    C -->|No explicit range| D[Apply default last 5 minutes]
    C -->|Explicit range| O[Preserve requested range]
    D --> W[Specialist worker]
    O --> W
    W --> T[Telemetry tool query]
    T --> R[Timestamped findings]
Loading

Reviews (4) · Last reviewed commit: "Merge branch 'nightly' into feat/sre-tel..."

Comment thread quickstart.toml
incidents, check oncall schedules, and gather incident context.

- Always use tools — do not guess or fabricate incident data.
- Unless explicitly requested otherwise, inspect incidents, alerts, and status over the last 5 minutes by default.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Lookback Hides Active Incidents

Applying the five-minute cutoff to incidents, monitors, alerts, and status can hide an active object that opened earlier. For example, an older but still-triggered PagerDuty incident could be omitted, causing incomplete triage or a false all-clear. Query active or open state independently, and apply the lookback only to the supporting telemetry timeline. This pattern also appears in the Datadog and Mezmo incident-response examples.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

@@ -67,6 +68,7 @@ mcp_filter = ["prometheus_query", "prometheus_targets", "prometheus_metric_metad
preamble = """
You are a Prometheus Analyst. Use your tools to query and analyze metrics.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Lookback Behavior Lacks Coverage

The SRE orchestration tests only check tool-name prefixes and do not inspect telemetry query arguments. A worker can therefore ignore the five-minute default or overwrite a user-requested range while all current tests still pass. Please add focused coverage for both the default window and an explicit override.

@dirtyren

Copy link
Copy Markdown
Author

I have read the CLA Document and I hereby sign the CLA

@Shearerbeard

Copy link
Copy Markdown
Collaborator

Hey @dirtyren - this PR looks good but only has some minor failings with fmt (we use nightly - you can see some of the setup in the .ci dir) and a commitlint run. There is a commitlintrc.js at the root that should be able to run with npm run commitlint.

jakedipity and others added 3 commits September 18, 2026 00:09
Stable releases run on main, where semantic-release lands a version
commit and hangs the release tag on it. nightly and beta never produce
that commit, so the main → channel back-merge is how they receive the
tag. A channel's version comes from the tags it can reach, so one still
waiting on that back-merge keeps numbering from the tag before it. After
main shipped 0.2.17, nightly would build 0.2.17-nightly.1, and the
moving `nightly` image tag would sort below `latest`.

sync-main.yml proposes those back-merges and has never opened one. The
job runs without a checkout, so `gh pr list` and `gh pr create` die with
"fatal: not a git repository" resolving the repository from a git
remote.

- GH_REPO hands gh the repository instead.
- A `release: released` trigger stops sync waiting up to six hours for
  the schedule, and skips beta's prereleases, which do not move main.
- An open sync pull request moves forward with main rather than going
  stale, unless its branch carries commits main does not. The update is
  a fast-forward rather than a force, so a commit pushed while the job
  runs refuses the move instead of being discarded.
- check-release-baseline.sh backstops all of it, refusing to release a
  channel that cannot reach main's latest release tag. Exit 10 is the
  stale verdict; any other non-zero fails the stage rather than reading
  as a releasable branch.
- scripts/tests holds its suite, wired into the make test hook.

See docs/design/release-channels.md.

Ref: mezmo#697
Signed-off-by: Jacob Hull <jacob@planethull.com>
Add guidance in the coordinator prompt and specialist worker preambles
to inspect the last 5 minutes of telemetry data (metrics, logs, traces,
alerts) by default during operational investigations, unless the user
explicitly specifies a different or longer time window.

This prevents agents from executing unbounded queries or missing recent
anomalies when triaging live cluster incidents.

Ref: mezmo#697
Add integration test coverage in sre_orchestration_test for inspecting
telemetry query arguments, verifying both the 5-minute default lookback
window and explicit user-requested time range overrides.

Also add unit tests in aura-config and aura ensuring shipped configs and
preamble builders preserve telemetry lookback policies.

Ref: mezmo#697
@dirtyren
dirtyren force-pushed the feat/sre-telemetry-lookback-policy branch from d310b95 to 8db9ffe Compare September 18, 2026 03:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants