Conversation
|
All contributors have signed the CLA ✍️ ✅ |
|
| incidents, check oncall schedules, and gather incident context. | ||
|
|
||
| - Always use tools — do not guess or fabricate incident data. | ||
| - Unless explicitly requested otherwise, inspect incidents, alerts, and status over the last 5 minutes by default. |
There was a problem hiding this comment.
Lookback Hides Active Incidents
Applying the five-minute cutoff to incidents, monitors, alerts, and status can hide an active object that opened earlier. For example, an older but still-triggered PagerDuty incident could be omitted, causing incomplete triage or a false all-clear. Query active or open state independently, and apply the lookback only to the supporting telemetry timeline. This pattern also appears in the Datadog and Mezmo incident-response examples.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
| @@ -67,6 +68,7 @@ mcp_filter = ["prometheus_query", "prometheus_targets", "prometheus_metric_metad | |||
| preamble = """ | |||
| You are a Prometheus Analyst. Use your tools to query and analyze metrics. | |||
There was a problem hiding this comment.
Lookback Behavior Lacks Coverage
The SRE orchestration tests only check tool-name prefixes and do not inspect telemetry query arguments. A worker can therefore ignore the five-minute default or overwrite a user-requested range while all current tests still pass. Please add focused coverage for both the default window and an explicit override.
|
I have read the CLA Document and I hereby sign the CLA |
|
Hey @dirtyren - this PR looks good but only has some minor failings with fmt (we use nightly - you can see some of the setup in the .ci dir) and a commitlint run. There is a commitlintrc.js at the root that should be able to run with |
Stable releases run on main, where semantic-release lands a version commit and hangs the release tag on it. nightly and beta never produce that commit, so the main → channel back-merge is how they receive the tag. A channel's version comes from the tags it can reach, so one still waiting on that back-merge keeps numbering from the tag before it. After main shipped 0.2.17, nightly would build 0.2.17-nightly.1, and the moving `nightly` image tag would sort below `latest`. sync-main.yml proposes those back-merges and has never opened one. The job runs without a checkout, so `gh pr list` and `gh pr create` die with "fatal: not a git repository" resolving the repository from a git remote. - GH_REPO hands gh the repository instead. - A `release: released` trigger stops sync waiting up to six hours for the schedule, and skips beta's prereleases, which do not move main. - An open sync pull request moves forward with main rather than going stale, unless its branch carries commits main does not. The update is a fast-forward rather than a force, so a commit pushed while the job runs refuses the move instead of being discarded. - check-release-baseline.sh backstops all of it, refusing to release a channel that cannot reach main's latest release tag. Exit 10 is the stale verdict; any other non-zero fails the stage rather than reading as a releasable branch. - scripts/tests holds its suite, wired into the make test hook. See docs/design/release-channels.md. Ref: mezmo#697 Signed-off-by: Jacob Hull <jacob@planethull.com>
Add guidance in the coordinator prompt and specialist worker preambles to inspect the last 5 minutes of telemetry data (metrics, logs, traces, alerts) by default during operational investigations, unless the user explicitly specifies a different or longer time window. This prevents agents from executing unbounded queries or missing recent anomalies when triaging live cluster incidents. Ref: mezmo#697
Add integration test coverage in sre_orchestration_test for inspecting telemetry query arguments, verifying both the 5-minute default lookback window and explicit user-requested time range overrides. Also add unit tests in aura-config and aura ensuring shipped configs and preamble builders preserve telemetry lookback policies. Ref: mezmo#697
d310b95 to
8db9ffe
Compare
Summary
When Aura operates as an SRE orchestrator or incident responder, queries to metrics, logs, and telemetry systems should focus on the most recent operational state by default. Without an explicit lookback window, agents or specialist workers can either issue unbounded queries or fail to focus on recent anomalies.
This PR establishes a default 5-minute lookback window policy across Aura's quickstart configuration, SRE orchestration integration tests, and example incident response agent profiles:
now-5mtonow) by default.incident-responder,metrics-analyst,log-analyst): Added default 5m time window guidance to preambles.Changes
quickstart.toml: Added default 5m lookback policy to coordinator delegation rules/principles and worker preambles (incident-responder,metrics-analyst,log-analyst).configs/integration-sre-orchestration.toml: Added 5m lookback guidance to coordinator routing andprometheus-analystpreamble.examples/complete/kubernetes-sre-orchestrated-multi-model.toml: Added 5m lookback guidance to coordinator routing andprometheus-analyst.examples/complete/kubernetes-sre.toml: Added 5m telemetry lookback step to cluster troubleshooting workflow.examples/complete/incident-response-mezmo.toml&examples/complete/incident-response-datadog.toml: Added default 5m telemetry lookback guideline to triage workflows.crates/aura-web-server/tests/sre_orchestration_test.rs:get_tool_callsandextract_telemetry_queryto inspect tool invocation arguments.test_sre_workers_use_domain_toolsto inspect arguments when telemetry queries execute.test_sre_telemetry_query_defaults_to_five_minute_windowto assert that telemetry query arguments default to a 5-minute lookback interval (5m).test_sre_telemetry_query_respects_explicit_override_windowto assert that user-specified overrides (e.g. 1 hour) are respected in query arguments and not overwritten by the 5-minute default.crates/aura/src/orchestration/config.rs: Added unit tests verifying preamble builders preserve telemetry lookback policies.crates/aura-config/src/config_test.rs: Added unit tests verifying shipped configurations preserve lookback guidance.Verification