Repository navigation
Walk a made-up CloudTrail incident from first query to reviewed report - #81
Merged
Merged
Conversation
The agent-native journey had never been run end to end on one realistic case. This adds a synthetic week of CloudTrail with a planted leaked key, the SQL and dashboards an agent would write, and a test that walks the agent's half and asserts what it finds. Running it found two things worth fixing. `schema` refused `.json.gz`, CloudTrail's own format, calling it not a DuckDB database. And a session that withheld SQL for possibly holding credentials then blamed the SQL for its unknown sources. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 3, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #71 and #64. This leaves #71 open, because nobody has run the window's half yet.
Why
Each part of the agent workflow has its own tests, but the whole journey had never been run on one realistic case. #71 asks for that, and for a record of where it hurts.
What
examples/cloudtrail_investigation/:data/cloudtrail.json.gz: one made-up week, 795 events, with a planted incident.generate.pyreproduces the same bytes. It uses AWS's documentation account IDs and keys, and RFC 5737 addresses.sql/,reports/investigation.dash,proposals/answer-new-user.dash: the investigation, the agent's first dashboard, and its answer to the person's question.README.md: the expected finding, the agent prompt and commands, the walkthrough in the window, and what happened.The planted finding: the build key
ci-deployappears at a new address, is refused 28 times out of 33 calls, creates an administratorsvc-backup, and that user reads 240 objects fromcustomer-exports.tests/cloudtrail_example.rswalks the agent's half on a copy and asserts those numbers. It also covers truncation, session doubts, the proposal leaving the report untouched, and a moved project.Two fixes it led to
schemarefused.json.gz, CloudTrail's own format, saying "This file is not a DuckDB database"schemaandsourceblocks.logs.json.gzbecomes a view calledlogsaccess_keymade the session drop the SQL, as designed, but the doubt then said the sources "could not be read from the SQL"The
.gzchange is insrc/db.rs, so it also reaches the window's file attach. That path has no test here and wasn't run.What this run recorded
schema --statssays little about a log that's one row of nestedRecords..dashfile has no shared view, so 5 queries repeat 13 lines of flattening.project initlists a proposal draft as a dashboard.access_key, and that's likely in security logs. I kept it, because it deliberately errs towards leaving text out.Not done
The window's half hasn't been run:
open --run --wait, drawing the dashboard, commenting, Review and Apply. On Windows the window can't start with its own state, so running it would have changed the saved workspace. The README says this step by step. Please run "The person's part" in the README, or tell me to and I'll back up the workspace first. Then #71 can close.The dashboard isn't checked under limits in this walkthrough, because
check --restrictedis in #80. Once both are in, step 4 of the README should use it.Testing
cargo test --locked -- --skip perf_probe::on Windows: 466 unit, 35 CLI, 3 example, 8 LSP and 4 network-path tests pass; clippy and fmt are clean.🤖 Generated with Claude Code