Skip to content

Walk a made-up CloudTrail incident from first query to reviewed report - #81

Merged
JetSquirrel merged 2 commits into
mainfrom
examples/cloudtrail-investigation
Nov 2, 2026
Merged

JetSquirrel merged 2 commits into
mainfrom
examples/cloudtrail-investigation

Conversation

@JetSquirrel

Copy link
Copy Markdown
Owner

Part of #71 and #64. This leaves #71 open, because nobody has run the window's half yet.

Why

Each part of the agent workflow has its own tests, but the whole journey had never been run on one realistic case. #71 asks for that, and for a record of where it hurts.

What

examples/cloudtrail_investigation/:

  • data/cloudtrail.json.gz: one made-up week, 795 events, with a planted incident. generate.py reproduces the same bytes. It uses AWS's documentation account IDs and keys, and RFC 5737 addresses.
  • sql/, reports/investigation.dash, proposals/answer-new-user.dash: the investigation, the agent's first dashboard, and its answer to the person's question.
  • README.md: the expected finding, the agent prompt and commands, the walkthrough in the window, and what happened.

The planted finding: the build key ci-deploy appears at a new address, is refused 28 times out of 33 calls, creates an administrator svc-backup, and that user reads 240 objects from customer-exports.

tests/cloudtrail_example.rs walks the agent's half on a copy and asserts those numbers. It also covers truncation, session doubts, the proposal leaving the report untouched, and a moved project.

Two fixes it led to

Found Fix
schema refused .json.gz, CloudTrail's own format, saying "This file is not a DuckDB database" Gzipped text now counts as a data file in schema and source blocks. logs.json.gz becomes a view called logs
A column aliased access_key made the session drop the SQL, as designed, but the doubt then said the sources "could not be read from the SQL" The doubt now gives the real reason

The .gz change is in src/db.rs, so it also reaches the window's file attach. That path has no test here and wasn't run.

What this run recorded

  • Manual handoffs: 4 expected, none run yet.
  • Friction: 7 items, in the README's table. Two are fixed above, one was a typo the error explained, and one has a workaround.
  • Still open, for you to decide if they deserve issues:
    • schema --stats says little about a log that's one row of nested Records.
    • A .dash file has no shared view, so 5 queries repeat 13 lines of flattening.
    • project init lists a proposal draft as a dashboard.
  • Worth a look: the credential word list costs the session its SQL for any query mentioning access_key, and that's likely in security logs. I kept it, because it deliberately errs towards leaving text out.

Not done

The window's half hasn't been run: open --run --wait, drawing the dashboard, commenting, Review and Apply. On Windows the window can't start with its own state, so running it would have changed the saved workspace. The README says this step by step. Please run "The person's part" in the README, or tell me to and I'll back up the workspace first. Then #71 can close.

The dashboard isn't checked under limits in this walkthrough, because check --restricted is in #80. Once both are in, step 4 of the README should use it.

Testing

  • cargo test --locked -- --skip perf_probe:: on Windows: 466 unit, 35 CLI, 3 example, 8 LSP and 4 network-path tests pass; clippy and fmt are clean.
  • I also ran the README's query and session commands by hand against the built binary.

🤖 Generated with Claude Code

JetSquirrel and others added 2 commits November 3, 2026 01:39
The agent-native journey had never been run end to end on one realistic
case. This adds a synthetic week of CloudTrail with a planted leaked key,
the SQL and dashboards an agent would write, and a test that walks the
agent's half and asserts what it finds.

Running it found two things worth fixing. `schema` refused `.json.gz`,
CloudTrail's own format, calling it not a DuckDB database. And a session
that withheld SQL for possibly holding credentials then blamed the SQL for
its unknown sources.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@JetSquirrel
JetSquirrel merged commit a2741a2 into main Nov 2, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant