From 8f7fcd4de3d2d753ad827159dd8ecb2340ffba0a Mon Sep 17 00:00:00 2001 From: incendiary Date: Fri, 31 Jul 2026 01:07:09 +0800 Subject: [PATCH] docs: add AGENTS.md orientation for AI agents The purpose of this tool is not recoverable from the code, and reading it the obvious way gets it wrong. Dangling CNAME detection is visible and prominent, so it reads as the point; the cloud IP matching looks like an attribute. It is the other way round in value. A dangling CNAME names a service, so intent can be read from DNS. A bare A record does not. If the address is a released cloud IP, whoever allocates it next controls what is served for that hostname, and DNS gives no signal that this is so. Finding which A records land in cloud ranges worth grinding, and where, is the differentiator. That makes provider, region and service the essential payload rather than decoration, and it makes the output files machine-readable interfaces for a downstream tool rather than reports. Also records the boundary that this tool identifies targets and never claims them, the two correctness rules recovered from shipped bugs, the output contracts, the environment traps (broken interpreter shim, both requirements files), the conventions that have bitten before (never tag, do not bump VERSION per PR, secret scanning is strict, branch protection wants job names), and the accepted limitations that follow from staying DNS-only. Named AGENTS.md as that is the file OpenAI Codex looks for at repo root. Co-Authored-By: Claude Opus 5 --- AGENTS.md | 232 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 232 insertions(+) create mode 100644 AGENTS.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..7f84b48 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,232 @@ +# AGENTS.md — orientation for AI agents working on DNSResolver + +Read this before changing anything. It records what the tool is *for*, which is not +obvious from the code, plus the conventions and traps that have already cost time here. + +Companion documents: [`REVIEW.md`](REVIEW.md) (findings), [`ROADMAP.md`](ROADMAP.md) (planned +work), [`docs/roadmap/AGENT-GUIDE.md`](docs/roadmap/AGENT-GUIDE.md) (per-task working rules). + +--- + +## 1. What this tool is for + +DNSResolver is passive DNS reconnaissance for offensive security. It takes a flat list of +candidate domains and **produces targets**. It has two jobs of equal weight: + +**Job 1 — dangling CNAME / subdomain takeover candidates.** A CNAME pointing at a service +name nobody owns any more. Well-covered ground; other tools do this too. + +**Job 2 — A records resolving into claimable cloud IP space.** This is the differentiator +and the part that is easy to under-value when reading the code. + +### Why Job 2 is the interesting one + +With a dangling CNAME you can *read the intent from DNS* — the target names a service +(`foo.s3.amazonaws.com`) and you can reason about whether it is claimable. + +With a bare A record you cannot. `1.2.3.4` says nothing. If that address is an AWS Elastic +IP that the owner released, whoever allocates it next controls what is served for that +hostname — but **DNS gives you no signal that this is the case**. The only way to find out +is to churn IP allocation in that provider and region until you are handed the address. + +So the tool's job is to answer: *which A records land in cloud ranges worth grinding, and +where?* That makes provider + region + service the essential payload, not decoration. + +### This tool does not claim anything + +It **identifies and hands off targets**. The allocation grinding is a separate tool. That +boundary is deliberate: + +- Do not add claiming, allocation, or any cloud-provider API calls here. +- Do not encode per-mechanism heuristics about what is "claimable". Surface the provider's + own published metadata faithfully and let the operator judge. +- The output files are therefore **machine-readable interfaces**, not prose reports. Treat + their formats as contracts — see §4. + +### Scope boundary (firm) + +DNS only. **No HTTP/HTTPS requests, no TLS inspection, no port scanning, no screenshots.** +Confirming a takeover needs an HTTP fetch; that belongs to a different tool. This keeps +DNSResolver passive, dependency-light, and safe to run broadly. Do not cross this line even +when it would resolve an ambiguity — several known limitations exist *because* of it, and +they are accepted trade-offs, not bugs. See §7. + +--- + +## 2. Architecture + +``` +resolver.py CLI entry point ────┐ +lambda_handler.py Lambda entry point ─┴─► run(env_manager) — shared async pipeline +``` + +| Module | Responsibility | +|---|---| +| `classes/environment_manager.py` | CLI environment: argparse glue, logging, domain file I/O. `fetch_external_ip()` is a module-level seam so the class can be built without a network. | +| `classes/lambda_environment_manager.py` | Lambda environment: env vars, stdout logging, `/tmp` I/O | +| `classes/config_resolver.py` | Parses CLI args, merges `config.json`. CLI always wins. | +| `classes/output_manager.py` | Builds the timestamped output tree; async writes via `aiofiles` | +| `classes/dns_handler.py` | Async resolution. Queries **A and AAAA concurrently**; a domain is unresolved only when *both* fail. aiodns primary, dnspython second opinion. | +| `classes/takeover_detector.py` | Dangling CNAME + NS takeover, depth-limited CNAME chain following | +| `classes/wildcard_detector.py` | Per-zone wildcard detection via random-label probes, cached per zone | +| `classes/domain_categoriser.py` | Regex classification of CNAME targets against `config.json`. **First match wins** — order is specific → general. | +| `classes/csp_ip_addresses.py` | Holds fetched provider ranges **and their region/service metadata** | +| `imports/cloud_ip_ranges.py` | Fetches AWS/GCP/Azure published ranges. Azure uses a three-stage fallback (live scrape → local cache → pinned URL). | +| `imports/cloud_service_provider_checks.py` | Matches resolved IPs to provider ranges; writes the handoff records | +| `imports/domain_processor.py` | Per-domain orchestration | +| `classes/run_summary.py` | End-of-run operator summary | + +Concurrency is `asyncio.gather` under a `Semaphore` (`--max-threads`). Failed domains are +collected and retried up to `--retries` times. + +--- + +## 3. Two correctness rules that were learned the hard way + +Both were shipped bugs. Do not reintroduce them. + +**A takeover candidate requires an actual CNAME.** A name that simply does not exist +(NXDOMAIN, no CNAME) has nothing dangling to claim. It must be reported as *unresolved*, +never as a candidate. An earlier version flagged every dead name, and a live run of 15 hosts +reported 9 candidates, all false. On a real enumeration — where most guessed names do not +exist — that noise buries the findings the tool exists to surface. + +**`self_referential` only applies at recursion depth > 0.** The check compares the current +name against the original, which is trivially true on entry. Gating on depth is what makes +it mean "a CNAME chain looped back", which is the only case it should describe. + +--- + +## 4. Output contracts + +Everything lands in a timestamped directory under `--output-dir`. **These are interfaces +consumed by other tooling — changing a format is a breaking change.** + +| File | Format | +|---|---| +| `resolution_results_*.txt` | `domain\|ip1\|ip2\|…` — IPv4 first, then IPv6. Prefixed `WILDCARD\|` when the name resolved only via a zone wildcard. | +| `takeover_candidates_*.txt` | `DANGLING\|origin\|target\|category\|recommendation\|evidence` and `NS_TAKEOVER\|origin\|domain` | +| `csp_matches_*.txt` | Cloud handoff records — see below | +| `unresolved_results_*.txt` | Domains that failed all retries | +| `environment_results_*.json` | Run metadata (command, external IP, Docker status) | +| `evidence/dns/` | `dig`/`nslookup` capture per flagged domain, when `--evidence` is set | + +The prefix convention (`WILDCARD|`, `DANGLING|`, `NS_TAKEOVER|`) is the house style for +tagging a line's type while keeping untagged lines backward compatible. Follow it. + +--- + +## 5. Environment and tooling + +**`python3` on the maintainer's machine is a broken pyenv shim.** Always use an explicit +interpreter: + +```bash +/opt/homebrew/bin/python3.12 -m venv .venv +.venv/bin/pip install -r requirements-dev.txt -r requirements-lambda.txt +``` + +Install **both** requirements files. `requirements-lambda.txt` supplies `boto3`; without it +the Lambda tests fail with `ModuleNotFoundError` — an environment gap, not a code bug. CI +installs both. + +**Before pushing**, run CI locally. A fail-closed `pre-push` hook enforces this; the +`pre-commit` slot chains gitleaks and ruff. + +```bash +bash ~/.claude/skills/local-ci/local-ci.sh --workflow ci.yml +.venv/bin/pytest --cov=classes --cov=imports +``` + +--- + +## 6. Conventions and traps + +- **Never run `git tag`.** Tags are created only when a release is cut. A tag for an + unreleased version breaks the version-sync baseline test on *every other branch*. +- **Do not bump `VERSION` per PR.** Version centrally, once per batch, at release time. + Parallel branches each bumping it conflict on every merge. +- **`VERSION` and `version.py` must agree**, and a baseline test enforces it. +- **Secret scanning is live and strict.** A gitleaks rule flags the maintainer's employer + name. Never put real client or employer names in code, comments, commits, or docs — refer + to "a client domain" or use `example.com`. This repository is public. +- **No GitHub pingback syntax.** Never write `owner/repo#123` for another repository; use a + full URL in backticks or plain prose. +- **Branch protection** requires the status checks `test`, `gitleaks`, `trufflehog` — these + are *job* names. Do not set required contexts to workflow names; that silently blocks + every PR forever. +- Conventional commit titles (`feat:`, `fix:`, `docs:`, `chore:`, `test:`), squash merge, + one logical change per PR. + +--- + +## 7. Known limitations (accepted, not bugs) + +- **Resolution covers A and AAAA only.** Hosts reachable solely by other record types read + as unresolved. +- **Classification is first-match-wins.** An unmatched CNAME target is reported `unknown` + rather than guessed at. +- **Wildcard detection cannot separate a real host from a catch-all** when the host shares + the wildcard's addresses. Shared hosting is the common case: a live site and a fabricated + name under the same platform return byte-identical DNS. Distinguishing them needs an HTTP + request, which §1 rules out. The flag means *"resolution proves nothing in this zone"*, + not *"this host does not exist"*. +- **Performance at scale is reasoned, not measured.** Known cliffs were removed (per-IP CIDR + parsing, O(n²) dedupe) but no benchmark has been run against a large list. + +--- + +## 8. Testing standards + +- **No test may perform a real DNS query or HTTP request.** Mock `aiodns`, `dns.resolver`, + `requests`, `urlopen`. Follow the patterns in `tests/conftest.py` and + `tests/test_dns_handler.py`. +- **Assert outcomes, not execution.** A test that only proves code "ran without raising" is + worse than no test, because it reads as coverage. Assert the classification string, the + precedence result, the graceful return value. +- **Coverage only goes up.** Currently ~95%. +- **Every bug fix gets a regression test.** + +### The verification lesson + +This is worth internalising. During a recent hardening pass, coverage rose from 85% to 95% +and caught **none** of the defects that mattered. All three were found elsewhere: + +- two by **running the tool against real DNS** (the false takeover candidates, and a wildcard + feature that passed all 16 of its unit tests while being completely non-functional on real + dual-stack zones); +- one by **attempting a real merge** (the branch-protection misconfiguration). + +Unit tests confirm the logic you thought of. They cannot tell you the premise was wrong. For +changes to resolution or classification, **run the tool against real domains** before +believing it works. + +--- + +## 9. In flight + +Region and service metadata capture is partially implemented on the branch +`feat/cloud-region-service`. All three providers publish it and the code currently discards +it at parse time: + +| Provider | Publishes | Historically kept | +|---|---|---| +| AWS | `ip_prefix`, `region`, `service`, `network_border_group` | prefix only | +| GCP | `ipv4Prefix`/`ipv6Prefix`, `service`, `scope` | prefix only | +| Azure | `addressPrefixes`, `region`, `systemService` | prefixes only | + +Without region and service a cloud match is not actionable downstream — the consumer needs +to know where to allocate and what the address belongs to. Of AWS's ~10,500 prefixes, over +half are the generic `AMAZON` tag, so "it is an AWS IP" is close to no information; `EC2` +in a named region is the useful signal. + +The work extends the fetchers to return `(ipv4, ipv6, metadata)` where metadata maps +CIDR → `(region, service)`, carries it on `CSPIPAddresses`, and writes one handoff record +per matched address: + +``` +domain|ip|provider|region|service|prefix +``` + +This changes the `csp_matches_*.txt` contract from prose to structured fields, and the +associated tests must be updated to assert the new contract rather than relaxed.