feat: capture cloud region and service, emit structured handoff records

AGENTS.md

@@ -203,30 +203,30 @@ believing it works.
 
 ---
 
-## 9. In flight
+## 9. Recently landed
 
-Region and service metadata capture is partially implemented on the branch
-`feat/cloud-region-service`. All three providers publish it and the code currently discards
-it at parse time:
+Region and service metadata capture is **done**. All three providers publish it and the code
+previously discarded it at parse time:
 
-| Provider | Publishes | Historically kept |
+| Provider | Publishes | Now captured as |
 |---|---|---|
-| AWS | `ip_prefix`, `region`, `service`, `network_border_group` | prefix only |
-| GCP | `ipv4Prefix`/`ipv6Prefix`, `service`, `scope` | prefix only |
-| Azure | `addressPrefixes`, `region`, `systemService` | prefixes only |
+| AWS | `ip_prefix`, `region`, `service` | region, service |
+| GCP | `ipv4Prefix`/`ipv6Prefix`, `service`, `scope` | scope → region, service |
+| Azure | `addressPrefixes`, `region`, `systemService` | region, systemService → service |
 
-Without region and service a cloud match is not actionable downstream — the consumer needs
-to know where to allocate and what the address belongs to. Of AWS's ~10,500 prefixes, over
-half are the generic `AMAZON` tag, so "it is an AWS IP" is close to no information; `EC2`
-in a named region is the useful signal.
-
-The work extends the fetchers to return `(ipv4, ipv6, metadata)` where metadata maps
-CIDR → `(region, service)`, carries it on `CSPIPAddresses`, and writes one handoff record
-per matched address:
+Fetchers return `(ipv4, ipv6, metadata)` where metadata maps CIDR → `(region, service)`.
+`CSPIPAddresses` carries it and exposes `describe(cidr)`. Each matched address is written as
+one handoff record:
 
 ```
 domain|ip|provider|region|service|prefix
 ```
 
-This changes the `csp_matches_*.txt` contract from prose to structured fields, and the
-associated tests must be updated to assert the new contract rather than relaxed.
+Why it mattered: over half of AWS's ~10,500 prefixes carry the generic `AMAZON` tag, so
+"it is an AWS IP" was close to no information. A live run against AWS-fronted hosts now
+reports `CLOUDFRONT` and `GLOBALACCELERATOR` rather than a flat provider tally — telling an
+operator at a glance that these are CDN edges, not EC2 addresses in a region, and so not
+worth pursuing.
+
+Fields are taken verbatim from the provider; where none is published they read `unknown`
+rather than being inferred. The tool does not judge what is claimable — §1 applies.

README.md

@@ -89,7 +89,7 @@ Each run creates a timestamped subdirectory under the output directory containin
 | `resolution_results_*.txt` | Successfully resolved domains and their IPv4/IPv6 addresses, pipe-delimited (`domain\|ip1\|ip2`). Lines prefixed `WILDCARD\|` resolved only via a zone wildcard — see [Wildcard DNS detection](#wildcard-dns-detection) |
 | `unresolved_results_*.txt` | Domains that could not be resolved after all retries |
 | `takeover_candidates_*.txt` | Takeover candidates — `DANGLING\|` lines (dangling CNAMEs with category, recommendation, evidence) and `NS_TAKEOVER\|` lines (unresolvable nameservers) |
-| `csp_matches_*.txt` | Domains resolving to cloud provider IP ranges (AWS, GCP, Azure — one line per match) |
+| `csp_matches_*.txt` | One handoff record per matched address: `domain\|ip\|provider\|region\|service\|prefix`. Region and service come from the provider's own published ranges — see [Cloud IP attribution](#cloud-ip-attribution) |
 | `environment_results_*.json` | Run metadata (command, external IP, Docker status) |
 | `evidence/dns/` | dig or nslookup output per flagged domain (when `--evidence` is set) |
 
@@ -117,6 +117,42 @@ lambda_handler.py        — Lambda entry point → run(env_manager)
 
 Domain processing uses `asyncio.gather` with a `Semaphore` cap (`--max-threads`) to run many domains concurrently without exhausting file descriptors or triggering DNS rate limits. Failed domains are collected after each pass and retried up to `--retries` times.
 
+## Cloud IP attribution
+
+Matching a resolved address to a cloud provider is only half an answer. `AWS` alone says
+little: of roughly 10,500 published AWS prefixes, over half carry the generic `AMAZON` tag,
+and the ones that matter operationally — `EC2` in a named region — look identical unless the
+provider's own metadata is kept.
+
+Each match is therefore written as a record carrying the provider's published region and
+service:
+
+```
+domain|ip|provider|region|service|prefix
+```
+
+```
+example.com|13.35.163.22|aws|GLOBAL|CLOUDFRONT|13.35.0.0/16
+example.com|3.11.53.7|aws|eu-west-2|EC2|3.8.0.0/14
+```
+
+The difference is the point: the first is a CDN edge address, the second an EC2 address in a
+specific region. Both are "AWS"; only one is a meaningful target.
+
+The end-of-run summary groups matches the same way:
+
+```
+  CSP matches — AWS: 6  GCP: 0  Azure: 0
+    by region and service:
+         4  aws  GLOBAL  CLOUDFRONT
+         2  aws  GLOBAL  GLOBALACCELERATOR
+```
+
+Metadata is taken verbatim from each provider (AWS `region`/`service`, GCP `scope`/`service`,
+Azure `region`/`systemService`). Where a provider publishes none, the fields read `unknown`
+rather than being inferred. DNSResolver does not judge which addresses are worth pursuing —
+it reports what the provider states and leaves that decision to the operator.
+
 ## Wildcard DNS detection
 
 A zone serving a wildcard record (`*.example.com`) answers for **every** name beneath it. Against an

classes/csp_ip_addresses.py

@@ -1,11 +1,28 @@
 class CSPIPAddresses:
-    def __init__(self, gcp_ipv4, gcp_ipv6, aws_ipv4, aws_ipv6, azure_ipv4, azure_ipv6):
+    def __init__(
+        self,
+        gcp_ipv4,
+        gcp_ipv6,
+        aws_ipv4,
+        aws_ipv6,
+        azure_ipv4,
+        azure_ipv6,
+        metadata=None,
+    ):
         self.gcp_ipv4 = gcp_ipv4
         self.gcp_ipv6 = gcp_ipv6
         self.aws_ipv4 = aws_ipv4
         self.aws_ipv6 = aws_ipv6
         self.azure_ipv4 = azure_ipv4
         self.azure_ipv6 = azure_ipv6
+        # CIDR -> (region, service). A match is only actionable downstream if the
+        # consumer knows where the address is allocated from and what it serves,
+        # so the publishers' own metadata is carried through rather than dropped.
+        self.metadata = metadata or {}
+
+    def describe(self, cidr):
+        """Region and service for a matched prefix, or unknowns if unpublished."""
+        return self.metadata.get(cidr, ("unknown", "unknown"))
 
     def get_gcp_ipv4(self):
         return self.gcp_ipv4

classes/run_summary.py

@@ -32,10 +32,21 @@ def display(self, total_input, failed_count):
             if ln.startswith("NS_TAKEOVER|")
         ]
 
+        # Handoff records: domain|ip|provider|region|service|prefix
         csp_lines = self._lines("csp")
-        aws_count = sum(1 for ln in csp_lines if "resolved to aws IPs" in ln)
-        gcp_count = sum(1 for ln in csp_lines if "resolved to gcp IPs" in ln)
-        azure_count = sum(1 for ln in csp_lines if "resolved to azure IPs" in ln)
+        csp_records = [ln.split("|") for ln in csp_lines if ln.count("|") >= 5]
+        aws_count = sum(1 for r in csp_records if r[2] == "aws")
+        gcp_count = sum(1 for r in csp_records if r[2] == "gcp")
+        azure_count = sum(1 for r in csp_records if r[2] == "azure")
+
+        # Region and service are what an operator acts on, so surface the
+        # breakdown rather than a bare provider tally.
+        csp_breakdown = {}
+        for record in csp_records:
+            provider, region, service = record[2], record[3], record[4]
+            csp_breakdown[(provider, region, service)] = (
+                csp_breakdown.get((provider, region, service), 0) + 1
+            )
 
         classified = {}
         unclassified = []
@@ -74,6 +85,12 @@ def display(self, total_input, failed_count):
         print(
             f"  CSP matches — AWS: {aws_count}  GCP: {gcp_count}  Azure: {azure_count}"
         )
+        if csp_breakdown:
+            print("    by region and service:")
+            for (provider, region, service), count in sorted(
+                csp_breakdown.items(), key=lambda kv: (-kv[1], kv[0])
+            ):
+                print(f"      {count:>4}  {provider}  {region}  {service}")
         print(thin)
 
         if total_candidates == 0:

imports/cloud_ip_ranges.py

@@ -4,7 +4,7 @@
 import json
 import os
 import re
-from typing import List, Tuple
+from typing import Dict, List, Tuple
 from urllib.request import urlopen
 
 import requests
@@ -13,82 +13,96 @@
 IPV6_KEYWORDS = ["ipv6Prefix", "ipv6_prefix", "addressPrefixes"]
 
 
-def fetch_ip_ranges_for_azure(url: str, extreme: bool) -> Tuple[List, List]:
+def fetch_ip_ranges_for_azure(url: str, extreme: bool) -> Tuple[List, List, Dict]:
     try:
         response = requests.get(url, timeout=10)
         if response.status_code != 200:
             print(
                 f"Failed to fetch IP ranges for Azure. Status code: {response.status_code}"
             )
-            return [], []
+            return [], [], {}
 
         data = json.loads(response.text)
 
-        ipv4_ranges = [
-            item
-            for value in data.get("values", [])
-            for item in value.get("properties", {}).get("addressPrefixes", [])
-            if ":" not in item  # Exclude IPv6 addresses
-        ]
-        ipv6_ranges = [
-            item
-            for value in data.get("values", [])
-            for item in value.get("properties", {}).get("addressPrefixes", [])
-            if ":" in item  # Only include IPv6 addresses
-        ]
+        ipv4_ranges = []
+        ipv6_ranges = []
+        metadata = {}
+
+        for value in data.get("values", []):
+            props = value.get("properties", {})
+            # Global tags carry an empty region; say so rather than inventing one.
+            region = props.get("region") or "global"
+            service = props.get("systemService") or value.get("name") or "unknown"
+            for item in props.get("addressPrefixes", []):
+                if ":" in item:
+                    ipv6_ranges.append(item)
+                else:
+                    ipv4_ranges.append(item)
+                metadata[item] = (region, service)
+
         if extreme:
             print("IPv4 Ranges:", ipv4_ranges)
             print("IPv6 Ranges:", ipv6_ranges)
 
-        return ipv4_ranges, ipv6_ranges
+        return ipv4_ranges, ipv6_ranges, metadata
 
     except requests.exceptions.RequestException as e:
         print(f"An error occurred while fetching the IP ranges: {e}")
-        return [], []
+        return [], [], {}
 
 
-def fetch_ip_ranges(url: str, extreme: bool = False) -> Tuple[List, List]:
+def fetch_ip_ranges(url: str, extreme: bool = False) -> Tuple[List, List, Dict]:
     try:
         response = requests.get(url, timeout=10)
         if response.status_code != 200:
             print(f"Failed to fetch IP ranges. Status code: {response.status_code}")
-            return [], []
+            return [], [], {}
 
         data = json.loads(response.text)
 
         if "prefixes" not in data:
             print(f"No 'prefixes' key in retrieved data: {data}")
-            return [], []
-
-        ipv4_ranges = [
-            prefix[keyword]
-            for prefix in data["prefixes"]
-            for keyword in IPV4_KEYWORDS
-            if keyword in prefix
-        ]
-        ipv6_ranges = [
-            prefix[keyword]
-            for prefix in data["prefixes"]
-            for keyword in IPV6_KEYWORDS
-            if keyword in prefix
-        ]
+            return [], [], {}
+
+        ipv4_ranges = []
+        ipv6_ranges = []
+        metadata = {}
+
+        for prefix in data["prefixes"]:
+            # AWS calls it 'region', GCP calls it 'scope'. Both name the place an
+            # address is allocated from, which is what an operator needs in order
+            # to act on a match.
+            region = prefix.get("region") or prefix.get("scope") or "unknown"
+            service = prefix.get("service") or "unknown"
+
+            for keyword in IPV4_KEYWORDS:
+                if keyword in prefix:
+                    cidr = prefix[keyword]
+                    ipv4_ranges.append(cidr)
+                    metadata[cidr] = (region, service)
+            for keyword in IPV6_KEYWORDS:
+                if keyword in prefix:
+                    cidr = prefix[keyword]
+                    ipv6_ranges.append(cidr)
+                    metadata[cidr] = (region, service)
+
         if extreme:
             print("IPv4 Ranges:", ipv4_ranges)
             print("IPv6 Ranges:", ipv6_ranges)
 
-        return ipv4_ranges, ipv6_ranges
+        return ipv4_ranges, ipv6_ranges, metadata
 
     except requests.exceptions.RequestException as e:
         print(f"An error occurred while fetching the IP ranges: {e}")
     except IOError as e:
         print(f"An error occurred while writing to the file: {e}")
 
-    return [], []
+    return [], [], {}
 
 
 def _fetch_and_save(
     url: str, filename: str, output_dir: str, extreme: bool
-) -> Tuple[List, List]:
+) -> Tuple[List, List, Dict]:
     ranges = fetch_ip_ranges(url, extreme)
     with open(os.path.join(output_dir, filename), "w", encoding="utf-8") as f:
         json.dump(ranges, f, indent=4)
@@ -97,7 +111,7 @@ def _fetch_and_save(
 
 def fetch_google_cloud_ip_ranges(
     output_dir: str, extreme: bool = False
-) -> Tuple[List, List]:
+) -> Tuple[List, List, Dict]:
     return _fetch_and_save(
         "https://www.gstatic.com/ipranges/cloud.json",
         "gcp_ip_ranges.json",
@@ -106,7 +120,9 @@ def fetch_google_cloud_ip_ranges(
     )
 
 
-def fetch_aws_ip_ranges(output_dir: str, extreme: bool = False) -> Tuple[List, List]:
+def fetch_aws_ip_ranges(
+    output_dir: str, extreme: bool = False
+) -> Tuple[List, List, Dict]:
     return _fetch_and_save(
         "https://ip-ranges.amazonaws.com/ip-ranges.json",
         "aws_ip_ranges.json",
@@ -123,21 +139,26 @@ def fetch_aws_ip_ranges(output_dir: str, extreme: bool = False) -> Tuple[List, L
 AZURE_CACHE_PATH = ".azure_ip_cache.json"
 
 
-def _save_azure_cache(ranges: Tuple[List, List]) -> None:
+def _save_azure_cache(ranges: Tuple[List, List, Dict]) -> None:
     try:
         with open(AZURE_CACHE_PATH, "w", encoding="utf-8") as f:
             json.dump(ranges, f, indent=4)
     except IOError as e:
         print(f"Warning: could not write Azure IP cache: {e}")
 
 
-def _load_azure_cache() -> Tuple[List, List]:
+def _load_azure_cache() -> Tuple[List, List, Dict]:
     with open(AZURE_CACHE_PATH, "r", encoding="utf-8") as f:
         data = json.load(f)
-    return data[0], data[1]
+    # Caches written before region/service capture hold only the two range lists.
+    # Read them rather than discarding a usable cache; metadata is simply absent.
+    metadata = data[2] if len(data) > 2 else {}
+    return data[0], data[1], metadata
 
 
-def fetch_azure_ip_ranges(output_dir: str, extreme: bool = False) -> Tuple[List, List]:
+def fetch_azure_ip_ranges(
+    output_dir: str, extreme: bool = False
+) -> Tuple[List, List, Dict]:
     confirmation_url = (
         "https://www.microsoft.com/en-us/download/confirmation.aspx?id=56519"
     )
@@ -160,7 +181,7 @@ def fetch_azure_ip_ranges(output_dir: str, extreme: bool = False) -> Tuple[List,
     # Step 2: if scrape succeeded, fetch and return
     if json_url:
         ranges = fetch_ip_ranges_for_azure(json_url, extreme)
-        if ranges != ([], []):
+        if ranges[0] or ranges[1]:
             _save_azure_cache(ranges)
             with open(
                 os.path.join(output_dir, "azure_ip_ranges.json"), "w", encoding="utf-8"
@@ -185,7 +206,7 @@ def fetch_azure_ip_ranges(output_dir: str, extreme: bool = False) -> Tuple[List,
         "This may be stale — update AZURE_PINNED_URL in cloud_ip_ranges.py if needed."
     )
     ranges = fetch_ip_ranges_for_azure(AZURE_PINNED_URL, extreme)
-    if ranges != ([], []):
+    if ranges[0] or ranges[1]:
         _save_azure_cache(ranges)
         with open(
             os.path.join(output_dir, "azure_ip_ranges.json"), "w", encoding="utf-8"
@@ -194,4 +215,4 @@ def fetch_azure_ip_ranges(output_dir: str, extreme: bool = False) -> Tuple[List,
         return ranges
 
     print("Azure IP fetch: all sources exhausted — no Azure ranges loaded.")
-    return [], []
+    return [], [], {}