UndREST-APISpy — API Request Inspector

AP👁️Spy is a Chrome/Edge DevTools extension that observes live Azure/Microsoft API calls in real time, classifying each one against the SpeQL API inventory to surface exact matches, version mismatches, unknown routes, and — with optional enrichment data — provider-known operations with plain-English risk summaries, without leaving the browser.

The extension ships with pre-bundled provider shards (updated nightly from UndREST-SpecQL) and supports ARM batch inspection, multi-select status filters, sort/quick-filter controls, enrichment-powered insights, column-level filters, clipboard/CSV export, and more.

Table of Contents

• Quick Start
• Installation
• Usage
• API Packs
• Azure Portal Sweep
• Shard Data
• Repository Structure
• Ecosystem
• Contributing
• License

Quick Start

1. Clone this repo
2. Load extension/ as an unpacked extension in Chrome/Edge (chrome://extensions → Developer mode → Load unpacked)
3. Open DevTools on any Azure Portal or Microsoft API-calling page
4. Navigate to the APISpy panel

Installation

Browser Extension

git clone https://github.com/UndREST-Labs/UndREST-APISpy.git

1. Open chrome://extensions (or edge://extensions)
2. Enable Developer mode (toggle, top-right)
3. Click Load unpacked
4. Select the extension/ directory from this repo

The extension installs with all provider shards pre-bundled in extension/data/shards/. These are updated automatically every night from the latest SpeQL export — see Shard Data.

See extension/README.md for full usage details, panel walkthrough, and configuration options.

Azure Portal Sweep Script

The Azure Portal sweep automation requires Python 3.8+:

pip3 install -r requirements.txt
python -m playwright install chromium

Usage

DevTools Panel

Once the extension is loaded, open DevTools on any page that makes Azure/Microsoft API calls (e.g., the Azure Portal) and click the APISpy tab.

The panel shows each outgoing API request with:

• Provider — e.g., Microsoft.Compute
• Match Status — Exact match / Version mismatch / Unknown
• Method + Path — Normalised API path
• API Version — Detected from the request

Use the filter controls to narrow by provider, status, or path pattern. Export to clipboard or CSV for offline analysis.

Demo

The panel classifying live ARM requests — showing exact matches, version mismatches, and unknown routes across provider namespaces.

API Packs

An API pack is a named collection of provider shards derived from one API platform's specifications. The extension manifest (extension/data/manifest.json, schema 2.0.0) groups shards by pack, which lets the extension load multiple API platforms simultaneously and gives users the ability to choose which pack(s) are active at runtime.

Pack ID	Platform	Source	Description
azure-rest-api-specs	Azure	Azure/azure-rest-api-specs	Azure Resource Manager provider shards (updated nightly via UndREST-SpecQL)

Selecting active packs

Click the Packs button in the APISpy toolbar to open the pack settings dialog. From there you can enable or disable individual packs. Your selection is saved to browser storage and persists across DevTools reloads.

Adding a new pack

See docs/ADDING_A_PACK.md for a step-by-step guide to adding shards for a new API platform (e.g. AWS, Google Cloud, a custom internal API).

The high-level steps are:

1. Generate shard files in the schema 3.0.0 format (one file per provider/service namespace)
2. Run scripts/prepare_data.py --pack-id <id> --merge to bundle the shards
3. Register a custom request normaliser if the new API uses non-ARM URL patterns
4. Extend lib/filters.js with the new platform's hosts/suffixes
5. Add a GitHub Actions workflow to keep the new pack up-to-date automatically

Azure Portal Sweep

scripts/azure_portal_sweep.py is an automation script that walks every service on the Azure Portal All Services page with the APISpy extension running, then exports all captured ARM API calls as a CSV — providing broad, automated coverage of real-world Azure API traffic across all services.

python3 scripts/azure_portal_sweep.py

Options:

--output-dir DIR      Output directory for CSV results (default: ./results)
--dwell-ms MS         Milliseconds to dwell on each service page (default: 3000)

Prerequisites:

pip3 install -r requirements.txt
python -m playwright install chromium

# Authenticate (device code flow — Azure CLI first-party app)
# The script will prompt you on first run

See scripts/PORTAL_SWEEP.md for full usage, output format, and troubleshooting.

Azure Provider-Operation Enrichment

scripts/provider_ops_sweep.py enumerates all provider operations across your Azure subscriptions and produces a structured JSON dataset with risk scores, capability tags, and plain-English summaries. scripts/prepare_provider_ops.py converts that output into the compact indexed format the extension loads at runtime.

# Enumerate provider operations (device code auth)
python3 scripts/provider_ops_sweep.py

# Convert to extension format
python3 scripts/prepare_provider_ops.py
# → extension/data/azure-provider-ops.json

When this file is present the extension classifies unmatched Azure ARM requests as 🔷 Provider known and shows an Insight panel with summary title, why-it-matters, risk score, severity, and tags. If the file is absent the extension falls back to standard behaviour with no degradation.

Shard Data

The extension's API inventory lives in extension/data/shards/. Each file covers a single provider namespace (e.g., Microsoft.Storage.min.json) and is generated by UndREST-SpecQL from the Azure/azure-rest-api-specs corpus.

Shards are grouped into packs in extension/data/manifest.json (schema 2.0.0). The built-in Azure pack stores shards in the flat shards/ directory. Additional packs store their shards in shards/<pack-id>/ subdirectories so filenames cannot collide.

Automatic nightly update:

1. UndREST-SpecQL runs its export pipeline at 05:00 UTC daily
2. It publishes the sharded zip to the shards-latest GitHub Release
3. This triggers the update-shards.yml workflow in this repo
4. That workflow downloads the zip, runs scripts/prepare_data.py, and commits the updated shards to extension/data/

Manual refresh:

Go to Actions → Update Extension Shards → Run workflow to pull the latest shards immediately.

Requires secret: SPEQL_READ_TOKEN — a fine-grained PAT with contents: read on UndREST-Labs/UndREST-SpecQL.

Generated Artefact Ownership

These artefacts are generated but intentionally committed so the extension works offline in fresh clones:

• extension/data/shards/ — generated by scripts/prepare_data.py, updated by update-shards.yml
• extension/data/manifest.json — generated by scripts/prepare_data.py (pack manifest)
• extension/data/azure-provider-ops.json — generated by scripts/prepare_provider_ops.py (optional enrichment index)
• demos/apispy-*.png — generated by scripts/generate_screenshots.py, updated by update-screenshots.yml

Do not hand-edit these files. Regenerate them with their owning script/workflow.

Repository Structure

UndREST-APISpy/
├── README.md
├── CONTRIBUTING.md
├── LICENSE
├── requirements.txt             # azure-identity, playwright
├── docs/
│   └── ADDING_A_PACK.md         # Guide for adding new API pack sources
├── extension/                   # Chrome/Edge DevTools extension
│   ├── data/
│   │   ├── manifest.json        # Pack manifest (schema 2.0.0) — lists all packs and shards
│   │   ├── azure-provider-ops.json  # (optional) Provider-op enrichment index; generated by prepare_provider_ops.py
│   │   └── shards/              # Per-provider API shards (flat for azure pack; subdir for others)
│   ├── lib/                     # filters, normalizer, loader, matcher, azure-enrichment modules
│   ├── icons/
│   ├── manifest.json
│   ├── devtools.html / devtools.js
│   ├── panel.html / panel.css / panel.js
│   └── README.md
├── scripts/
│   ├── prepare_data.py          # Converts SpecQL export zip/dir into extension/data/shards/
│   ├── azure_portal_sweep.py    # Playwright sweep of Azure Portal → CSV of ARM calls
│   ├── provider_ops_sweep.py    # Enumerates Azure provider operations → timestamped JSON
│   ├── prepare_provider_ops.py  # Converts provider_ops_sweep output → extension/data/azure-provider-ops.json
│   ├── generate_screenshots.py  # Demo screenshot generation
│   └── PORTAL_SWEEP.md
├── tests/
│   ├── test_filters.js
│   ├── test_loader.js
│   ├── test_matcher.js
│   ├── test_normalizer.js
│   └── vhs/
│       ├── 08-apispy-portal-sweep.tape
│       └── helpers/
│           └── mock_portal_sweep.sh
├── demos/
│   └── apispy-*.png
└── .github/
    └── workflows/
        ├── update-shards.yml        # Triggered by UndREST-SpecQL on new shard publish
        └── update-screenshots.yml   # Auto-generates demos/ screenshots on push to main

Ecosystem

APISpy is part of the UndREST Labs ecosystem:

Project	Repo	Description
SpeQL	UndREST-SpecQL	Query and reason about API behaviour; the engine that feeds APISpy
APISpy	(this repo)	Real-time visibility into API calls in the browser
Atlas	(future)	Mapping API ecosystems at scale

Observe → Understand → Map → Evolve

Contributing

See CONTRIBUTING.md for guidelines on contributing to the extension, portal sweep script, and shard preparation pipeline.

For SpeQL (query engine, inventory export, CodeQL queries), see UndREST-SpecQL.

License

See LICENSE.