Graphclone

Scoping a client site by hand is slow, and the estimates it produces are hard to defend — they rest on a mental model of the site that nobody else can see. Graphclone makes the map mechanical: it crawls a site and canonicalizes its routes, then hands the structured result to an LLM agent that reconstructs each page as a typed wireframe — so engineering scope is anchored in artifacts instead of assumptions.

Sample output

Graphclone crawled a live job-board site and canonicalized its routes; an LLM agent then reconstructed each page as a typed wireframe. Two of the generated pages (branding neutralized to a placeholder, Acme Jobs):

Route reconstruction	Detail template

Features

URL canonicalization

• Applies structural heuristics to normalize dynamic URLs, so parameterized routes (/product/1, /product/2, …) collapse to a single mapped template instead of exploding the crawl.
• Configurable thresholds for predictable, repeatable output.

Authenticated session handoff

• With WAIT_FOR_LOGIN enabled, a visible browser opens and you log in yourself, exactly as you normally would.
• Your password is never entered into Graphclone — the crawl continues on your authenticated session, so gated content maps like anything else.

Structural crawling

• Traverses nested directory structures and dynamic routing patterns while preventing infinite loops and redundant traversal.

Snapshot capture

• Raw HTML snapshot, page screenshot, and canonicalized route metadata for every mapped page.

LLM-driven wireframe reconstruction

• The structured crawl output (HTML, screenshots, canonical routes) is fed to an LLM agent (.github/agents/graphclone-builder.md) that generates a Next.js + TypeScript app in /wireframe — one typed React page per canonical route, reconstructing layout and component hierarchy.
• The generated app renders interactive wireframes and exports them to PDF for review, documentation, and scoping.

Architecture

A three-stage pipeline: a deterministic crawl produces structured artifacts, an LLM turns those artifacts into a typed app, and the app renders and exports.

• Scraper (main.py) — asynchronous Python driving Playwright browser automation. Writes HTML snapshots, screenshot.png, and canonical route metadata to /scrape-results.
• Builder agent (.github/agents/graphclone-builder.md) — an LLM agent that reads the crawl artifacts and generates the /wireframe app: one typed React page per canonical route.
• Wireframe app (/wireframe) — the generated Next.js + TypeScript project that renders interactive wireframes and supports PDF export.

Setup & installation

1. Install Python dependencies

pip install -r requirements.txt
playwright install chrome

2. Set up the wireframe builder

cd wireframe
npm install

Usage

1. Run the mapper

python main.py <URL>

Results are written to /scrape-results.

2. Authenticated mode (optional)

In config.py, set:

WAIT_FOR_LOGIN = True

When enabled, a visible browser window opens. Log into the site as you normally would, then return to the terminal and press Enter — crawling continues on the authenticated session. Authenticated runs are written to a _logged_in directory to keep them separate from public crawls.

3. Build the wireframes

Feed the scrape results into an LLM of your choice (.github/agents/graphclone-builder.md is recommended) to generate output in /wireframe.

4. Generate the product requirements document

python generate_prd.py

Tests

Three layers, gated by pytest markers so the default run stays fast — slower layers drive a real browser and a real Next.js dev server.

Command	Runs	Time
python -m pytest tests -q	unit tests only	~1s
python -m pytest tests -m integration	crawl loop against a local fixture site	~30s
python -m pytest tests -m e2e	wireframe app → PDF against a real dev server	~10s
python -m pytest tests -m ""	everything	~35s

Unit (tests/test_canonicalize.py, tests/test_generate_prd.py) — pure functions: URL canonicalization heuristics, Next.js route discovery, PDF HTML assembly. No browser, no network.

Integration (tests/test_crawl_integration.py) — runs main.crawl_and_screenshot against an in-process fixture site (tests/fixture_site.py) with real Chrome. Proves the crawl loop itself builds correct cardinality/nav-signal data from live pages, rather than just asserting the heuristics given hand-built inputs.

E2E (tests/test_prd_e2e.py) — runs generate_prd.py's full pipeline against a minimal fixture Next.js app (tests/fixtures/mini_wireframe/): boots a real dev server, screenshots every route, and asserts a real PDF with valid embedded PNGs comes out. Requires npm install to have been run in wireframe/ (it borrows that node_modules); skips itself if missing.

The true end-to-end — a target URL all the way to a PDF — isn't automated, since the crawl and the wireframe build are joined by a human-in-the-loop LLM step (.github/agents/graphclone-builder.md).