Skip to content

RAG-1: Live docs scraper script #66

Description

@nonogv

Context

Part of the v2 RAG milestone. See BACKLOG.md for the full dependency chain.

What

Add scripts/scrape-live-docs.ts, runnable as npm run scrape-docs.

Crawls and chunks the following into data/live-docs.json:

Source Content
ableton.com/en/manual/ Full Ableton Live manual — instrument/audio effect/MIDI effect reference pages
makingmusic.ableton.com Intermediate music-making concepts (server-rendered, fetchable)
ableton.com/en/blog/ Selected tutorials (tag-filtered: tutorials, sound-design)

Each chunk:
```ts
interface DocChunk {
id: string; // stable hash of url + offset
url: string;
title: string; // page title or section heading
body: string; // ~400 token text excerpt
tags: string[]; // e.g. ['device', 'Operator', 'synthesis']
source: 'manual' | 'makingmusic' | 'blog';
scrapedAt: string; // ISO date
}
```

Output: data/live-docs.json (target ≤ 5MB) and data/live-docs-diff.json (added/removed/updated URLs + unchanged count).

The scraper must be idempotent — re-running updates changed pages, keeps unchanged ones.

Notes

  • learningmusic.ableton.com is a JS SPA; static fetch returns a shell. Skip it.
  • Ableton's manual uses anchor sections (e.g. #operator) on three reference pages, not per-device URLs. The abledtonManualUrl() function in web-search.ts already maps these correctly — the scraper should use the same anchor extraction logic (extractDeviceSectionFromHtml).
  • Scraper exits non-zero on hard failures so CI can catch structural changes.

Acceptance criteria

  • npm run scrape-docs produces data/live-docs.json with ≥ 50 chunks
  • Re-running is idempotent (same content = same chunk ids, no churn)
  • data/live-docs-diff.json is written with added/removed/updated/unchanged counts
  • Hard failures (network outage, 404 on all three reference pages) exit non-zero

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestv2v2 power features

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions