Context
Part of the v2 RAG milestone. See BACKLOG.md for the full dependency chain.
What
Add scripts/scrape-live-docs.ts, runnable as npm run scrape-docs.
Crawls and chunks the following into data/live-docs.json:
| Source |
Content |
ableton.com/en/manual/ |
Full Ableton Live manual — instrument/audio effect/MIDI effect reference pages |
makingmusic.ableton.com |
Intermediate music-making concepts (server-rendered, fetchable) |
ableton.com/en/blog/ |
Selected tutorials (tag-filtered: tutorials, sound-design) |
Each chunk:
```ts
interface DocChunk {
id: string; // stable hash of url + offset
url: string;
title: string; // page title or section heading
body: string; // ~400 token text excerpt
tags: string[]; // e.g. ['device', 'Operator', 'synthesis']
source: 'manual' | 'makingmusic' | 'blog';
scrapedAt: string; // ISO date
}
```
Output: data/live-docs.json (target ≤ 5MB) and data/live-docs-diff.json (added/removed/updated URLs + unchanged count).
The scraper must be idempotent — re-running updates changed pages, keeps unchanged ones.
Notes
learningmusic.ableton.com is a JS SPA; static fetch returns a shell. Skip it.
- Ableton's manual uses anchor sections (e.g.
#operator) on three reference pages, not per-device URLs. The abledtonManualUrl() function in web-search.ts already maps these correctly — the scraper should use the same anchor extraction logic (extractDeviceSectionFromHtml).
- Scraper exits non-zero on hard failures so CI can catch structural changes.
Acceptance criteria
Context
Part of the v2 RAG milestone. See
BACKLOG.mdfor the full dependency chain.What
Add
scripts/scrape-live-docs.ts, runnable asnpm run scrape-docs.Crawls and chunks the following into
data/live-docs.json:ableton.com/en/manual/makingmusic.ableton.comableton.com/en/blog/Each chunk:
```ts
interface DocChunk {
id: string; // stable hash of url + offset
url: string;
title: string; // page title or section heading
body: string; // ~400 token text excerpt
tags: string[]; // e.g. ['device', 'Operator', 'synthesis']
source: 'manual' | 'makingmusic' | 'blog';
scrapedAt: string; // ISO date
}
```
Output:
data/live-docs.json(target ≤ 5MB) anddata/live-docs-diff.json(added/removed/updated URLs + unchanged count).The scraper must be idempotent — re-running updates changed pages, keeps unchanged ones.
Notes
learningmusic.ableton.comis a JS SPA; static fetch returns a shell. Skip it.#operator) on three reference pages, not per-device URLs. TheabledtonManualUrl()function inweb-search.tsalready maps these correctly — the scraper should use the same anchor extraction logic (extractDeviceSectionFromHtml).Acceptance criteria
npm run scrape-docsproducesdata/live-docs.jsonwith ≥ 50 chunksdata/live-docs-diff.jsonis written with added/removed/updated/unchanged counts