From 2fcef136eb2fbbed1a5b99cf838afd21b00a9d6c Mon Sep 17 00:00:00 2001 From: lars20070 Date: Fri, 31 Jul 2026 08:27:55 +0200 Subject: [PATCH 01/15] Add Cursor config --- .cursor/mcp.json | 17 +++++++++++++++++ .cursor/settings.json | 11 +++++++++++ 2 files changed, 28 insertions(+) create mode 100644 .cursor/mcp.json create mode 100644 .cursor/settings.json diff --git a/.cursor/mcp.json b/.cursor/mcp.json new file mode 100644 index 0000000..142745e --- /dev/null +++ b/.cursor/mcp.json @@ -0,0 +1,17 @@ +{ + "mcpServers": { + "context7": { + "command": "npx", + "args": [ + "-y", + "@upstash/context7-mcp@latest" + ] + }, + "github": { + "url": "https://api.githubcopilot.com/mcp/", + "headers": { + "Authorization": "Bearer ${env:GITHUB_TOKEN}" + } + } + } +} \ No newline at end of file diff --git a/.cursor/settings.json b/.cursor/settings.json new file mode 100644 index 0000000..0fc8894 --- /dev/null +++ b/.cursor/settings.json @@ -0,0 +1,11 @@ +{ + "mcpServers": { + "context7": { + "disabled": false + }, + "github": { + "disabled": false + } + }, + "remote.containers.reopenFolderInContainer": false +} \ No newline at end of file From fb680e6dddfac6736955845b3996ff94189b88a9 Mon Sep 17 00:00:00 2001 From: lars20070 Date: Fri, 31 Jul 2026 10:31:39 +0200 Subject: [PATCH 02/15] Plan for scraping Google style guide --- .../plans/google_style_guide_scraper_plan.md | 313 ++++++++++++++++++ 1 file changed, 313 insertions(+) create mode 100644 .cursor/plans/google_style_guide_scraper_plan.md diff --git a/.cursor/plans/google_style_guide_scraper_plan.md b/.cursor/plans/google_style_guide_scraper_plan.md new file mode 100644 index 0000000..67e7214 --- /dev/null +++ b/.cursor/plans/google_style_guide_scraper_plan.md @@ -0,0 +1,313 @@ +# Coding Agent Specification: Google Style Guide Single-File Markdown Generator + +## 1. Problem Statement + +### 1.1 Overview +The **Google Developer Documentation Style Guide** (hosted at `https://developers.google.com/style`) is an essential reference for technical writers, software engineers, and AI prompt engineers. However, Google publishes this guide across **hundreds of discrete web pages** without providing an official single-file download (such as a consolidated PDF or Markdown document). + +### 1.2 Target Audience & Use Case +A single-file Markdown (`.md`) document is required to: +1. Provide **offline accessibility** and fast local text searching. +2. Serve as a **clean context document** for Large Language Models (LLMs) and RAG (Retrieval-Augmented Generation) pipelines. +3. Enable easy version-controlled local archival and custom printing. + +### 1.3 Technical Challenges +- **Navigation Hierarchy:** Page order matters. The scraper must parse the left-hand navigation sidebar to retain the intended reading sequence. +- **Boilerplate & Noise Removal:** Google DevSite pages contain extensive non-content UI (navigation bars, search boxes, header/footer, rating widgets, sidebars, cookie banners). +- **Custom DOM Elements:** DevSite uses custom tags and CSS classes for callouts (`aside class="note"`, `aside class="caution"`), custom code tabs, and structured tables. +- **Internal Link Disruption:** Relative links pointing to `https://developers.google.com/style/...` will break unless converted into local section anchors (`#...`). +- **Rate Limiting & Politeness:** Fetching hundreds of pages sequentially requires request throttling, proper User-Agent headers, and retry logic. + +--- + +## 2. Solutions Overview + +### Option Comparison + +| Approach | Pros | Cons | Decision | +|---|---|---|---| +| **A. Static HTML Scraper (`httpx` + `BeautifulSoup` + `markdownify`)** | Fast, lightweight, pure Python, highly customizable AST conversion. | Requires manually mapping custom DOM elements. | **RECOMMENDED** | +| **B. Headless Browser (`Playwright` + Node/Python + Turndown)** | Renders JavaScript-heavy elements automatically. | Slower, heavier resource usage, unnecessary overhead for static text pages. | Secondary fallback | +| **C. Shell Pipeline (`wget` + `pandoc`)** | Simple one-liner execution. | Messy output, preserves UI clutter, mangles navigation order, poor callout handling. | Rejected | + +### Recommended Solution Architecture +A modular Python script operating in 6 sequential stages: +1. **Discover:** Extract page hierarchy from sidebar navigation. +2. **Fetch:** Retriable HTTP fetches with concurrent batching and caching. +3. **Clean:** Target main content containers and strip site chrome/UI. +4. **Transform:** Convert HTML AST to clean Markdown with custom rules for code and callouts. +5. **Normalize:** Update internal cross-references to point to document anchors. +6. **Compile:** Assemble master TOC, frontmatter, and output single `.md` file. + +--- + +## 3. Detailed Staged Implementation Plan + +``` +[ Stage 1: Setup ] ──> [ Stage 2: Crawl Nav Tree ] ──> [ Stage 3: Fetch & Clean HTML ] + │ +[ Stage 6: Validation ] <── [ Stage 5: Compile & Normalize ] <── [ Stage 4: HTML -> MD ] +``` + +### Stage 1: Environment & Dependency Setup +Set up a python environment with required libraries: +- `httpx` (Async HTTP client with HTTP/2 and retry capabilities) +- `beautifulsoup4` + `lxml` (Fast HTML parsing and tree navigation) +- `markdownify` (HTML to Markdown converter with custom subclassing) +- `pyyaml` (Frontmatter generation) + +#### Required Directory Layout +```text +scraper/ +├── cache/ # Downloaded raw HTML files (prevents refetching) +├── output/ # Output directory for final MD file +├── config.py # CSS selectors, user agents, rate limits +├── parser.py # Custom HTML-to-MD rules +└── main.py # CLI entrypoint and orchestrator +``` + +--- + +### Stage 2: Sitemap & Hierarchy Discovery +**Objective:** Parse the left navigation menu on `https://developers.google.com/style` to build an ordered list of URLs with section levels. + +#### Step-by-Step Instructions: +1. Fetch the main index page `https://developers.google.com/style`. +2. Locate the navigation tree element: + - Primary selector: `ul.devsite-nav-section-list` or `nav.devsite-section-nav`. +3. Extract all `` tags with `href` attributes starting with `/style`. +4. Store URLs sequentially in a list of structured dicts: + ```python + page_item = { + "title": "Voice and tone", + "url": "https://developers.google.com/style/voice", + "level": 2, # Depth in nav hierarchy + "slug": "voice" + } + ``` +5. Deduplicate links while strictly maintaining first-seen order. + +--- + +### Stage 3: Fetching & DOM Cleaning +**Objective:** Fetch raw HTML for each page, store in cache, and strip non-article elements. + +#### Step-by-Step Instructions: +1. **Caching:** Check if `cache/{slug}.html` exists. If not, fetch via HTTP request with `User-Agent: Mozilla/5.0 ...`. +2. **Rate Limiting:** Implement a 200–500ms delay between requests to avoid 429 throttling. +3. **Locate Core Content:** + - Target container: `article.devsite-article` or `div.devsite-article-body`. +4. **De-cluttering (Removal List):** + Remove the following selectors before processing: + - `nav`, `header`, `footer` + - `.devsite-rating-container` (Feedback widgets) + - `.devsite-toc` (In-page right-hand TOC) + - `.devsite-content-footer` + - `script`, `style`, `noscript` + - Buttons like "Copy code" or print triggers. + +--- + +### Stage 4: Custom HTML-to-Markdown Conversion +**Objective:** Transform cleaned HTML DOM into clean Markdown while handling Google-specific elements. + +#### Custom Tag Handling Rules: + +1. **Headings (`

` to `

`):** + - Shift header ranks down by 1 level if using `# Title` for the main page document. + - Inject HTML anchor IDs or explicit slug targets: ``. + +2. **Callout Boxes / Notices (`