RFC: Evolve Veneer Spec (.vnr) into a Declarative Web Scraping Language

[watashi-00]

Description & Vision

Currently, the Veneer Spec (.vnr) is used primarily for browser runtime UI reconstruction inside the SPM Chrome Extension. However, the core compiler, AST parser, and resolver built into spm-cli have the architectural foundation to be evolved into a pure, declarative Web Scraping and Data Extraction Language.

The core value proposition is shifting from imperative spaghetti scripts (large Node/Python scripts filled with custom DOM queries, try-catches, and cleanup code) to declarative data contracts written in .vnr. The user writes a declarative layout blueprint, and a runtime engine parses the contract, navigates the target, applies extraction pipelines, and outputs structured JSON.

---

Core Proposals & Architectural Extensions

1. Runtime Modes (Static vs. Dynamic)

To scrape efficiently, the spec must distinguish between server-rendered HTML and client-rendered SPAs:

• Static Mode (static): Simple, fast HTTP GET requests parsing raw HTML. Suitable for SSR pages, with zero memory overhead.
• Dynamic Mode (dynamic): Instructs the runner to boot a headless browser (e.g. Puppeteer/Playwright) to await dynamic elements before extracting:

  extractor "ProductScraper" {
      mode: "dynamic";
      wait_for: ".product-grid";
  }
  

2. Advanced Value Extraction & Cast Pipelines

Extending the parser to support type-casting, string processing, and regex extraction inside the propsMap (or custom extraction directives):

• Casting: span.price | text | toFloat
• Boolean Mapping: .is-available | exists | toBoolean
• Regex Extraction: div.meta | text | regex("Author: (.*)") | trim

3. Nested/Array Extraction

Web scraping requires parsing repetitive items (like lists, tables, cards). We propose introducing a nesting block syntax:

array reviews: ".review-card" {
    author: ".review-author | text";
    rating: ".review-stars | attr:data-rating | toInt";
    comment: ".review-body | text | optional";
}

4. Pagination & Crawling Control

Declarative hooks to extract the next page URL and control depth:

pagination {
    next_page: "a.next-btn | attr:href";
    max_depth: 5;
}

5. Resilience & Fallbacks

If a website layout changes, the scraper should degrade gracefully instead of crashing:

• Default fallbacks: img.avatar | attr:src || "/placeholder.png"
• Optional attributes: .promo-badge | text | optional

---

Implementation / Runtime Discussion

To execute these contracts, we need a vnr-runner execution engine. We present two architectural approaches:

1. Pure C++ Runner:
   • Pros: Minimal binary size, ultra-fast parsing, extremely low memory usage (perfect for high-throughput containerized scraping services).
   • Cons: Integrating a headless JS/Chromium engine natively in C++ is highly complex and increases compilation times.
2. Hybrid Approach (Recommended):
   • Use spm-cli (C++) as the compiler to compile .vnr specs into structured JSON manifest contracts.
   • Build a decoupled runner in Go (compiled binary, excellent concurrency model, native engines like go-rod) or Node.js (native Playwright/Puppeteer support) which consumes the JSON contract and handles target navigation.

---

Proposed Scraping .vnr Blueprint

extractor "ProductCatalog" {
    targetUrl: "https://example-store.com/products/*";
    mode: "dynamic";
    wait_for: ".catalog-loaded";

    product_name: "h1.product-title | text | trim";
    price: ".price-amount | text | regex('([0-9.]+)') | toFloat";
    in_stock: ".in-stock-badge | exists | toBoolean";

    array images: ".gallery-thumbnail" {
        url: "img | attr:src";
    }

    pagination {
        next: "a.pagination-next | attr:href";
        max_depth: 10;
    }
}