SourceMender

SourceMender is a deterministic self-healing evidence layer for public-web collection. Bright Data discovers and collects records; SourceMender validates the returned bytes against a versioned contract, derives a bounded repair request from named failures, and stops at an explicit approval boundary before any proposed collector repair is applied.

Every payload, validation report, repair request, proposal, approval, and final document is fingerprinted. sourcemender signal verify re-hashes the stored evidence and replays the validator instead of trusting a receipt.

Videos

Start here: the SourceMender story

▶ Watch “SourceMender: The Scraper That Heals Itself” on YouTube

This presentation is the explanation-first introduction: why self-healing scrapers need a deterministic evidence gate, how SourceMender works with Bright Data, and what the verified repair looks like end to end.

Then see the product work

▶ Watch “SourceMender in 64 Seconds: Run → Approve → Verify” on YouTube

The short demo is the complete Run → Approve → Verify walkthrough. The 1080p YouTube version contains no MiniMax/H3 animation or other generated interstitial. The exact H.264 MP4 remains checked into this repository for the offline interactive evidence experience, where English captions are included.

The 4K presentation master is intentionally kept outside this source repository; the repository contains the deterministic collection, repair, and verification code used by both demos. See the current readiness audit and form draft.

A self-contained alternative lives in this repository: the interactive React evidence experience replays the verified live run (Run → Approve → Verify), and embeds the same narrated, captioned 64-second Playwright-only judge walkthrough served locally from apps/evidence-experience/public/media/, with its checked-in provenance alongside it and the source-review receipts in the candidate-04 review set. The long presentation stays on YouTube; the short functional demo needs no network at all.

The additive interactive evidence experience rebuilds that same verified run as a React interface: a playable five-stage repair story, filterable field diff, source-excerpt drawers, explicit product responsibility map, and expandable verification ledger. It imports the checked-in run evidence directly and leaves every static report and media candidate untouched.

Verified live evidence

The repository includes a preserved August 17, 2026 Bright Data run against the official Into the Scrape-Verse page. In one controlled contract-change test, required-field completeness for one record rose from 5/9 (55.6%) to 9/9 (100%) after an explicitly approved collector repair. The accepted result contained six source excerpts, passed all 14 replay checks, and passed a separate post-repair collection.

See the live-evidence guide, reproducible metrics, and claim ledger. The claim ledger marks the limits of this evidence, including what the project does not claim.

How Bright Data Scraper Studio is used

Bright Data Scraper Studio is the collection and repair engine at the center of the project:

1. sourcemender signal source create turns the versioned field contract into a custom Scraper Studio collector.
2. sourcemender signal collect or signal run invokes that Collector ID and receives structured JSON from the public page.
3. SourceMender validates the exact returned bytes and blocks incomplete data.
4. sourcemender signal heal converts the named validation failures into a bounded repair request for Scraper Studio.
5. The proposed repair is fingerprinted and held for explicit approval. Once approved, the same Collector ID is run again and the complete evidence chain is replayed by sourcemender signal verify.

The live proof used custom collector c_msxk8bqf29gzigb16j. Its preserved creation response, approved repair, and 14-check verification are checked in.

Example structured output

The approved live re-collection produced the complete accepted document. The same run also produced a self-contained HTML evidence report that walks the six-stage story — collect, gate, bounded repair, explicit approval, re-run, verify — and states its own claim limits. Earlier report variants are preserved unmodified beside it with their provenance. One exact excerpt is:

{
  "title": "Into the Scrape-Verse",
  "organizer": "WeMakeDevs with Bright Data",
  "prize_type": "mixed",
  "hardware_prize": "NVIDIA DGX Spark, Apple iPad, Keychron Keyboard",
  "source_evidence": [
    {
      "field": "title",
      "source_url": "https://www.wemakedevs.org/hackathons/scrape-verse",
      "source_excerpt": "Into the Scrape-Verse"
    }
  ]
}

Quick start

Build the Rust CLI:

cargo build --release -p sourcemender-cli

Run the complete offline fixture flow without network access, credentials, or Bright Data credits:

mkdir -p /tmp/sourcemender-demo/fixtures
cp crates/sourcemender-core/fixtures/*.json /tmp/sourcemender-demo/fixtures/

target/release/sourcemender signal init \
  --project-dir /tmp/sourcemender-demo \
  --url https://www.wemakedevs.org/hackathons/scrape-verse \
  --collector c_sourcemenderdemo01 \
  --fixtures fixtures

target/release/sourcemender signal run \
  --project-dir /tmp/sourcemender-demo \
  --auto-approve-heal

target/release/sourcemender signal verify \
  --project-dir /tmp/sourcemender-demo

target/release/sourcemender signal report \
  --project-dir /tmp/sourcemender-demo \
  --out /tmp/sourcemender-demo/report.html

Fixture runs are permanently labelled as fixture evidence. Live runs resolve the official Bright Data CLI, clear the ambient environment, forward only an allowlist, bound runtime and output, and redact credentials before persistence.

For a live custom collector, authenticate Bright Data and initialize a project without --fixtures or an existing Collector ID:

npx -p @brightdata/cli bdata login

target/release/sourcemender signal doctor --json

target/release/sourcemender signal init \
  --project-dir /tmp/sourcemender-live \
  --url https://www.wemakedevs.org/hackathons/scrape-verse

target/release/sourcemender signal source create \
  --project-dir /tmp/sourcemender-live \
  --name sourcemender-live

target/release/sourcemender signal run \
  --project-dir /tmp/sourcemender-live

If the run stops at the approval boundary, review the proposed Scraper Studio repair before applying it, then prove the repaired result from stored bytes:

target/release/sourcemender signal approve \
  --project-dir /tmp/sourcemender-live

target/release/sourcemender signal verify \
  --project-dir /tmp/sourcemender-live

target/release/sourcemender signal report \
  --project-dir /tmp/sourcemender-live \
  --out /tmp/sourcemender-live/report.html

Architecture

• sourcemender-foundation: hashes, UTC timestamps, and privacy-safe paths.
• sourcemender-core: pure contracts, validation, repair state, receipts, verification, and HTML evidence reports.
• sourcemender-cli: bounded Bright Data execution and the public workflow.

The deterministic core performs no network access, subprocess execution, or ambient clock reads. See docs/architecture.md.

Development checks

cargo fmt --all -- --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace

AI assistance

Built with assistance from Claude Code and OpenAI Codex. The deterministic decision boundary remains reviewable and testable without either assistant.