A self-healing web-data mesh, built on Bright Data Scraper Studio.
Hadalpoint Software Solutions LLC
Scrapers work in testing. Then a site renames a class at 2am, the extraction returns an empty column instead of an error, and nobody finds out until the report is wrong on Thursday.
ScalyFoot fixes the whole failure, not just the first half. You describe each field once, in plain English. A custom Bright Data Scraper Studio collector fetches the page and the DOM it parsed. When a field stops coming back, ScalyFoot finds it again from your description — and if it cannot find a replacement it trusts, it refuses to guess and tells you so.
The scaly-foot snail lives on hydrothermal vents three kilometres down and builds itself a shell plated with iron sulfide. It is the only animal known to armour itself in metal. It seemed like the right name for this.
$ scalyfoot run examples/contracts/laptop_catalog.yml --variant before
▸ HEALTHY 24 records price ← .price
# overnight, the site is redesigned and `.price` no longer exists
$ scalyfoot run examples/contracts/laptop_catalog.yml --variant after
● SURFACE 0 m fetched via replay
● PHOTIC 0–200 m 24 records, 6 fields
● TWILIGHT 200–1 000 m 1 drift signal(s)
● MIDNIGHT 1 000–4 000 m 1 critical
● ABYSSAL 4 000–6 000 m 1 repaired, 0 refused
● HADAL 6 000–11 000 m sounding #4 committed
price healthy ████████████ 100% [data-test="price"]
◆ PROMOTED price
was .price
now [data-test="price"]
Repaired: '[data-test="price"]' now fills 100% of records (was 0%),
values match 100% of the time, and its durability is 0.97.
16 candidate(s) considered.
▸ HEALTHY 24 records 23 msNothing downstream saw a gap. The next run uses the healed selector and does not re-diagnose anything.
Any system can find an answer. The interesting question is whether it knows when to stop.
Run the same contract against a redesign where the price survives only inside
machine-generated class names (.css-1x2y3z), with no semantic hook left:
$ scalyfoot run examples/contracts/laptop_catalog.yml --variant after_hostile
price quarantined ░░░░░░░░░░░░ 0% .price
rating quarantined ░░░░░░░░░░░░ 0% .rating
◆ REJECTED price
was .price
now <unchanged>
No repair was good enough to trust, so nothing was changed and the field
is quarantined. Keeping the previous setting means this column is
reported as broken rather than quietly filled with the wrong value.
▸ DRIFTING 24 recordsprice comes back null, not wrong. The run exits non-zero. The refusal is
recorded in the ledger with the same weight as a success, including every
candidate that was considered and the reason each was rejected.
That is the "break nothing" half, and it is the reason the "scrape anything" half is safe to use.
Scraper Studio is not a fetching library bolted on the side — it is the half of the system that touches the internet, and it does two jobs.
The custom collector in scraper_studio/ is written against
no site in particular. It takes a contract at runtime — a list of fields, each
with a plain-language description — so one collector serves every feed.
It returns the parsed rows and page_html, the DOM it parsed them from.
That second part is the entire reason healing is possible: you cannot repair an
extraction from its empty output, only from the page that produced it. A
collector that returns just rows makes its own outages unfixable.
The same collector has a second mode. Given a field's description and a page
that no longer yields it, mode: "probe" searches the live DOM and returns
ranked candidate selectors with the evidence for each.
Running probe from the same collector is deliberate. A candidate derived from differently-rendered markup is a candidate that breaks the moment it goes live, so the proposal has to come from the page as Bright Data renders it — through the same proxying, browser rendering, and unblocking as the real run.
ScalyFoot never trusts a proposal on authority. Whatever Scraper Studio
returns is re-run against the real rows and scored, and is rejected if it does
not beat the incumbent. Scraper Studio proposes; the verifier disposes. Every
heal event records source, so the ledger can tell you whether a given repair
was Scraper Studio's idea or the local fallback's.
| File | Scraper Studio pane | What it does |
|---|---|---|
scraper_studio/collector.js |
Interaction code | Navigates, waits, scrolls lazy listings, dispatches by mode |
scraper_studio/parser.js |
Parser code | Extracts fields + returns page_html; or ranks candidates in probe mode |
scraper_studio/README.md |
— | Step-by-step: create the collector, wire the token, run live |
Setup instructions are in scraper_studio/README.md.
Without credentials the engine runs identically against a checked-in fixture
corpus (--backend replay), which is how the tests and the demo stay
deterministic.
Needs Python 3.11+. No credentials required.
git clone <your-fork> && cd ScalyFoot
python3 -m venv .venv && source .venv/bin/activate
pip install -e core
# two clean runs — drift is judged against a feed's own history, and a
# baseline needs two samples before there is any history to judge against
scalyfoot run examples/contracts/laptop_catalog.yml --variant before
scalyfoot run examples/contracts/laptop_catalog.yml --variant before
# the site is redesigned; watch it repair itself
scalyfoot run examples/contracts/laptop_catalog.yml --variant after
# a redesign it should refuse to repair
scalyfoot run examples/contracts/laptop_catalog.yml --variant after_hostileSeed only one baseline run and the redesign takes the cold-start path instead, where only required fields at zero coverage are escalated — optional ones are left alone on purpose, since an empty optional field on a first run may simply be legitimately absent. Useful behaviour, easy to mistake for a healing failure.
Then the dashboard:
scalyfoot serve # JSON API on :8787
cd dashboard && npm install && npm run dev # UI on :3000Everything above runs offline against the fixture corpus. To go live, create
the collector (scraper_studio/README.md), fill in
.env from .env.example, and run the one contract that
targets a real host:
$ scalyfoot run examples/contracts/books_live.yml --backend live
● SURFACE 0 m fetched via live
● PHOTIC 0–200 m 11 records, 4 fields
● HADAL 6 000–11 000 m sounding #1 committed
title healthy ████████████ 100% h3 a
price healthy ████████████ 100% .price_color
availability healthy ████████████ 100% .instock.availability
detail_url healthy ████████████ 100% h3 a
▸ HEALTHY 11 records 16942 msReal output in examples/output/books_live.json:
prices arrive as floats (£45.17 → 45.17) and the sounding records
backend: live alongside the Bright Data collector and snapshot ids, so a live
run is auditable the same way a replayed one is.
Everything below this section assumes you are happy authoring YAML. Most people are not, and the contract is the only thing standing between them and a working scraper — so ScalyFoot will write it for you from a URL and a sentence.
$ scalyfoot discover https://example-store.invalid/laptops \
--want "the price, the rating, and whether it's in stock"
page https://example-store.invalid/laptops
found 24 records matching div.card
what you could track
price currency ██████████░░ 81%
.price
· $2,795.34
rating number ██████████░░ 81%
.rating
· 3.7 / 5
stock text ██████████░░ 81%
.availability
· In stockAdd --save site.yml and you have an ordinary contract you can run, edit, and
commit. The YAML does not go away — it gets generated. There is no second
format and no hidden state; the builder's output is the same artefact the rest
of this README describes.
The same thing in the browser, for people who will never open a terminal:
scalyfoot serve
cd dashboard && npm run dev # then open /newPaste a URL, say what matters in plain English, tick the fields (each shown with three real values off your own page), and press Start watching.
not found on this page
delivery date
Nothing on this page clearly matched "delivery date". It may be on the
individual detail pages rather than this one, or loaded by JavaScript
after the page renders — or it may be here under a different name.
nearest: link, thumb_image, titleDiscovery will not hand back the tidiest column on the page and call it your delivery date. A confidently wrong field is worse than a missing one, because you have no reason to check it — this is the same refusal the healer makes, applied at authoring time. You find out in ten seconds instead of next Thursday.
Saving a site schedules it. Then:
$ scalyfoot watch --once
HEALTHY laptops 24 records 3 changed
· price ↓ 3215.34 → 2795.34
· availability Backorder → In stock
· price ↑ 1467.84 → 1727.84A change is not a drift, and ScalyFoot keeps them apart on purpose:
| means | recorded in | |
|---|---|---|
| drift | the extraction broke — a column stopped coming back | the trench |
| change | the extraction worked and the value moved | the studio |
Two consequences fall out of that split, and both are the point:
- Records are matched across runs by an identity key (a product link, a permit reference), chosen automatically. A catalogue that reorders itself reports nothing, where a positional diff would report every row as changed.
- A field that drifted or quarantined this run emits no change events. A broken column must never read as "the price changed to nothing" — that turns one outage into a thousand false alarms at the moment you most need to see the outage.
A repair is invisible here too. When .price heals to [data-test="price"]
and the number is the same, nothing changed, and the change feed says so.
This is the only file you author — or the file scalyfoot discover writes for
you, in exactly this format.
name: laptop_catalog
domain: retail
target:
url: https://example-store.invalid/laptops
row:
selector: .product-grid > .card
description: each individual product card in the results grid
fields:
- name: price
description: the current selling price shown on the card, in dollars
type: currency
required: true
selector: .priceThe description is not a comment. It is the only thing that survives a
redesign, and it is what the healer searches with. ScalyFoot refuses a contract
whose descriptions are too thin to heal from — better to fail at authoring time
than to discover a field is unfixable at the exact moment it breaks.
type describes a shape (a number, a date, a link), never a subject. To the
engine, a laptop price, an advertised salary, and a permit fee are the same
thing. That is what makes one binary work on all three.
The same engine, unchanged, across three domains that share no vocabulary:
| Contract | Domain | Redesign it survives | Records |
|---|---|---|---|
laptop_catalog.yml |
Retail | .price replaced by a data-test hook |
24 |
job_board.yml |
Hiring | Record wrapper renamed; fields moved to schema.org microdata | 18 |
permit_register.yml |
Public records | A <table> rebuilt as <div>s — the row selector itself is destroyed |
30 |
The third is the hardest: nothing about table.permits tbody tr survives, so
the repeating structure has to be rediscovered before any field can even be
judged. ScalyFoot repairs the row selector first, precisely so that one root
cause does not produce five separate alarms.
There is no branch anywhere in the engine on domain, on a field name, or on a
site. test_engine.py::TestDomainAgnosticism asserts this directly.
Candidates are proposed cheaply and judged expensively, on four independent axes:
| Axis | Question | Why it alone is not enough |
|---|---|---|
| Coverage | Does it find a value at all? | A selector matching everything scores perfectly |
| Validity | Is it the shape you described? | A title containing "14" is not a price |
| Alignment | Exactly one value per record? | Three matches means the first is a coin flip |
| Durability | Will it survive the next redesign? | div:nth-child(3) works today and breaks Friday |
Durability is the axis most self-healing systems omit, and leaving it out is the difference between healing and thrashing. A repair that works today but is anchored to a position on the page will break again next week, and the system will "heal" nightly forever while looking healthy on a dashboard.
[data-testid="price"] 0.97 an explicit contract for automation
[itemprop="price"] 0.90 machine-readable by design
.product-price 0.68 semantic — describes the content
.text-sm.px-4 0.30 presentational — moves when the design does
.css-1x2y3z 0.12 compiler output — changes every build
div:nth-child(3) > span 0.15 positional — breaks if anything moves
A candidate is promoted only if it clears all the gates and beats the
incumbent's coverage by the contract's min_improvement. Otherwise the field is
quarantined and a human is told.
The ledger — the Trench — is append-only, and that is enforced by SQLite triggers rather than by convention:
$ scalyfoot verify
✓ append-only enforced
The database refused a test UPDATE:
the trench is append-only: soundings cannot be updated
History in this ledger cannot be rewritten, including by ScalyFoot.A run is written once, complete, in a single transaction — there is no
open-then-close lifecycle a crash can leave half-finished. Rejected repairs are
stored with the same weight as successful ones; a healer that logs only its
successes cannot be audited. And the "current" selector for a field is not a
flag that gets toggled, it is simply the newest row in selector_versions, so
lineage falls out for free:
$ scalyfoot history laptop_catalog
PROMOTED price
.price → [data-test="price"]Every event renders at three depths, from the same stored data:
$ scalyfoot explain 4 --level plain
We collected 24 records. The page had changed, so 1 thing was repaired
automatically, recovering 24 values. Nothing downstream saw a gap.
$ scalyfoot explain 4 --level why
16 candidate(s) tested against the page that broke; 4 rejected.
'.meta' — only 12% of what it found is an amount of money (needs 50%)plain contains no jargon at all — and that is enforced by a test
(test_explain.py) that fails the build if words like selector, DOM, or
null leak into it. scalyfoot explain --glossary defines every term the
system uses; --pipeline explains the six stages.
The same three-level toggle is in the dashboard.
┌────────────────────────────────────────────┐
contract │ BRIGHT DATA SCRAPER STUDIO │
(plain │ custom collector — collector.js/parser.js │
language) │ │
│ │ collect → rows + page_html │
└─────▶│ probe → ranked candidate selectors │
└────────────────┬───────────────────────────┘
│
┌──────────────────────────▼───────────────────────────┐
│ SURFACE acquire the page, keep the DOM │
│ PHOTIC extract fields via active selectors │
│ TWILIGHT validate shape + coverage │
│ MIDNIGHT detect drift against this feed's history │
│ ABYSSAL propose → score → promote *or refuse* │
│ HADAL commit atomically to the Trench │
└──────────────────────────┬───────────────────────────┘
│
┌─────────────────────┼─────────────────────┐
▼ ▼ ▼
typed records CLI + JSON API Next.js dashboard
(json/csv/ndjson) :8787 :3000
The zone names are the ocean's own names for depth, and they are used identically in the CLI, the dashboard, and the ledger — learn them once and you can follow a run end to end.
| Module | Responsibility |
|---|---|
contracts.py |
The user's plain-language spec; refuses undescribable fields |
brightdata.py |
Scraper Studio client (live) + fixture replay, one interface |
extract.py |
Selectors → rows; reports how many matched, not just whether |
fieldtypes.py |
Shape parsing that knows nothing about subject matter |
drift.py |
Coverage vs. validity, judged against each feed's own history |
selectors.py |
Candidate proposal + the durability metric |
heal.py |
Scoring, promotion gates, and the refusal path |
engine.py |
Ordering: rows before fields, re-measure after every promotion |
ledger.py |
The append-only Trench |
explain.py |
Three depths of explanation, no jargon in the shallow one |
drivers.py |
One set of SQL over both SQLite and Postgres |
discover.py |
A URL → a proposed contract, with the evidence for each field |
wishlist.py |
Plain-English wants → fields, or an honest "not on this page" |
watch.py |
Value changes across runs — deliberately not drift |
studio.py |
Saved sites, quotas, change history. Writable; not the trench |
fetch.py |
The SSRF guard on fetching a URL somebody typed |
server.py used to be able to say that no route writes, because every route was
a read over the trench. The builder needs writes, so the claim narrowed to one
that is structural rather than incidental:
Nothing reachable over HTTP can write to the trench.
The route tables are separated by which database they can open. The GET table
takes a Trench; the studio table accepts writes and takes a Studio — a
different file, holding preferences rather than evidence. No handler gets both,
a write matching no studio route is a 404 rather than a fallthrough, and
test_studio_api.py attacks the boundary from outside rather than trusting the
arrangement.
The trench is still append-only, still trigger-enforced, and still written only by a run.
Real files, generated by the commands above — see examples/output/:
{
"contract": "laptop_catalog",
"sounding_id": 4,
"status": "healthy",
"row_count": 24,
"records": [
{
"title": "Tessera Pro 14 (2024)",
"price": 2795.34,
"rating": 3.7,
"availability": "In stock",
"product_url": "https://example-store.invalid/products/lp-1000",
"image": "https://example-store.invalid/img/lp-1000.jpg"
}
]
}Values are typed, not strings: price is a float, rating is a float,
product_url is absolute. Written regionally as $2,795.34 or 2.795,34 €,
both parse to the same number.
| File | What it shows |
|---|---|
laptop_catalog.json |
Normal typed output |
permit_register.csv |
CSV export, public-records domain |
sounding_healed.json |
A full run including the heal event and every rejected candidate |
sounding_quarantined.json |
A run where healing refused, and price is null rather than wrong |
$ cd core && pytest -q
411 passed, 27 skipped in 21.86sThe suite runs the whole pipeline against real markup through the real ledger — nothing is stubbed. Replay removes the network, not the logic.
The builder's tests are mostly about restraint, because that is where it can
fail invisibly: that discovery does not offer a concatenated wrapper or a
navigation menu as a field, that a want with no answer on the page is refused
rather than given the tidiest column, that a reordered list reports nothing,
and that a quarantined field emits no change events. test_fetch_ssrf.py is
the one to read first — an SSRF hole is the only failure here that looks like
success from the outside.
Three test classes are named TestRegression... and are worth reading.
Two cover bugs found by running the engine against the hostile fixture, and both
had the same shape — validation too permissive. One let a product title parse
as a price ("Meridian Pro 14 (2024)" → 14.0) and promote onto the wrong
element at 100% reported confidence. They are the tests that keep "break
nothing" honest.
The third covers a bug replay could never have caught, because replay never
makes the call: the live client triggered Bright Data's queued flow and then
polled the immediate flow's result endpoint. Those endpoints do not share an
auth domain, so every live run would have failed with 401 Invalid credentials
and sent whoever debugged it hunting for a bad API token instead of a wrong URL.
test_brightdata.py stubs the transport rather than the network, which pins the
request shapes without needing an account.
ScalyFoot is built for openly published pages: catalogues, public job listings,
government registers. The shipped contracts target reserved .invalid fixture
domains and collect no personal information — the permit register deliberately
takes reference numbers, site addresses, decision dates, and outcomes, and not
applicant names or contact details.
The tool has no login, session, cookie, or credential handling of any kind, by
design: there is no code path by which it can reach content behind an
authentication wall. Respect each site's terms and robots.txt.
The builder accepts an arbitrary URL and shows you what came back, which is the
exact shape of a server-side request forgery: the server sits inside a network
you do not, and http://169.254.169.254/ asks it to read something on your
behalf and print the result.
In production every user URL goes to Bright Data, whose network is not ours —
that removes the class of attack rather than filtering it. For local
development without credentials there is a direct backend, and it is
fetch.py's whole job to make that safe:
- off unless
SCALYFOOT_ALLOW_DIRECT_FETCH=1, so it cannot become a production default by accident; http/httpsonly, and every resolved address checked against loopback, RFC1918, link-local, CGNAT, multicast and their IPv6 equivalents — including an IPv4 address smuggled inside IPv6 notation;- the connection is pinned to the address that was approved, because checking a hostname and then handing the hostname to the HTTP client is a check that does nothing (DNS can answer differently the second time). TLS is still verified against the real hostname, so this does not trade an SSRF hole for a man-in-the-middle one;
- every redirect is re-checked from scratch — a public URL that 302s to
127.0.0.1is the same attack wearing a hat; - size and time caps, and per-account quotas on how many pages can be looked up in a day.
robots.txt is checked at build time and reported, not enforced — the
answer is shown to whoever is adding the site, and recorded against it, so a
person decides.
core/scalyfoot/ engine, CLI, API core/tests/ 411 tests
core/fixtures/ corpus + generator scraper_studio/ the custom collector
examples/contracts/ four domains + live examples/output/ example output
dashboard/ Next.js + TypeScript docs/ judges' walkthrough
Reviewing this? docs/JUDGES.md is a hands-on walkthrough
for checking these claims yourself — including the ones that do not flatter us.
docs/DEMO.md is the video script.
MIT — see LICENSE.