feat: mirror Stairway to Hell archive into S3 by mattgodbolt-molty · Pull Request #688 · mattgodbolt/jsbeeb

mattgodbolt-molty · 2026-05-03T17:32:52Z

Summary

Adds tooling and a manual GitHub Actions workflow to snapshot the upstream
Stairway to Hell BBC Micro, Acorn
Electron, and sideways-ROM archives into s3://bbc.xania.org/archive/sth/.
The mirror exists so jsbeeb's sth: URLs and the in-app archive browser
keep working if the volunteer-run upstream site disappears — STH has been
effectively frozen since around 2008 (see meta/bbc-disklog.txt /
meta/bbc-tapelog.txt once mirrored), so this is a one-shot snapshot
rather than a continuous sync.

The client (src/sth.js) is intentionally not changed in this PR. That
swap follows in #689 once the mirror has been populated and spot-checked.

What gets mirrored

Category	Files	Approx size	Source
`diskimages/`	1,608	26 MB	`bbc/archive/diskimages/reclist.php?sort=name&filter=.zip`
`tapeimages/`	1,502	24 MB	`bbc/archive/tapeimages/reclist.php?sort=name&filter=.zip`
`sthcollection/`	143	~30 MB est.	`bbc/sthcollection.html`
`other/educational/`	75	1.7 MB	`bbc/other/educational/reclist.php?sort=name&filter=.zip`
`roms/`	50	~3 MB	`roms/homepage.html` (BBC + Electron sideways ROMs)
`electron/uefarchive/`	888	15 MB	`electron/uefarchive/reclist.php?sort=name&filter=.zip`
`electron/dfs/`	231	TBD	`electron/dfs/homepage.html`
`electron/adfs/`	23	TBD	`electron/adfs/homepage.html`
`electron/multiplexing/`	1	small	`electron/multiplexing/homepage.html`
`electron/t2p3/`	4	small	`electron/t2p3/homepage.html`
`meta/*`	10	<250 KB	upstream changelogs and index pages

Total ~4,525 objects, expected ~80 MB after HEAD requests fill in sizes
for the homepage-indexed categories. Still trivial for S3.

The Electron categories and the roms/ category are mirrored even though
jsbeeb does not currently emulate the Electron and already ships its own
copy of the BBC ROMs in public/roms/ — the goal is a complete archival
snapshot of what STH hosts, not just what the running app uses today.

Layout in S3

Top-level prefix is archive/sth/ so future mirrors of other archives can
sit alongside as archive/<id>/ with their own top-level manifests.
Within archive/sth/ the layout mirrors STH's own URL structure exactly:

s3://bbc.xania.org/archive/sth/
  manifest.json                                    # top-level: lists categories
  diskimages/manifest.json
  diskimages/Acornsoft/Elite.zip
  ...
  tapeimages/manifest.json
  tapeimages/Hewson/Uridium-FIXED_B.zip
  ...
  sthcollection/manifest.json
  sthcollection/A1_r1.zip
  ...
  other/educational/manifest.json
  other/educational/Selective/MickeysMagicMixture.zip
  ...
  roms/manifest.json
  roms/ACPE00.zip
  roms/ADFS-1.30.zip
  ...
  electron/uefarchive/manifest.json
  electron/uefarchive/Hewson/Uridium-FIXED_E.zip
  ...
  electron/dfs/manifest.json
  electron/adfs/manifest.json
  electron/multiplexing/manifest.json
  electron/t2p3/manifest.json
  meta/bbc-disklog.txt
  meta/bbc-tapelog.txt
  meta/bbc-{homepage,diskimages,tapeimages,sthcollection,other}.html
  meta/roms-homepage.html
  meta/electron-{homepage,other}.html

Manifest schema (schemaVersion: 1) and per-category manifest layout are
documented in tools/README-sth-mirror.md.

Safety choices worth flagging

No --delete on the S3 sync. Strictly additive uploads only. STH files
don't actually disappear (frozen archive), and a typo on the destination URL
with --delete could wipe the live app served from the same bucket. If
pruning ever becomes needed, add it behind an explicit prune: true input
in a follow-up.
workflow_dispatch only, with a dry_run input that defaults to
true. First run scrapes only, prints the top-level manifest summary, and
stops short of touching S3. Re-run with dry_run: false once the dry-run
output looks right.
Cache-Control: zips get public, max-age=31536000, immutable (paths
are content-stable); manifests + meta/* get max-age=300.

Test plan

npm run ci-checks is clean
Local smoke: node tools/mirror-sth.js --out /tmp/sth-mirror --quick
parses all 10 catalogs (1608/1502/143/75/50/888/231/23/1/4 entries
— counts match hand-counted upstream totals)
End-to-end on smallest categories: full download of electron/t2p3
(4 files) and sthcollection sample, byte-identity verified by
sha256sum against upstream URLs
Resumability: re-running after partial download skips files already
present at the expected size
URL encoding: paths with [ ] (e.g. Unreleased/Daxis[droids]-demo.zip)
round-trip cleanly via the scraper's encodePath
parseHomepage correctly filters ../ cross-refs (the one
../electron/uefarchive/... link on /roms/homepage.html doesn't
get double-counted; that file lives in the electron/uefarchive
category)
Run workflow with dry_run: true to confirm AWS creds resolve and
the top-level manifest summary looks sane
Run workflow with dry_run: false and spot-check
https://bbc.xania.org/archive/sth/manifest.json and a sample of
zips from a few categories

Open question for review

The existing update-web deploy uses jakejarvis/s3-sync-action which does
not add --delete (verified by reading the action's entrypoint.sh), so
the regular deploy will leave anything under archive/sth/ alone. But this
PR assumes the IAM user behind AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY
has write access to the archive/sth/ prefix as well as the bucket root
used by update-web. If the policy is scoped tighter than that, the
dry_run: false run will 403 and you'll need to widen the policy first.

🤖 Generated with Claude Code

Adds tooling and a manual GitHub workflow to snapshot the upstream STH BBC Micro software archive into s3://bbc.xania.org/archive/sth/. The mirror exists so jsbeeb's sth: URLs and the in-app archive browser keep working if the volunteer-run upstream site disappears. The client (src/sth.js) is intentionally not changed in this commit; that swap will follow in a separate PR after the mirror has been populated and spot-checked. * tools/mirror-sth.js scrapes four categories (disk images, tape images, sthcollection, educational) plus a handful of changelog/index files for provenance, writes per-category and top-level manifests in a versioned JSON format, and is resumable by size. * .github/workflows/mirror-sth.yml is workflow_dispatch only with a dry_run input that defaults to true. The S3 sync is strictly additive — no --delete — to make a typo on the destination URL safe against wiping the live app served from the same bucket. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Adds six more categories so the snapshot covers the full set of downloadable archive areas STH hosts, not just the BBC Micro disc and tape sections: * roms/ — 50 BBC + Electron sideways ROMs from /roms/homepage.html * electron/uefarchive/ — ~890 Electron tape image zips, ~15 MB, via reclist.php with the same parser as the BBC tape archive * electron/dfs/ — ~230 Electron DFS disk images * electron/adfs/ — ~23 Electron ADFS disk images * electron/multiplexing/ — single curiosity file * electron/t2p3/ — 4 curiosity files The total mirror grows from ~3,330 to ~4,500 objects and from ~50 MB to ~80 MB; still trivial for S3. S3 paths follow the upstream layout exactly (e.g. archive/sth/electron/uefarchive/Hewson/Uridium-FIXED_E.zip), so provenance is preserved. Generalises parseSthCollection into a reusable parseHomepage that handles any hand-rolled HTML index page, filtering out external, absolute, and upward (../) hrefs and optionally stripping a directory prefix. While doing this, fixes a latent bug where sthcollection's fileBase was set to /bbc/ instead of /bbc/sthcollection/, which would have produced 404s on the first non-quick scrape — not caught by the earlier smoke test because that test only ran the educational category. Adds three new meta files (roms-homepage.html, electron-homepage.html, electron-other.html) and renames the existing bbc/* meta files with a bbc- prefix to disambiguate from the new electron- and roms- ones. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

mattgodbolt-molty mentioned this pull request May 3, 2026

feat: load STH discs and tapes from the bbc.xania.org mirror #689

Draft

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: mirror Stairway to Hell archive into S3#688

feat: mirror Stairway to Hell archive into S3#688
mattgodbolt-molty wants to merge 2 commits into
mainfrom
claude/sth-mirror

mattgodbolt-molty commented May 3, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

mattgodbolt-molty commented May 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What gets mirrored

Layout in S3

Safety choices worth flagging

Test plan

Open question for review

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

mattgodbolt-molty commented May 3, 2026 •

edited

Loading