Skip to content

Let admins update RetroDB with one click from GitHub Releases - #23

Merged
milnet01 merged 9 commits into
mainfrom
claude/modest-goodall-5faq39-updater
Oct 29, 2026
Merged

milnet01 merged 9 commits into
mainfrom
claude/modest-goodall-5faq39-updater

Conversation

@milnet01

@milnet01 milnet01 commented Oct 27, 2026 •

Copy link
Copy Markdown
Owner

What

RetroDB checks GitHub Releases for a newer version. Admins see it on the dashboard and in Settings > System > Updates, with the release notes. Download fetches and verifies the release, and Install and restart applies it. If the new version does not come up, RetroDB rolls back to the old version by itself. Both source and standalone installs are covered.

This PR is built on #21's branch and targets main so that CI runs on it. Until #21 merges, this diff also shows #21's commits; merge #21 first.

Why

This is PASS-59-169, requested 2026-10-27. Your decisions:

  • check for updates, plus a one-click install (never unattended);
  • GitHub Releases as the source;
  • source and standalone installs both;
  • write the spec, then go straight to code, with the code as the review.

The spec is docs/specs/PASS-59-169-self-updater.md.

How

Checking. services/updater.py asks releases/latest at most once a day, or when you click Check now. Draft releases are invisible to that endpoint, so publishing the draft is what offers the update to users.

Downloading.

  • It picks RetroDB-v{V}-{Platform}.zip with its line in SHA256SUMS.txt, or …-Standalone.zip with its .sha256.
  • Download URLs must start with https://github.com/milnet01/RetroDB/releases/download/.
  • Size caps: 3 GiB downloaded, 8 GiB unpacked.
  • Extraction refuses absolute paths, .. and symlinks, and keeps Unix permission bits.

Installing.

  • Install backs up the database to database/backups/pre_update_*.db, writes a plan, and starts the new version's helper. For source installs that is updater_apply.py; for standalone, retrodb --apply-update, which the top of app.py hands over before importing anything.
  • The helper waits for the old process to exit, then swaps files with an undo record:
    • source: file by file, with a manifest so files a release dropped are removed;
    • standalone: the whole _internal/ folder plus the program.
  • For source installs it runs the hash-locked dependency install when requirements.lock changed.
  • It then restarts RetroDB and waits for /health to report the new version. Any failure restores the files and the database and restarts the old version. The outcome is written for the admin to see.

Never written: database/, logs/, data/ settings and tokens, scraped media. config.py holds only app constants, including APP_VERSION, so it is replaced from the new config.example.py. The old copy stays in the rollback folder.

Not one-click: git checkouts and LocalWebServerManager-managed installs see the notice and a link, but are not updated.

Threat model (spec §7): the checksum catches corruption and a file swapped in transit. Checking the cosign signatures release.yml already attaches is roadmap 59.170.

Mandatory-workflow checklist

  • Version bumped: no, 3.33.0 is still open.
  • Changelog entry: "RetroDB updates itself", in 3.33.0.
  • CSS rebuilt: new features/updates.css, added to build_css.py.
  • JS rebuilt: new page script updates.js, registered in build_js.py; the i18n manifest is regenerated.
  • Tests run: 2309 passed, including 32 in tests/test_updater.py.
  • Lockfile: not applicable.
  • CLAUDE.md updated: updater contracts; .claude/rules/distribution.md notes that release asset names are now load-bearing.

Verification

End to end, on a real source install of this branch. A copy of the tree with no .git ran on port 5088. I ran the real helper exactly as Install does, using the installed version's own services/updater.py for the plan, backup and spawn:

  • 3.33.0 → 3.33.1: done in 5 s. /health reports 3.33.1, config.py reads 3.33.1, the staging tree is cleaned up and rollback-3.33.0 is kept.
  • 3.33.1 → a broken 3.33.2: a release whose app exits on start. Rolled back in 7 s. 3.33.1 is serving again with its own files and config.py, the new file the release added is removed, the database is restored from the backup, and the manifest is unchanged.
  • Bug found by the first run, fixed: the restarted process loaded a stale config.pyc. Python checks a .pyc by the source's size and whole-second mtime, and copy2 carried over the staged mtime, so a same-size config.py edit was not recompiled. Files are now copied with a fresh mtime and __pycache__ is cleared. A regression test reproduces it.

Browser walk (Chromium, 1400px and 375px):

  • the dashboard banner shows for admins;
  • the Settings card shows the release notes as plain text (HTML in them stays text), and shows the git-checkout reason with no install buttons;
  • Check now reached the real GitHub API: the latest published release is 3.32.0, so it reports "latest version";
  • the daily-check toggle saves;
  • /settings#updates opens the card;
  • no page errors.

The walk also caught .btn overriding [hidden] on the buttons; fixed.

Not verified:

  • a standalone (PyInstaller) update on a real bundle; only the swap logic is unit-tested, with a fake bundle;
  • Windows and macOS;
  • the Install button's restart overlay in a browser (this checkout is a git tree, so Install is refused);
  • semgrep's registry rules (blocked locally; CI runs them).

Roadmap 59.169 lists trying it against the first real published release after this ships.

🤖 Generated with Claude Code

https://claude.ai/code/session_013s5ztLFVCcdTjjTeVcTnJr

- Screenshot dedup scans/deletes under IMAGE_PATH, not the bundle STATIC_PATH
- Multi-disc scan lists each disc file once (case-insensitive FS double-count)
- Default-controller auto-assign tries longest folder pattern first (snes != nes)
- /api/controllers: one systems query instead of one per controller
- Calculate storage: one batched write instead of a commit per game
- Local search clamps negative limit; similar-games escapes LIKE wildcards
- AI Fill invalidates the Library filter cache
- Case-only ROM renames allowed (samefile check)
- Database optimize closes its connection when VACUUM fails
- ROM-tools task cleanup iterates a snapshot; browse-folders single scandir

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013s5ztLFVCcdTjjTeVcTnJr
services:
- Clear scraped data commits the filename-title reset (close() rolled it back)
- Analytics "not started" sums NULL and 'not_started' groups
- query/execute helpers close connections they open outside an app context
- JPEG save converts P/LA/PA/I/F modes, not only RGBA
- Orphan-media scan looks basenames up in a set (was orphans x references)

jobs:
- Per-system RA refresh accepted system_name (TypeError -> 500 before)
- Swapped/demoted bulk scrapes drop their 'queued' job_queue row
- Bulk scrape settings load inside the try (no stuck 'running' job)
- HLTB / alt-titles jobs commit per game, not holding the writer lock
  across 25 network calls
- Player-core worker re-runs a request that raced its shutdown
- Museum image fetch: import/connect failures end the job cleanly
- Recovery dedup no longer collapses distinct multi-system scrapes
- GUI installer surfaces worker errors; .desktop Exec paths are quoted

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013s5ztLFVCcdTjjTeVcTnJr
- Card grid: one-column phone layout no longer gets inline row lines;
  re-layout when a resize changes the column count; measure all cards
  before placing (one forced layout instead of one per card)
- Game detail modal: late replies no longer paint a closed/other game;
  error message no longer replaces the spinner for good; AI Fill only
  reopens the game still shown
- Job toasts: a slow status reply is no longer aborted by the next poll
- Stale-reply guards: museum status, version search, log viewer,
  official-names proposals; Kill closes the Now-playing popover
- CHD converter/verify escape disk file names in innerHTML
- Replace undefined CSS variables with existing tokens (skip-link focus
  outline, settings tab bar, rename modal backgrounds were invisible)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013s5ztLFVCcdTjjTeVcTnJr
- The last active admin can no longer be demoted or deactivated
- /api/psn/games clamps page/per_page (per_page=0 was a 500)
- Steam/Xbox/PSN library fetch: duplicate check uses one index built
  from one query instead of re-reading the system per fetched title
- /trophies reads and cleans PlayStation titles once, not per trophy set;
  PSN full sync cleans each title once
- PSN single-game sync writes trophies in one batch; detail-page debug
  logging moved from INFO to DEBUG
- Collector trophies refresh: one read and one transaction (~140 round
  trips before); sidebar rank cached per request

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013s5ztLFVCcdTjjTeVcTnJr
- IGDB age ratings read IGDB's single global rating enum (CERO/USK/GRAC/
  ClassInd/ACB were mapped as 1-5) and the newer organization /
  rating_category shape; hybrid fetch requests those fields
- IGDB players: null offlinemax no longer aborts the merge; no made-up 1
- ScreenScraper primary reads the flat get_game_info dict on a cache miss
- ScreenScraper media extensions come from the URL path, not the query
  string (which carried account credentials into file names)
- TGDB lookup tables: a failed fetch is retried, not cached empty;
  unknown IDs are dropped instead of written as bare numbers
- TGDB / RAWG / ES-DE no longer invent players=1 (ES-DE overwrote
  curated players/modes through COALESCE)
- ES-DE gamelist: full-path match wins over a same-filename match
- RetroAchievements: curly apostrophes normalised; search title once
- Scans keep an existing system logo; scan_roms finds SVG logos

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013s5ztLFVCcdTjjTeVcTnJr
- all-games-controller: the resize handler called debounce() at load,
  before the deferred core bundle defines it, so the Library and System
  pages threw and rendered no cards. Use a local timer.
- The column count read from computed grid-template-columns includes the
  implicit tracks the cards' own inline placement created, so a grid
  narrowed to one column (phone) kept laying out two columns with a huge
  gap. A full layout clears inline placement first; a ResizeObserver on
  the grid re-lays out once its width settles (a window-resize timer
  fired mid sidebar animation and read the wrong count).
- screenscraper_password / screenscraper_devpassword join
  SECRET_API_KEY_FIELDS: masked on GET /api/scraper-settings and the
  settings page, kept on a masked echo.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013s5ztLFVCcdTjjTeVcTnJr
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013s5ztLFVCcdTjjTeVcTnJr
- /api/jobs/dismiss checks the same role / ownership as resume
  (_may_manage_job, shared); any signed-in user could dismiss any job
- The dashboard recovery banner leaves out jobs the user may neither
  resume nor dismiss (a player no longer sees an interrupted bulk scrape)
- Demoting or deactivating the last admin gets its own message;
  catalogs regenerated
- CLAUDE.md: a release needs at least 5 changelog items, no upper limit
  (user, 2026-10-27)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013s5ztLFVCcdTjjTeVcTnJr
Admins see a newer published release on the dashboard and in Settings >
System > Updates with its notes. Download fetches the release asset for
this install (source zip or standalone bundle), checks it against the
release's SHA-256 file and unpacks it safely (no path escape, no links,
executable bits kept). Install backs up the database, writes a plan and
starts the NEW version's helper (updater_apply.py, or the standalone
binary with --apply-update, dispatched before app.py imports anything),
then shuts down. The helper waits for the old process, swaps the files
(source: per file with a manifest of installed files; standalone: the
whole _internal/), installs changed dependencies from requirements.lock,
restarts and waits for /health to report the new version. Any failure
puts files and database back and restarts the old version; the outcome
is shown to the admin.

User data is never written: database, logs, data/*.json settings and
tokens, scraped media. config.py (app constants, APP_VERSION) follows
the new config.example.py, the old one kept for rollback. Copied files
get a fresh mtime and __pycache__ is cleared, after an end-to-end run
showed a same-size config.py loading its stale .pyc.

Git checkouts and LocalWebServerManager installs see the notice but are
not updated. Daily check toggle: update_check_auto.

Spec docs/specs/PASS-59-169-self-updater.md; roadmap 59.169, and 59.170
for cosign signature verification.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013s5ztLFVCcdTjjTeVcTnJr
@milnet01
milnet01 changed the base branch from claude/modest-goodall-5faq39 to main October 27, 2026 13:09
@milnet01
milnet01 merged commit f81fe17 into main Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants