Skip to content

Regional release dates, previous/next game, processed uploads, RA card and unscraped-count fixes - #25

Merged
milnet01 merged 2 commits into
mainfrom
claude/admiring-clarke-y0aljw
Oct 29, 2026
Merged

milnet01 merged 2 commits into
mainfrom
claude/admiring-clarke-y0aljw

Conversation

@milnet01

Copy link
Copy Markdown
Owner

What

  • Release dates for every region. New games.release_dates column (migration 024). The AI is asked for it in a scrape's gap-fill and in AI Fill. The game page and the quick view show the dates.
  • Previous / Next game on the game page. The links follow the page you came from: the Library, or the game's console page. They keep that list's order, filters and search.
  • Uploaded covers, fanart and screenshots get the same image processing as scraped ones. That means the configured format, resizing and the small/medium copies, and for screenshots the duplicate check.
  • Fix: RA card missing on the game page for games that show the RA badge in the Library (GameCube: Metroid Prime, Metroid Prime 2).
  • Fix: Systems page "unscraped" count didn't match the console's Unscraped filter (Sony PlayStation: 2 unscraped, filter empty).

This PR is built on #24's branch and targets main so that CI runs on it. Until #24 merges, this diff also shows #24's commit; merge #24 first. I built on it because migrations are numbered by position: #24 adds migration 023, so release_dates has to be 024. Building on it also avoids conflicts in scrape_ai.py, hybrid_scraper.py, the game page and the quick view, which both PRs change.

Why

All five were requested on 2026-10-27.

  • Unscraped count: the Systems page counted bonus discs. The console page's own count and its game list (and so the Unscraped filter) leave bonus discs out. The two PlayStation games were bonus discs.
  • RA card: the game page ignored the stored RA link and searched RA by title on every page load. The RA refresh had matched these games by ROM name, which that title search misses. So the Library badge, which reads the stored link, showed RA while the page found nothing.
  • Uploads: only 2D and 3D box uploads were resized. Fanart and screenshots were saved untouched, nothing was converted to the configured format, and uploaded screenshots had no duplicate check.

How

  • Unscraped count: routes/systems.py missing_count now leaves out is_bonus_disc rows, the same rule as /system/<id> and the dashboard.
  • RA card: get_retroachievements_info(game) builds the card from ra_game_id, ra_achievement_count and ra_points. It asks RA by ID (get_game_info) only when no count is stored. The page also no longer waits on an RA request each time it loads.
  • Uploads (services/game_media_service.py):
    • save_upload(..., image_type=...) names the file with the configured extension (WebP by default), as scraped files are. It then runs finalize_downloaded_image: format check, resize and -sm/-md variants.
    • save_screenshots processes each screenshot the same way, then runs keep_screenshot_if_unique against the game's existing screenshots and the earlier files in the same upload. It returns how many were dropped, and the page tells you that number.
    • A replaced cover's old file is deleted when no other game uses it. remove_media_file now also deletes the -sm/-md files. Before this, a re-upload left the old file on disk.
    • try_standardize is replaced by this path.
  • Previous / Next:
    • AllGamesController saves each list's filter and sort params for the current tab (sessionStorage, one key for the Library and one per console) via RetroDB.saveGameNavContext.
    • gameDetailUrl now adds ?from=system on console pages.
    • The game page sends the saved params to the new GET /api/games/neighbors. It uses the same _build_games_query as /api/games and returns {found, position, total, prev, next}. It is capped like /api/games/ids.
    • A regional version that the list folds into its group's card gets that card's place in the list.
    • A game outside the saved filters, for example one opened from global search, steps through the whole console or Library list.
    • Pages opened from ROM Reports, Achievements or trophies show no links.
  • Release dates:
    • services/release_dates.py reads what the AI returns: region codes, region names such as "Japan" or "PAL", or a list of {region, date}. It drops unknown regions and impossible dates.
    • The column stores compact JSON in region order. Each region may have a full date, a year and month, or a year only.
    • Pages list the dates earliest first, with region names from CLDR and dates in the user's language, so these need no new translations.
    • They are saved fill-only (COALESCE), the same as units_sold, and cleared by "Clear scraped data".
    • release_date is unchanged; it stays the date of that ROM's own release.
    • I left IGDB's per-region dates out on purpose. You asked for the AI, and IGDB's need platform matching per entry.

Version / changelog: this opens 3.35.0. #24's 3.34.0 entry already has 7 items, so these 5 make a new release. Expect small conflicts with #21–#24 in config.example.py, at the top of data/changelog.yaml and in the translation catalogs. Keep 3.35.0 above 3.34.0, and regenerate the catalogs with the CLAUDE.md §8 command. The per-locale changelogs are not updated; they go in at release time, as done before.

Mandatory-workflow checklist

  • Version bumped in config.example.py (APP_VERSION 3.35.0; APP_LAST_UPDATE is already 2026-10-27). config.py is untracked.
  • Changelog entry added at the top of data/changelog.yaml (3.35.0, 5 items)
  • CSS rebuilt
  • JS rebuilt
  • Tests run: 2286 passed, 1 skipped (the same skip as on main)
  • Lockfile: not applicable
  • CLAUDE.md updated: notes on release_dates and the Previous/Next wiring. Also updated: docs/specs/api-contracts.md (/api/games/neighbors), migrations.md, scrapers.md, and retrodb.spec (lists migration 024).
  • i18n catalogs regenerated; check_i18n_fresh.py passes. Three new strings; the region names come from CLDR.

Verification

  • Tests: pytest -n 8, ruff check ., node --check on the changed JS, and scripts/ci_local.sh --docs.
    • New tests: tests/test_game_neighbors.py, test_release_dates.py, test_upload_image_pipeline.py (real image bytes), test_game_page_ra_card.py and test_systems_unscraped_count.py.
    • The new tests also pass on a tree with only the staged files.
    • The Systems-page tests fail on the old routes/systems.py.
  • Browser (Chromium, dev server on a throwaway database: PlayStation with 2 unscraped bonus discs, GameCube with Metroid Prime and Prime 2 linked to RA):
    • Systems: PlayStation no longer shows "2 unscraped", which matches its empty Unscraped filter.
    • From the Library: quick view, then View full page, shows "3 / 7" with Previous and Next. Clicking Next goes to the next Library game with ?from=library.
    • From the GameCube page searched for "Metroid": shows "1 / 2" and Next goes to Echoes. The breadcrumb reads Systems › Nintendo GameCube.
    • Opened with no list saved: steps through the whole console list ("2 / 3").
    • Error paths: a page opened from ROM Reports has no links, and an unknown id returns found: false.
    • RA card: both Metroid games show it ("0 / 61", "0 / 54") with no RA account set up, so it comes from the stored link.
    • Release dates: shown on the game page and in the quick view: "North America: Nov 17, 2002 · Japan: Feb 28, 2003 · Europe: Apr 3, 2003".
    • Phone (375px): no horizontal scroll; Previous/Next wrap under the breadcrumb.
    • No console errors on any page.
    • Upload through the game page's Edit window:
      • a 2000×2800 PNG cover was saved as 9_custom.webp, 1440×1080, with -sm and -md copies;
      • of three screenshots, two the same image, two were kept as WebP and the page said "Screenshots not added because they match ones the game already has: 1";
      • replacing a cover named 9_tgdb.jpg deleted it and its two variants;
      • a cover another game also uses was kept.
  • Not verified:
    • a live AI provider call for release dates; the prompt, parser and save path are covered by unit tests only;
    • real RA data.

Screenshots (UI changes only)

Taken in the session, not attached: the game page at 1400px and 375px with Previous/Next, the release dates and the RA card.


Generated by Claude Code

… width fixes (3.34.0)

RPCS3 compatibility
- services/rpcs3_compat.py reads the local RPCS3's
  GuiConfigs/compat_database.dat (the rpcs3.net export RPCS3 downloads via
  Manage > Databases > Download Compatibility Database), re-read only when
  the file changes, and looks games up by disc serial. Never stored, never
  downloaded by RetroDB. Local PS3 ROMs only; imported games are skipped.
- Shown on Library and console cards, the quick view and the game page;
  the game page hints how to get the list when RPCS3 has none.
- Serials: PARAM.SFO for PSN/installed folders, and a serial in a file or
  folder name ([BLUS30443]); missing serials are read in the background.
- _rpcs3_dirs also looks beside rpcs3.exe (Windows), in a portable folder
  and in macOS Application Support, matching RPCS3's get_config_dir().

Sales figures
- games.units_sold (migration 023), asked of the AI in the scrape's
  gap-fill and in AI Fill, from free published sources; parse_units_sold()
  normalises "2.4 million", "~850k" and the like. Shown on the game page
  and in the quick view. AI Fill now uses AI_FILLABLE_FIELDS instead of a
  copy of it.

Fixes
- Franchise / Series and Platform Availability switches in the edit
  windows did nothing: their <label for> pointed at the hidden text input,
  not the checkbox. Their labels are now translated.
- Achievements tabs each capped the page from their own card size
  (RA/Steam/Xbox 1463px, PSN 1563px, RPCS3 1845px); all now share
  .achievements-page's tokens.
- AI Fill's rating inference called .strip() on integer columns.
- Library card completion and score tooltips are translated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013s5ztLcay7qP1ezTvc85UA
…RA card and unscraped-count fixes (3.35.0)

Release dates by region
- games.release_dates (migration 024): a JSON object of region code -> date
  ({"JP": "2002-11-17", "NA": "2002-11"}), asked of the AI in the scrape's
  gap-fill and in AI Fill, fill-only. services/release_dates.py parses what a
  model returns (codes, region names, {region, date} lists), drops unknown
  regions and impossible dates, and writes region names and dates in the
  user's language (CLDR). Shown on the game page and in the quick view.

Previous / next game
- The game page shows Previous / Next and the game's position in the list
  it was opened from: ?from=library steps through the Library, otherwise
  its console. AllGamesController saves each list's filters and sort per tab
  (RetroDB.saveGameNavContext); the page sends them to the new
  /api/games/neighbors. A version folded into its group's card is placed at
  that card; a game outside the saved filters steps through the whole list.

Uploads
- Uploaded covers, 3D boxes, fanart and screenshots go through the scraper's
  pipeline: the configured image format, size standardization and the
  -sm/-md variants. Uploaded screenshots get the perceptual-hash duplicate
  check; dropped duplicates are reported. A replaced cover's old file and
  variants are removed when no other game uses it; remove_media_file now
  removes variants too.

Fixes
- The game page's RA card read a fresh RA title search on every load, which
  missed games the RA refresh had matched by ROM name, so games with the
  Library's RA badge had no card (Metroid Prime / Prime 2 on GameCube). It
  now uses the stored ra_game_id link and counts, asking RA by id only when
  no count is stored.
- The Systems page counted unscraped bonus discs, which the console list
  and its Unscraped filter never show (Sony PlayStation: 2 unscraped, filter
  empty).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013s5ztJjyQyt3pKwNu4uwpS
@milnet01
milnet01 merged commit 05271af into main Oct 29, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants