Skip to content

pso_gpu: the profile names the GPU families the offline pipeline archive is made for - #653

Closed
oliverames wants to merge 2 commits into
seedds:mainfrom
oliverames:profile-pso-gpu
Closed

oliverames wants to merge 2 commits into
seedds:mainfrom
oliverames:profile-pso-gpu

Conversation

@oliverames

@oliverames oliverames commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

The offline pipeline archive follows the device's GPU family, but the defaults in both build paths named only one. This change adds an optional game-profile key, pso_gpu, that names the GPU architectures the app gets the archive for, read by dm/games/game.py, dm/ios/tools/build_ipa.sh and scripts/build_ipa_from_nsp.py. The problem it answers is a hardcoded one: a Duo foldable reports MTLDevice.architecture applegpu_g19p, while the hardcoded default g18p is the A19 / A19 Pro of the iPhone 17 family, so an app built without an explicit --gpu/DM_PSO_GPUS value carried an archive made for a family the device never asked for. The profile key lets a profile name the families it serves instead (pso_gpu g18p g19p names both in one build), where previously a per-build flag was the only way.

The key and its validation

In dm/games/game.py the key validates as whitespace-separated tokens, each matching [A-Za-z0-9]+ with the applegpu_ prefix optional, at most 1–4 tokens; none is not a value. When the key is absent the existing default applies unchanged: it is the absence of the key that means unset.

In dm/ios/tools/build_ipa.sh the existing chain DM_PSO_GPUS env → g18p when pso_mtl4 on → none now consults the key's list before the hardcoded g18p step. scripts/build_ipa_from_nsp.py applies the same fallback order at its --gpu default; the --gpu build flag still wins over all defaults. Help text and comments updated in both places.

The build order

The order is exactly preserved: the --gpu flag first, then DM_PSO_GPUS in the shell chain, then the pso_gpu key, then g18p for a profile with pso_mtl4 on, then none. Nothing a profile sets displaces an explicit build-flag value; the key only takes the place g18p held alone.

Why a profile key

Each family's archive costs its own metal-tt run, so a profile names exactly the devices it serves, not a superset built on spec. The archive side is proven for the new family: running dm/gfx/tools/pso_offline.py --gpu g19p over Super Mario Bros. Wonder's actual pair tables on this Mac (Apple M2 Pro, toolchain metalfe-32023.922) compiled 3,697 bases into a 67 MB .mtl4a in 113 s, planning 27.5 s of that, with 1 pair left to the device and 0 function-constant pairs. Both families, pso_gpu g18p g19p, therefore cost two such runs in one build, where the key makes that a property of the profile rather than of every invocation's flags.

Tests

The profile loader accepts profiles with and without the key. The validation cases the loader already had are extended where the neighbouring test pattern allowed: a single token, two tokens, an invalid token rejected and the absent key read as unset. The NSP script now consults pso_gpu only when pso_mtl4 is on, matching the shell build. A regression check covers both profile modes, whitespace-separated GPU families, explicit --gpu values and the PSO_GPUS setting. The new dm/tools/tests/test_game_profile.py covers the rejection cases on a synthetic profile (none, punctuation in a token, five tokens, comma-joined tokens).

dm/games has no runtime code in this change; scripts/regress.py check --quick runs over Breath of the Wild's stored route (the only local game with a route and a baseline) as the header/packaging check of the same loaders, and its report lands here as the baseline-vs-candidate compare of the merge base and this branch.

What this alone does not do

No shipped game changes: no profile in dm/games sets the key yet; the companion Super Mario Bros. Wonder profile PR does, and its device checks carry the open-on-device run. The device-side stamp/coverage probe of a g19p archive is the same mechanism every family's archive goes through; its run on the device belongs to that profile PR. One Oct 28 deployment-log note in the companion repository claimed g19p archives did not compile; the current toolchain compiles them fine (the run above), so that note is a record of an older attempt, not a current limit of this change.

The branch is rebased onto a662f29, which includes the upstream t_fpslock sanitizer fix. The 32 pipeline-archive, profile-validation, documentation and environment-switch tests pass (one existing skip), and the pinned lint ratchet reports 392 baseline findings with no new findings.

@oliverames

Copy link
Copy Markdown
Contributor Author

regress.py check (quick), this branch as the candidate

scripts/regress.py ... check --quick --games 01007EF00011E000 — baseline 682fdd26726d (the merge base), candidate this branch (ca47dbf53cfb), quick mode, Apple M2 Pro, macOS 27.2, Xcode 27.2, 818.8 s of chain.

game status shaders (base → cand) newly failing newly building changed MSL changed variants interface (new / known)
The Legend of Zelda Breath of the Wild 01007EF00011E000 boot ERROR 19420/19420 → 19420/19420 0 0 0 343 0 / 0

The boot-smoke leg could not run: saves/outdoors (this game's smoke route's save) does not exist in this checkout of the game folder (an empty zora/ since Oct 27); the same error marks the baseline's own leg when run here, and it is the state the sibling PR comments report.

The changed-variants count needs the read that follows: every one of the 343 differs in bytes only — changed MSL 0, reflection 0 — and a spot comparison of one of the listed files across the two chain build dirs shows the same size, the same metalfe-32023.922.4 toolchain tag, and file mtimes on two different days (Oct 27 13:12 vs Oct 28 10:13), i.e. two compiler runs over identical MSL, whose output the toolchain does not reproduce byte for byte; the harness compares the variant libraries' bytes the way chain does, so any pair of rebuilds of the same key produces exactly this class, and it is not this change's — the branch itself holds dm/games/game.py, dm/ios/tools/build_ipa.sh, scripts/build_ipa_from_nsp.py, the tests and the docs rows, none of which the shader build inputs read. Every profile in dm/games loads under the new pso_gpu validation, and dm/games headers generated unchanged (checked on the two profiles this checkout can build headers for).

seedds added a commit that referenced this pull request Nov 1, 2026
The iOS archive default targeted only g18p, so another GPU family needed
an explicit build override. Read optional pso_gpu targets in both packers
behind the existing pso_mtl4 opt-in gate, retaining explicit overrides
and the previous defaults when the key is absent.

Keep game settings unchanged. Record the contributor's Wonder archive
probe in its game note and clarify archive selection and device validation.

Validation: 50 targeted tests run, one Metal compiler test skipped; all
121 profile headers unchanged; shell syntax valid. Pinned Python lint
reports the same pre-existing reportArgumentType at
dm/tools/tests/test_shader_toolchain_cache.py:134 on fresh origin/main
and this commit (393 findings, 392 in its baseline); no additional finding.
Regression risk is low: no shipped profile uses pso_gpu and the host and
shader chain are unchanged.

Co-authored-by: Oliver Ames <7366664+oliverames@users.noreply.github.com>
@seedds

seedds commented Nov 1, 2026

Copy link
Copy Markdown
Owner

Landed on main as 105f417b, retaining the implementation and crediting Oliver Ames as co-author.

Both iOS packaging paths now read pso_gpu behind the existing pso_mtl4 on gate, with explicit build overrides taking precedence. Existing defaults and every shipped game setting remain unchanged. The documentation now states that gate consistently, and the Wonder-specific archive measurement lives in Wonder's note with its Mac-only evidence and outstanding device checks clearly identified.

Validation after rebasing onto fresh origin/main:

  • 50 targeted tests: 49 passed, one skipped because the iOS Metal compiler was unavailable.
  • All 121 profile headers are unchanged against the parent revision; Wonder still has Metal 4 off.
  • Shell syntax, CLI help and whitespace checks passed.
  • Pinned Python lint reports the same pre-existing reportArgumentType at dm/tools/tests/test_shader_toolchain_cache.py:134 on both fresh main (bbc3b737) and the landing commit: 393 findings against a 392-finding baseline. No additional finding was introduced by this change. The lint ratchet therefore remains failing on that upstream issue.

Regression risk is low: no shipped profile uses the new key, and this change modifies neither the host nor the shader chain, so no multi-game regression run was performed. Confidence is high in the tested packaging behavior. This is a generalized archive-target selection fix at the packaging source, with no game-specific host branch or runtime workaround; it makes no claim of measured on-device performance or memory improvement.

The original implementation and design survive meaningfully, so the commit includes Co-authored-by: Oliver Ames <7366664+oliverames@users.noreply.github.com>.

#651 remains deferred pending device performance and memory validation. iOS remains opt-in.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants