Skip to content

Add an MCP server for scripted in-game testing - #1025

Open
nicholasgower wants to merge 6 commits into
NotRexed:mainfrom
nicholasgower:feature/mcp-ingame-testing
Open

nicholasgower wants to merge 6 commits into
NotRexed:mainfrom
nicholasgower:feature/mcp-ingame-testing

Conversation

@nicholasgower

@nicholasgower nicholasgower commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

What this adds

scripts/hookshot_mcp.py, a Model Context Protocol server that turns a built Hookshot into
something a test can drive. It is a separate Python process: nothing is linked into hookshot.exe, and it needs no game
files of its own.

  • launch starts the game headless, muted, on Vulkan, script-only (no physical pad, keyboard or mouse), with its own run
    folder, save folder and shader/pipeline cache.
  • press / stick / sequence write the game's existing live-input file (HOS_INPUT_LIVE) and wait for the game to log
    that it read the line.
  • screenshot asks the renderer for one PNG of the next presented frame (Vulkan RX_VK_SHOT_FILE, WebGPU
    RX_SCREENSHOT_FILE) and returns it inline as MCP image content plus a path on disk, with the present number.
  • status / logs / wait read the process state and the recent game log, and advance wall time.
  • stop asks the game to quit with the !quit live command, then terminates if it does not.
  • Resources: hookshot://log, hookshot://last-screenshot, hookshot://status.

Game-side seams added for this: HOS_INPUT_SCRIPT_ONLY, the analog stick tokens LX=, LY=, RX=, RY=, the !quit
live command, HOOKSHOT_NO_ENV_FILE, and the one-shot screenshot request files.

Docs: docs/mcp.md (install, host configuration, tools, launch options, what the server sets, limits). The tool and
resource tables in it are checked against the live server by tests/test_docs.py.

Why this matters for testing

Today the project has two testing speeds: game-free unit tests that run in seconds, and a person playing the game. Anything
in between — start the game, reach the title, open a menu, look at the frame — is done by hand, one run at a time, and
cannot be repeated after a change.

That is the gap this closes:

  • Regressions can be found without playing. A change to the renderer, the HLE runtime, input routing or the builder
    can be checked by the same scripted sequence: it either produces the expected frames or it does not. The log, the input
    lines and the PNGs are left in a run folder to compare.
  • It is the base for every further automated test. Input plus a screenshot plus a log is enough to test menus,
    cutscenes, the pause and file menus, resolution and frame-generation settings, controller mappings and save paths
    without a human at the keyboard.
  • Any MCP host can run the test, including agents. Claude Desktop, OpenCode, the MCP Inspector or a CI wrapper speak the
    same protocol; the tools are plain JSON calls and the frame comes back as an image, so "is the title screen drawn?" is a
    question a model can actually answer about a real build.
  • It unblocks dependent work. The save editor (feat: add standalone save editor and read-only diagnostics #784) needs scripted input to reach a save point; it is on hold until
    this lands.
  • It is cheap to keep running. One process, no window, ~10 GB, a clean exit code, and no dialog boxes anywhere.

Rebased, and the previous PR

This is the same branch as #764, which was closed to keep the queue clear with "please rebase on current main and reopen".
It is now rebased onto main at b2765ba (v2.0.6 and the merges after it), and the game has been rebuilt from this branch
and driven end to end.

What I tested

Not only the unit tests: the server was run against a real build of this branch, over its real stdio transport, and every
tool was exercised. Build: a normal builder layout (hookshot.exe launcher, bin\, romfs), Vulkan back end, Windows,
13 min 47 s, 0 failed modules, 100.0000% of 17,646,501 instructions translated.

Game-free suite (what CI runs):

Check Result
tests/test_mcp_server.py PASS — protocol in memory and over stdio, tool and resource catalogue, input-line encoding, environment handling, game_dir, fake-game end to end
tests/test_docs.py PASS — links, documented commands, MCP tables checked against the live server
tests/test_pr_check.py PASS
scripts/pr_check.py --base main PASS — 19 changed paths, none forbidden; LF endings; Conventional subjects and both trailers

The full game-free suite gave 107 passed, 5 failed, 11 skipped. The five failures need tools this PC does not have — Vulkan
and SPIR-V headers, the Windows SDK headers for this compiler, and symlink privileges — and none of them touch a file in this
PR. The three new checks above pass.

Every tool against the real game, over stdio (34 checks):

Tool What was verified
launch cold run reached its first present at 85 s while 38,809 programs were prepared, warm run ~4 s; pid, run dir, log, input and request paths returned; wait_ready works
status correct before launch, while running (renderer Vulkan (hookshot_vk.dll), uptime, last screenshot) and after stop
screenshot 1280x720 PNG inline and to a chosen path, present parsed (e.g. "present 116"); the picture is the presented frame
press A acknowledged in the game log, and it started New Game — the frame changed from the title screen to the intro cutscene
stick left and right acknowledged; LX=, LY=, RX= in the log; the frame changed
sequence a three-step line queued and acknowledged (0:DOWN/120,400:DOWN/120,800:A/150); the frame changed
wait returns status; uptime advances
logs filtered (hid: live) and unfiltered; every injected line appears
stop clean !quit, exit code 0, process gone; a second stop is safe
resources status (JSON), log (text/plain), last-screenshot (image/png, 1280x720)
error paths before launch, duplicate launch, unknown button, centred stick, reserved extra_env, after stop — all clear messages

The visual half of that run was checked too: title screen, then the cutscene after A, then further changes after stick and
sequence — 12/12 visual checks. One detail worth knowing (now in the docs): the very first wait_ready screenshot of a cold
run is a black boot frame, because it returns on the first present while the shaders are still being prepared; the run above
waited for the title before asserting on the picture. The screenshots stayed on the development disk; no game-derived file is
in this PR.

Finally the server was installed as a project MCP server in OpenCode (V2 mcp.servers configuration) and the nine tools were
called through OpenCode itself — status, launch (with wait_ready), wait, press, stick, sequence, screenshot,
logs, stop — all returned the expected results, and the run ended with exit code 0.

Bug found and fixed by this testing

--game-dir and HOOKSHOT_MCP_GAME_DIR selected the folder's hookshot.exe. In a folder made by the builder or
make_game.py that is the setup launcher, which opens the installed builder (builder/setup/setup.c), not the game in
bin\; launch would have started the builder setup and no screenshot would ever have arrived. resolve_paths now prefers
bin\hookshot.exe and falls back to the top of the folder for a development build, and tests/test_mcp_server.py covers
both layouts.

docs/mcp.md also now gives the V2 OpenCode configuration (mcp.servers, no enabled field) and notes that wait_ready
returns on the first present, which on a cold cache is a boot frame while the shaders are still being prepared.

On the CLI-first feedback in #764

The points raised there — a CLI, files instead of inline images, state in run folders, waits on game state, a GPU lock — are
reasonable, and this keeps the door open for them. Every tool is a thin call into GameSession; the run state already lives
in a run folder (log.txt, input.txt, shot.request, shots/); screenshot(inline=false) returns a path instead of image
content; and input waits for the game's acknowledgement line rather than a fixed sleep. A CLI front end over the same
functions, a wait-on-log-state tool and a GPU lock are the natural next steps, and none of them need this PR to change shape.

Settings and the cheat menu (added since this PR opened)

  • settings() reads every row of the Insert menu — Graphics, Gameplay and Controls — with its key, label, value, range
    and the text the player sees. When the cheat menu is on it also returns the Cheats page's eleven rows, the pouch's
    items, the 31 teleport destinations and the inventory's 85 items with their counts.
  • set_setting(name, value) goes through the menu's own setter: it clamps, applies a quality preset or a control layout,
    marks the settings Custom and writes hookshot_options.ini. A row that says it needs a restart still does.
  • cheat(name, value, item) runs a cheat row directly: heart protection, no-clip flight and the free camera; mark and
    return; the clock; teleports by name or index; the age; the pouch; the inventory (a wanted count, or -1 to
    unlock/relock).
  • launch(cheats=true) turns the cheat menu on with HOS_OOT_CHEATS=1, which is otherwise a line in hookshot_env.txt
    before start.

On the game side these are three live-input commands — !settings, !setting <key> <value>, !cheat <name> <value> <extra> — answered in the game log, which the server reads back. The rows cross the renderer table (API 8) from
gpu/vk/vk_menu.cpp through host/rx_backend.c to host/nn_hid.c. The WebGPU back end has no Insert menu and says so.

Testing found and fixed one real defect: two cheat actions in quick succession overwrote each other (the menu kept a single
atomic), so an inventory edit sent right after a pouch edit was lost. They are a queue now, and both are taken.

Real-game verification against a build of this branch (Vulkan, a copied save): 23 settings listed with ranges and text;
set_setting("frame_pacing", 1), set_setting("text_speed", 3) and set_setting("frame_generation", 0) applied, shown by
settings() and written to hookshot_options.ini; an unknown key refused; heart protection and the free camera toggled
and read back; cheat("set_time", 18) made the game log "cheats: requested 18:00"; cheat("pouch", item="Bombs", value=5)
logged "cheats: pouch Item_Bomb +5"; cheat("teleport", item="Kakariko Village") moved Link from Death Mountain Trail to
Kakariko Village (screenshot, and "cheats: teleport child to KakarikoVillage_Child ... request sent"); mark_position
logged the position; two actions sent back to back both ran.

The game-free suite: 134 passed, 11 failed, 14 skipped — the same eleven environment failures as origin/main in a
clean worktree.

Not included

No game files, dumps, builds, saves, logs or screenshots. Test runs stayed under work/, which is ignored.

@hckrman101

Copy link
Copy Markdown
Collaborator

Hi! This PR has merge conflicts with main now (a lot has landed today). Could you merge main in or rebase and resolve them? Happy to give it a proper review once it's up to date.

nicholasgower and others added 6 commits November 9, 2026 11:06
Add scripts/hookshot_mcp.py, a stdio MCP server using the official Python SDK. It launches Hookshot, drives HOS_INPUT_LIVE, requests on-demand screenshots and returns them inline.

Game-side seams: HOS_INPUT_SCRIPT_ONLY, analog stick tokens (LX/LY/RX/RY), !quit, HOOKSHOT_NO_ENV_FILE, RX_VK_SHOT_FILE and RX_SCREENSHOT_FILE.

Tests: tests/test_mcp_server.py plus script-only and analog cases in the input fixture. Docs include MCP host configuration and the new switches.

Assisted-by: DeepSeek V4.1 Flash

Co-Authored-By: DeepSeek V4.1 Flash <noreply@deepseek.com>
scripts/pr_check.py audits the branch before a pull request: game-derived paths, LF endings, Conventional Commit subjects and trailers, then the full game-free suite. It never stages, commits, pushes or calls GitHub.

tests/test_pr_check.py builds throwaway repositories with a bad and a clean commit, and the docs tables and contributor checklist now mention the command.

Assisted-by: DeepSeek V4.1 Flash

Co-Authored-By: DeepSeek V4.1 Flash <noreply@deepseek.com>
tests/test_shot_request.py extracts read_shot_request() from vk_replay.cpp and compiles a fixture for the missing, empty, trimmed, one-shot and buffer-edge cases behind RX_VK_SHOT_FILE.

Assisted-by: DeepSeek V4.1 Flash

Co-Authored-By: DeepSeek V4.1 Flash <noreply@deepseek.com>
tests/test_docs.py resolves relative Markdown links, verifies documented python and powershell commands name tracked files, and compares docs/mcp.md's tool/resource tables with the live MCP server. Vendored third_party docs are skipped.

Assisted-by: DeepSeek V4.1 Flash

Co-Authored-By: DeepSeek V4.1 Flash <noreply@deepseek.com>
A game folder made by HookshotBuilder or make_game.py keeps the game in
bin\, with hookshot.exe at the top being the setup launcher that opens the
installed builder (builder/setup/setup.c). --game-dir and
HOOKSHOT_MCP_GAME_DIR chose the launcher, so launch started the builder
setup instead of the game and no screenshot ever arrived.

resolve_paths now prefers bin\hookshot.exe and falls back to the top of the
folder for a development build, with a regression test for both layouts.

docs/mcp.md gives the V2 OpenCode configuration (mcp.servers, no enabled
field) and says that wait_ready returns on the first present, which on a
cold cache is a boot frame while the shaders are still being prepared.

Assisted-by: DeepSeek V4.1 Flash

Co-Authored-By: DeepSeek V4.1 Flash <noreply@deepseek.com>
scripts/hookshot_mcp.py gains settings(), set_setting() and cheat(), and
launch takes cheats=true (HOS_OOT_CHEATS=1) so the cheat menu is on without
editing hookshot_env.txt first.

The game side answers three live input commands. !settings dumps every
Insert menu row (its key, label, tab, value, range and the text the player
sees) and, when the cheat menu is on, its eleven rows with the pouch's
items, the teleport destinations and the inventory's items and counts.
!setting <key> <value> goes through the menu's own setter, so it clamps,
applies presets and control layouts and writes hookshot_options.ini, and a
row that says it needs a restart still does. !cheat <name> <value> <extra>
runs a cheat row the way Enter and the arrows do: the toggles, mark and
return, the clock, teleports, the age, the pouch and the inventory.

The rows cross the renderer table (settings_dump, setting_set and cheat in
MkvVkRenderer, MKWVK_API_VERSION 8) from gpu/vk/vk_menu.cpp through
host/rx_backend.c to host/nn_hid.c; the WebGPU back end has no Insert menu
and says so. Cheat actions are now a queue rather than one atomic: two
Enter presses in a frame, or an MCP cheat right after another, are both
taken instead of the second overwriting the first (found by testing the
tools against a running game).

Tests: the fake game answers the three commands and the tools are exercised
over MCP, tests/input_background_test.c covers the live command plumbing,
and native_seams checks the new functions. docs/mcp.md documents the tools,
the launch option and the commands.

Assisted-by: DeepSeek V4.1 Flash

Co-Authored-By: DeepSeek V4.1 Flash <noreply@deepseek.com>
@nicholasgower
nicholasgower force-pushed the feature/mcp-ingame-testing branch from 58da1b7 to 894961e Compare November 9, 2026 19:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants