Skip to content

Add MCP server for scripted in-game testing - #764

Closed
nicholasgower wants to merge 4 commits into
NotRexed:mainfrom
nicholasgower:feature/mcp-ingame-testing
Closed

nicholasgower wants to merge 4 commits into
NotRexed:mainfrom
nicholasgower:feature/mcp-ingame-testing

Conversation

@nicholasgower

@nicholasgower nicholasgower commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

What this adds

  • scripts/hookshot_mcp.py: a stdio MCP server built on the official Python SDK (mcp==2.3.0).
    • Launches a built Hookshot headless, muted, with a dedicated run/cache/save directory.
    • Tools: launch, stop, status, press, stick, sequence, screenshot, wait, logs.
    • Screenshots are returned inline as MCP image content and saved to the run directory.
    • Resources: hookshot://log, hookshot://last-screenshot, hookshot://status.
  • Game-side test seams:
    • HOS_INPUT_SCRIPT_ONLY=1 ignores physical pads, keyboard and mouse.
    • Analog stick tokens LX=, LY=, RX=, RY=.
    • !quit live command for a clean stop.
    • HOOKSHOT_NO_ENV_FILE=1 lets the MCP server own the environment.
    • RX_VK_SHOT_FILE and RX_SCREENSHOT_FILE request one PNG of the next present from the Vulkan and WebGPU renderers.
  • Tests: tests/test_mcp_server.py (protocol over stdio, input encoding, env handling, native seams, fake-game end to end), plus analog and script-only cases in tests/test_input_background.py.
  • Docs: docs/mcp.md with host configuration and tool usage; updated development, design, contributing and Vulkan notes.
  • CI installs mcp==2.3.0 for the new test.

Follow-up checks

  • scripts/pr_check.py: a read-only pre-PR gate. It rejects game-derived paths, CRLF endings, non-Conventional commit subjects and missing commit trailers, then runs the game-free suite unless --no-tests is given. It never stages, commits, pushes or calls GitHub.
  • tests/test_pr_check.py: throwaway-repository coverage for clean and bad PRs, including a .nup, CRLF content, missing trailers and an untracked game file.
  • tests/test_shot_request.py: extracts the production read_shot_request() from vk_replay.cpp and compiles a fixture for the missing, empty, trimmed, one-shot and buffer-edge cases behind RX_VK_SHOT_FILE.
  • tests/test_docs.py: resolves relative Markdown links, verifies documented python/powershell commands name tracked files, and compares this page's MCP tool/resource tables with the live server.
  • CONTRIBUTING.md and docs/development.md now point at scripts/pr_check.py; gpu/vk/VULKAN.md and docs/mcp.md point at the new fixtures.

Verification

  • python -I -B tests/test_mcp_server.py
  • python -I -B tests/run_all.py mcp --strict
  • python -I -B tests/test_input_background.py --cc gcc (all cases, including analog and script-only)
  • python -I scripts/hookshot_mcp.py --check --game-dir <fake-folder>
  • python -I -B tests/test_pr_check.py
  • python -I -B tests/test_shot_request.py --cxx g++
  • python -I -B tests/test_docs.py
  • python -I -B tests/run_all.py pr_check shot_request docs --strict

The Vulkan/WebGPU C++ changes could not be compiled in this environment because there is no game build, Vulkan headers or dump here. The input-parser changes were compiled and exercised through the existing fixture; the extracted shot-request parser was compiled and tested separately.

The full local tests/run_all.py --strict run gave 71 passed, 11 failed, 9 skipped; the failures are environment/tool gaps here (missing SPIR-V/Vulkan headers, clang/clang++, and Windows symlink privileges in nx2/test_nup_paths.py), not the new tests. All three new checks passed.

Known issue in the first commit

The first commit e9c2b0d predates the new pr_check gate. It is not a Conventional Commit and lacks the Co-Authored-By trailer, so python -I -B scripts/pr_check.py --base main --no-tests reports it. The three follow-up commits pass. I did not rewrite the published branch; if a rebase/amend is acceptable, the subject should become feat: add the Hookshot MCP server for scripted in-game testing and the body should end with both trailers from docs/design.md section 8.

No game files, builds, saves or generated screenshots are included.

@nicholasgower

Copy link
Copy Markdown
Contributor Author

I'm also working on a save file editor for this project, and I've decided I will not continue work on this until after the MCP server is merged. Easily automating user input is impossible without a functional MCP server, and I'd rather not combine PRs prematurely. #784

@hckrman101

Copy link
Copy Markdown
Collaborator

Nice work on this; the game-side seams in particular (script-only input, !quit, the one-shot screenshot request) are things I've wanted too.

I've been running something similar locally for a while, but as a few small command-line tools instead of an MCP server, so I wanted to share how that's turned out. I think the core of this would be more useful as a CLI, with MCP (if wanted) as a thin layer on top.

What I use, roughly:

  • oot hookshot [--dir D] starts a build with its own data folder (XDG_DATA_HOME), then gets into the newest save by itself. It appends presses to the HOS_INPUT_LIVE file, and decides when to press from the log rather than from sleeps: the per-second a frame: N draws line is about 2 in the intro, 15–70 in menus, about 5 while loading, and hundreds in play.
  • A run harness: every run gets its own folder with a copy of a pristine save, a log, and screenshots (RX_VK_SHOTS). Results are files, plus one summary line per run in a text file (fps, flash count from the frame log, …), so two builds can be A/B'd by diffing.
  • oot-status is a small GPU queue/lock. With several builds and test runs going at once, I needed runs to stop stepping on each other:
    • exclusive (alone on the GPU), shared (screenshot runs), timing (exclusive, and no builds running beside it, so fps numbers mean something), and build (no GPU slot);
    • FIFO with priorities;
    • "only while I'm not playing" for background runs.
  • hookshot-build wraps the build with a shared object cache across checkouts. A second checkout of the same commit builds in about 20 s instead of about 12 min.
  • oot-compare loads the same save in Hookshot and in an emulator and takes side-by-side screenshots. The save files are the same format in both. I wrote it up here: https://gist.github.com/hckrman101/701e15b643b7ce32f778116f3004b1a4

Why a CLI rather than only MCP:

  • Anyone can use it: a person in a terminal, CI, a shell script, or any coding agent, since all of them can run commands. An MCP server needs an MCP host to be useful, and it's harder to poke at by hand when something goes wrong.
  • It composes. Exit codes, stdout, and files on disk mean you can write launch && wait-for play && shot && stop, loop over builds, or diff two runs' logs, without a protocol in between.
  • Screenshots as files, not inline images. A path is cheap. Returning the PNG as image content each time is expensive for an agent's context and gone once the session ends. With files, you can look at the ones that matter, keep them for a PR, or compare them later.
  • State lives in a run folder, not in a long-lived server. If the tool or the agent restarts, the run is still there: pid, live-input file, log, shots. status just reads it back.
  • Easier to test. Each command is a plain function with arguments and output.

What I'd suggest: keep the seams as they are, and make scripts/hookshot_mcp.py's tools into subcommands of one script. For example, hookshot-test launch|press|stick|sequence|shot|wait|logs|status|stop --run <dir>, where launch writes the run folder and the rest read it. Then hookshot_mcp.py can be ~100 lines that call those same functions. Two things I'd add that have saved me a lot of time:

  1. wait on game state (log signals: draw counts, "loading", the first present) instead of seconds.
  2. A lock around GPU use, so parallel runs and builds don't skew each other's timings.

Happy to compare notes or help with any of it.

nicholasgower and others added 4 commits November 8, 2026 16:50
Add scripts/hookshot_mcp.py, a stdio MCP server using the official Python SDK. It launches Hookshot, drives HOS_INPUT_LIVE, requests on-demand screenshots and returns them inline.

Game-side seams: HOS_INPUT_SCRIPT_ONLY, analog stick tokens (LX/LY/RX/RY), !quit, HOOKSHOT_NO_ENV_FILE, RX_VK_SHOT_FILE and RX_SCREENSHOT_FILE.

Tests: tests/test_mcp_server.py plus script-only and analog cases in the input fixture. Docs include MCP host configuration and the new switches.

Assisted-by: DeepSeek V4.1 Flash

Co-Authored-By: DeepSeek V4.1 Flash <noreply@deepseek.com>
scripts/pr_check.py audits the branch before a pull request: game-derived paths, LF endings, Conventional Commit subjects and trailers, then the full game-free suite. It never stages, commits, pushes or calls GitHub.

tests/test_pr_check.py builds throwaway repositories with a bad and a clean commit, and the docs tables and contributor checklist now mention the command.

Assisted-by: DeepSeek V4.1 Flash

Co-Authored-By: DeepSeek V4.1 Flash <noreply@deepseek.com>
tests/test_shot_request.py extracts read_shot_request() from vk_replay.cpp and compiles a fixture for the missing, empty, trimmed, one-shot and buffer-edge cases behind RX_VK_SHOT_FILE.

Assisted-by: DeepSeek V4.1 Flash

Co-Authored-By: DeepSeek V4.1 Flash <noreply@deepseek.com>
tests/test_docs.py resolves relative Markdown links, verifies documented python and powershell commands name tracked files, and compares docs/mcp.md's tool/resource tables with the live MCP server. Vendored third_party docs are skipped.

Assisted-by: DeepSeek V4.1 Flash

Co-Authored-By: DeepSeek V4.1 Flash <noreply@deepseek.com>
@NotRexed

NotRexed commented Nov 9, 2026

Copy link
Copy Markdown
Owner

Closing to keep the queue clear — please rebase on current main and reopen; thanks!

@nicholasgower

Copy link
Copy Markdown
Contributor Author

Rebased onto current main (b2765ba) and opened again as #1025, with the real-game verification and the --game-dir fix from that testing included.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants