test: add GUT test suite, generator golden files and headless CI - #138
Merged
Conversation
Adds GUT (Godot Unit Test) 9.7.1 at addons/gut/, enables the plugin, and
adds a .gutconfig.json pointing at res://test/unit/.
Vendored rather than pulled from the Asset Library or a submodule:
- The Asset Library has a single Godot 4 entry and it tracks 9.6.1 for
Godot 4.6. Twitcher targets 4.7, which needs 9.7.1 from the godot_4_7
branch.
- Upstream explicitly discourages submodule use ("GUT's file structure is
not organized to be used as a Git submodule"), and the symlink workaround
is fragile on Windows runners.
Vendoring also keeps CI free of a network dependency.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014crH3h7Nz27xtYCULUJKbx
Twitcher leaks process-global state aggressively, so suite isolation needs explicit help rather than good intentions. StaticStateGuard snapshots and restores 17 slots between tests: - singleton `instance` vars on TwitchService, TwitchAPI, TwitchEventsub, TwitchChat, TwitchBot and TwitchMediaLoader - static registries that production code never clears: ALL_COMMANDS, all_rotational_messages, _open_tracked_redemptions, HTTPServer._servers, TwitchLoggerManager.log_registry - the six static logger callables installed by the set_logger cascade — TwitchAuth._init() alone rewrites three of them as a construction side effect - ProjectSettings keys under twitcher/, written whenever any TwitchLogger is constructed Without this, suites pass alone and fail together, which is the worst kind of failure to debug. TwitcherTest wires the guard into before_each/after_each and adds scratch directories under user:// (so token caches and key stores stay off the developer's real filesystem), fixture loading, and two assertions the upcoming DTO round-trip suites need. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014crH3h7Nz27xtYCULUJKbx
The filter predicate was inverted:
used_scopes.filter(func(s): return scopes.find(s) != -1)
That keeps exactly the scopes named for removal and discards everything
else — the precise inverse of the contract. remove_scopes([A]) on [A, B]
returned [A].
The scope set builds the OAuth authorization URL, so the user was prompted
for the wrong permissions and the failure surfaced later as a 401 from an
unrelated endpoint.
Adds 19 tests covering ssv_scopes, add_scopes, remove_scopes, round-tripping
and signal emission. One of them pins that remove_scopes emits
scopes_changed twice (property setter plus explicit emit) — harmless today,
but a decision rather than an accident.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014crH3h7Nz27xtYCULUJKbx
TwitchScope.CHAT_READ was defined with value "chat:edit" and the "Send chat messages" description, and CHAT_EDIT with "chat:read" and the "View chat messages" description. SCOPE_MAP maps "chat:read" to CHAT_READ, so SCOPE_MAP["chat:read"].value was "chat:edit" — anyone requesting read access to chat was silently granted write access instead. Rather than asserting only the two swapped entries, the new suite checks key/value consistency across all ~90 SCOPE_MAP entries, plus uniqueness, non-null definitions and non-empty descriptions. The next copy-paste error in a hand-maintained table of that size is then caught for free. Also covers Definition.get_category, get_all_scopes/get_grouped_scopes partitioning, and sort determinism. 16 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014crH3h7Nz27xtYCULUJKbx
RegexUtil.escape is the purest function in the addon and feeds TwitchCommandContains' word-boundary matching, so a miss there means a chat command silently stops firing — or a user-supplied trigger gets interpreted as a regex. 10 tests: every metacharacter table-driven, backslash-before-everything-else ordering, and a round-trip asserting the output both compiles and matches its source literally and nothing more. Two are documentation rather than bug reports: - hyphen and forward slash are deliberately not escaped (only special inside character classes, which escape() never emits) - a trigger ending in a metacharacter can never match under match_word, because \b cannot anchor between '+' and a space. escape() handles "c++" correctly; the \b...\b wrapping applied on top is what breaks. Flagged here so the TwitchCommandContains suite either fixes it or records it as accepted. Also adds 9 meta-tests for StaticStateGuard. It is the load-bearing piece of the harness — if it silently stops restoring a slot, failures become order-dependent — so it is tested like production code. One test asserts every declared slot still resolves to a real script, so a rename cannot quietly disable protection. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014crH3h7Nz27xtYCULUJKbx
Sets up Godot 4.7 via chickensoft-games/setup-godot, imports the project, runs the unit suite headless and publishes JUnit results. Two non-obvious pieces: - The import step is mandatory and tolerates failure. Without a populated .godot/ the global class list is empty, so class_name lookups and GUT's script doubling fail with parse errors. The first import routinely exits non-zero on benign warnings, so the GUT step is the real gate. - GODOT_DISABLE_LEAK_CHECKS=1, because Godot reports leaked ObjectDB instances at shutdown and Twitcher's @tool scripts trip it on a green run. GUT exits 1 on any failure, so no extra wiring is needed to fail the build. Note it also exits 1 on an unknown CLI flag — a typo would fail the build with a passing suite. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014crH3h7Nz27xtYCULUJKbx
.gitattributes export-ignores everything outside /addons, so vendoring GUT at addons/gut/ would have shipped the whole test framework to anyone installing Twitcher from the Asset Library. It has to live under /addons for Godot to load it as a plugin, so exclude it explicitly instead. test/ was already covered by the blanket /** rule. Also commits the .uid sidecars Godot generates for the new test scripts, matching how the rest of the repo tracks them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014crH3h7Nz27xtYCULUJKbx
Twitcher ships ~44k lines across 289 generated files. Covering that by testing
the output would mean 289 near-identical test scripts; covering the two
generators that produce it costs three small OpenAPI fixtures.
The seam is already in TwitchAPIParser.parse_api():
if definition == {}:
definition = await _load_swagger_definition()
Assign definition up front and the HTTP fetch never happens. Everything
downstream — component_code, group_code, from_json_code, path_code, iter_code,
get_type — is pure String -> String. GeneratorHarness wraps that, and never
calls generate_api() or write_output_file(), so the real
addons/twitcher/generated/ is never at risk from a test run.
It also frees the BufferedHTTPClient that TwitchAPIParser creates in a variable
initialiser. That client is only ever added to the tree inside
_load_swagger_definition(), which the injected definition skips, so it would
otherwise leak as an orphan on every test.
Three fixtures: spec_minimal (scalar type mappings, required fields),
spec_grouped (Response/Opt grouping, typed and primitive arrays, inline
sub-objects, pagination) and spec_renamed_fields (the full _update_name table).
regenerate_goldens.gd shares GeneratorHarness with the test suites, so a golden
can never be produced by a different code path than the one asserting on it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014crH3h7Nz27xtYCULUJKbx
32 tests over the parser's object model plus snapshot tests on generated source. test_api_parser.gd pins the model rather than the output, so a mapping bug reports as "String became Variant" instead of a hundred-line diff: the full JSON-Schema type table, required/optional splitting, typed vs primitive arrays, inline objects promoted to sub-components, the allOf-takes-first limitation, the field rename table with its _original_name counterpart, and TwitchGenParameter.sort. The sort tests matter more than they look: parameter order is part of the public signature of every generated method, and sort_custom is not documented as stable. One test asserts the comparator is antisymmetric for distinct names, which is what makes stability irrelevant. test_golden_files.gd compares emitted source against checked-in goldens, and additionally asserts that every .golden and spec_*.json on disk is claimed by a case — a stale fixture fails the build rather than rotting quietly. The diff reporter prints the first differing line with context instead of dumping two files into the test log. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014crH3h7Nz27xtYCULUJKbx
TwitchEventsubGenerator is a ~90% duplicated fork of TwitchAPIGenerator.
Three of the differences are defects rather than decisions. These tests fail
when a divergence is fixed as well as when a new one appears — each is a
choice that should be made explicitly, and every assertion says which kind it
is.
1. Renamed fields lose their wire name. TwitchGenField renames identifiers
that are illegal in GDScript (animated -> animated_format, 1 -> _1,
source-only -> source_only) and keeps the original. The API generator emits
the wire name on both the track_data and d.get() sides; EventSub emits the
sanitised name on both. Latent — no EventSub schema hits the rename table
today — but it becomes silent data loss the day Twitch adds one.
2. Array fields are never tracked. Both generators fill arrays with
result.x.append(...), which mutates in place and so never runs the property
setter. The API generator compensates with an explicit track_data after the
loop; EventSub does not. This one is live in shipped code — see the
companion round-trip test.
3. Dead fully-qualified-name path. EventSub's get_type() reads
component.get_meta("fqdn"), and nothing anywhere calls set_meta("fqdn"), so
full_qualified is indistinguishable from false. The API generator uses a
Callable that is actually assigned.
Also pins the intentional differences (TwitchES prefix, suffix sets, output
folders) and two cosmetic ones (untyped `var path`, an off-by-one page
boundary in EventSub's iterator) so that unifying the two generators shows up
as a visible change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014crH3h7Nz27xtYCULUJKbx
TwitchData is the base class all 289 generated files extend, so its behaviour is the contract for ~44k lines. 20 tests over track_data/to_dict/to_json. The load-bearing case is null-erase: null means "not set", not "set to null", which is what keeps to_dict() sparse so a PATCH-style body does not clobber unset fields. Three tests also pin surprises that would otherwise be found in a debugger — to_dict() returns the live dictionary rather than a copy; to_json() sorts keys alphabetically because JSON.stringify defaults to sort_keys=true, while to_dict() keeps insertion order; and Godot's JSON parser has no integer type, so a round-tripped int comes back as float. test_generated_dto_roundtrip.gd samples the shipped output and demonstrates that the missing track_data on EventSub arrays is not theoretical: ESChatNotification.Event parses a two-element badges array correctly and reads back fine via the property, but to_dict() omits it entirely, while a scalar field on the same object survives. Anything that re-serialises an EventSub DTO — logging, caching, forwarding — loses every array field. Same for primitive arrays like fragment.format. Left failing-as-documented rather than fixed: the fix belongs in TwitchEventsubGenerator.from_json_code() followed by regenerating 94 files, which is its own change. The test asserts current behaviour and says what to flip when that lands. Also registers test/generator with the runner and CI. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014crH3h7Nz27xtYCULUJKbx
GUT Results126 tests 126 ✅ 0s ⏱️ Results for commit d77a71c. ♻️ This comment has been updated with latest results. |
The earlier commit titled "vendor GUT 9.7.1" set up .gutconfig.json and
enabled res://addons/gut/plugin.cfg in project.godot, but never actually
added the framework, so every test invocation died on:
ERROR: Can't load script: res://addons/gut/gut_cmdln.gd
These are the addons/gut/ files from the godot_4_7 branch at 9.7.1
(bitwes/Gut aeb5d4f), copied verbatim with no local modifications.
.gitattributes already export-ignores addons/gut/**, so this does not
reach Asset Library consumers. Note that export-ignore only affects
git archive output — it never had any bearing on CI, which clones.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a GUT-based test suite to Twitcher, plus a headless CI job that runs it on
push and pull request. 126 tests across 9 scripts, 509 asserts, green on CI.
Two of the commits are production fixes surfaced while writing the tests, and
are the parts worth reviewing closely:
fix: remove_scopes kept the scopes it was asked to remove—OAuthScopes.remove_scopes()inverted its filter.
fix: CHAT_READ and CHAT_EDIT held each other's values— swapped constants inTwitchScope.The rest is additive test infrastructure:
addons/gut/— GUT 9.7.1, vendored verbatim from the upstreamgodot_4_7branch. Vendored rather than taken from the Asset Library, whose single Godot 4
entry tracks 9.6.1 for Godot 4.6 while Twitcher targets 4.7.
.gitattributesexport-ignores it so it does not ship to Asset Library consumers.test/helpers/—TwitcherTest(extend this, notGutTest) andStaticStateGuard,which snapshots and restores 17 slots of process-global state between tests: singleton
instancevars, static registries production code never clears, the six static loggercallables installed by the
set_loggercascade, andProjectSettingskeys written as aside effect of constructing any
TwitchLogger. Without it, suites pass alone and failtogether.
test/generator/— golden-file tests for the two code generators. Twitcher ships~44 000 lines across 289 generated files; rather than 289 near-identical test scripts,
these test the generators that produce them, via a harness that pre-assigns the swagger
definition so no HTTP happens and
addons/twitcher/generated/is never written.test/unit/— mirrorsaddons/twitcher/. Covers the OAuth scope handling, the scopemap,
RegexUtil.escape,TwitchDataand generated DTO round-trips, and the state guard itself..github/workflows/tests.yml— Godot headless, with import-artifact caching andJUnit results published as a check.
Test plan
Run locally against Godot 4.7.2 and on CI against 4.7.0; 126/126 pass on both.
test/README.mddocuments the harness, the conventions, and the gotchas — notably thatheadless is canonical, since
Engine.is_editor_hint()gates behaviour in six files and theeditor panel can take different code paths than CI.
🤖 Generated with Claude Code