10/11 Make CAT tests hermetic (no live API), ~13 min → ~15 s - #644
Open
mcfrank wants to merge 1 commit into
Open
10/11 Make CAT tests hermetic (no live API), ~13 min → ~15 s#644mcfrank wants to merge 1 commit into
mcfrank wants to merge 1 commit into
Conversation
… live API The five language CAT tests (english/french/dutch/japanese/spanish) replayed a recorded item/response/theta sequence but reached the live production R API to obtain each next item — so the suite required a network connection and was coupled to whatever parameters happen to be deployed. That coupling is exactly what had already broken the Spanish sequence test (the deployed API drifted to pre-rename item names), which was skipped as a result. A new cat_api_mock helper patches the view's cdi_cat_api to replay the recorded sequence (and to answer startItem from the start-items CSV) — the recording is the ground truth the API is meant to return, so this exercises the full view flow (render, store answer, advance, stop, persist theta) with no network. The theta the live API returned was rounded to 4 decimals; the mock matches that so the est_theta assertions hold. Each class is pinned to CAT_ENGINE=remote so it deterministically tests the remote view path. Result: the CAT suite drops from ~13 min (network-bound) to ~15 s, the previously-skipped Spanish sequence test is restored, and the tests pass even with CAT_API_URL pointed at a dead address (proving no live call). Also renames the Dutch class's copy-pasted test_spanish_* methods to test_dutch_*. Addresses #628. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Continues the stack (targets the security-round-2 branch; merge after #643). Addresses #628.
The five language CAT tests replayed a recorded item/response/theta sequence but reached the live production R API for each next item — so the suite needed a network connection and was coupled to whatever parameters are deployed. That coupling already broke the Spanish sequence test (deployed API drifted to pre-rename item names), which had been skipped.
cat_api_mockhelper patches the view'scdi_cat_apito replay the recorded sequence (and answerstartItemfrom the start-items CSV). The recording is the ground truth the API is meant to return, so this exercises the full view flow — render, store answer, advance, stop, persist theta — with no network.est_thetaassertions hold. Each class is pinned toCAT_ENGINE=remote.test_spanish_seq_1, and renames the Dutch class's copy-pastedtest_spanish_*methods totest_dutch_*.Result
CAT_API_URLpointed at a dead address (127.0.0.1:1) — a real call would fail connection-refused.🤖 Generated with Claude Code