test(byllm): deslop the test suite: one fake LLM, table-driven clusters, and per-PR rules

[kugesan1105]

test(byllm): deslop the test suite: one fake LLM, table-driven clusters, and per-PR rules

Problem

The byLLM suite under jac/jaclang/byllm/tests/ is 244 tests in 6,308 lines plus 60 fixture programs. About 210 of those tests landed in 2026 across roughly 55 PRs, and only 2 PRs in that period consolidated anything. Every fix PR appends a sibling test and often a new fixture, so the suite grows at ~26 tests a month with no shared architecture.

What that produced:

• Six independent ways to fake the LLM: module-level MockLLM outputs lists (43 fixtures), patch.object(model, "model_call_no_stream" / "model_call_with_stream") with hand-built SimpleNamespace responses, patch("litellm.completion"), monkeypatched dispatch_no_streaming on a MockLLM, ad-hoc _serve closures, and _FakeTC objects that bypass the model. Root cause: MockLLM only scripts the non-streaming path, so anything touching streaming, tool-call chunks, or usage has to invent its own fake.
• Three fake MTRuntime scaffolds (_MockMTRun, make_fake_mt_run, and the seven-argument MTRuntime(...) constructor pasted 16 times), then re-wrapped by _mk_run, _mk_forced_run, _finish_env, _usage_from.
• Five builders for the same OpenAI chat-completion response shape (_finish_env, _mk_forced_run, _tc_chunk/_text_chunks, _chunk/_frags/_plain, _response). _finish_env (PR fix(byllm): structured-output robustness for finish_tool arguments and json_object-downgrading backends #8246) and _mk_forced_run (PR fix(byllm): finish_tool-related calls stop shrinking the tools array (Anthropic cache-prefix fix) #8386, one week later) are near-verbatim copies.
• In-file boilerplate: step_a and finish_tool defined 9 times in test_compaction.jac; capture_cb defined 8 times in test_telemetry_.jac; 45 tests in test_byllm.jac open with the identical five-line stdout-capture prelude; 10 fixtures assert inside the fixture and print a _PASS sentinel so the test checks only a string.
• Sibling tests where a row was needed: api key resolution (9 tests), output conversion errors (7), litellm debug config (7), model pool (6 tests over 5 fixtures), typed retry (5), issue 7657 (4, named after the ticket), tools-array cache prefix (4 with two duplicate helper families).
• Near-duplicate fixtures: react_max_iterations.jac vs react_max_iterations_finish_tool.jac (one output line differs), five model_pool_*.jac sharing one router mock, three logging_stream_*.jac sharing one skeleton.
• No TESTING.md for byLLM and an empty fixtures/__init__.py, so each author starts from scratch in whichever file they touch.

The good patterns already exist in the repo and are not being reused: the positions row table in jac/tests/langserve/test_server.jac, the LanguageServerTestHelper object in jac/tests/langserve/server_test/test_lang_serve.jac, extend-by-assert in jac/tests/language/test_language.jac, parametrize() from jaclang.testing.test, and within byLLM itself test_usage_cost.jac (5 dispatch paths, 2 tests, one PATHS table and one runner).

Target architecture

One support annex, tests/support_tests.jac (not test_* so it is never collected), that owns every fake:

• FakeLLM: a BaseLLM subclass that records every outgoing params dict and serves scripted replies on all five dispatch paths (dispatch_no_streaming, adispatch_no_streaming, dispatch_streaming, dispatch_streaming_with_tools, adispatch_streaming). Replies are declared as text, ToolCall, raw JSON, chunk stream, or an exception. This replaces the six faking seams. Where it makes sense, the same capability goes into the product's MockLLM so users get it too.
• response(...), tool_call_reply(...), chunks(...): the single OpenAI-shape builders.
• mk_run(...): the one MTRuntime factory with keyword overrides.
• run_fixture(name) -> str: imports a fixture and returns captured stdout; fixtures stop printing sentinels and the test asserts.
• capture_logs(...): the log-capture context used by the 7657 and debug tests.

Suite shape after the work: test_byllm.jac splits by subsystem (config_tests, output_tests, finish_tool_tests, streaming_tests, async_tests), each cluster is one table-driven test (a row list with a why label, or parametrize() when per-row reporting matters), and a regression is a new row or a new assert in the existing test for that behaviour.

Rules for every byLLM PR from now on (reviewers check these)

• A regression test is a new row or assert in the existing test for that behaviour, not a sibling test named after the issue.
• No new fake LLM, response builder, or MTRuntime constructor in a test file. Use support_tests.jac; extend it if a seam is missing.
• No new fixture program when an inline def ... by llm(...) in the test body covers the scenario. A fixture is for multi-module or compile-time behaviour only.
• Fixtures do not assert or print sentinels. The test asserts on returned values or captured output.
• A new test file only for a new subsystem, never for a new PR.
• PR description names which existing test each new row went into.

Phases

Phase 0 (P0, blocks everything else): the seam

• Add tests/support_tests.jac with FakeLLM, response, tool_call_reply, chunks, mk_run, run_fixture, capture_logs. (PR test(byllm): one fake model for the suite, and the first three clusters onto it (#8930 phase 0 and 1) #8936; the helpers are say/call/calls/finish/fail plus scripted, mk_run, run_fixture, load_fixture, capture_logs, tool_names.)
• Extend the product MockLLM so it scripts tool-call chunks and usage on the streaming paths Superseded: FakeLLM covers this for the suite and runs the real dispatch paths. Upgrading the user-facing MockLLM is a separate decision. Related: fix(byllm): async streaming crashes on every BaseLLM subclass without its own model_call_with_stream_async (MockLLM, LocalLLM, ModelPool) #8932 (async streaming crashes on every BaseLLM subclass without its own async stream call).
• Add tests/TESTING.md for byLLM Dropped by decision in PR test(byllm): one fake model for the suite, and the first three clusters onto it (#8930 phase 0 and 1) #8936; the rules live in this issue and in the support_tests.jac docstring.
• Migrate test_usage_cost.jac and test_truncated_tool_calls.jac onto the seam as the reference examples. (PR test(byllm): one fake model for the suite, and the first three clusters onto it (#8930 phase 0 and 1) #8936)

Phase 1 (P0): collapse test_byllm.jac (114 tests, 2,438 lines)

• api key resolution: 9 tests to 3 (precedence table, dispatch guard, format_prompt never prints the key). (PR test(byllm): one fake model for the suite, and the first three clusters onto it (#8930 phase 0 and 1) #8936)
• output conversion + configuration errors: 7 tests to 3 tables on mk_run. (PR test(byllm): one fake model for the suite, and the first three clusters onto it (#8930 phase 0 and 1) #8936)
• litellm debug config and exception logging: 7 tests to 3 (flag table, config defaults, provider-error table). (PR test(byllm): debug-config, typed-retry, and issue-7657 clusters become tables; non-streaming replies are real ModelResponses (#8930 phase 1) #8938)
• finish_tool healing + tools-array cache prefix: 9 tests and 7 private helpers to 4 table-driven tests; the unusable-argument case now also runs on the async and recovery paths. (PR test(byllm): one fake model for the suite, and the first three clusters onto it (#8930 phase 0 and 1) #8936)
• issue 7657 cluster: 4 tests to 2 tables on capture_logs. (PR test(byllm): debug-config, typed-retry, and issue-7657 clusters become tables; non-streaming replies are real ModelResponses (#8930 phase 1) #8938)
• typed retry: 5 tests to 2 via load_fixture. (PR test(byllm): debug-config, typed-retry, and issue-7657 clusters become tables; non-streaming replies are real ModelResponses (#8930 phase 1) #8938)
• stdout-capture tests: replace the 45 copies of the prelude with run_fixture done for 40 in PR test(byllm): one fake model for the suite, and the first three clusters onto it (#8930 phase 0 and 1) #8936; the 5 that capture warnings, loguru output, a nested fixture dir, or expect an exception keep their own handling. Still open: move the 10 _PASS sentinel assertions out of the fixtures into the tests.
• Split the remaining file by subsystem so a reader can find an existing cluster before adding one.

Phase 2 (P1): fixtures (60 programs, 43 building their own MockLLM)

• Merge react_max_iterations.jac and react_max_iterations_finish_tool.jac into one fixture with two entry functions.
• Merge the five model_pool_*.jac fixtures into one with a shared router mock; 6 tests to 1 table.
• Merge the three logging_stream_*.jac fixtures into one.
• Convert single-function fixtures (llm_params_temp, api_key_verbose, plain_text_recovery, on_iteration_callback, conversation_param, and similar) to inline by llm defs in the test body.
• Delete every fixture nothing references after the merges; add a guard test that fails on an unreferenced fixture.

Phase 3 (P1): the other files

• test_compaction.jac: hoist step_a / finish_tool (9 copies) to module level; one compaction_llm(outputs, ctx_window) helper for the 10 MockLLM constructions; ctx_window priority (4 tests) to 1 table.
• test_telemetry_.jac: one with_clean_registry() helper and one capture_cb factory (8 copies); callback registry tests (3) to 1; parent_invocation_id tests (2) fold into the record-fields table.
• test_parallel.jac (42 tests): move _FakeTC, _FakeTool, _MockMTRun into support; serialize/default/timeout attribute tests (8) to 1 table; gate-logic tests (4) to 1 table; hint/annotation tests (5) to 1 table.
• test_mtir_integration.jac: scope-name tests (3 trailing-letter cases) to 1 table; enum extraction tests (5) to 1 table.

Phase 4 (P2): boundaries

• Reconcile parent_invocation_id coverage between byllm/tests/test_telemetry_.jac and scale/tests/misc/test_llm_telemetry.jac so each behaviour is tested once at the layer that owns it.
• Decide the fate of validate_schema.jac (needs a live model and an API key; it is collected by directory runs). Either gate it behind an env var explicitly or move it to an examples/benchmarks folder.
• Record before/after wall-clock for jac test -d jac/jaclang/byllm/tests in this issue.

Acceptance

File	Now	Target
test_byllm.jac	114 tests / 2,438 lines	split into ~5 files, ~50 tests total
test_parallel.jac	42	~25
test_mtir_integration.jac	29	~20
test_compaction.jac	14	~10
test_telemetry_.jac	13	~8
fixtures	60 programs	~35
fake-LLM seams	6	1
MTRuntime constructors in tests	16 inline + 4 wrappers	1

Coverage must not drop: every deleted test maps to a row or assert in the PR that deletes it, and the PR description lists the mapping.

Progress

• PR test(byllm): one fake model for the suite, and the first three clusters onto it (#8930 phase 0 and 1) #8936 (merged 2026-09-04): Phase 0 seam plus Phase 1 clusters 1 to 3 and the prelude replacement. test_byllm.jac 2,438 to 1,960 lines, 114 to 99 tests; suite 186 passed. Review left two deferred items for the next Phase 1 PR: build non-streaming replies as real litellm.ModelResponse objects so both halves of the seam have the same fidelity, and give the seam exports without callers (load_fixture, capture_logs, exhausted) a first user or drop them.
• PR test(byllm): debug-config, typed-retry, and issue-7657 clusters become tables; non-streaming replies are real ModelResponses (#8930 phase 1) #8938 (merged 2026-09-04, c08bef2): Phase 1 second batch, 16 tests to 7; response() is now a real litellm.ModelResponse and the seam exports have callers, closing both deferred review items from test(byllm): one fake model for the suite, and the first three clusters onto it (#8930 phase 0 and 1) #8936. test_byllm.jac 1,963 to 1,803 lines, 99 to 90 tests, jac check clean.
• fix(byllm): async streaming crashes on every BaseLLM subclass without its own model_call_with_stream_async (MockLLM, LocalLLM, ModelPool) #8932 filed for the product bug the seam exposed; the FakeLLM async-stream override goes away when it lands.

Non-goals

• No behaviour changes in jaclang.byllm beyond extending MockLLM.
• No rewrite of validate_schema.jac content; only how it is gated.
• No changes to the scale telemetry store tests beyond the overlap in Phase 4.

[MalithaPrabhashana]

Opened #9002 to track the execution side of this, with measurements and verified mechanics rather than a competing plan. It does not replace this issue; three things in it are additive:

1. What the suite verifies, measured. 66 asserts live inside fixtures guarded by only 10 sentinel checks, so a fixture can be deleted and 12 checks vanish with the test count unchanged. Also: MockLLM replaces dispatch_no_streaming, above make_model_params and parse_response, so two functions with opposite sem strings return identical output and fixtures/llm_semstrings.jac has a -> int returning a str in CI today.

2. Verified conversion mechanics. Probed before writing code. by llm() does resolve its model global at call time, so a test can own the fake and the fixture stays pure, but the fixture needs glob llm: any = None;. Jac.jac_import returns a cached module, which contradicts the comment above test_byllm.jac:1618. Routing replies are generated handles (Agent_alpha), not indices. And load_fixture() plus root ++> g.Desk() fails jac check with E1097, so graph fixtures need a static import instead.

3. A gate as phase 0. The measured reason: this issue's phase-1 PRs removed 280 lines, and fix(byllm): usage/cost reaches every by llm() invocation shape #8479 added 1,046 back the same day on the old seams, deleting test_usage_cost.jac (the file test(byllm): one fake model for the suite, and the first three clusters onto it (#8930 phase 0 and 1) #8936 had just migrated and this issue cites as the pattern to copy). Net so far is +766 lines. A TESTING.md plus a ratchet guard test is about 60 lines and is the only piece that stops that.

Two suggestions for this issue's acceptance table, whichever tracker we keep:

• Add "every consolidated cluster asserts on llm.sent(...) or routes its reply through parse_response". Without it, 53 tautological fixture tests can be folded into one 53-row table and every number in the table is met while the suite checks the same things.
• Add an assert-count floor (baseline 842) alongside the test count, for the reason in point 1.

Two corrections to the problem statement, both from a local run:

• validate_schema.jac is not collected by directory runs. It has no test_ prefix, and the collected count is 200 without it.
• There are no orphaned fixtures. All 66 are referenced; most of the ones that look dead are named by path string from test_mtir_integration.jac.

Happy to fold #9002 into this issue and close it if you would rather keep one tracker.

[MalithaPrabhashana]

Correction to my comment above. I cited #8479 as inflow that ignored the new annex. It was opened 2026-08-22, twelve days before #8936 existed, so its author could not have used it. That example was unfair and I have withdrawn it from #9002.

The evidence that actually matters is the live queue. Nine open PRs touch byllm/tests/ right now, adding about 2,600 lines with 4 new fixtures, 15 SimpleNamespace fakes and 3 MockLLM constructions. Two of the nine (#8953, #8955) use the annex. Most predate it, so nobody is ignoring a rule; nothing tells an author at rebase time that the rule changed. Table in #9002 section 2.4.