[houman44]

Goal card: goal_cejel_v27_a1_test_count_overmatch_2026-11-01.md (propose-only; land with a merge commit).

CONSTRAINTS-VERSION: 2026-08-01.5

Done: A. Name every removed file
Done: B. Find the over-matches
Done: C. Ratio consistency

Everything here is behind the v27 gate (usesV27FalseAssuranceClosure) or a default argument. v17 and every other rubric are unchanged.

A. Every file a v27 rule rates hollow is named

Under witan-rubric-v27-prospective-2026-10-19, A1's note gets one sentence that names each file a v27 rule moved from non-hollow to hollow, with the rule. For example:

2 files that v17 rates non-hollow are rated hollow here because they run no test: tests/test_cache.py (run-wide skip: hook in tests/conftest.py), tests/test_lru.py (run-wide skip: hook in tests/conftest.py).

• Rules named: conftest, run-wide skip: hook in <conftest.py>, run-wide skip: pytestmark in <file>, every test skipped.
• Note length: the note stays within the 1000-character limit. It lists as many files as fit and ends with , and N more not listed. Beside a long unresolved-coverage note, that note shortens through its own machinery so that the first named file still fits.
• Evidence: A1's evidence cites each such file, the first eight, with the label Rated hollow under this rubric: runs no test (<rule>). For every test skipped the citation is at the line of the first skip.
• What was missing before: before this change a module-level pytestmark skip produced no finding and no citation at all.

B. Over-matches found

These are synthetic fixtures in src/witan/__tests__/fixtures/v27-a1-removal-fixtures.ts. Before this change, v27 rated a file hollow (or a whole suite skipped) in each of these, though the condition did not hold:

1. conftest: python_files names conftest.py, so pytest collects its tests. The conftest.py was still removed and rated hollow.
2. run-wide skip, hook: pytest_collection_modifyitems(session, config) takes no items parameter and loops over a module-level items = []. The whole suite was read as skipped, and A1 went critical.
3. run-wide skip, hook: the hook runs items.clear() and then restore_sorted(items, saved), a helper that puts the items back. The whole suite was read as removed, and A1 went critical.
4. run-wide skip, pytestmark: pytestmark = pytest.mark.parametrize("backend", [pytest.param("redis", marks=pytest.mark.skip), "memory"]) skips one parameter only. The file was rated hollow.
5. run-wide skip, pytestmark: the same as 4, with the parametrize mark in a list.
6. every test skipped: every test_* is skipped, but python_functions = ["test_*", "check_*"] also collects a live check_get. The file was rated hollow.
7. every test skipped: every test is skipped, but addopts = "-p no:skipping" disables the skip marks. The file was rated hollow.

Fixes:

• Each rule now withholds where the setting it depends on may differ from pytest's default: python_files, python_functions / python_classes, or no:skipping. These are read from pytest.ini, .pytest.ini, pyproject.toml, tox.ini, setup.cfg or a CI workflow, and only where a Python test-shaped file exists.
• A hook must name an items parameter on its def line.
• A clear is not read where the hook hands the list itself to other code (a bare items that is not subscripted, a method call, in items, or the keyword name items=).
• A skip in pytestmark counts only as the value itself or as an item of the list or tuple the value opens with. A skip inside another mark's arguments does not count.

Already handled before this change, kept as regression tests:

• conftest-like names: test_conftest.py, conftest_test.py, tests/conftest/test_x.py. A non-root conftest.py that defines test_ functions under the default python_files is correctly rated hollow, and is now named.
• hooks: filters on a marker, a name or a comprehension; conditions on sys.platform, os.environ or an option (both if … return and run_all or …); a skipif marker; an autouse fixture that skips conditionally.
• pytestmark: a list with a conditional skip, a list with a skipif, a skipif on a name, a skip inside a class, parametrize ids containing "skip".
• per-test skips: one skipped test among several; parametrize ids containing "skip"; python_classes alone.
• JavaScript: describe.skip beside a live suite; it.skip beside a local it( helper or an imported helper; it.each with a case named skip.

The hook regression fixtures did differ from v17 before this change, but only in test_to_source_ratio, through their conftest.py. That difference is item C, not the hook rule.

Left open (not a v27 rule): a JS file holding a describe.skip beside a live describe is rated hollow by v17's SUITE_LEVEL_SKIP_PATTERN, under every rubric. The constraints keep v17 byte-identical and do not let v27 rise above v17, so this card cannot fix it. I tried to file a later: issue for it, but the harness needed approval for gh issue create, so the issue was not filed. It needs an issue opened by the operator.

C. Ratio consistency

The only rule that can lower test_to_source_ratio is the conftest exclusion: the other two rules touch only the non-hollow share. The operator ruled that both real-world ratio drops were wrong, so I concluded the ratio should not move for a conftest.py.

• What the ratio counts: the ratio measures test surface, and a conftest.py's fixtures are test surface. Under v27 the ratio is now measured over v17's test list, as the non-hollow share already was.
• What still changes under v27: a conftest.py is still not cited as a test file, and an asserting one is still rated hollow (now named).
• The conftest-only case: where conftest.py files are the only test-shaped files, the note says the ratio counts them although no concrete test file was found.
• The declaration: the v27 declaration in rubric-version.ts now says the same thing, and a test checks that it does.

This is my reading of the ruling, and the operator can reverse it: reverting the metric line to allTestFiles.length restores the old drop.

v27 pins that moved (v27 only)

• In false-assurance-v27.test.ts, three tests pinned v27's old conftest ratio: the ratio without the conftest, or 0. They now pin v17's ratio. The conftest-only test also pins the new note sentence.
• In v27-no-rise.ts, lostFindings exempted A1's "Test suite files are present" finding using "v27 ratio is 0" as a stand-in for "the only test-shaped files are conftest.py". Item C made that stand-in false. The exemption now reads the same premise from v27's A1 evidence: no Detected test file citation, and a pytest configuration (conftest.py), not a test file citation. The exemption is no broader than before.
• No behaviour fingerprint moved: rubric-behaviour-fingerprint.test.ts passed unchanged.

Verification

• Type check: pnpm run typecheck is clean.
• Build: pnpm run build succeeds.
• Lint: this repository has no lint script in package.json and no lint step in CI, so lint was not run.
• New tests: src/witan/__tests__/v27-a1-removals.test.ts (85 tests) failed first. The red commit e7b1e4f86 had 25 failing, then the fix made them pass.
• Existing tests: the existing v27 suites pass, including false-assurance, the shards, declared scope, the generated no-rise suites and the note-limit tests.
• Full suite (pnpm exec vitest run) on bb7b6f3f8: 6165 passed, 1 skipped, 2 failed. Neither failure touches A1:
  • src/__tests__/maeve-lesson-delivery.test.ts: a seed already on main, PENDING_cejel_name_the_repo_on_external_commits_in_preregistrations_2026-10-23.json, is 8 days old against the 7-day bound. That is the courier-stopped condition, and it needs a manual harvest.
  • src/witan/__tests__/ingest.test.ts ("rejects a non-regular explicit ingest candidate"): on this host tmpdir() is /tmp, a symlink, so the scan refuses with "symlinked ingest directory" before the non-regular check runs. It is host-specific.

Lesson

Staged as docs/orchestration/maeve-unanchored-lessons/PENDING_cejel_a-rule-that-removes-from-a-count-must-name-what-it-removed_2026-11-01.json (the pre-commit validator accepted it). A rule that removes items from a count cannot be checked until it names what it removed. A rule that reads a tool's default must withhold wherever the repository may set that default.

{"dunstan": "0.1", "headCommit": "bb7b6f3f80d4795b4daa6847edfba74ed8134822", "filesChanged": ["docs/orchestration/maeve-unanchored-lessons/PENDING_cejel_a-rule-that-removes-from-a-count-must-name-what-it-removed_2026-11-01.json", "src/witan/__tests__/false-assurance-v27.test.ts", "src/witan/__tests__/fixtures/v27-a1-removal-fixtures.ts", "src/witan/__tests__/v27-a1-removals.test.ts", "src/witan/__tests__/v27-no-rise.ts", "src/witan/repo-signals.ts", "src/witan/rubric-version.ts"]}

No tests member: this repository has no turbo pipeline and no ci-full.yml JUnit record, and its one package's suite is the full suite, so no matching record exists to declare.

🤖 Generated with Claude Code

[houman44]

Review at head bb7b6f3

This is an independent adversarial pass: 31 probe repos scored at this head, at the merge base 206074f and at current main, with pytest 9.2.0 run for the ground truth. I reproduced finding 1 myself. Two blockers. Both are new false negatives this PR introduces, and both hand back v17's full A1 credit with nothing in the note.

What "withhold" means here. A withheld rule falls back to v17's reading of the file: full credit, not an abstention, and no disclosure. On a hook that skips every test, A1 goes from 1.7 critical to 2.3 verified. So every withhold below is a silent credit.

Blockers

1. The common deselect-everything hook is no longer caught. In handsOnPytestItems (repo-signals.ts, diff +416-430), any reference to items other than items[, items., in items or items= cancels the clear. Reproduced with the same repo and only the conftest hook changed:

   tests/conftest.py hook	pytest	v27 A1
   items.clear()	no tests run	1.5 critical
   dropped = list(items), items.clear(), config.hook.pytest_deselected(items=dropped)	2 deselected, no tests run	2.1 verified

   • The reviewer saw the same with len(items) logged after the clear, and with a no-op noop(items) call.
   • The merge base caught all three (1.7 critical).
   • Suggest treating a name read as an argument as harmless unless the call could write to the list after the clear. At least allow list(items), tuple(items), len(items) and items[:]. Pin the deselect idiom as an excluded fixture.

2. Any mention of a pytest setting cancels the rules.

   • readPytestCollectionSettings (+329-344) runs bare \bpython_files\b, \bpython_(functions|classes)\b and \bno:skipping\b searches over the whole text of every settings file and CI workflow. A comment, an unrelated section or an echo string all count.
   • Each of the following returned v17's credit (A1 1.7 critical to 2.3 verified, or the share 2/3 to 3/3):
     • a setup.cfg with [metadata] and the comment # addopts = -p no:skipping;
     • a docs workflow run: echo "see pytest -p no:skipping docs";
     • a CI line run: pytest # python_classes left default;
     • python_functions = "test_*" (pytest's own default) in pyproject;
     • [pytest] python_files = test_*.py (the default) in tox.ini.
   • Suggest:
     • ignore comments;
     • read only the [pytest], [tool:pytest] and [tool.pytest.ini_options] sections, and workflow lines that run pytest or set PYTEST_ADDOPTS;
     • treat a default value as default;
     • say in A1's note when a rule was withheld.

Should-fix (over-matches checked against pytest; they predate this PR, which leaves them)

• --noconftest is not read. With addopts = "--noconftest", pytest ran "2 passed", but v27 still rates both files hollow (A1 2.3 to 1.7 critical). It is the same kind of setting as no:skipping and should withhold the hook rule.
• A root conftest pytest never loads. With a repo-root conftest.py that skips everything, pkg/pytest.ini, and CI running cd pkg && pytest, pytest ran "1 passed" because confcutdir is the ini's directory. v27 rates the test hollow.
• python_files set where Cejel does not look, such as a Makefile pytest -o python_files=... or a noxfile. pytest collects the conftest's tests ("3 passed"), but v27 still rates the conftest hollow.
• No RUBRIC_CHANGELOG.md or CHANGELOG.md entry. leaderboard/RUBRIC_CHANGELOG.md says a scoring change without one "is a bug, not a release".

Nits

• A black-wrapped hook def line (with items on the next line) is now ruled out explicitly by pytestHookTakesItems. The base missed it too.
• The note's file list is ambiguous when a path contains , or (. Only the first eight evidence pointers carry exact paths.
• items.clear(); items.__iadd__(saved) and pytestmark = [pytest.mark.skip for _ in ()] are still rated hollow. Both are contrived.

Holds

• Tests. The PR's tests pass (1031). The mutations each caught: dropping the items-parameter requirement, dropping the python_files withhold, and disabling handsOnPytestItems.
• Other rubrics. v17 to v26 are byte-identical against both the merge base and current main: 310 reports, 31 probes × 10 rubrics. No v27 score rises above v17 on any probe.
• Over-matches fixed. Confirmed: a nested parametrize skip, a hook with *args or no items parameter, a subset deselect, importorskip, a skipif list, and a skipped class beside a live function.
• Still caught. A hook with a keyword-only or annotated items parameter.
• Note and ratio. Files are named in file order with their rule, unicode survives, and evidence is capped at eight. test_to_source_ratio now equals v17's.
• Public surface. No internal names in the added lines.

Follow-ups filed

• later: v17 rates a JS test file hollow when it holds describe.skip beside a live describe #623: the v17 describe.skip over-match this PR's author could not file.
• fix(action): recognise the v27 "Unrated" abstention labels; main is red since #618 and #619 #624: the Action label fix. build-test here is red only from the main breakage that fix(action): recognise the v27 "Unrated" abstention labels; main is red since #618 and #619 #624 fixes.

Merging is the operator's call. I'd hold this PR for the two blockers.

[houman44]

Held for the two blockers in the review; superseded by goal_cejel_v27_a1_test_count_overmatch_v2_2026-11-02, which starts from this branch.