engineering-patterns.json

[
 "np.array(list_of_tuples) silently builds a 2-D array, not a 1-D object array of tuples; iterating it yields row-arrays and dict/DataFrame lookups then fail with KeyError. Fix the wrapping (keep a plain list, or fill an explicit object array) rather than the labels.",
 "pandas MultiIndex columns mean column labels are TUPLES. Code that assumes scalar labels (string keys, str() calls, np.array wrapping) breaks on them. Handle tuple labels as first-class, preserve them.",
 "Never fix a lookup error (KeyError/IndexError) by changing the IDENTITY of the data — stringifying tuples, joining keys, renaming labels. That cures the symptom by amputation and violates the library contract. Fix the code path that mishandles the identity instead.",
 "Before submitting, re-read your diff and ask: does this patch change the TYPE or identity of anything public (attribute, return value, label)? If yes, the fix is probably wrong even though your reproduction passes.",
 "Prefer the smallest patch that makes the failing case pass: a fix that deletes or bypasses the problematic wrapper is usually more correct than one that transforms the data to survive the wrapper.",
 "Agent scaffold: when a verification GATE auto-runs a check and then REJECTS the agent's declare/submit, feed back the check's SPECIFIC diagnostic, never a generic \"gate not passed\". A generic rejection gives the agent nothing to act on, so it re-declares or thrashes the same tool until its turn/token budget dies (observed: sphinx phase-1 bootstrap burned 20-36 run_smoke_test turns into a 50-turn budget death on envs that were actually fine). Capture the checker's result and surface the actionable reason (missing test dependency, uncollectable suite, no green test). Keep the payload byte-identical when there is no extra detail so unrelated gate paths (e.g. a different phase) are provably unaffected.",
 "Test-runner flags must be version-gated to the tool era of the code under test. A benchmark harness runs each instance against ITS OWN pinned test-framework version, so a CLI flag that is valid for your newest instances can be a fatal usage error for older ones. Concretely: pytest's --no-header/--no-summary exist only in pytest>=6.0; on pytest 4.x/5.x they are 'unrecognized arguments' -> exit code 4 with ZERO tests collected/run. If that flag lives in THE single test path used for SCORING, every old-framework instance is silently auto-scored UNRESOLVED regardless of whether the patch is correct (a whole false-negative family). Detection signature: misses whose captured tail is a bare framework header line (e.g. 'rootdir: ...') with no pass/fail counts, clustered in the repo that pins the oldest framework. Fix: probe the framework major version once per environment (CACHE KEYED ON THE ENV DIR, never on a relative interpreter path that is identical across instances in a long-running multi-instance process) and include era-specific flags only when supported. Verify by running the SHIPPED scoring function (not an ad-hoc command) on one affected instance and confirming the result flips, plus a newest-version instance to prove no regression, interleaved in one process to expose cache-key collisions.",
 "A test suite that fails to COLLECT (ImportError / VersionRequirementError / \"needs at least <framework> vX\" / \"found no collectors\") instead of failing an assertion is an ENVIRONMENT problem, not your code — never edit source to silence it. Testing an OLD framework version requires its plugin/theme/support ecosystem (e.g. sphinxcontrib-*, alabaster, pyparsing) pinned to that version’s era; the newest release of a support package often hard-requires a newer framework and aborts collection. Let the harness own env setup and get a green baseline (run_smoke_test with no args) before trusting any repro.",
 "Warnings-as-errors suite still dies at COLLECTION after you pinned the obvious too-new dep? Look for a SECOND too-new pure-python dep that raises a DeprecationWarning during import — especially version machinery. A newer package can HIJACK a library's entry point (e.g. a 'vcs-versioning' fork taking over setuptools_scm's version-scheme entry points) and turn a config value the project still uses (an old scheme name like 'release-branch-semver') into a fatal deprecation at import. Downgrading the primary package is NOT enough while the hijacking sibling stays installed — you must UNINSTALL it, not just pin. Reproduce the reference env by clearing EVERY too-new build/version dep, not only the first. Whether it bites can be version-dependent (only builds that invoke the machinery at import), so an env fix that is a no-op for some instances is still correct for all.",
 "Auditing a coding-benchmark scorer for FALSE POSITIVES needs a two-state CAUSATION test, not a bare re-run of the withheld checks. SWE-bench resolution = all FAIL_TO_PASS flip to pass AND all PASS_TO_PASS stay green; a home scorer that runs only FAIL_TO_PASS can mark a solve resolved even if the patch broke a P2P. But you CANNOT detect that by just re-running P2P at home: many P2P tests fail in the home env even at base commit (home/gold ENV DISCREPANCY), so a bare P2P failure misclassifies a correct solve. Only P2P that PASSES at base+test_patch and FAILS at base+test_patch+model_patch is a genuine model-induced regression. Corollary: naively enforcing PASS_TO_PASS in the home scorer would trade a rare false positive for many false negatives -> enforce P2P ONLY in the authoritative Docker eval (end-of-run audit), symmetric to the false-negative reclaim workflow. Empirically (140-run, 56 resolved w/ surviving work-dirs): 47 fully green, 9 P2P-fail, ALL 9 = env discrepancy, 0 genuine regressions.",
 "Benchmark tests that hardcode an EXTERNAL service URL (e.g. requests' suite defaults HTTPBIN to http://httpbin.org/) become a false-negative family in a network-isolated scorer: every such FAIL_TO_PASS raises ConnectionError regardless of patch correctness, so a correct patch is scored UNRESOLVED. Detection signature: misses in the repo whose F2P are network endpoints (httpbin('get')/'post'/'basic-auth'/...), with either a LONG runtime (DNS/connect timeouts, tens of seconds) or a fast connection-refused; confirm the host is unreachable and no local stub is installed. General fix: provide a LOCAL stand-in (pip install pytest-httpbin, start its bundled httpbin Flask app on 127.0.0.1, export HTTPBIN_URL before pytest -- the module reads HTTPBIN at import time so a fixture is too late) -- do NOT edit source. Verification method that also discriminates real misses: re-run F2P against the local service; genuine misses still fail with real assertions/TypeErrors (requests-2317's patched compat.py raises TypeError even with httpbin up) while env-FNs flip to all-pass (requests-2148 10/10). Caveat: a test that scheme-swaps http->https on the SAME host needs a dual-scheme local server, else it SSL-fails locally though it passes against a real dual-scheme host.",
 "When a benchmark test reads configuration (e.g. a service URL like HTTPBIN_URL) from the ENVIRONMENT at MODULE-IMPORT time, the earliest robust injection point is a repo-root conftest.py MODULE BODY: pytest imports conftest before collecting/importing test modules, so setting os.environ there (and starting any needed local server) beats the test's import-time read, whereas a session-scoped fixture runs too late. Mark any background server thread daemon (+atexit stop) so a never-stopped serve_forever cannot hang pytest at exit. Keep the conftest UNTRACKED so it stays out of the candidate's git-diff and carries no answer/instance data, and write it AFTER applying the scoring test patch so a suite-provided conftest is never clobbered. General, leakage-safe way to de-network an external-service test family (shipped for psf/requests as ensure_local_httpbin, commit 9bab933).",
 "A test RUNNER's default PARALLELISM can corrupt a benchmark scorer, independent of the code under test. django's tests/runtests.py defaults to a multiprocessing worker pool; on ANY failing test the worker cannot pickle the traceback, so django prints 'tracebacks cannot be pickled, making it impossible for the parallel test runner to handle this exception cleanly', the pool is torn down uncleanly (Pool state=RUN at __del__), and the terminal OK/FAILED result summary is DROPPED while the exit code can be perturbed by the teardown exception. Two harms: (1) score_tail telemetry becomes a bare 'ResourceWarning: unclosed running multiprocessing pool' instead of 'FAILED (failures=N)'/'OK', defeating false-negative triage; (2) a genuinely PASSING patch can score as a miss if the 'OK' line is suppressed or the exit code is perturbed (false negative). Fix: force SERIAL execution at scoring time (django: '--parallel 1'), which matches the authoritative SWE-bench django harness and makes results deterministic. Detection signature: a django (unittest) miss whose score_tail is the pool ResourceWarning with no result summary; passing runs are unaffected because no traceback pickling occurs, so the bug hides on green instances and only bites on runs containing a failure/error. GENERAL RULE: pin a test runner to serial/deterministic mode for scoring, and version-gate any runner flag on the runner actually supporting it (static source read / --help probe, cached) so an older codebase never receives an unrecognized flag -- the same guard the harness uses for pytest '--no-header' (_pytest_major).",
 "A REGRESSION/ACCEPTANCE signal in a bug-fix harness is only as good as WHICH tests it samples. Sampling 'neighbor' tests by the first-N in pytest collection order (top-of-tree/alphabetical) means that in large repos (django/sympy) the sampled files are unrelated to the model's edit, so a locally-introduced regression is almost never exercised and the signal is near-worthless. Bias the sample toward the FAULT AREA: take the in-repo traceback frames of the reproduction as hint paths and rank candidate test files by proximity to them (same directory >> shared leading package segments >> module-name match, e.g. test_expr.py near expr.py). Keep it a STABLE reorder so that with no hints every score is 0 and the result is byte-identical to the old first-N selection -> the change is provably behavior-preserving where no fault signal exists. Leakage-safety falls out for free: FAIL_TO_PASS tests fail pre-patch so they are auto-excluded from a 'passing-baseline' sample, and the neighbor node ids surfaced to the model are PASS_TO_PASS-like (green pre-patch), never the graded hidden tests. Make the ranker a PURE function (paths in, ordered ids out) so it is unit-testable offline and cannot touch the scoring path.",
 "A 'warnings-as-errors' test suite makes EVERY too-new tooling dependency a latent false-negative source, and the TEST RUNNER ITSELF is one of those deps. matplotlib sets filterwarnings=error, so a fresh env where the resolver pulls the LATEST pytest (9.x / >=8.4) dies at COLLECTION: pytest raises PytestRemovedIn10Warning for any same-file test that passes a generator to @parametrize (e.g. test_rcparams.py::test_validator_valid) -> fatal -> the whole module fails to collect -> 'found no collectors' -> a genuinely correct target patch is scored a miss. This is the SAME NO_COLLECTORS symptom as too-new pyparsing and the setuptools-scm/vcs-versioning hijack — a THIRD sub-cause in the same family. Fix by pinning the runner to the era of the code under test (pytest<8, era-appropriate 7.x, which still supports the harness's --no-header) alongside the other pure-python pins. Verify BOTH directions before shipping: reproduce the collection error under the too-new runner, confirm the pin restores collection (target test runs to a real verdict), AND reconstruct a currently-RESOLVED instance (base+model+test) to confirm the downgrade keeps it green (no regression). Note the pin can surface a real miss (the masked verdict may still be a FAIL) — restoring collection is about scorer CORRECTNESS, not about manufacturing reclaims.",
 "Before hypothesizing an ENVIRONMENT false-negative for a repo's miss cluster, VERIFY by re-running FAIL_TO_PASS in the retained work-dir venv (model+test patch applied): a repo can be dominated by verification-frontier (DECLARED-WRONG) misses with ZERO env-FN family. It is ENV when the failure is a collection error / skip / network timeout / fatal warning from a too-new dep; it is GENUINE when env_ok holds, the interpreter is healthy, and 'N failed' comes with a deterministic assertion/logic error in <1s -- frequently with the model's OWN self-verification passing. sympy is the second case: 6/6 verifiable self-verified misses (24102/22005/21847/20639/21171/21379) reproduce as genuine parser/solver/printing edge-case failures the model's happy-path self-verify never covered (plus 23191 where the model edited only a test file). The lesson is symmetric: a home audit both RECLAIMS false-negatives AND RULES OUT a whole class of env investigation, redirecting effort to the real bottleneck -- a non-self acceptance signal. Corollary: model self-verification is systematically over-optimistic on parse/print/format tasks; weight it accordingly.",
 "A self-verifying agent's RED->GREEN reproduction gate is only as strong as the reproduction's ASSERTION. A script that goes RED via an uncaught exception and GREEN once that exception stops firing proves only 'no crash', NOT 'correct output'. Reproductions whose sole check is a constant (assert True/1) or that carry no value assertion are a systematic self-deception vector for OUTPUT-correctness bugs (parse/print/format/compute): the agent declares success while hidden acceptance tests that check the actual value still fail. Measured on 147 SWE-bench traces: 68% of self-verified MISSES had such weak reproductions vs 55% of self-verified RESOLVES -- real and directional but NOT a clean classifier (many correct crash-fixes legitimately have no value assertion). Therefore act on it as ADVISORY STEERING, never a hard gate (a hard block would manufacture false-negatives on genuine crash-fixes). Implementation: a pure, leakage-safe strength classifier over the model's OWN script (value_check / vacuous_constant / weak); at GREEN, if not value_check, surface a note urging an expected-VALUE assertion. Keep it OUT of the score path (the scorer runs the hidden tests directly) so it can only steer, never change a verdict. (swe_fix_tools _reproduction_strength, commit 644f3b2)",
 "A subprocess that can DEADLOCK (not merely run slow) needs a PROGRESS-WATCHDOG, not just a hard timeout: watch a strictly-growing progress signal and abort in ~grace seconds when it FREEZES, keeping the hard timeout only as a backstop. A lone hard timeout forces a bad choice between killing legitimately-slow work early and silently burning the whole budget on a hang. Localize WHERE a hang lives before trying to fix it: all worker threads parked in futex_do_wait at ~0% CPU with ZERO downstream side effects (no build-cache growth, no new images/containers, empty run logs) == an ORCHESTRATION/lock deadlock, DISTINCT from a network or client-handshake stall. Prove the distinction cheaply and POSITIVELY instead of assuming: SWE-bench run_evaluation hung on pop-os 2026-07-15 even though the docker-py 7.1.0 handshake returned in 0.02s and CLI + low-level + high-level SDK builds of a trivial image all finished in <0.1s -- so the hang was inside the eval build orchestration, not the registry (pulled fine) or the SDK. Corollary: a recurring \"blocked until the cache/network warms\" assumption should be TESTED, not deferred cycle after cycle -- the real blocker may be internal and warming will never clear it. General hygiene: fail fast + fail visibly (distinct exit code for stall vs error), verify the watchdog decision logic with a dependency-free selftest (stalled child killed within grace; progressing child left alone), and restore any scratch state (e.g. throwaway test images) you created while probing.",
 "Before calling a low-CPU/futex/no-artifacts subprocess DEADLOCKED, dump ALL its threads and read the parked frame: that signature is ALSO an ordinary streaming socket read (e.g. a multi-GB docker image PULL in socket.readinto), which is NOT a deadlock and WILL finish. Ptrace-free method (works under yama ptrace_scope=1 where py-spy sibling-attach is blocked): launch via a wrapper doing faulthandler.register(signal.SIGUSR1, all_threads=True) then kill -USR1 <pid>. CASE 2026-07-15: SWE-bench run_evaluation was mislabeled an orchestration deadlock for ~10 cycles; the dump showed a ~2.5GB prebuilt-image pull and an unguarded run then completed in 259s (resolved=true). Watchdog lesson: sample a signal that moves in the CURRENT phase; a token blind to the phase (image-count/containers/logs are all frozen during a pull) FALSE-ABORTS legitimate slow work. Prefer widening grace or PRE-STAGING the slow resource over a noisy proxy (network RX) a shared host can mask. Corollary to #18: test the deferred assumption -- 'cache will warm' was wrong, and so was 'it is a deadlock'; a direct measurement settled it.",
 "SWE-bench Docker reclaim path -- two lessons (2026-07-15). (1) PRE-STAGE the prebuilt image before a guarded authoritative eval: `docker pull swebench/sweb.eval.x86_64.<instance-id, __->_1776_, lowercased>:latest` FIRST, so run_evaluation's build_container images.pull returns instantly ('Found N existing instance images. Will reuse them.') and the stall-watchdog only measures the real test phase (its image/container/log token is blind to an in-flight pull). Now AUTOMATIC in docker_eval_guard.sh (default on; --no-prepull to disable; --prepull-timeout N). (2) The AUTHORITATIVE scorer can itself be NON-DETERMINISTIC / externally-coupled: psf/requests prebuilt images run bare `pytest test_requests.py` with NO local httpbin and NO HTTPBIN_URL, so the suite hits PUBLIC http://httpbin.org/. requests-2148 ran twice, both resolved=false for EXTERNAL reasons: run1 hit httpbin.org 503 throttling (36 failed); run2 (httpbin up) passed ALL 10 FAIL_TO_PASS but failed 1 PASS_TO_PASS (test_redirect_with_wrong_gzipped_header, a gzip ContentDecodingError = modern httpbin.org behavior DRIFT vs the 2014 test). LESSON: a home audit that CHANGES the env (spin up a local pytest-httpbin) proves the patch CAN pass but does NOT establish the authoritative verdict, which depends on a flaky external service -- do not label such instances 'clean false negatives'. The conservative reclaim gate (flip only docker_confirmed=true) correctly protected the results. For network-coupled families neither a home local-stub pass nor a single Docker resolved=false is decisive.",
 "Bootstrap env acceptance must NOT depend on the model picking a passing smoke test (2026-07-15). In a model-driven env bootstrap with a turn budget (BOOTSTRAP_BUDGET=50), the model can burn the ENTIRE budget repeatedly choosing smoke tests that fail to COLLECT while the env is actually HEALTHY -> the instance is discarded env_ok=False, phase1_reason=budget (a TOTAL loss, strictly worse than a scored miss). Root example: 5 sphinx-doc instances (8801/8721/8474/8282/7975) lost this way. The uv-provisioned standalone CPython (python-build-standalone) OMITS the CPython-internal _testcapi C-extension, so any test module that does `import _testcapi` (e.g. sphinx tests/test_util_inspect.py) fails collection under that interpreter -- an ENV ARTIFACT, not a broken env (a benign sibling, tests/test_util_typing.py, gives 19 passed in the same venv). FIX (swe_agent_v2.py run_one, commit b5b197c): when the model exhausts the budget with sanity_ok=True but smoke_ok=False, run ONE deterministic harness-driven auto_verify_env pass and re-check env_ready before bailing. GENERAL PRINCIPLE for safe overnight edits: design the change to be STRICTLY NON-WORSENING -- fire ONLY on instances already destined to be a loss, and REUSE accept logic already trusted elsewhere -- so it can only convert losses into fix-phase attempts (never lower the resolved count) and is safe to ship even without a full end-to-end reproduction. META-LESSON: a trace in traces_v2/<id>.trace.json can be STALE / MISMATCH the current results row (sphinx-8801 trace showed phase1_reason=declared/env_ok=true from an earlier attempt while results_full300 recorded env_ok=false/budget) -- always reconcile trace[outcome] with the results entry before trusting a trace as root-cause evidence.",
 "Preflight ALL required schema keys of an input BEFORE any expensive/irreversible side effect (multi-GB image pull, build, upload), not merely that the file parses (2026-07-15). Concrete miss: a guarded SWE-bench eval validated preds were a non-empty JSON list with instance_id+model_patch, pulled the 11.2GB matplotlib prebuilt image (~30min), THEN swebench.harness.run_evaluation died with KeyError: model_name_or_path (it reads prediction[KEY_MODEL]) -- a 1-second schema check would have caught it. General rule: a downstream KeyError/validation that can only fire AFTER a slow side effect is a preflight GAP; assert every field the slow consumer will dereference (here instance_id + model_patch + model_name_or_path) up front. Fixed in docker_eval_guard.sh preflight.",
 "Structural 'can this patch even resolve?' gates need a REPO-AGNOSTIC test-path detector AND must enforce the requirement independently (2026-07-15). A patch that touches ONLY test/test-infra files can never resolve a SWE-bench instance (the scoring FAIL_TO_PASS tests are held out of the working tree), so it must be refused at edit time and must fail the submit gate. Bug in swe_fix_tools.py: the detector regex (^|/)tests?/|/test_|_test\\.py$ caught tests/ and test_*.py but MISSED pytest's own suite dir testing/ (pytest stores tests under testing/, not tests/) and conftest.py -> in the pytest repo a test-only patch slipped past both the h_patch refusal and the gate; separately, the gate _diff_nonempty only checked `git diff --stat` was non-empty (ANY change), trusting the upstream refusal instead of independently requiring a non-test change. Fix: shared _is_test_path() with (^|/)(tests?|testing)/|(^|/)test_|_test\\.py$|(^|/)conftest\\.py$ used by BOTH the refusal and a rewritten _diff_nonempty (git diff --name-only; require >=1 changed NON-test file). Verified do-no-harm: 0/145 completed resolved predictions are test-only under the new detector; 20/20 path unit-cases pass incl. negatives src/_pytest/ (pytest SOURCE), latest/, contest.py, src/testutils.py. General rule: encode every repo's test-dir convention, and verify a structural precondition directly rather than trusting an upstream filter.",
 "Validate an ADVISORY signal on labeled data before promoting it to a (soft) gate -- measure precision, recall AND the false-reject cost on genuinely-correct declares, not a single aggregate rate, and re-check that it has not drifted (2026-07-15). The fix-loop repro_strength note (weak vs value_check) looked directional in aggregate (68% weak on self-verified MISSES) but verification_signal_audit.py showed: (a) the discriminator DECAYED to nil over the run -- self-verified RESOLVES rose to the SAME 68% weak-rate -- so a broad flag-if-strength!=value_check gate would false-flag 67% of correct declares while catching only 66% of declared-wrong; (b) only the vacuous_constant subset (repro whose sole assertion is assert True/assert 1) is high-precision: 0/51 correct declares vs 2/29 declared-wrong, so a gate must be scoped to THAT subset, never the broad weak class; (c) the neighbor-regression signal is ORTHOGONAL to insufficient-fix misses (0/29 fired on declared-wrong -- insufficient fixes do not regress neighbors), so do not promote it for the verification frontier. General lesson: an aggregate looks-predictive rate hides class-conditional false-reject cost and can drift; build the confusion matrix + a soft-gate cost/benefit sim from the model OWN artifacts (leakage-safe) before touching a gate, and prefer a high-precision narrow trigger over a high-recall noisy one.",
 "Recurring triage: when a classification step runs EVERY cycle and its ad-hoc inline re-implementation keeps missing categories, commit ONE single-source-of-truth classifier instead of re-deriving the regex each time (2026-07-15). The SWE-bench overnight loop re-wrote a score_tail miss-triage inline every cycle; the throwaway regex twice failed to bucket real families (django unittest FAILED-failures=N summaries and pytest<6 bare rootdir: lines), stalling the loop. swe_fn_triage.py fixes this with three properties any recurring classifier should have: (1) an EXPLICIT ordered rule table mapping family -> kind (reclaimable_fn / env_fail / network_coupled / genuine_miss / review) so the taxonomy is versioned, not folklore; (2) a hard count-RECONCILIATION assert sum(buckets)==total so a dropped or misrouted row fails loudly instead of silently shrinking a family; (3) a REVIEW new-category radar bucket that catches any signature NOT matched by a known rule (e.g. a bare 'N error in Xs' collection error) and annotates it with cheap row discriminators already present -- self_verified is a WEAK hint only -- Docker ground truth shows self_verified=False instances CAN be real FNs (Docker-confirmed FNs pytest-dev__pytest-5227 and scikit-learn__scikit-learn-25570 both have self_verified=False), so treat any collection/env anomaly as a POSSIBLE FN and let the Docker eval -- not self_verified -- adjudicate -- so the ambiguous case is both surfaced AND pre-triaged. Make it repo-AGNOSTIC where the mechanism is universal (warnings-as-errors 'found no collectors' is not matplotlib-specific) so a new repo hitting the same failure lands in the family, not in REVIEW. Keep it read-only and leakage-safe: consume only score_tail + public flags + the Docker-confirmed manifest, never gold_patch / test_patch / FAIL_TO_PASS.",
 "When you have an authoritative oracle (e.g. the SWE-bench Docker eval labels), VALIDATE any cheap heuristic filter against it BEFORE encoding it as fact. The overnight loop repeatedly asserted 'a real scoring false-negative requires self_verified=true' and even printed it as REVIEW adjudication guidance, yet the Docker-confirmed FN manifest holds two instances (pytest-dev__pytest-5227, scikit-learn__scikit-learn-25570) with fix_verified_by_model=False -- the model did NOT self-verify but the patch was correct and only an env/collection artifact hid it. Gating the reclaimable_fn classifier on self_verified would have EXCLUDED real FNs the oracle catches (a silent false-negative-of-false-negatives). Lesson: a plausible discriminator can be flatly contradicted by ground truth -- keep such hints ADVISORY (surface the split, never filter on it), and when a heuristic and an oracle disagree, the oracle wins. Process corollary: probing the actual labels BEFORE editing turned a would-be regression into a corrected do-no-harm change; always size a proposed filter against the labeled set first.",
 "A miss-triage / reclaim-accounting view must net out EVERY terminal adjudication state, not just the reclaim-positive one. swe_fn_triage.py computed its 'pending Docker-confirm' reclaim backlog as (reclaimable-family members - Docker-confirmed FNs) but never subtracted the manifest's confirmed_real_misses, so 6 instances already adjudicated as genuine misses whose score_tail still carries a reclaimable-family signature (matplotlib-22835/23299/23476/24265 NO_COLLECTORS + scikit-learn-25500/25638 importorskip-skip) kept showing as an OPEN reclaim backlog (21 reported vs 15 true), and a prior cycle's NEXT ACTION literally proposed 'Docker-confirm a pending FN' -- i.e. re-litigating settled negatives. Fix (commit cc460a2): also load confirmed_real_misses and report per family docker_confirmed / real_miss / pending with pending = members - confirmed_FN - real_misses. General rule: any 'open/pending' bucket over a workflow with multiple TERMINAL outcomes must subtract ALL of them or it silently inflates and misdirects effort. Prove do-no-harm by showing the taxonomy buckets+ids are byte-identical old-vs-new and ONLY the pending sub-split moves (21->15, adjudicated_real_miss=6); the tool stays read-only and answer-leakage-safe (public ids + manifest only).",
 "When you fix a buggy pattern, SEARCH THE REST OF THE FILE/MODULE for other occurrences of the same pattern and fix every one. A correct change applied to only one of several identical sites leaves the bug alive and the tests red -- you were right and stopped early. Before declaring done, grep the symbol/expression you just changed.",
 "Prefer the MINIMAL change that extends an existing structure over adding a new special-case branch. If the code already has a condition enumerating cases, add your case to that condition rather than writing a separate block in front of it. Large defensive blocks tend to be wrong in ways the existing structure would have prevented.",
 "Handle the DEGENERATE case explicitly: zero, empty, None, single element, identity. A formula that is right in general is often wrong exactly at the boundary (x/abs(x) is correct for every x except 0). If your fix is a formula, ask what it does at the edge before you declare it verified.",
 "Agent scaffold: before concluding a model 'ignores its own history' or 'thrashes', read the message the model ACTUALLY RECEIVES, not the value the tool returned -- traces log tool output PRE-clip, so scaffold starvation is invisible from the trace and can hide behind months of unrelated work. A loop that serializes each prior tool result and HEAD-clips it to a fixed size is not showing the agent the result, it is showing the agent the preamble. Test runners, compilers and tracebacks BACK-LOAD their diagnosis -- pytest puts 'platform linux -- Python 3.9.20, pytest-7.4.0' first and the 'E ValueError: ...', the failing file:line, and the 'short test summary' LAST -- so a head-only clip preserves boilerplate and discards exactly the part the agent needs to act. The agent then re-issues the identical call, because nothing in what it received distinguishes this attempt from the last, and a loop-detecting critic fires with no effect because the agent has no new information to act on. Detection signature: the same tool called with identical args 4+ times with byte-identical results, and/or repeated re-reads of the same range, and/or a tool returning a bare rejection flag (registered_as_reproduction: False) whose REASON sits past the clip boundary. Observed live in LLMOS: results clipped to 1800 chars against a 131072-token window (~27K of a 131K context, and the wrong 27K) -- django-16820 called verify_fix at turns 44/48/49/54/57/63 and reproduce eight times with the same script. Note the tell that it was knowable all along: the SCORER already read 'score_tail' because we knew the tail carried the signal, while the MODEL was fed the head. Fix: clip head+tail (head keeps identifying fields, tail keeps the diagnosis) with an EXPLICIT elision marker in the text so the agent can narrow its next request instead of blindly re-reading -- silent truncation is strictly worse than announced truncation, because the agent believes it saw everything. Budget by RECENCY (generous for recent turns, tight for old) rather than one flat cap, so a 160-turn run still fits the window. GENERAL RULE: the diagnostic question is never 'how big is the context window' but 'how much of what each tool returned reaches the model, and from which end'. Measure the prompt you send, not the data you have. When an agent looks stupid, dump its prompt first.",
 "A knowledge base is worth only what its APPLICATION is worth: the moment you write a pattern down, GREP THE CODEBASE FOR EVERY OTHER INSTANCE OF ITS SHAPE. We wrote pattern #6 ('when a verification gate rejects the agent, feed back the SPECIFIC diagnostic, never a generic rejection -- a generic rejection gives the agent nothing to act on, so it thrashes the same tool until its budget dies') about a sphinx bootstrap gate, and left the byte-identical bug sitting in h_reproduce for the whole benchmark. Measured: 799 reproduce calls, 442 rejected, 380 of them (86%) carried no reason at all. Writing it down is not helpful if you don't read it -- and reading it is not helpful if you only apply it where you found it. TWO GENERAL CODE SMELLS FALL OUT. (1) An if/elif chain that attaches a user-facing EXPLANATION must have an else. A missing else is a silent failure that fires only in one late state -- here, `if registered: note=... elif not state['seen_red']: note=...` with no else, so the explanation vanished exactly AFTER a reproduction was registered, i.e. exactly in the late-run state where budget death happens. That selectivity is why it survived every review: the happy path and the early path both look fine. (2) A STATE-INDEPENDENT SYSTEM PROMPT paired with a STATE-DEPENDENT TOOL will trap the agent. Our FIX_SYSTEM_PROMPT says 'If your script exits 0, it does not demonstrate the bug -- rewrite it', which is correct BEFORE a reproduction is registered and actively wrong after it (once red is registered, the right move is verify_fix, not another reproduce). The prompt herded the model into the exact loop the tool then refused to explain. When a tool's meaning depends on hidden state, the tool -- not the prompt -- must say which state you are in and name the call that advances from here. DETECTION: dump the tool messages the model actually received and count how many rejections carry no reason field. If a rejection can be returned with no explanation, it eventually will be, in the worst possible state.",
 "Agent determinism: a per-request sampler seed does NOT make an agent reproducible, and when identical-config runs diverge, diff the two transcripts and find the FIRST differing byte -- it names the leak precisely. Observed leak class: a tool that uses an LLM internally (a judge/ranker) and embeds the judge's sampled PROSE in its result -- a reason string, discard notes. Two runs made the IDENTICAL decision, same top_hit, but worded the explanation differently; the wording entered the main agent prompt, and at temperature 1.0 the trajectories never re-converged. Environment text (timings, addresses, tmp paths) leaks the same way. RULE: a tool may consult an LLM, but its RESULT must carry only the decision -- paths, lines, counts, booleans -- never the sampled wording of it. Then equal decisions give byte-identical results and runs diverge only where a real decision differs, which is the only divergence worth having. Corollary: reduce an agent's stochasticity by removing CHOICE POINTS, not by cooling the sampler -- scaffold-executed forced moves (auto verify after patch, auto submit after green tests) and stage-scoped tool menus eliminate dice entirely, which beats loading them.",
 "Test-time scaling, done honestly: a stochastic agent at temperature 1.0 is a SAMPLER, and multiple attempts per problem is legitimate published methodology (EntroPO+R2E, arXiv 2509.12434: entropy is deliberately PRESERVED in training because diverse rollouts + a verifier beat one greedy rollout; their 30B hits 49.67 on SWE-bench Lite exactly this way). The integrity line: the decision to retry, and the choice among attempts, must read ONLY agent-side signals -- its own red->green verification, patch nonempty -- NEVER the benchmark score or hidden tests; the verdict of withheld tests is answer-key information even as a single bit. Corollary discovered independently: our stitched best-of-N reruns were accidental TTS. Two roads lead from the same model ceiling: squeeze variance for one reliable attempt (determinism work) or spend variance across N attempts and verify (TTS). Both are scaffold-legitimate and a system can do both -- deterministic procedure inside an attempt, sampling across attempts. The missing organ in ours is the HYBRID VERIFIER (execution checks plus an execution-free patch-plausibility judge returning a decision enum): the stronger the agent-side verifier, the more of the model's latent solve-rate TTS can bank without leaking anything.",
 "THE THREE QUESTIONS (Mikey's debugging invariant -- ask them at every stage, every failure, every verification): 1. WHAT WERE WE TRYING TO GET? State the goal as a checkable artifact (triage done_criteria, the worksheet's trying-to-get line) -- if you cannot state it, you are not debugging, you are wandering. 2. CAN WE GET IT? Feasibility and progress: does any attempt move toward it, is it within reach of the tools and the model, did the last change bring it closer or not (same_failure_as_last answers this mechanically). 3. IF WE GET IT, IS IT IN THE RIGHT FORMAT? Getting the substance wrong-shaped is still a miss: flask-5063 produced working subdomain output labeled with the wrong word -- capability delivered, format failed. Format includes vocabulary (label outputs after the fields they display), types, ordering, and the exact contract the consumer checks. The questions recur fractally: each pipeline stage asks them of its own output (env: got an interpreter? the right one? importable?), each tool result, each patch, each verification. A failure is always a NO at one of the three -- find which one FIRST, because the repairs differ: no-goal needs triage, no-progress needs a different approach, wrong-format needs the consumer's contract read back into the work.",
 "Per-repo structural-map protocol: for a given repo/framework, build the structural map its bugs habitually need BEFORE patching -- unconditionally, whether this instance turns out to need it or not. It is cheap and often it IS the fix. For a web framework (flask): build the OUTPUT MAP of any structured output in play (CLI table, header set, message, serialized response) -- each field's SOURCE expression, its header DERIVED from the source name (attr.title(), never copied from the reporter's mock), and any MODE flag that selects between alternative sources (record both branches). Then render from the map; the model decides only WHICH fields, the scaffold computes the spelling / branches / alignment. This is the neuro-symbolic split made concrete: derivable output is derived, not sampled. Generalizes -- each repo has a map-kind its failures cluster around (django ORM: the queryset-to-SQL shape; sympy: the expression-tree transform; matplotlib: the artist attribute being repr'd). The per-repo KB should name that map-kind and mandate building it during triage.",
 "Different problem types need different VERIFICATION INSTRUMENTS -- and verifying with the wrong one silently discards correct fixes. The gate verified a fix by rerunning the agent's standalone `python -c` reproduction. But a bug INSIDE a framework (pytest's import machinery, collection, fixtures, plugins; a test-runner's own semantics) cannot be reproduced by a standalone script -- the reproduction is pinned RED forever, so a CORRECT patch never self-verifies. Consequence chain, confirmed on pytest-11148: repro can't go green -> gate never fires -> the model is blind to its own success -> leak-safe best-of-N selection (which must NOT read the hidden score) can't tell the winning attempt from a losing one -> a patch that PASSED the hidden tests (resolved=True on disk) is recorded as a MISS. The fix is not a better prompt; it is a second verification path that runs the reproduction THROUGH the framework (write it as a pytest test, run via `pytest`), selected by problem type. Keep it leak-safe: it is still the agent's OWN reproduction, run with the right tool; write the temp test OUTSIDE the repo so it never enters the git diff; exit 0=green, 1=red, 5(no collection)=invalid-not-red. GENERAL RULE: when an attempt cannot self-verify, do not assume model failure -- check whether the verification instrument can even observe this bug class; and give the selector a fallback signal (fault-location touched, framework test flipped) so unrecognized-but-correct fixes are not thrown away.",
 "Symbolic tools for operations with a known-correct form -- provide the correct implementation and always route through it, so the model cannot reinvent it wrong. An operation whose correctness is an INVARIANT, not a judgment call, should never be left to token sampling: it will be right some draws and wrong others (the coin-flip we watched on pytest-11148). IMPORT is the canonical case -- 'correct import' means exactly: return the cached module if present, else build once and cache before exec; anything else makes two objects for one logical module. So: a symbolic import_tool (cached_import / import_from_path) the agent uses in reproductions, and a per-repo fix protocol that mandates MIRRORING that contract in the patch (the sys.modules cache-check). The same shape recurs for other invariant operations: header spelling = attr.title() (never a synonym), path joining, version comparison, set/dict de-duplication, encoding fallback. For each: identify the invariant, implement it once, and either (a) hand the agent the primitive to call, or (b) encode the canonical patch shape in the KB and lint the diff against it. The LLM should decide WHICH operation is needed and WHERE; it should never re-derive an operation whose correct form is already known. This is the neuro-symbolic split at the primitive level: fuzzy choice from the model, exact form from a tool.",
 "ONE CANONICAL REGISTRY -- no duplicate stores of the same identity unless they serve a genuinely distinct purpose, and then one must be canonical and the other must defer to it. Symptom family: 'these two things should be the same object but aren't' -- double imports, a cache that disagrees with its source, two config objects, a singleton that isn't, duplicated session/connection pools. The bug is almost never 'add more state'; the state you need ALREADY EXISTS (for imports it is sys.modules). The bug is a second, competing store that writes to the canonical one but never READS it first. Fix shape: before constructing, look up the canonical store -- AND look it up under the SAME KEY everyone else uses (this is the half that gets missed: pytest-11148's fix works only because module_name_from_path produces the canonical dotted name, so the lookup hits the entry Python's own import placed there; a self-consistent internal key passes the narrow unit test and still fails the integration test). When triaging such a bug, write the IDENTITY AGREEMENT TABLE: canonical store / canonical key / who computes the key / which code path bypasses it.",
 "MODES MUST PRESERVE INVARIANTS -- a mode/flag/alternate path that skips the canonical registry is a DEFECT IN THAT MODE, not a configuration choice, and 'switch back to the safe mode' is a retreat, not a fix. pytest's --import-mode has prepend/append (which delegate to Python's import_module and therefore consult sys.modules for free) and importlib (which hand-builds via sys.meta_path and WRITES to sys.modules without ever reading it). Modes are supposed to differ in HOW they find a thing, never in WHETHER the thing is unique -- uniqueness is an invariant every mode owes the caller. Telling the user to revert to the default takes away the capability they deliberately chose (PEP 420 namespace packages, here). DEBUGGING MOVE: ask early 'is there a switch?' -- it sorts the problem instantly. Either the behavior is configurable and you are done, or the switch itself is where the bug lives and you now know exactly which code path to read. Then enumerate the modes and ask, for each, which invariant it bypasses.",
 "DON'T DO THE SAME THING TWICE UNLESS THERE IS A REASON. Loading, creating, connecting, parsing, computing -- if the work has already been done and the result is still valid, reuse it; only redo it when the situation genuinely calls for a fresh one, and if you redo it, be able to say why. Doing it twice is not merely wasteful: when the operation creates an IDENTITY (a module, a connection, a client, a config object, a cache entry), doing it twice creates TWO of something that everyone else assumes is ONE -- and then state written on one copy is invisible on the other, which is a bug that looks like magic. So before you construct, check whether it already exists; look for the place the system keeps track of it, and look under the same name everyone else uses. If the codebase offers several ways to do the operation, each way must obey this equally -- a mode that skips the check is broken, not configurable. HOW TO TEST IT: obtain the thing TWO WAYS and assert they are the SAME OBJECT -- identity (`is`), never equality (`==`), because two fresh copies usually compare equal and that is exactly the bug hiding. Get it the way the subsystem under test gets it, AND the way ordinary code gets it; those are the two that must agree. A test that obtains it twice through the SAME path only proves the path agrees with itself -- it passes while the real bug survives. Stronger form: set a marker on the first copy and read it back through the second; if the marker is missing you have two objects, whatever the names suggest. Run the test where the duplication actually happens -- if the duplicating machinery is a framework's, the test must run under that framework, not standalone.",
 "ALWAYS IMPORT time AT MODULE SCOPE so timestamping is available everywhere without thinking about it. Anything you log, cache, collect or compare later wants a timestamp, and the moment you reach for one is usually inside an exception handler or a rarely-taken branch -- the worst place to discover the name is not bound. A missing `import time` inside a try/except that swallows exceptions produces the worst failure there is: the feature does nothing and looks fine. Corollary, and the more important half: NEVER let a helper fail silently. `except Exception: pass` around a collector, a logger or a cache write makes a broken component indistinguishable from a working one. Not breaking the caller and failing silently are DIFFERENT requirements -- return the error and let the caller surface it, so 'collected nothing' can never look identical to 'collecting fine'.",
 "WHEN YOU EMIT A SPEC-GOVERNED ARTIFACT, THE SPECIFICATION DECIDES -- NOT THE SURROUNDING CODE. If your change produces something a FOREIGN consumer must read -- LaTeX or HTML markup, SQL, a CSV or JSON string, an HTTP header, a date, a URL, an encoding -- then correctness is defined by that format's published rules, which live OUTSIDE this repository. The code around you is one implementation of those rules, and an incomplete one: sibling functions routinely 'get away with' a wrong construction because their inputs never exercise the case that breaks it. So matching the neighbours is the right default for NAMING and STRUCTURE, and the wrong default for FORMAT CORRECTNESS. When local habit and the published rule disagree, the published rule is why the output is judged wrong. HOW TO SPOT IT: ask who consumes this string. If the answer is a renderer, a parser, a database, a browser or another program rather than a human reading your source, you are implementing a spec. Then ask what that spec says about the specific construct you are emitting -- nesting, escaping, quoting, sizing, precedence, ordering -- because those are exactly the clauses implementations get wrong. HOW TO CHECK IT: emit the artifact in BOTH the simple and the composed case (the bare value and the same value wrapped, nested, quoted or escaped) and compare them. Bugs of this class never show in the simple case; they appear only when the construct is nested inside another of its own kind. A construction that is correct alone and wrong when nested is the signature.",
 "WRITE IN THE LANGUAGE'S PUBLISHED STYLE, NOT AN INVENTED ONE. Every mainstream language has an authoritative style guide -- PEP 8 for Python, the Google or LLVM guides for C++, gofmt for Go, rustfmt for Rust, the standard style for JavaScript. Follow it: naming (snake_case functions, CapWords classes, UPPER constants in Python), indentation, line length, import order, whitespace around operators, docstring form. This is not cosmetic. A patch that violates the house style is visibly foreign, gets rejected in review, and in projects that run a linter in CI it FAILS THE BUILD -- a formatting violation and a logic error cost exactly the same there. Where the local file and the published guide disagree on layout, follow the LOCAL file: consistency inside one file beats global conformance, and the maintainers chose it. Where the local file has no opinion, follow the guide. And keep the diff minimal: do not reformat lines you did not need to change, because unrelated style churn hides the actual fix and is the fastest way to have a correct patch rejected.",
 "A SELECTION PREDICATE AND THE BRANCH IT SELECTS ARE ONE CONTRACT. Any function that routes inputs among output forms, handlers or modes pairs each condition with a consumer. Changing the condition changes what BOTH sides must handle: inputs newly excluded fall through to another branch -- verify their output there is acceptable; inputs newly admitted reach this consumer -- verify it can handle them, which sometimes means the consumer must change too. The reported failing input is ONE point in the space; the fix is judged on every class the changed condition re-routes. After editing any routing condition, enumerate the input classes on each side and run one specimen of each through the real code."
]