This matrix records what the repository test suite proves and, equally importantly, what it does not prove. The governing rule is Test doubles and integration proof: a double may replace a collaborator beyond the production owner under test, but it may not replace that owner and still be cited as integration evidence.
- Unit or policy: pure parsing, normalization, ordering, rendering, validation, or orchestration. A replaced boundary owner limits the claim to wiring.
- Boundary integration: the production owner crosses a real filesystem, Git, HTTP, process, persistence, or protocol boundary; a controlled peer may sit beyond it.
- Installed integration: a built wheel or sdist-derived wheel runs in a clean environment with hostile import and path conditions.
- External qualification: the actual external component is executed. This evidence is version-specific and is not replaced by a fake response.
- Static contract: repository configuration, workflows, public examples, package contents, or documentation are inspected without claiming execution of the external service.
| Requirement or boundary | Production owner and entry point | Required observable result | Evidence | Double boundary and claim limit | State |
|---|---|---|---|---|---|
| #87 capture current protected target identity | providers.gitlab.acquire_review_snapshot |
MR head, target project, protected branch, and target SHA agree through bounded HTTPS | test_gitlab_snapshot_crosses_real_https_adapter_and_binds_protected_target; identity, redirect, size, and deadline variants in the same module |
local TLS GitLab peer is beyond the production urllib adapter | proven |
| #87 source cannot select policy and concurrent movement cannot switch the captured object | review_runner._prepare_policy_context; GitRepositoryReader.fetch_commit |
exact captured SHA is fetched without moving checkout refs; unavailable or unsafe objects fail closed | test_bounded_fetch_gets_exact_commit_without_moving_refs; policy input negative cases; complete review-preflight E2E |
real local bare remote is beyond Git plumbing; no Git reader mock | proven |
| #87 exact repository-owned rules and operator-owned external rules | _prepare_policy_context; write_private_bytes; GitRepositoryReader.read_blob |
exact policy-commit blob becomes mode-0600 private input; checkout is unchanged; external absolute path is preserved | exact-policy transport test and complete review-preflight E2E | no owner is replaced | proven |
| #87 unchanged diff range with actual OCR rule consumption | ocr_compat.run_contracts invoking real ocr review --preview |
the same base/head excludes the synthetic extension without the rule and selects it with target rules | target_rule_selection_probe in the OCR compatibility harness; committed OCR 1.9.4 evidence after rerun |
actual OCR binary; no LLM needed for deterministic selection | proven |
| #87 target guidance and decisions, not base/source policy | collect_repository_evidence(..., policy_ref=...); store; MCP |
policy records bind policy SHA and changed-path applicability, source policy has no authority | collector policy tests; schema-v4 hostile readback; installed wheel/sdist MCP; complete review-preflight E2E | real Git/store/stdio in integration tests | proven |
| #87 schema compatibility | EvidenceStore.read/write |
v4 policy identity round-trips; v1-v3 retain explicit legacy semantics and hostile extensions fail | schema tests in test_evidence_model.py |
persisted files are real; mutation of fixture data is beyond loader | proven |
| #88 changed-template priority | template plugin plus core/store admission | added, modified, removed, renamed, and over-limit changed templates precede unchanged inventory without raising limits; partial remains explicit | test_evidence_framework_plugins.py priority cases and collector/store limit cases |
temporary Git repositories and real store | proven |
| #88 installed queryability and version CLI | installed ocr-ci; installed stdio MCP |
direct wheel and sdist-derived wheel expose typed late template facts and centralized version under hostile import/PATH conditions | test_installed_policy_e2e.py |
clean built artifacts; real child process and stdio protocol | proven |
| #89 bounded provider projection | GitLab adapter plus normalize_merge_request_context |
only title, description, labels, branch, identity and statuses survive; limits, controls, redaction and collisions are enforced | real-TLS adversarial provider test plus normalizer boundary tests | local TLS peer beyond adapter | proven |
| #89 source-head binding and quality-signal degradation | acquire_review_snapshot; run_evidence_review |
mismatched head is rejected; no fabricated intent enters the store | real-TLS mismatch and complete preflight E2E | no provider-owner mock | proven |
| #89 persistence, hostile readback, bootstrap omission, MCP query | store/readback/project/MCP | closed invocation-trust descriptor round-trips; raw values are absent from bootstrap; OCR-materialized optional fields do not break summary/list/get; raw values are retrievable only by an explicit list/get path | test_review_context.py; direct wheel and sdist-derived wheel stdio MCP with OCR-1.9.4-shaped arguments; complete preflight E2E |
real files and stdio; direct dispatcher test alone is component evidence | proven |
| #100 context selector privacy and receipt v3 | parse_review_context_mode; provider snapshot; run_evidence_review |
off never normalizes mutable MR text; metadata has closed completeness; receipt binds identities/context/capability/evidence without raw provider or transport values |
parser/provider/hostile receipt tests; real-TLS provider and complete preflight E2E | synthetic child is beyond real subprocess launcher; local TLS peer is beyond production HTTP owner | proven |
| #100 metadata/approval/MCP authority | separate provider projection, receipt-v3 policy, profiled MCP composition, post-time GitLab identity readback | complete metadata is eligible but non-authoritative; degraded metadata or any external MCP blocks; source/author movement and self-approval write nothing | review-context, approval, runtime-helper, and complete preflight tests | posting/API mocks prove policy and ordering, not live GitLab mutation; real config/persistence/subprocess boundaries are exercised separately | proven for toolkit authority |
| #89 matching, contradictory, absent/ambiguous intent and objective-defect review semantics | toolkit-authored bootstrap guidance consumed by OCR/model | model output demonstrates calibrated outcomes without a follow-up question | the single OCR 1.9.4 run did not read MR context: its 70 union-shaped calls all selected summary and failed on an inactive materialized field; deterministic guidance and corrected installed stdio retrieval are not substituted for model evidence |
a fake LLM would replace the behavior being claimed; no second OCR run is permitted for this release | not qualified in 0.6.1 |
| #90 OCR 1.9.4 CLI/result/selection compatibility | compatibility harness against checksum-verified asset | actual binary passes version, help, preview, target-rule selection, deterministic local-gateway review, and result consumer contracts | hosted qualification plus local probe-local; compatibility evidence must be regenerated after adding selection probe |
local gateway is beyond OCR's HTTP client and proves protocol/result behavior, not general model quality | proven |
The complete run_evidence_review synthetic test deliberately uses one controlled child executable beyond the production subprocess launcher. The child reads the exact generated rule artifact, starts the configured production MCP server over stdio, queries summary/policy guidance/MR context, and only then emits tool-call counts. It proves orchestration and boundary composition; it does not claim to be the real OCR selector. The separate compatibility probe supplies that real-consumer proof.
| Requirement or boundary | Production owner and entry point | Required observable result | Evidence | Double boundary and claim limit | State |
|---|---|---|---|---|---|
| Official OCR 1.9.5 identity and consumed CLI/result compatibility | scripts/ocr_compat.run_contracts invoking the checksum-verified ocr executable |
version/help, JSON preview, target-rule selection, clean JSON review, structured comment fields, and toolkit result consumers agree | hosted run 32000131436; repeated local Darwin arm64 probe-local; committed evidence and manifest asset hashes |
deterministic local HTTP server replaces only the external LLM API beyond OCR's real HTTP/agent/result boundary; OCR itself is not mocked | proven |
| Aggregate review budget preserves usable work and exposes lost coverage | scripts/ocr_compat._budget_result_probe entering through the official ocr review --max-tokens-budget CLI |
one real run completes two of three selected files, preserves their findings, emits summary.budget_exceeded, token_budget_reached, and one manifest failed(budget) record, and normalizes to partial |
checksum-verified local OCR 1.9.5 qualification recorded in compatibility/evidence/ocr-1.9.5.json |
controlled local LLM peer reports deterministic usage; it does not replace OCR dispatch, usage accounting, manifest creation, JSON serialization, or toolkit parsing | proven |
| Synthetic GitLab budget reaches OCR argv without shell interpretation | review_runner.run_review production subprocess launcher |
quoted --max-tokens-budget and its value cross the real process boundary with private artifacts |
test_run_review_crosses_real_subprocess_boundary_with_private_artifacts; static GitLab example contracts |
synthetic executable is beyond the real launcher and proves transport only; actual OCR budget semantics are separately qualified above | proven |
| Budget-stopped results remain partial and approval-ineligible | result_contract.parse_result_outcome; posting and approval policy |
completed findings remain publishable; budget attribution and manifest partition agree; outcome never looks clean or authorizes approval | test_result_contract.py, budget cases in test_review_runner.py, test_posting_helpers.py, and test_posting_approval.py |
fixture-driven parser/policy evidence is not cited as OCR integration; official executable qualification supplies that boundary | proven |
| Scan-only 1.9.5 signal does not widen toolkit scope | toolkit strategy, compatibility contract, backlog, and executable example | the toolkit continues to invoke diff review, does not add scan, and does not duplicate OCR telemetry |
static contract tests and complete architecture diff review | static evidence proves ownership/scope, not OCR scan behavior | proven |
The no-release qualification used official OCR 1.9.5 with SHA-256 459d3986e59fed5ed8ad6a97bc02d2eb995a89106b3fe6a6fcf74bb69cab1b73, a disposable synthetic repository, owner-only isolated homes, a local OpenAI-compatible peer, and a real stdio MCP protocol peer. The private transient payload and peer were deleted after a bounded observation summary was extracted.
| Requirement or boundary | Production owner and entry point | Required observable result | Evidence | Double boundary and claim limit | State |
|---|---|---|---|---|---|
| Tool-name allowlist and object-authorization limit | mcp_config composition -> real ocr-ci review -> OCR RegisterAll/Provider.Execute |
only configured names reach model; model-generated cross-tenant argument reaches allowed peer unchanged | one real peer advertised allowed and hidden/reserved tools; the gateway exposed only allowed external tenant_read/tenant_fail; stdio peer recorded the exact synthetic object ID |
local model peer chooses deterministic calls; proves transport/authorization absence, not safe provider semantics | proven |
| Server-authored description/schema trust and phase exposure | OCR 1.9.5 MCP client/tool-definition projection | hostile markers reach plan/main context and the same external tools are available in both phases | real model requests contained description/schema markers in plan prompt and main tool definitions; session records included plan_task and main_task |
controlled model peer observes exact OCR request; no claim about general model susceptibility | proven |
| Text-result boundary | OCR MCP CallTool/contentToText -> real model request |
complete bounded high-volume result reaches the next request without toolkit-side truncation at the tested size | real stdio response marker and complete synthetic payload appeared in the subsequent model request | bounded payload proves no toolkit cap was enforced below the tested size in this path, not absence of all caps or behavior for unbounded/exhaustive input; separate configuration/source audit establishes the missing operator cap | proven |
| OCR session persistence | OCR session.jsonlWriter reached by real review |
prompts, responses, tool arguments, results, plan/main task identity persist in owner-only JSONL | one mode-0600 session contained all named fields and the complete synthetic result | isolated home and synthetic data only; does not prove cleanup because current toolkit has none | proven |
| Failure/degradation and mandatory evidence | OCR optional MCP init/tool error plus toolkit receipt gate | unavailable server and isError tool result do not become false successful context evidence; mandatory evidence remains required |
tool-error run completed; unavailable-server run warned and completed with one evidence call; receipt omitted unavailable server | local unavailable path proves degradation, not every network/provider failure | proven |
| Reserved names, collisions, and receipt limits | mcp_config.parse/compose; _mcp_usage_receipt |
reserved server/tool and cross-server collision fail closed; receipt counts known calls only | production parser/composer rejected all three; real result receipt contained evidence=1/external=2 and no arguments/results | parser checks are policy evidence; real receipt proves serialization only, not authorization/completeness/content safety/model correctness | proven with stated limits |
| Requirement or boundary | Production owner and entry point | Required observable result | Evidence | Double boundary and claim limit | State |
|---|---|---|---|---|---|
Closed context selection and identity-only off |
evidence.review_context.parse_review_context_mode; review_runner.run_evidence_review; GitLab provider snapshot |
invalid/enriched fails before OCR without raw value; off validates source/protected-target/author while mutable text never enters normalizer/store/bootstrap/result/log |
test_review_context.py, test_gitlab_provider.py, test_review_runner.py including local TLS/full subprocess path |
local TLS and synthetic OCR child sit beyond real HTTP/subprocess owners; no model-intent judgment claim | proven |
| Receipt-v3 approval identity and self-approval prevention | review_runner._review_receipt; posting.approval; posting.gitlab_approval.execute_approval |
exact closed hostile readback; complete metadata eligible; degraded/external/legacy blocked; current source+author required; bot author performs zero writes | test_review_context.py, test_posting_approval.py, test_review_runner.py |
mocked GitLab API proves adapter policy/ordering only; no live approval mutation is claimed | proven for deterministic toolkit policy |
| GitLab-MR MCP topology | mcp_config.parse_mcp_servers/compose_mcp_servers; run_evidence_review |
external remote HTTPS only; stdio/setup fields rejected; inherited registry revalidated; fixed built-in remains independent | test_runtime_helpers.py, complete review preflight |
parser/persistence path is real; no claim that OCR tool annotations prove semantic read-only behavior | proven with annotation-enforcement non-claim |
| Create outcome and marker transport | posting.gitlab.api_write_url_detailed; inline create callers; posting.payloads/markers |
`posted | invalid_position | definite_failure | ambiguous_create`; endpoint-specific 2xx identity; independent marker survives character/UTF-8 budgets; no blind create retry |
| One-shot reconciliation and explicit transaction rollback | posting.reconciliation; PostingTransaction; posting.snapshot.rollback_current_run_comments |
complete endpoint pagination; exact marker+author one-match recovery; zero/multiple/foreign/malformed unavailable stay ambiguous; exactly-once draft publish; rollback only explicit IDs absent from baseline | test_posting_helpers.py real paginated local HTTP read, workflow state matrices, baseline rollback tests |
mocked item sets prove hostile shape/state branches; local peer proves production GET pagination, not GitLab service semantics | proven with no live GitLab write claim |
The architecture checkpoint assigned every claim to a production owner before implementation. This matrix now promotes only claims exercised through that owner and keeps final real-OCR, external-service, stable-release, and model-judgment limits explicit. Unit mocks cannot prove the broker, adapter, protocol, persistence, subprocess, Git, OCR, or model behavior whose owner they replace.
The M5 negative suite is an attacker matrix, not merely a replay of observed review findings. It crosses malformed scalar/container types and recursion; ordering, exact/changed duplicates, and admission-budget accounting; no-read/timeout/partial-frame/pagination mutation; replay, expiry, wrong-run/policy, and retry accumulation; tenant/service identity confusion; HTML/entity/Unicode/inline-reference-autolink Markdown laundering; marker smuggling and duplicate multiplicity; repository-owned executable resolution; and request/byte/character/line/item/work limits. Those deterministic cases do not make an adapter truthful, detect semantic paraphrase, or protect same-owner memory/files after host compromise.
| Threat/property | Production control and owner | Evidence | Current state |
|---|---|---|---|
| Source branch expands access | review_runner._prepare_enriched_context and context.policy.load_protected_policy bind the captured policy SHA and exact policy path |
real immutable Git loader/wiring tests plus exact-call, source-policy, duplicate/UTF-8/size/schema/object-type cases | proven for deterministic Git/policy wiring; live forge service remains outside local evidence |
| Source rules select public setup wording | review_runner._record_rules_path_setup plus pre_execution atomic envelope/hostile read and posting.workflow.post_pre_execution_status |
real immutable Git and local-TLS GitLab identity prove absence at base/policy plus metadata-only regular source blob; hostile file/identity matrix and static posting tests cover malformed, stale, unknown, symlink/hard-link, oversize, emoji, stderr exclusion, and strict/advisory behavior | deterministic setup classification and source-to-sink wording proven; same-CI-identity compromise remains a host residual risk |
| Generated background exceeds current OCR guidance | review_runner._qualify_review_background invokes the resolved preflight-qualified OCR with the exact production refs, selection arguments, rules, and background under review --preview; result finalization owns the post-DLP advisory and pre_execution owns hard-rejection persistence |
exact-argv, bounded output, cleanup, near-miss, advisory spoof/schema/receipt/approval tests, hard-rejection/no-model, hostile status, and static posting tests pass; the exact supported OCR executable independently owns and returns its current recommendation/rejection thresholds | deterministic toolkit orchestration and closed reporting are proven; the numeric advisory is not a warning, DLP/coverage/receipt input, telemetry signal, or approval blocker, while the subsequent actual review remains OCR's independent production revalidation |
| BOLA/confused deputy/service credential mismatch | context.adapters.authorize_and_resolve plus context.broker.acquire_external_records before handle minting |
real stdio and local-TLS peers exercise operator tenant/resource allowlists, request/run mismatch, uniform unavailable, replacement, and cross-tenant denial | proven for toolkit broker/protocol; adapter service truth remains a deployment non-claim |
| Arbitrary URL/ID, SSRF, redirects, traversal, writes | context.recognizers.recognize, broker admission, redirect-free fixed transports, and context.mcp.call_context_tool |
fixed-grammar/malformed-authority/config/redirect tests plus real one-process stdio MCP prove only committed handles reach context_get; arbitrary IDs, URLs, malformed/tampered cursors, unknown args/tools and source-bound cursors fail before store access |
deterministic toolkit path and real local MCP proven; the real OCR run advertised both tools but made zero context calls, so model-driven list/get is not claimed |
| Oversize, omission, TOCTOU, replay | separate request/admission/streaming limits, repeated bounded GitLab digest, aggregate orchestration, and context.store.ContextStore binding |
real child/TLS no-read, request-write timeout, oversize, partial/multiple frame and bounded-pagination mutation; authorization-before-budget, exact/changed duplicate, hard-link/persistence/recursive-readback, version, lifetime and replay cases | proven through production transport owners, committed store, and local MCP readback |
| PII/secret leakage and output laundering | context.dlp.check_text, review_runner._publication_projection, and ocr_result.transform_ocr_result own one inode-checked sink/private-retention projection and receipt replacement after cleanup; explicit local preservation may add only the bounded value-free _private_dlp_decisions attribution |
multibyte/line/PII/secret, bare SHA/build and separator-bearing technical-ID false-positive classes, whole-value, 24-character exact excerpt, hidden-only/nested HTML source, inline/reference/autolink Markdown, private-key/value sanitization, path/decision bounds, and work-bound tests cover retrieval and raw OCR result; unsafe sink units disappear while private-only fields are sanitized without losing a valid manifest; prior review, one-for-one fingerprint matching, approval blocking, summary marker and log event remain | corrected publication behavior and value-free local attribution are proven deterministically through the production transform/sidecar/posting owners. The real triggering run predates the final correction and is not cited as exact-final-tree proof; operators must compare sidecar digests to separately retained private data before classifying a conservative match |
| Identity spoofing | providers.gitlab_discussions.acquire_discussions provider class plus run-local pseudonym; unknown fails closed |
repeated real-TLS provider classification, ordered pagination, unknown actor, mutation, and display-identity exclusion | proven for GitLab acquisition |
| Same-revision CI identity and scope | providers.gitlab_ci.acquire_gitlab_ci_outcomes plus context.ci_outcomes.prepare_ci_outcome_records; protected policy v3 owns exact check names and path prefixes |
exact-head/current-versus-other-pipeline, stale, duplicate retry, pagination, mutation, hostile payload, DLP, store replay, MCP list/get, bootstrap, and approval-independence tests | proven for bounded GitLab acquisition and provider-neutral projection; job truth and model use remain service/model non-claims |
| Prompt injection and authority escalation | fixed adapter fields, static bootstrap authority text, exact marker preambles, and fixed context_list/context_get; context cannot alter policy/tools/commands/suppression/approval |
hostile schema/description and marker-smuggling tests, fixed MCP declarations, separate evidence-call accounting, and receipt approval tests pass; the real OCR run saw the fixed three-tool declaration but did not call context tools | deterministic authority boundaries proven; model judgment and model context consumption remain non-claims |
| Model skips mandatory repository evidence | evidence.project.build_bootstrap places an unclippable static summary-call instruction before repository-derived detail; review_runner._attach_receipt requires OCR-recorded usage |
exact minimum-budget ordering/rendering/truncation tests and zero-call receipt rejection; self-query is independently recorded and never counted; final real OCR 1.9.8 output recorded 44 evidence calls | deterministic bootstrap/receipt gate and real model compliance at the reviewed pre-remediation head proven |
| GitLab CI unresolved inheritance invents defaults | protected examples/gitlab/rules.json GitLab CI rule; optional compiled facts use only the M5 document adapter/broker |
four synthetic cases freeze unknown/proven expectations; real stdio broker records prove compiled true/false delivery; checksum-verified OCR 1.9.8 receives the exact rule and field set during a real review | rule selection/delivery proven; finding judgment remains explicitly non-deterministic and no compiled-config fetch is claimed |
| Session leakage | review_runner.run_evidence_review owner-only OCR HOME, safe absolute non-repository executable resolution, termination deferral across finally cleanup, and atomic result transformation |
orchestration and real stdio MCP prove HOME isolation/context removal, failure/interruption cleanup and cleanup-gated receipt; inode replacement rejects races. Final real OCR completed one pass and left no context/session artifact | deterministic success/failure/interruption containment proven; real OCR success cleanup proven at the reviewed pre-remediation head |
| Second review engine/model semantics | review_runner.run_evidence_review invokes one exact OCR process; native OCR is required for any future separate adjudication |
orchestration tests assert one process/model loop; the final real run completed once with 44 evidence calls, while expected finding/context-use scenarios remain model-dependent | one-engine execution proven; model semantics remain unqualified |
| Requirement or boundary | Production owner and entry point | Required observable result | Evidence | Double boundary and claim limit | State |
|---|---|---|---|---|---|
| Explicit completion/output cap reaches OCR 1.9.10 wire request | provider_config.provider_config_from_environment -> configure.build_config_updates -> generated OCR config -> actual ocr review |
unset preserves OCR's observed max_completion_tokens=58888; explicit 4096 replaces only that wire field |
checksum-verified Darwin arm64 OCR 1.9.10 no-LLM local-gateway probe; reusable compatibility probe; generated-config and installed wheel/sdist tests | controlled local gateway is beyond OCR's real HTTP client and observes request shape only; it does not prove model quality, spending policy, or that 4096 suits every provider |
proven for exact qualified OCR wire contract |
| Configure/preflight provider boundary stays canonical and secret-safe | provider_config.ProviderConfig shared by configure and preflight |
explicit protocol, normalized HTTPS root/terminal endpoint, query handling, auxiliary URL, headers, and request-body controls agree; credentials/fragments/mismatches fail closed | test_runtime_helpers.py, test_environment_contract.py, and installed-artifact tests |
local metadata peers and environment fixtures prove parser/transport wiring, not a live provider's endpoint policy | proven |
| Non-zero retry diagnostics become a closed GitLab failure note | bounded result owner -> provider_failure.parse_retry_report_failure -> posting renderer |
only validated class/phase/status/terminal outcome selects static text; classified raw result/stderr stays private; previous review remains; findings and approval are unreachable | test_provider_failure.py, test_posting_helpers.py, and test_review_runner.py, including the required status/failure/malformed/oversize/privacy matrices |
result fixtures and mocked GitLab writes prove strict parsing, control flow, and rendered payload; no live GitLab write or provider semantic claim | proven for deterministic toolkit policy |
| Requirement or boundary | Production owner and entry point | Required observable result | Evidence | Double boundary and claim limit | State |
|---|---|---|---|---|---|
| Exact OCR 1.10.0 review contract | compatibility gateway -> hosted Linux binary -> manifest/evidence promotion | path-aware comments, semantic grouping/filtering, default-medium two rounds, token/tool accounting, partial budget, inherited 16384 and explicit 4096 completion caps, and closed numeric CLI/effective-value boundaries |
hosted run 32825123658; checksum-pinned compatibility/evidence/ocr-1.10.0.json; independent Darwin arm64 no-LLM gateway probes including actual max-tools loop exhaustion |
deterministic gateway proves the consumed wire/result/loop contract, not provider quality or production spending policy | proven for exact qualified OCR wire contract |
| Normalized max-tools preview reaches enriched production orchestration safely | run_evidence_review collection -> context/store -> MCP composition/self-query -> exact production OCR preview argv |
exact recognized 30 -> 50 diagnostic crosses the gate, emits only a closed operator notice, and does not enter result warnings or approval inputs; unknown output remains rejected |
parser adversarial matrix plus complete enriched/MCP production-caller regression in test_review_runner.py |
a subprocess double beyond the production launcher emits the qualified OCR diagnostic and stops at the model-review boundary; the exact binary probe separately proves its grammar and effective rounds | proven for toolkit orchestration without a model-quality claim |
| Operator effort stays closed and source-independent | configure.review_effort -> generated root effort -> OCR CLI precedence |
unset/empty becomes medium; only `low |
medium | highis accepted; explicit OCR--effort` remains the per-run override |
environment/configuration, runtime-helper, integration, compatibility, and installed wheel/sdist tests |
| Group/round diagnostics cannot gain posting or approval authority | private result -> recursive DLP -> canonical projection -> receipt v6 -> approval | safe group data leaves the canonical projection and eligibility unchanged; PII/secrets are privately sanitized; receipt extensions fail closed | test_review_runner.py, test_posting_approval.py, and BL-017 source-to-signal audit |
result fixtures prove deterministic toolkit isolation, not the semantic quality of OCR grouping | proven |
| Result-file ownership remains toolkit-only | review argument validation -> pre-opened result descriptor -> atomic finalization/cleanup | caller --output, --output=..., -o, and attached short forms fail before preview |
test_review_runner.py and installed-artifact tests |
subprocess doubles prove rejection timing; the exact OCR binary independently proves the new flag exists | proven |
| Failed qualification retains bounded coordination evidence without becoming green | qualification status owner -> always-run issue/artifact handling -> terminal failure step | closed phase/reason/version/run reaches the canonical issue/artifact; raw exception remains job-log-only; aggregate cannot run from failure | compatibility CLI/workflow tests and the repaired workflow contract | static Action tests prove control flow; the next real failed candidate run remains operational readback evidence | proven for workflow contract |
| Requirement or boundary | Production owner and entry point | Required observable result | Evidence | Double boundary and claim limit | State |
|---|---|---|---|---|---|
| Exact OCR 1.10.1 review contract | compatibility gateway -> hosted Linux binary -> manifest/evidence promotion | result/manifest, inherited 16384 and explicit 4096 completion caps, default-medium two rounds, semantic grouping, and behaviorally observed max-tools values remain compatible |
hosted run 32955196785; checksum-pinned compatibility/evidence/ocr-1.10.1.json; independent checksum-verified Darwin arm64 local gateway probe |
deterministic gateways prove the consumed wire/result/loop contract, not provider quality or production spending policy | proven for exact qualified OCR wire contract |
| New upstream rule selection | exact OCR preview -> allowlist and built-in rule resolver | .m input is reviewable and content selects MATLAB or Objective-C behavior without changing the toolkit rule file |
isolated Darwin arm64 no-LLM ocr rules check readback for MATLAB and Objective-C plus upstream semantic audit |
selection evidence proves language/rule routing, not finding quality | proven |
| Bounded upstream session keys stay private | OCR runtime private cache/session owner -> toolkit result boundary | new bounded keys do not enter result, receipt v6, toolkit telemetry, GitLab text, or approval | upstream semantic audit plus unchanged result/manifest probes | source and result evidence prove the consumed boundary, not every upstream internal cache implementation | proven |
| Requirement or boundary | Production owner and entry point | Required observable result | Evidence | Double boundary and claim limit | State |
|---|---|---|---|---|---|
| Exact OCR 1.10.2 review contract | compatibility gateway -> hosted Linux binary -> human-reviewed manifest/evidence promotion | result/manifest, inherited and explicit completion caps, default-medium rounds, semantic grouping, and effective max-tools remain compatible; grouping uses the inherited 16384 cap |
hosted run 33055459209; checksum-pinned compatibility/evidence/ocr-1.10.2.json; adjacent source audit; isolated checksum-verified Darwin arm64 version/help and no-LLM selection/rule checks |
deterministic gateways and previews prove consumed wire/result/selection contracts, not provider quality or production spending policy | proven for exact qualified OCR contract |
| Feature-bearing patch notes cannot silently promote | release-note classifier -> qualification lane -> protected reviewed promotion | a patch containing Features or a conventional feat(...) entry requires human review even when maintenance keywords are also present |
focused classifier regression plus the reviewed correction of run 33055459209 evidence | static release-note classification proves routing, while human review owns semantic impact | proven |
| New language Rules are selected without widening toolkit authority | exact OCR preview/rule resolver -> recommended built-in Rules | .sol and .vy inputs are reviewable and resolve Solidity/Vyper Rules; no toolkit result, DLP, receipt, telemetry, or approval contract changes |
isolated Darwin arm64 no-LLM preview/rule readback plus upstream semantic audit | selection proves language/rule routing, not finding quality | proven |
| Upstream Action/session additions do not claim backlog completion | source audit -> BL-021 and M6 ownership records | Action checkpoint ranges remain upstream-Action-only; session comparison and private grouping/grace records are not toolkit forge parity or telemetry | adjacent source/release audit and backlog/strategy reconciliation | source evidence classifies overlap only; no second forge integration is claimed | proven as partial overlap, backlog remains open |
| Requirement or boundary | Production owner and entry point | Required observable result | Evidence | Double boundary and claim limit | State |
|---|---|---|---|---|---|
| Exact OCR 1.11.0 review contract | compatibility gateway -> hosted Linux binary -> human-reviewed manifest/evidence promotion | status-first grouping inventory preserves status/churn; result/manifest, inherited 16384, an explicitly tested positive override, medium rounds, and effective max-tools remain compatible |
hosted run 33158664020; checksum-pinned compatibility/evidence/ocr-1.11.0.json; strict historical-1.10/1.11.0 inventory fixtures; adjacent source audit |
deterministic gateways prove structural wire/result/loop contracts, not provider quality or production reasoning replay; the tested override is evidence, not a toolkit default or recommendation | proven for exact qualified OCR contract |
| Max-tools help correction does not imply runtime drift | exact numeric CLI and effective loop probes -> public operator docs | omitted/0 remain effective 100; 49 reports 50 but stays effective 100; 50 stays effective 100; 101 becomes effective 101 |
identical hosted 1.10.2 and 1.11.0 evidence plus upstream #1077 source/release audit | behavioral probes prove current embedded-template ownership; they do not promise future OCR defaults | proven unchanged |
| Private reasoning and request controls do not widen toolkit schemas | OCR isolated session -> canonical result projection -> DLP/publication/receipt/approval | provider-native reasoning and tool_choice remain absent from every toolkit public or authority-bearing sink; cleanup precedes publication |
upstream adapter/session audit, result-contract probe, toolkit hostile projection/cleanup regressions, and external configured-environment checklist | local tests prove toolkit containment, while real provider replay remains an external qualification item | toolkit containment proven; provider replay pending external qualification |
| New selection behavior stays at the OCR edge | exact no-LLM preview/rule resolver -> built-in OCR Rules | repository-relative/backslash file_find fallback and .hbs/.mustache rule selection work without changing toolkit evidence authority |
adjacent source audit plus isolated checksum-verified Darwin version/help, rule readback, and soft/hard background previews | selection proves routing, not review finding quality or evidence-pack completion | proven |
| Contextual security-signal classification | DLP-checked published finding -> closed metadata/term matcher -> reviewer guide | explicit vulnerability classes remain promoted while neutral injection domains do not increase the count or effort | issue #153 reproducer, closed positive/neutral separator matrix, deterministic ranking, and immutability regressions in test_posting_helpers.py |
guide analytics only; severity, finding lifecycle, suppression, and approval inputs remain unchanged | proven |
| Requirement or boundary | Production owner and entry point | Required observable result | Evidence | Double boundary and claim limit | State |
|---|---|---|---|---|---|
| One effective reviewed source identity | review_identity.effective_reviewed_sha shared by review, posting, and pre-execution status validation |
a lowercase 40-hex non-zero MR SHA wins; only an absent or all-zero MR SHA may use the exact CI_COMMIT_SHA; malformed populated values, stale receipts, and cross-boundary mismatches fail closed |
strict scalar/type/value matrix, detached setup/background status cases, provider diff-ref assertions, and stale/malformed receipt cases in test_gitlab_provider.py, test_posting_helpers.py, and test_review_runner.py |
environment and local-TLS fixtures prove deterministic parser and production-provider wiring, not GitLab's truthfulness or runner integrity | proven for deterministic identity selection and binding |
| Evidence action receipt cannot be forged or skipped | fixed evidence MCP owner -> owner-only action receipt v3 -> review_receipt.verified_evidence_actions -> inode-checked atomic result publication |
MCP-received attempts cannot exceed OCR by-tool totals, OCR-preparse residuals become count-only unattributed attempts, and only completed actions establish successful use; missing, malformed, incomplete, unwritable, type-confused, overflowed, unattributed, completion-without-attempt, or per-tool-mismatched action attribution blocks finalization before findings, advisories, or approval; a failed or malformed attempt cannot satisfy the mandatory summary | real local MCP action-receipt writes plus missing/extra-key, old-schema, malformed/failed action, concurrency, bounds, type, per-tool, producer-validation, formatting, and approval regressions in test_evidence_actions.py, test_evidence_mcp.py, test_review_runner.py, test_posting_helpers.py, and test_posting_approval.py |
local protocol and hostile files prove toolkit ownership and aggregate reconciliation; they do not prove model judgment, per-call identity, or an uncompromised same-user host | proven for exact attempt/completion integrity boundary |
| Explicit unprotected target remains constrained and comment-only | providers.gitlab.acquire_review_snapshot -> review_runner constrained preparation -> receipt-v8 validation -> posting/approval workflow |
secure default remains protected-only; explicit unprotected mode accepts only actual unprotected state with off or bounded metadata, immutable repository evidence, and exact-target Rules; enriched context, adapters, protected policy, external MCP, accepted decisions, structured guidance, and approval mutation remain unreachable |
real local HTTPS protection-state matrix, real immutable Git Rules reads, privileged-context/external-MCP rejection ordering, omitted target-policy records, hostile receipt fields, all outcome renderings, and unreachable approval-executor regressions across provider, evidence, runner, posting, and approval tests | controlled GitLab and OCR peers prove deterministic toolkit controls, receipt provenance, and call ordering; they do not establish branch governance, live GitLab merge policy, or semantic finding quality | proven for constrained toolkit path; operational protection remains external |
| Exact OCR 1.11.2 compatibility contract | compatibility gateway -> hosted Linux binary -> human-reviewed evidence promotion | prior result/manifest/budget/provider contracts remain compatible; .mjs/.cjs select JavaScript Rules, .cxx/.hxx select C++ Rules, and .svh remains excluded |
hosted run 33508349494; checksum-pinned compatibility/evidence/ocr-1.11.2.json; independent checksum-verified Darwin arm64 no-LLM qualification |
deterministic gateways and rule checks prove consumed structural and selection contracts, not provider-backed review quality | proven for exact qualified OCR contract |
Every top-level test module is classified below. A module can contain more than one evidence class; the strongest class applies only to the named boundary, never to all tests in that file.
| Test module | Primary owners and evidence | Doubles and non-claims |
|---|---|---|
test_actions_cleanup.py |
cleanup planning and bounded deletion policy; static workflow contract | API replacement tests prove classification/idempotence, not live GitHub deletion |
test_cli.py |
parser/dispatch unit contract; source version identity | patched dispatch is wiring only; installed CLI is proven in installed E2E |
test_common_helpers.py |
pure redaction/Markdown/config parsing | environment patching supplies hostile input, not an external integration claim |
test_distribution_contents.py |
real wheel/sdist archive contents | no registry publication claim |
test_evidence_ansible.py |
real temporary Git collection, typed store, MCP dispatcher | dispatcher calls are component evidence, not stdio; installed stdio is elsewhere |
test_evidence_categorize.py |
pure deterministic categorization | no boundary claim |
test_evidence_collectors.py |
parsers plus real immutable Git/store collection and deltas | monkeypatches around read counters or constrained stores prove batching/admission policy only |
test_evidence_composer.py |
parser semantics plus real Git/store/MCP component projection | no Composer execution or Packagist claim |
test_evidence_ecosystems.py |
static architecture/dependency ownership | no runtime integration claim |
test_evidence_framework_plugins.py |
pure static plugin contracts plus real Git/core/store priority cases | patched limits are boundary-condition inputs; no framework runtime execution claim |
test_evidence_go.py |
parser plus real Git/store/MCP component projection | no Go toolchain execution claim |
test_evidence_infrastructure.py |
parser plus real Git/store/MCP component projection | no container/CI execution claim |
test_evidence_invocation.py |
closed environment-to-identifier projection | synthetic mappings, no provider API claim |
test_evidence_javascript.py |
parser plus real Git/store/MCP component projection | no npm/Yarn/pnpm execution or registry claim |
test_evidence_mcp.py |
dispatcher abuse tests and real stdio child-process protocol launch | in-memory serve tests are component evidence; process tests prove stdio/import/PATH |
test_evidence_model.py |
real persistence, atomic replacement, hostile readback, schema and budgets | patched os calls prove error handling/ordering where the filesystem owner is not claimed |
test_evidence_policy.py |
pure closed policy grammar, matching and bounds | no repository acquisition claim |
test_evidence_repository.py |
real Git objects/plumbing, private files, collection/store | subprocess wrappers used for counting/corruption limit those cases to orchestration/parser rejection; neighboring real Git tests prove plumbing |
test_gitlab_provider.py |
local TLS provider transport, real Git fetch/object reads, full read-only review preflight through real store/config/stdio/subprocess | local peer and synthetic child sit beyond production owners; actual OCR selection is separately qualified |
test_install_local_artifact.py |
exact requirement/hash generation unit policy | monkeypatched hash/metadata inputs do not prove pip installation |
test_installed_policy_e2e.py |
clean wheel and sdist-derived wheel, isolated imports, private files, real Git and stdio MCP | package installer/venv are real; no OCR model claim |
test_integration_contracts.py |
static public example/workflow/rules contracts | “integration” here means repository integration configuration, not execution |
test_ocr_compat.py |
qualification policy/unit tests; committed evidence validation | mocked GitHub/download responses prove bounds/retries only; hosted and local harness runs prove actual asset execution |
test_ocr_result_contract.py |
fixed upstream-result parser compatibility | fixture parsing, not OCR execution |
test_operations_docs.py |
static public documentation/workflow contract | no operator or provider execution claim |
test_posting_approval.py |
approval policy, exact-SHA request construction, ordering and fail-closed workflow | API owners are replaced; no live GitLab approval integration is claimed because writes are unsafe in tests |
test_posting_helpers.py |
pure formatting/workflow policy, real Git reads, real local HTTP create serialization and paginated reconciliation reads, real result-file boundaries | mocked GitLab API owner cases prove response/error/workflow behavior only; local peers prove transport/completeness mechanics, not live GitLab semantics |
test_posting_suggestions.py |
pure proof-bound suggestion decisions | fake readers are collaborators beyond the pure decision owner; no Git blob integration claim |
test_provider_failure.py |
strict bounded retry-report v1 parsing and closed provider-neutral reason projection | fixture records prove the toolkit parser only; actual OCR request behavior is separately observed by the compatibility probe |
test_python_support.py |
static metadata/CI support range | supported interpreters are proven by the quality matrix, not this test alone |
test_quality_script.py |
real synthetic Git history for Gitleaks range plus static wrapper policy | fake scanner proves wrapper invocation/range, not secret-detection efficacy; pinned real Gitleaks runs before push |
test_release_authorization.py |
pure authorization rules plus real bounded helper subprocess/filesystem behavior | API response fixtures do not prove GitHub state; release closure requires live readback |
test_release_notes.py |
parser and repository changelog structure | no GitHub Release publication claim |
test_release_receipt.py |
receipt schemas, descriptor-safe files, release workflow policy | mocked provider requests prove request sequencing/parsing only; stable closure requires live registry/GitHub readback |
test_result_contract.py |
pure normalized outcome parser | no OCR execution claim |
test_review_context.py |
real store persistence/hostile reload and MCP component projection | direct dispatcher is not stdio; stdio proof is installed/full-preflight E2E |
test_review_runner.py |
private result/filesystem owner, real child-process launcher, receipt parser; separate wiring tests | patched subprocess/orchestration tests are explicitly unit evidence only |
test_runtime_helpers.py |
config filesystem boundaries and real local preflight HTTP transport; MCP/config parsing | mocked binary and URL_OPENER cases prove version/request/error policy only, not executable/network integration |
test_testpypi_preview.py |
registry-manifest parser and static workflow contract | fixture index payloads do not prove publication; live TestPyPI/PyPI verification is a release gate |
The suite intentionally does not perform live GitLab comment, discussion, cleanup, or approval writes; live GitHub issue/release mutations; or PyPI publication. Their tests prove closed payloads, ordering, fail-closed decisions, transport serialization, and receipt parsing. Release completion requires independent live readback as defined in docs/release.md.
Likewise, a deterministic local LLM gateway proves OCR request/result integration but cannot prove general model judgment. The one OCR 1.9.4 release run exposed an actual MCP argument-shape incompatibility and never read review.merge_request_context; its four code findings therefore do not qualify matching, contradictory, absent/ambiguous, or objective-defect intent calibration. Corrected real installed stdio summary/list/get paths prove transport and queryability only. Model-dependent intent calibration remains a named future qualification item; no mock-selected finding can close it.
| Requirement or boundary | Production owner and entry point | Required observable result | Evidence | Double boundary and claim limit | State |
|---|---|---|---|---|---|
| Exact OCR 1.11.3 compatibility contract | compatibility gateway -> hosted Linux binary -> reviewed evidence promotion | 1.11.2 contracts and ten-extension language probe remain stable; official assets and adjacent release metadata agree | hosted run 33725971286; checksum-pinned compatibility/evidence/ocr-1.11.3.json; independent Darwin arm64 checksum/version readback; adjacent source audit |
deterministic qualification proves consumed wire, structural, and selection contracts, not provider-backed finding quality | proven for exact qualified OCR contract |
| Raw traffic capture cannot be inherited | review owner -> preview and actual OCR child environments | OCR_RAW_LOGGING is absent from both children while other provider configuration remains available |
subprocess-boundary regressions in test_review_runner.py |
child doubles prove exact environment projection; no provider request is made | proven |
| Additive failed-tool diagnostics cannot erase useful review signal | OCR result -> diagnostic parser -> action receipt v3 -> DLP -> receipt v8 -> posting transaction | absent, malformed, hostile, non-zero, and contradictory diagnostics retain valid findings, summary, manifest, and posting; details are bounded redacted console-only data; completed toolkit evidence remains authoritative | hostile parser, local/provider finalization, DLP, formatter, approval, and actual posting-workflow regressions across test_review_runner.py, test_posting_helpers.py, and test_posting_approval.py |
synthetic OCR results and mocked/local GitLab writes prove toolkit control flow and payloads, not upstream tool success or live GitLab semantics | proven for deterministic toolkit policy |
| Decision diagrams stay aligned with executable contracts | public documentation -> runtime/test owner comparison | canonical Mermaid flow distinguishes integrity errors from additive warnings and separates publication from later-action eligibility with stable terminal colors | docs/review-decision-flow.md, fenced/static structure contract tests, and holistic source review |
static diagrams explain implemented decisions; they do not replace runtime validation or a renderer | proven when release gates pass |