FOIL 0.5.1: evidence estimator, enforcement boundary, policy kernel, preregistered four-config benchmark - #27
Draft
Kitahl wants to merge 7 commits into
Draft
FOIL 0.5.1: evidence estimator, enforcement boundary, policy kernel, preregistered four-config benchmark#27Kitahl wants to merge 7 commits into
Kitahl wants to merge 7 commits into
Conversation
…y, guard, broker
Evidence core
- tools/foil_evidence.py: Beta(Jeffreys)-posterior classifier replaces the
count rule (D1); evidence tiers REAL_WORK/SCREEN/ASSISTED/UNVERIFIED with
min_effective_n in real-work units (D2); 180-day recency weight with stale-
only evidence -> INSUFFICIENT_EVIDENCE, never POSSIBLE_GAP (D4); SPRT
log-likelihood cross-check with Wald boundaries; exact false-classification
calculators; SUFFICIENCY_TOLERANCE closes a knife-edge at exactly
min_effective_n caused by sub-microsecond recency decay.
- tools/foil_assistance.py: single Assistance enum (A0-A4, append-only) with
legacy aliases, ExecutionOwner {USER, SHARED, TOOL}, and
independent_mastery_eligible(); generated ladder block for SKILL.md (D3).
- tools/foil_interventions.py: time-ordered, refutable ledger; most-recent
admissible outcome per phase; content-addressed ids; verifier required.
- All four duplicate classifiers (profile, assessment, layer2, calibration)
now call foil_evidence.summarize; screen reports expose screen_signal for
probe routing so a two-item screen routes but never asserts competence.
Profile schema v2
- tools/foil_profile.py: verified/verifier/execution_owner on observe();
conservative v1->v2 migration on load (legacy rows verified=false,
assessment rows -> SCREEN tier); derived counters recomputed from events;
migration preserves migrated_from provenance and is idempotent.
Registry, guard, broker
- tools/foil_capabilities.py + foil_tool_policy.py: REQUIRED_ATTRIBUTES
validated; select_provider(require_write=True) raises CapabilityWriteError;
route_claim returns REFUSED_WRITE (D5).
- tools/foil_task_guard.py: guarded_operation() brackets every spend;
reservation holds budget immediately; refund on error; SHA-256 hash-chained
events with attest(); fcntl.flock on POSIX and msvcrt LK_NBLCK retry loop on
Windows, kernel-released on process death; single descriptor close;
timeout now forwarded on the Windows path (D7).
- tools/foil_tool_broker.py: PreToolUse hook that enforces the budget and the
write boundary at actual dispatch; inert only when FOIL_TASK_RUN is unset;
any partially configured or broken run fails closed for brokered tools (D6,
closed only under the hook; the ledger stays advisory elsewhere).
- .claude/settings.json registers the broker first in PreToolUse; .gitignore
adds .foil/ and benchmark_runs/**/browsecomp_review_private.json.
Tests: 46 -> 171 (unittest discover, 1 expectedFailure reserved for the
SKILL.md contract-drift test landing in the docs lane); ruff and compileall
clean. validation/FOIL_LEDGER_AUDIT_2026-08-23.md records the audit of the
external update ledger, including the re-scored vNext evidence (35/36, not
32/36) and the corrected V2 locator (9540860).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…, ledger B-items - tools/foil_policy.py: RuntimePolicyV2 ported verbatim (post-docstring byte-identical) from experiments/foil_vnext/runtime_policy_v2.py at 9540860; V2 mechanisms are implemented hypotheses, efficacy NOT MEASURED. tests/test_foil_policy_v2.py ports the 12 V2 assertions. - benchmarks/harness/bench_foil_session.py: session bench with an explicit bucket->kernel adapter (_to_policy_inputs) whose values derive from V2's own gate thresholds; decisions taken with/without profile so budget and verifier deltas are measured, not clamped; positive control routed_states (336) guards against vacuous invariants; REFERENCE fallback exits 1 unless --allow-reference. 14/14 invariants over 1,161,440 decide() calls. Falsification controls: 13 of 14 invariants fail under mutant kernels (I10 is tautological under this adapter and is documented as such). - tools/foil_models.py + foil_setup.py: provider-neutral adapters; cli family gains output_parser "claude_json" (returns .result, records session_id, total_cost_usd, num_turns, duration_ms; ModelError on non-JSON or missing result); prompt via stdin when no {prompt} token; doctor exits 1 with a named finding when primary == reviewer. docs/FOIL_MODEL_SETUP.md. - Ledger B-items: B1 migration receipt + derivation_version (profile_sha256 excludes wall-clock fields so migration stays deterministic); B3 hardened compact_context/hook (closed vocabulary, sanitized free text incl. angle brackets, 4,000-char budget, as_of/profile_sha256/derivation_version header, fail-soft hook); B4 gap_kinds_contract_block + drift test (expectedFailure until SKILL.md carries the block); B6 guard binding records model/effort/allowed_tools/isolation_session_id with a sidecar session claim that fails closed on reuse. - benchmark_runs/2026-08-23/foil_session_receipt.json: policy_source foil_policy.RuntimePolicyV2, gold_opened=false. Tests: 171 -> 248 (unittest discover), 2 expectedFailures owned by the SKILL.md lane; ruff and compileall clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ered four-config benchmark
Skill and docs
- skills/foil/SKILL.md: generated ladder (A0-A4 + ExecutionOwner) and gap-kind
blocks pasted verbatim; both contract-drift tests now pass (no
expectedFailure left in the suite); estimator, freshness gate, broker
boundary and task-property routing described; benchmark identity is
receipt metadata only.
- CHANGELOG [0.5.1] with the D1-D11 closure table and the not-adopted /
deferred list; VERSION 0.5.1; CITATION.cff; README paragraph;
docs/ARCHITECTURE.md module map + broker boundary; docs/
FOIL_EVIDENCE_ESTIMATOR.md with measured error tables and the SPRT cross-
check (20/1 -> 12.764 vs 2.944); research/FOIL_RESEARCH_BASIS.md locator
ledger (12 locators flagged not re-fetched, 3 fetched).
Freshness gate (D4, research ruling)
- tools/foil_evidence.py: EvidencePolicy.freshness_horizon_days = 360 (2x
half-life, UNRESOLVED); when admissible real-work evidence exists but none
is inside the horizon the verdict is INSUFFICIENT_EVIDENCE with
stale_only=True. Measured regression: 80 misses aged 3600 d were
POSSIBLE_GAP under the weight floor alone; now INSUFFICIENT_EVIDENCE, and
one fresh miss restores the decision (positive control).
Preregistered benchmark (D10)
- benchmarks/CLAUDE_FOUR_CONFIG_PROTOCOL.md: BASE vs FOIL on the same items
under {sonnet,opus} x {low,high}; sealed condition map (sha preregistered,
LF digest); identical flags except --append-system-prompt-file and the
"/foil solve" prefix; fresh cwd, settings containing only the broker hook,
child env stripped; GPQA-Diamond 24 (no tools), BrowseComp 12 (12 search +
12 follow-up budget via the broker); one replicate, declared EXPLORATORY
and NOT POWERED with the measured power figures; mid-p McNemar primary,
exact conditional sensitivity, Wilson CIs, Holm across configs; gold never
opened before predictions are committed.
- benchmarks/harness/paired_stats.py (stdlib McNemar exact/mid-p, Wilson,
Holm) and claude_four_config_runner.py (prepare/--check-only, run/
--dry-run, score that refuses uncommitted predictions); GPQA answers must
match ^[A-D]$ or the unit is INVALID; is_error and subtype surfaced by the
claude_json parser and both consulted.
- docs/BENCHMARKS.md: new section for the unrun contract test; the vNext
evidence recorded as the re-scored descriptive 35/36 with caveats, never
as validated.
- foil-benchmark.yml: offline --check-only step + paired_stats tests.
Tests: 248 -> 321 (unittest discover, 0 expected failures); ruff,
compileall, validate_soul_gauntlet_public.py and prepare --check-only
(14/14) clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… tip, leak scan From the adversarial review of a7f39e0 (every finding reproduced before repair): - B1 (medium): the broker's PreToolUse matcher omitted MCP tools, so MCP retrieval spent no budget; matcher now includes mcp__.* and a wiring test pins it (exec-form args substitution confirmed from the hooks docs). - C1 (medium): attest() accepted tail truncation; state now records event_count + chain head on every append, and attest() replays the spend events and fails on count, head, or used/replay mismatch. - B2 (low): malformed hook stdin denied ("hook payload unreadable; failing closed") when a run is configured; inert when FOIL_TASK_RUN is unset. - B4 (low): classify_tool strips and case-folds tool names consistently with the IGNORECASE MCP patterns. - D1: private-leak scan scoped to tracked files; path-shaped AppData and OneDrive patterns plus an enumerating test that permits only the %APPDATA% environment references the Windows profile path needs. - E1: score refusal on uncommitted predictions is a one-line REFUSED message. Tests: 321 -> 350 OK; ruff, compileall, public validator clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
10 tasks
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
…symlink) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…wcase validate_showcase.py hard-coded 0.5.0, so the 0.5.1 bump failed the release_identity_consistent check in CI. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two more 0.5.0 literals (showcase-revision separation and visuals VERSION consistency) failed CI after the 0.5.1 bump. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Kitahl
added a commit
that referenced
this pull request
Aug 23, 2026
* FOIL 0.5.1 phase 1+2: evidence estimator, vocabulary, ledger, registry, guard, broker
Evidence core
- tools/foil_evidence.py: Beta(Jeffreys)-posterior classifier replaces the
count rule (D1); evidence tiers REAL_WORK/SCREEN/ASSISTED/UNVERIFIED with
min_effective_n in real-work units (D2); 180-day recency weight with stale-
only evidence -> INSUFFICIENT_EVIDENCE, never POSSIBLE_GAP (D4); SPRT
log-likelihood cross-check with Wald boundaries; exact false-classification
calculators; SUFFICIENCY_TOLERANCE closes a knife-edge at exactly
min_effective_n caused by sub-microsecond recency decay.
- tools/foil_assistance.py: single Assistance enum (A0-A4, append-only) with
legacy aliases, ExecutionOwner {USER, SHARED, TOOL}, and
independent_mastery_eligible(); generated ladder block for SKILL.md (D3).
- tools/foil_interventions.py: time-ordered, refutable ledger; most-recent
admissible outcome per phase; content-addressed ids; verifier required.
- All four duplicate classifiers (profile, assessment, layer2, calibration)
now call foil_evidence.summarize; screen reports expose screen_signal for
probe routing so a two-item screen routes but never asserts competence.
Profile schema v2
- tools/foil_profile.py: verified/verifier/execution_owner on observe();
conservative v1->v2 migration on load (legacy rows verified=false,
assessment rows -> SCREEN tier); derived counters recomputed from events;
migration preserves migrated_from provenance and is idempotent.
Registry, guard, broker
- tools/foil_capabilities.py + foil_tool_policy.py: REQUIRED_ATTRIBUTES
validated; select_provider(require_write=True) raises CapabilityWriteError;
route_claim returns REFUSED_WRITE (D5).
- tools/foil_task_guard.py: guarded_operation() brackets every spend;
reservation holds budget immediately; refund on error; SHA-256 hash-chained
events with attest(); fcntl.flock on POSIX and msvcrt LK_NBLCK retry loop on
Windows, kernel-released on process death; single descriptor close;
timeout now forwarded on the Windows path (D7).
- tools/foil_tool_broker.py: PreToolUse hook that enforces the budget and the
write boundary at actual dispatch; inert only when FOIL_TASK_RUN is unset;
any partially configured or broken run fails closed for brokered tools (D6,
closed only under the hook; the ledger stays advisory elsewhere).
- .claude/settings.json registers the broker first in PreToolUse; .gitignore
adds .foil/ and benchmark_runs/**/browsecomp_review_private.json.
Tests: 46 -> 171 (unittest discover, 1 expectedFailure reserved for the
SKILL.md contract-drift test landing in the docs lane); ruff and compileall
clean. validation/FOIL_LEDGER_AUDIT_2026-08-23.md records the audit of the
external update ledger, including the re-scored vNext evidence (35/36, not
32/36) and the corrected V2 locator (9540860).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* FOIL 0.5.1 phase 3: policy kernel port, session bench, model adapters, ledger B-items
- tools/foil_policy.py: RuntimePolicyV2 ported verbatim (post-docstring
byte-identical) from experiments/foil_vnext/runtime_policy_v2.py at
9540860; V2 mechanisms are implemented hypotheses, efficacy NOT MEASURED.
tests/test_foil_policy_v2.py ports the 12 V2 assertions.
- benchmarks/harness/bench_foil_session.py: session bench with an explicit
bucket->kernel adapter (_to_policy_inputs) whose values derive from V2's
own gate thresholds; decisions taken with/without profile so budget and
verifier deltas are measured, not clamped; positive control routed_states
(336) guards against vacuous invariants; REFERENCE fallback exits 1 unless
--allow-reference. 14/14 invariants over 1,161,440 decide() calls.
Falsification controls: 13 of 14 invariants fail under mutant kernels
(I10 is tautological under this adapter and is documented as such).
- tools/foil_models.py + foil_setup.py: provider-neutral adapters; cli family
gains output_parser "claude_json" (returns .result, records session_id,
total_cost_usd, num_turns, duration_ms; ModelError on non-JSON or missing
result); prompt via stdin when no {prompt} token; doctor exits 1 with a
named finding when primary == reviewer. docs/FOIL_MODEL_SETUP.md.
- Ledger B-items: B1 migration receipt + derivation_version (profile_sha256
excludes wall-clock fields so migration stays deterministic); B3 hardened
compact_context/hook (closed vocabulary, sanitized free text incl. angle
brackets, 4,000-char budget, as_of/profile_sha256/derivation_version
header, fail-soft hook); B4 gap_kinds_contract_block + drift test
(expectedFailure until SKILL.md carries the block); B6 guard binding
records model/effort/allowed_tools/isolation_session_id with a sidecar
session claim that fails closed on reuse.
- benchmark_runs/2026-08-23/foil_session_receipt.json: policy_source
foil_policy.RuntimePolicyV2, gold_opened=false.
Tests: 171 -> 248 (unittest discover), 2 expectedFailures owned by the
SKILL.md lane; ruff and compileall clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* FOIL 0.5.1 phase 4+5: skill contract, docs, freshness gate, preregistered four-config benchmark
Skill and docs
- skills/foil/SKILL.md: generated ladder (A0-A4 + ExecutionOwner) and gap-kind
blocks pasted verbatim; both contract-drift tests now pass (no
expectedFailure left in the suite); estimator, freshness gate, broker
boundary and task-property routing described; benchmark identity is
receipt metadata only.
- CHANGELOG [0.5.1] with the D1-D11 closure table and the not-adopted /
deferred list; VERSION 0.5.1; CITATION.cff; README paragraph;
docs/ARCHITECTURE.md module map + broker boundary; docs/
FOIL_EVIDENCE_ESTIMATOR.md with measured error tables and the SPRT cross-
check (20/1 -> 12.764 vs 2.944); research/FOIL_RESEARCH_BASIS.md locator
ledger (12 locators flagged not re-fetched, 3 fetched).
Freshness gate (D4, research ruling)
- tools/foil_evidence.py: EvidencePolicy.freshness_horizon_days = 360 (2x
half-life, UNRESOLVED); when admissible real-work evidence exists but none
is inside the horizon the verdict is INSUFFICIENT_EVIDENCE with
stale_only=True. Measured regression: 80 misses aged 3600 d were
POSSIBLE_GAP under the weight floor alone; now INSUFFICIENT_EVIDENCE, and
one fresh miss restores the decision (positive control).
Preregistered benchmark (D10)
- benchmarks/CLAUDE_FOUR_CONFIG_PROTOCOL.md: BASE vs FOIL on the same items
under {sonnet,opus} x {low,high}; sealed condition map (sha preregistered,
LF digest); identical flags except --append-system-prompt-file and the
"/foil solve" prefix; fresh cwd, settings containing only the broker hook,
child env stripped; GPQA-Diamond 24 (no tools), BrowseComp 12 (12 search +
12 follow-up budget via the broker); one replicate, declared EXPLORATORY
and NOT POWERED with the measured power figures; mid-p McNemar primary,
exact conditional sensitivity, Wilson CIs, Holm across configs; gold never
opened before predictions are committed.
- benchmarks/harness/paired_stats.py (stdlib McNemar exact/mid-p, Wilson,
Holm) and claude_four_config_runner.py (prepare/--check-only, run/
--dry-run, score that refuses uncommitted predictions); GPQA answers must
match ^[A-D]$ or the unit is INVALID; is_error and subtype surfaced by the
claude_json parser and both consulted.
- docs/BENCHMARKS.md: new section for the unrun contract test; the vNext
evidence recorded as the re-scored descriptive 35/36 with caveats, never
as validated.
- foil-benchmark.yml: offline --check-only step + paired_stats tests.
Tests: 248 -> 321 (unittest discover, 0 expected failures); ruff,
compileall, validate_soul_gauntlet_public.py and prepare --check-only
(14/14) clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* FOIL 0.5.1 review repairs: broker matcher, fail-closed payload, chain tip, leak scan
From the adversarial review of a7f39e0 (every finding reproduced before repair):
- B1 (medium): the broker's PreToolUse matcher omitted MCP tools, so MCP
retrieval spent no budget; matcher now includes mcp__.* and a wiring test
pins it (exec-form args substitution confirmed from the hooks docs).
- C1 (medium): attest() accepted tail truncation; state now records
event_count + chain head on every append, and attest() replays the spend
events and fails on count, head, or used/replay mismatch.
- B2 (low): malformed hook stdin denied ("hook payload unreadable; failing
closed") when a run is configured; inert when FOIL_TASK_RUN is unset.
- B4 (low): classify_tool strips and case-folds tool names consistently with
the IGNORECASE MCP patterns.
- D1: private-leak scan scoped to tracked files; path-shaped AppData and
OneDrive patterns plus an enumerating test that permits only the
%APPDATA% environment references the Windows profile path needs.
- E1: score refusal on uncommitted predictions is a one-line REFUSED message.
Tests: 321 -> 350 OK; ruff, compileall, public validator clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Compare resolved paths in the settings-file test (macOS /private/var symlink)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Derive showcase release identity from VERSION; show v0.5.1 in the showcase
validate_showcase.py hard-coded 0.5.0, so the 0.5.1 bump failed the
release_identity_consistent check in CI.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Showcase validator: derive every software-version token from VERSION
Two more 0.5.0 literals (showcase-revision separation and visuals VERSION
consistency) failed CI after the 0.5.1 bump.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Typed-runtime overlay on FOIL 0.5.1 (stacked candidate)
Applies the vNext typed-runtime overlay (source commit 9e876ed, authored
against v0.5.0) on top of FOIL 0.5.1 (a7f39e0) with three-way resolution:
- tools/foil_hook.py: FOIL's budgeted, sanitized, fail-soft context path is
kept in full; the overlay's typed-runtime bridge (record_prompt_adaptation,
typed_runtime status in the task line) is merged on top. The overlay's
unbudgeted print path is not taken.
- docs/ARCHITECTURE.md and docs/RUNTIME_SETUP.md: FOIL sections retained;
overlay additions folded in.
- .claude/settings.json: FOIL tool broker stays first in PreToolUse.
- portability.yml gate job + test_portability_has_stable_required_gate;
pinned action SHAs and --require-hashes untouched; requirements-lock.txt
byte-unchanged (zero new dependencies).
Repairs to the overlay as shipped: docs/VNEXT_RUNTIME_PIPELINE.md was
missing the SPEC/STATE/ACTION/RECEIPT/VERDICT paragraph its own validator
requires (restored from 9e876ed); 32 ruff violations in overlay-new files
fixed without changing any assertion; LF line endings restored per
.gitattributes.
Verified on Windows: unittest 361 OK (321 FOIL + 39 egrt + 1 gate),
validate_vnext_runtime 27/27, validate_soul_gauntlet_public PASS, ruff and
compileall clean, private-leak and release-hardening tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Close false-CLEARED paths in the typed release gate
Audit of the vNext typed runtime found ~10 ways an obligation could report
CLEARED without evidence. Fixes, each guarded by a test in
tests/test_egrt_gates.py that fails against the pre-fix code:
- egrt_store: monotonic seq in each receipt body; order by (stored_at, seq)
so a clock tie no longer lets an older CLEARED beat a newer ISSUE. Task
file gains a content_hash and read_task verifies it. write_receipt stamps
task_id centrally so the gate can reject a receipt from another task.
- soul_runtime.release_gate: module ownership defaults to MODULE_FOR_KIND
(no fail-open None); verdict aggregation ISSUE > UNAVAILABLE > UNKNOWN >
CLEARED so UNAVAILABLE/ISSUE cannot be masked.
- gauntlet_runtime: an empty trace is "not-applicable" UNKNOWN, not a green
ASSURANCE receipt (7 monitor branches).
- time_runtime: CLEARED only when the candidate is significantly better
(Holm-adjusted across a multiplicity family); worse -> ISSUE, inconclusive
-> UNKNOWN. Exact McNemar via Fraction/comb (no OverflowError at large n,
never returns p=0). Frozen item_ids manifest; duplicate/off-list exclusions
raise.
- council_runtime: commit stores only a nonce-bound commitment; the
submission is unreadable until a matching reveal. Control must match the
council obligation_id; finalize refuses after CLOSE.
- power_runtime: verifier argv[0] resolved (shutil.which); python -m modules
and pytest -p plugins restricted unless EGR_POWER_ALLOW_CUSTOM_COMMANDS=1.
- egrt_hook/foil_hook: PostToolUseFailure hook emits action.failed; non-dict
stdin no longer crashes; FOIL clears ADAPTATION only on an explicit alias,
not on every prompt.
- private_io: os.replace retry + advisory lock (no lost obligations under
concurrent add); double-close removed.
- validate.yml runs validate_vnext_runtime.py; test_runtime_tools no longer
writes repo-root .egrt.
421 tests OK (390 + 31 new gate tests); ruff clean; both validators pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Fix mobile showcase overflow
* Trigger main-branch validation
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Codex Integration Check <codex-integration@local.invalid>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Closes the ten execution-derived FOIL defects D1–D10 (non-monotone classifier, two-observation verdicts, disjoint assistance vocabularies, no recency/refutation, dead
writesmetadata, advisory budget guard, stale lock/double close, prose patches, unfalsifiable receipt invariants, n=5 benchmark) plus D11 model-agnostic adapters, following the 2026-08-23 research report and the audited update ledger (validation/FOIL_LEDGER_AUDIT_2026-08-23.md). Preregisters — but does not run — a paired four-config Claude contract test.Mechanism
tools/foil_evidence.py: Beta(Jeffreys) posterior classifier with evidence tiers,min_effective_nin real-work units, 180-day recency weight and a freshness gate (stale-only evidence →INSUFFICIENT_EVIDENCE, never a gap), SPRT cross-check.tools/foil_assistance.py/foil_interventions.py: singleAssistance+ExecutionOwnerenums, generated SKILL.md blocks with drift tests; refutable time-ordered ledger.tools/foil_profile.py: schema v2 (verified/verifier/execution_owner), conservative v1→v2 migration with receipt; hardened, budgetedcompact_context/hook.tools/foil_capabilities.py/foil_tool_policy.py: registry governs writes (CapabilityWriteError).tools/foil_task_guard.py/foil_tool_broker.py: hash-chained ledger (tip + replay attestation) and a PreToolUse broker that enforces budget/write boundaries at dispatch (fail-closed; inert only whenFOIL_TASK_RUNis unset).tools/foil_policy.py:RuntimePolicyV2ported byte-identical fromexperiment/foil-vnext5-vnext@9540860;benchmarks/harness/bench_foil_session.pyenumerates 14 invariants over 1,161,440 states with a positive control (routed_states=336).tools/foil_models.py/foil_setup.py: provider-neutral adapters incl.claude_json.benchmarks/CLAUDE_FOUR_CONFIG_PROTOCOL.md+ runner +paired_stats.py: preregistered BASE vs FOIL on the same items under {sonnet,opus}×{low,high}; sealed condition map (sha pinned); gold never opened before predictions are committed.Evidence / verification
Baseline / ablation
Baseline v0.5.0 (
ba03be5): count rule (correct=20, incorrect=1 -> UNCERTAIN), noverifiedfield,O_EXCLlock, advisoryauthorize(). Each defect has a pinned regression test showing the old behaviour fails and the new passes. The four-config benchmark is not yet run; the vNext evidence indocs/BENCHMARKS.mdis a descriptive re-score (35/36) and is labelled non-validated.Limitations and residual risk
claude -pauth); the protocol is declared exploratory and not powered (n=24/12, 1 replicate).Compatibility
Reproducibility
CHANGELOG.mdupdated if user-visible🤖 Generated with Claude Code