Skip to content

benchmark.md

Latest commit

 

History

History
75 lines (48 loc) · 4.29 KB

benchmark.md

File metadata and controls

75 lines (48 loc) · 4.29 KB
Raw

Deterministic v0.2 microbenchmark

The checked-in benchmark is a deterministic local governance microbenchmark. It exercises candidate selection, passthrough decisions, receipt creation, and protected-fact preservation. It does not call a model or provider API.

Reproduce

From the repository root with Python 3.10-3.14:

python -m benchmarks.run --manifest benchmarks/fixtures/manifest.json --output .tgl/benchmark-v0.2.0.json

The runner validates the manifest integrity hash, fixed seed, result schema, exact case identities, expected actions/reasons, receipt expectations, and preservation evidence before writing canonical JSON. Compare the generated bytes with the checked-in result:

python -c "from pathlib import Path; a=Path('.tgl/benchmark-v0.2.0.json').read_bytes(); b=Path('benchmarks/results/v0.2.0.json').read_bytes(); raise SystemExit(0 if a == b else 1)"

Verify both README result bindings:

python -m benchmarks.check_readme --results benchmarks/results/v0.2.0.json --readme README.md
python -m benchmarks.check_readme --results benchmarks/results/v0.2.0.json --readme README.zh-CN.md

The checked-in result is benchmarks/results/v0.2.0.json; its SHA-256 is 8146af8b74d92abc4356654dbab124956c0527243fc928876d97214adf0fe18b.

Fixtures

benchmarks/fixtures/manifest.json uses fixed seed 20260715 and contains 14 named cases:

Group Case IDs Contract
Registered transforms repetitive_log, test_output, build_output, protected_middle, long Transform, create a receipt, and preserve protected evidence
Secret passthrough secret_first, secret_middle, secret_end Detect secret-like text at different positions, passthrough, no receipt
Conservative passthrough source_passthrough, json_passthrough, unicode, empty, malformed, large Reject unsupported, not-smaller, malformed, or unmatched candidates without a receipt

Fixture secret strings are synthetic detector inputs, not credentials. The large case represents an unmatched larger fixture; it is not intended to reach the runtime maximum-payload boundary.

Result schema

benchmarks/schema/result-v1.schema.json defines a closed result-v1 object. Top-level fields include:

  • version, label, seed, manifest digest, methodology, and portable environment contract;
  • one row per manifest case with action, risk, strategy, reason, preservation booleans, estimated counts, protected-fact evidence, and receipt creation;
  • aggregate case and estimated-token totals.

The runner performs schema validation with repository code and adds cross-field checks that JSON Schema alone does not express: exact ID/order correspondence, fixed version/seed/digests, valid closed enum values, expected receipt behavior, per-case arithmetic, aggregate arithmetic, and successful preservation.

Methodology

Each case is converted into the same GovernanceRequest used by runtime adapters and executed through a local GovernanceEngine with an isolated temporary SQLite ledger. For each case the runner records:

  1. whether the engine transformed or passed through;
  2. the selected registered strategy and fixed reason/risk values;
  3. the engine's independent preservation result;
  4. a second evidence comparison against the manifest's protected facts;
  5. local estimated tokens before, after, and the difference;
  6. whether a transformed-only receipt was created.

estimate_tokens is the repository's deterministic Unicode word/punctuation estimator. It is intentionally not a Claude tokenizer and is labeled estimated_candidate_microbenchmark in the result.

Checked-in result

The aggregate in v0.2.0 is:

Cases Transformed Passthrough Estimated before Estimated after Estimated candidate saved
14 5 9 348 307 41 (11.7816%)

All case-level preservation fields are true. The result demonstrates only the behavior of these fixed fixtures under the local estimator.

Limitations

This benchmark does not measure provider-billed tokens, prompt caching, model context accounting, latency under production load, downstream task success, answer quality, restoration frequency, or behavior across arbitrary repositories. It is not an A/B model evaluation and must not be converted into a provider cost or task-quality claim.