feat(workflows): evidence operations loop for agent session optimization - #414
Merged
Conversation
Adds four operator-facing workflows, each producing an auditable artifact rather than an estimate: - `entroly audit` — content-blind accounting over historical agent sessions, counting per-session usage without double-counting cumulative totals. - `entroly trial` — paired baseline/optimized experiments. One consented arm is recorded at a time; agent tasks are stateful and costly, so the same task is never auto-run twice. Fewer than three balanced runs stays "insufficient evidence" rather than a win. - `entroly shrink` — bounded command-output envelopes, streamed rather than buffered whole. - `entroly browser` — structure-aware selection over accessibility context. Supporting surfaces: reversible response contracts, three extractive codecs for diff/search/markup routes, and marker-gated installers for Codex, Claude and Gemini bundles that uninstall into a recoverable directory instead of deleting. Malformed trial evidence is quarantined rather than averaged in: integer "booleans", non-finite costs, and receipts without a valid command digest cannot influence a comparison. Validation: 32 feature tests and 88 affected-surface tests pass, ruff is clean, and the four commands are reachable from `python -m entroly`.
…parses
Adapter selection keyed off the *root label*. `custom_roots` labels whatever
the operator points at `custom-N`, so a Codex export audited through
`--history-root` matched neither the codex nor the claude adapter and fell
through to the generic scanner — which does not reach
`payload.info.total_token_usage`. The file contributed nothing and was not even
recorded as unknown-semantics.
Measured on one rollout, same bytes both ways:
default root blocks=3 input=900
--history-root blocks=0 input=0
Silence is the worst outcome available here. No error was raised; the report
still rendered, under a claim boundary that reads as though something had been
audited. A zero that looks like a finding.
Recovers the adapter from the record's shape when the root label is not already
a known agent, so a recognised adapter's semantics are never reinterpreted.
Both paths now agree at blocks=3 input=900, and the cumulative contract still
holds through the custom path — the peak, not the sum of every reported total
(900, not 1140).
Found by driving the workflow end to end rather than trusting that the command
parsed. Removing the inference fails the new test.
`provider_input_token_difference` was gated on `comparable` (>= 1 balanced run)
while `status` required three. A single run therefore reported:
status insufficient-evidence
provider_input_token_difference 4200
A difference is a comparison, so it must not exist while the status says the
comparison is unsupported. Any consumer reading the number without the status —
a dashboard, a README, a launch claim — renders one run as a 4,200-token win,
which is exactly what the claim boundary printed beside it forbids: "Three
matched runs permit a directional operational comparison only."
Measured before and after, same receipts:
runs/arm status difference (before -> after)
1 insufficient-evidence 4200 -> None
2 insufficient-evidence 8400 -> None
3 directional 12600 -> 12600
Per-arm totals stay populated at any run count; those are observations. It is
their difference that is a claim.
Found by driving the workflow rather than reading it. Re-gating on `comparable`
fails the new test.
`_resolved_wrap_env` now calls `environment_contract()`, which resolves
`contract_path()` and therefore `Path.home()`. `Path.home()` picks its flavour
from `os.name`, so any caller that has swapped it makes `Path` construction
raise on POSIX — `pathlib.UnsupportedOperation` on 3.13+, plain
`NotImplementedError` earlier, both reported as "cannot instantiate
'WindowsPath' on your system".
`tests/test_cli.py::test_wrap_never_retries_user_arguments_through_a_shell`
does exactly that to exercise the Windows shim path, and `cli.os` *is* the `os`
module, so the swap is process-wide. On main that test constructed no Path and
passed; adding the contract lookup to the wrap path turned an optional
environment pointer into a hard failure of the wrap command on every Linux
wheel build:
FAILED tests/test_cli.py::test_wrap_never_retries_user_arguments_through_a_shell
pathlib._abc.UnsupportedOperation: cannot instantiate 'WindowsPath'
1 failed, 359 passed
Resolution failures now fold into the empty result the function already returns
when no contract exists, debug-logged rather than silent. A wrapped agent
losing an optional pointer is not a reason to fail the user's command.
This could not be reproduced on a Windows host, where `WindowsPath` is the
native flavour — the guard is unit-tested directly instead of relying on the
platform to raise.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four operator-facing workflows, each producing an auditable artifact rather than an estimate.
Commands
entroly auditentroly trialentroly shrinkentroly browserThe design decision that matters
trialdoes not auto-run the same coding task twice. Agent tasks are stateful and costly, so re-running one is not a clean counterfactual. It records one explicitly consented arm at a time and compares matched receipts; fewer than three balanced runs stays "insufficient evidence", not a win.Malformed trial evidence is quarantined rather than averaged in — integer "booleans", non-finite costs, and receipts without a valid command digest cannot influence a comparison.
Supporting surfaces
os.replace,0o600, temp cleanup infinally).Verification
main: 67 passed, including release-surface and distribution checks.ruffclean acrossentroly/,tests/,scripts/.python -m entroly— checked against the CLI parser, not just imported.NotImplementedError. The three barepassstatements are narrow cleanup idioms (except FileNotFoundErroraround a temp unlink afteros.replace, and a best-effort proxy-bypass restore infinally) — none are fail-open verification paths.mainwas merged in before pushing; no conflicts.