Skip to content

feat(loop): builder/verifier convergence controls (#680) - #748

Merged
kunaldhongade merged 1 commit into
mainfrom
feature/680-builder-verifier-loop-close
Aug 6, 2026
Merged

feat(loop): builder/verifier convergence controls (#680)#748
kunaldhongade merged 1 commit into
mainfrom
feature/680-builder-verifier-loop-close

Conversation

@kunaldhongade

@kunaldhongade kunaldhongade commented Aug 6, 2026

Copy link
Copy Markdown
Member

Summary

  • Finish independent builder/verifier loop controls: AC terminals verified/shallow-proof, verifier hypothesis proposals (advisory only), path/model/wall budgets, protected-path + oscillation stops, and inspectable .codedecay/local/loop-audit persistence with resume counters.
  • Record repair / challenge / current-tree-reverify decisions in the loop state machine; verifier still cannot edit or self-verify.
  • Add UAT-LOOP-1/2/4/5 coverage with deterministic fake builder/verifier processes.

Closes #680

Test plan

  • vitest run packages/harness/test/loop.test.ts
  • vitest run packages/cli/test/loop.test.ts
  • vitest run packages/cli/test/loop-e2e.test.ts
  • tsc -p tsconfig.base.json --noEmit
  • CI green on this PR

Made with Cursor

@kunaldhongade
feat(loop): finish builder/verifier convergence controls for #680
532673f 
Emit verified/shallow-proof terminals, wire verifier hypotheses, enforce
path/model/wall budgets with oscillation stops, persist loop audit, and cover
UAT-LOOP-1/2/4/5 with deterministic fake agents.
@github-actions github-actions Bot added documentation Improvements or additions to documentation type: test Test coverage, fixtures, or verification improvements area: cli CLI package or command behavior area: docs README, community files, or documentation area: harness Agent and tool harness interfaces labels Aug 6, 2026
@github-actions

github-actions Bot commented Aug 6, 2026

Copy link
Copy Markdown

CodeDecay PR Check

Lead catch: Api area changed — packages/harness/src/loop/controller.ts:3

packages/harness/src/loop/controller.ts touches a api area and should be reviewed for regression impact.

Risk: High · Merge 100/100 · Decay 54/100 · Security 0/100

Full CodeDecay report

CodeDecay Report

Overall risk: High

Score Value
Merge risk 100/100
Decay risk 54/100
Security risk 0/100
Findings Count
High 8
Medium 10
Low 16

Changed Files

  • docs/loop.md modified (+10/-8)
  • packages/cli/src/commands/loop.ts modified (+7/-0)
  • packages/cli/src/docs/command-docs/orchestration.ts modified (+9/-2)
  • packages/cli/src/parsers/loop.ts modified (+78/-0)
  • packages/cli/src/types/loop.ts modified (+7/-0)
  • packages/cli/test/loop-e2e.test.ts modified (+3/-3)
  • packages/cli/test/loop.test.ts modified (+5/-5)
  • packages/harness/src/loop/audit.ts added (+139/-0)
  • packages/harness/src/loop/budgets.ts added (+141/-0)
  • packages/harness/src/loop/controller.ts modified (+297/-34)
  • packages/harness/src/loop/hypotheses.ts added (+66/-0)
  • packages/harness/src/loop/index.ts modified (+9/-0)
  • packages/harness/src/loop/render.ts modified (+2/-0)
  • packages/harness/src/loop/types.ts modified (+10/-0)
  • packages/harness/test/loop.test.ts modified (+111/-9)
  • scripts/end-user-demo.mjs modified (+3/-1)

Likely Impacted Areas

  • High API surface (api): packages/harness/src/loop/controller.ts
  • Low Documentation (docs): docs/loop.md, packages/cli/src/docs/command-docs/orchestration.ts
  • Low Source code (source): packages/cli/src/commands/loop.ts, packages/cli/src/parsers/loop.ts, packages/cli/src/types/loop.ts, packages/harness/src/loop/audit.ts, packages/harness/src/loop/budgets.ts, packages/harness/src/loop/hypotheses.ts, packages/harness/src/loop/index.ts, packages/harness/src/loop/render.ts, packages/harness/src/loop/types.ts, scripts/end-user-demo.mjs
  • Low Tests (test): packages/cli/test/loop-e2e.test.ts, packages/cli/test/loop.test.ts, packages/harness/test/loop.test.ts

Normalized Impact Graph

Graph artifact: .codedecay/local/impact-graph.json (4423 node(s), 8998 edge(s))

  • Confidence: Direct: 8998, inferred: 0, heuristic: 0
  • codedecay-js-babel-symbols via @babel/parser (available, adapter 1.0.0)
    • Limitation: Call expressions are not connected to target symbols in this adapter version.
    • Limitation: Only JavaScript and TypeScript files parsed by @babel/parser are represented.
    • Limitation: Static import resolution does not resolve runtime dependency injection or dynamic imports.
  • codedecay-python-lezer via @lezer/python (available, adapter 1.0.0)
    • Limitation: Dynamic imports, dependency injection, decorators without literal routes, and framework route registries are not resolved.
    • Limitation: Python impact evidence uses the @lezer/python grammar and conservative module-to-file resolution.
    • Limitation: Static test imports do not prove execution or assertion quality.
  • Graph limitation: A static test import does not prove the symbol executed or that assertions cover its behavior.

Symbol Impact Evidence

Graph artifact: .codedecay/local/symbol-impact-graph.json (935 file(s), 5589 edge(s))

  • packages/cli/src/commands/loop.ts#runLoopCommand -> packages/cli/src/commands/registry.ts
  • packages/cli/src/docs/command-docs/orchestration.ts#ORCHESTRATION_COMMAND_DOCS -> packages/cli/src/docs/commands.ts
  • packages/cli/src/parsers/loop.ts#parseLoopArgs -> packages/cli/src/commands/loop.ts, packages/cli/src/parsers/args.ts
  • packages/cli/src/types/loop.ts#LoopOptions -> packages/cli/src/parsers/loop.ts, packages/cli/src/types.ts, packages/cli/src/types/index.ts
  • packages/harness/src/loop/audit.ts#appendLoopAuditRecord -> packages/harness/src/loop/controller.ts, packages/harness/src/loop/index.ts
  • packages/harness/src/loop/audit.ts#createAuditRecordFromRound -> packages/harness/src/loop/controller.ts
  • packages/harness/src/loop/audit.ts#defaultLoopAuditPath -> packages/harness/src/loop/controller.ts, packages/harness/src/loop/index.ts
  • packages/harness/src/loop/audit.ts#loadLoopAuditResumeState -> packages/harness/src/loop/controller.ts, packages/harness/src/loop/index.ts
  • packages/harness/src/loop/audit.ts#LoopAuditResumeState -> packages/harness/src/loop/index.ts
  • packages/harness/src/loop/audit.ts#LoopAuditRoundRecord -> packages/harness/src/loop/index.ts
  • packages/harness/src/loop/audit.ts#writeLoopAuditSummary -> packages/harness/src/loop/controller.ts, packages/harness/src/loop/index.ts
  • packages/harness/src/loop/budgets.ts#checkChangedFileBudgets -> packages/harness/src/loop/controller.ts
  • ...and 29 more symbol impact(s)

Language And Parser Coverage

  • Source files classified: 15
  • Fully supported parser files: 15
  • Limited files: 0
  • Unsupported files: 0

Merge Risk Breakdown

  • Score: 100/100
  • Raw score before dampeners: 100/100
  • Adjusted score before severity cap: 100/100
  • Highest contributing severity: High

Top contributors:

  • +30 Api area changed (direct): packages/harness/src/loop/controller.ts touches a api area and should be reviewed for regression impact.
  • +18 Change size (structural): Changed lines amplify review cost across 16 file(s).
  • +12 File spread (structural): Change breadth spans 16 file(s).
  • +4 Docs area changed (heuristic): docs/loop.md touches a docs area and should be reviewed for regression impact.
  • +4 Docs area changed (heuristic): packages/cli/src/docs/command-docs/orchestration.ts touches a docs area and should be reviewed for regression impact.

Notes:

  • Untrusted memory context is visible but contributes 0 score until trusted evidence corroborates it.

Decay Risk Breakdown

  • Score: 54/100
  • Raw score before dampeners: 100/100
  • Adjusted score before severity cap: 84/100
  • Highest contributing severity: High
  • Evidence mode: heuristic-only

Top contributors:

  • +18 High complexity in changed function (heuristic): parseLoopArgs has estimated cyclomatic complexity 49.
  • +18 High complexity in changed function (heuristic): runCodeDecayLoop has estimated cyclomatic complexity 32.
  • +18 Large changed function (heuristic): parseLoopArgs spans 258 lines, which increases review and regression risk.
  • +18 Large changed function (heuristic): runCodeDecayLoop spans 389 lines, which increases review and regression risk.
  • +10 Broad unrelated change set (heuristic): This PR changes 14 files across 2 top-level areas and 3 risk categories.

Dampeners:

  • -16 Heuristic-only dampener: Decay stays conservative until direct evidence exists.

Notes:

  • Heuristic-only decay is capped at 54/100 until direct evidence exists.

Security Risk Breakdown

  • Score: 0/100
  • Raw score before dampeners: 0/100
  • Adjusted score before severity cap: 0/100

Security Matcher Coverage

  • Changed source files scanned: 12
  • Security candidates found: 0
  • Skipped files: 0

Test Evidence

  • Mode: heuristic-only
  • Sources: none
  • Changed source coverage:
  • packages/cli/src/commands/loop.ts: not measured (no measurable changed lines)
  • packages/cli/src/docs/command-docs/orchestration.ts: not measured (no measurable changed lines)
  • packages/cli/src/parsers/loop.ts: not measured (no measurable changed lines)
  • packages/cli/src/types/loop.ts: not measured (no measurable changed lines)
  • packages/harness/src/loop/audit.ts: not measured (no measurable changed lines)
  • packages/harness/src/loop/budgets.ts: not measured (no measurable changed lines)
  • packages/harness/src/loop/controller.ts: not measured (no measurable changed lines)
  • packages/harness/src/loop/hypotheses.ts: not measured (no measurable changed lines)
  • Notes:
  • No runtime coverage artifact was found. Test audit remains heuristic-only.

Changed Path Test Proof

Status Count
Runtime-proven 0
Static-only 30
Weakened by mocks 0
Unproven 12
  • Static-only packages/cli/src/commands/loop.ts#runLoopCommand (static-reference, deterministic)
    • Evidence: Referenced by packages/cli/test/loop-e2e.test.ts, packages/cli/test/loop.test.ts, packages/harness/test/loop.test.ts, but no runtime coverage artifact proves changed lines executed.
    • Static references: packages/cli/test/loop-e2e.test.ts, packages/cli/test/loop.test.ts, packages/harness/test/loop.test.ts
    • Repair task: Strengthen packages/cli/test/loop-e2e.test.ts so it executes packages/cli/src/commands/loop.ts#runLoopCommand with assertions; static import alone is not proof.
  • Unproven packages/cli/src/docs/command-docs/orchestration.ts#ORCHESTRATION_COMMAND_DOCS (missing-proof, heuristic)
    • Evidence: No runtime coverage or static test reference was found for this changed path.
    • Repair task: Add an integration test that reaches packages/cli/src/docs/command-docs/orchestration.ts#ORCHESTRATION_COMMAND_DOCS without mocking the changed boundary.
  • Static-only packages/cli/src/parsers/loop.ts#parseLoopArgs (static-reference, deterministic)
    • Evidence: Referenced by packages/cli/test/loop-e2e.test.ts, packages/cli/test/loop.test.ts, packages/harness/test/loop.test.ts, but no runtime coverage artifact proves changed lines executed.
    • Static references: packages/cli/test/loop-e2e.test.ts, packages/cli/test/loop.test.ts, packages/harness/test/loop.test.ts
    • Repair task: Strengthen packages/cli/test/loop-e2e.test.ts so it executes packages/cli/src/parsers/loop.ts#parseLoopArgs with assertions; static import alone is not proof.
  • Static-only packages/cli/src/types/loop.ts#LoopOptions (static-reference, deterministic)
    • Evidence: Referenced by packages/cli/test/loop-e2e.test.ts, packages/cli/test/loop.test.ts, packages/harness/test/loop.test.ts, but no runtime coverage artifact proves changed lines executed.
    • Static references: packages/cli/test/loop-e2e.test.ts, packages/cli/test/loop.test.ts, packages/harness/test/loop.test.ts
    • Repair task: Strengthen packages/cli/test/loop-e2e.test.ts so it executes packages/cli/src/types/loop.ts#LoopOptions with assertions; static import alone is not proof.
  • Static-only packages/harness/src/loop/audit.ts#appendLoopAuditRecord (static-reference, deterministic)
    • Evidence: Referenced by packages/harness/test/loop.test.ts, but no runtime coverage artifact proves changed lines executed.
    • Static references: packages/harness/test/loop.test.ts
    • Repair task: Strengthen packages/harness/test/loop.test.ts so it executes packages/harness/src/loop/audit.ts#appendLoopAuditRecord with assertions; static import alone is not proof.
  • Static-only packages/harness/src/loop/audit.ts#createAuditRecordFromRound (static-reference, deterministic)
    • Evidence: Referenced by packages/harness/test/loop.test.ts, but no runtime coverage artifact proves changed lines executed.
    • Static references: packages/harness/test/loop.test.ts
    • Repair task: Strengthen packages/harness/test/loop.test.ts so it executes packages/harness/src/loop/audit.ts#createAuditRecordFromRound with assertions; static import alone is not proof.
  • Static-only packages/harness/src/loop/audit.ts#defaultLoopAuditPath (static-reference, deterministic)
    • Evidence: Referenced by packages/harness/test/loop.test.ts, but no runtime coverage artifact proves changed lines executed.
    • Static references: packages/harness/test/loop.test.ts
    • Repair task: Strengthen packages/harness/test/loop.test.ts so it executes packages/harness/src/loop/audit.ts#defaultLoopAuditPath with assertions; static import alone is not proof.
  • Static-only packages/harness/src/loop/audit.ts#loadLoopAuditResumeState (static-reference, deterministic)
    • Evidence: Referenced by packages/harness/test/loop.test.ts, but no runtime coverage artifact proves changed lines executed.
    • Static references: packages/harness/test/loop.test.ts
    • Repair task: Strengthen packages/harness/test/loop.test.ts so it executes packages/harness/src/loop/audit.ts#loadLoopAuditResumeState with assertions; static import alone is not proof.
  • ...and 34 more changed path proof entries

Untrusted Memory Context

  • Project invariant may be impacted (packages/cli/src/commands/loop.ts:70): Untrusted memory context: invariant "No hidden cloud or model call" applies to this change. The OSS CLI must remain useful without telemetry, API keys, hosted services, required LLM calls, or CodeDecayCloud.
  • Project invariant may be impacted (packages/cli/src/commands/loop.ts:70): Untrusted memory context: invariant "Commands are explicit" applies to this change. CodeDecay must not run project commands unless they are configured and safety.allowCommands is true.
  • Project invariant may be impacted (docs/loop.md:37): Untrusted memory context: invariant "Tool evidence is separate from AI suggestions" applies to this change. Reports must not present agent/model suggestions as verified evidence unless backed by deterministic checks or command output.
  • Project invariant may be impacted (docs/loop.md:37): Untrusted memory context: invariant "Output must be actionable" applies to this change. Redteam reports and agent bundles should say what behavior to verify, which test proof is weak or missing, and what task a coding agent should perform.
  • Architecture note applies (packages/cli/src/commands/loop.ts:70): Untrusted architecture context: CLI is the published surface: The public npm package is @submuxhq/codedecay and the binary is codedecay. Internal workspace packages are implementation details.

High Risk Findings

  • High complexity in changed function (packages/cli/src/parsers/loop.ts:6): parseLoopArgs has estimated cyclomatic complexity 49.
  • High complexity in changed function (packages/harness/src/loop/controller.ts:49): runCodeDecayLoop has estimated cyclomatic complexity 32.
  • Large changed function (packages/cli/src/parsers/loop.ts:6): parseLoopArgs spans 258 lines, which increases review and regression risk.
  • Large changed function (packages/harness/src/loop/controller.ts:49): runCodeDecayLoop spans 389 lines, which increases review and regression risk.
  • Api area changed (packages/harness/src/loop/controller.ts:3): packages/harness/src/loop/controller.ts touches a api area and should be reviewed for regression impact.

Medium Risk Findings

  • Broad unrelated change set: This PR changes 14 files across 2 top-level areas and 3 risk categories.
  • Duplicated added logic (packages/cli/src/docs/command-docs/orchestration.ts:259): A similar block of added logic appears 4 times across 1 file(s).
  • Duplicated added logic (packages/harness/src/loop/budgets.ts:94): A similar block of added logic appears 3 times across 1 file(s).
  • Duplicated added logic (packages/harness/src/loop/controller.ts:6): A similar block of added logic appears 2 times across 2 file(s).
  • Duplicated added logic (packages/harness/src/loop/controller.ts:223): A similar block of added logic appears 7 times across 1 file(s).
  • Duplicated added logic (packages/harness/src/loop/controller.ts:224): A similar block of added logic appears 7 times across 1 file(s).
  • High complexity in changed function (packages/harness/src/loop/controller.ts:985): nextStepsForStatus has estimated cyclomatic complexity 14.
  • High complexity in changed function (packages/harness/src/loop/render.ts:11): renderLoopMarkdown has estimated cyclomatic complexity 13.
  • High complexity in changed function (scripts/end-user-demo.mjs:339): runLoopConvergenceChecks has estimated cyclomatic complexity 12.

Low Risk Findings

  • Docs area changed (docs/loop.md:37): docs/loop.md touches a docs area and should be reviewed for regression impact.
  • Docs area changed (packages/cli/src/docs/command-docs/orchestration.ts:259): packages/cli/src/docs/command-docs/orchestration.ts touches a docs area and should be reviewed for regression impact.
  • Source area changed (packages/cli/src/commands/loop.ts:70): packages/cli/src/commands/loop.ts touches a source area and should be reviewed for regression impact.
  • Source area changed (packages/cli/src/parsers/loop.ts:189): packages/cli/src/parsers/loop.ts touches a source area and should be reviewed for regression impact.
  • Source area changed (packages/cli/src/types/loop.ts:20): packages/cli/src/types/loop.ts touches a source area and should be reviewed for regression impact.
  • Source area changed (packages/harness/src/loop/audit.ts:1): packages/harness/src/loop/audit.ts touches a source area and should be reviewed for regression impact.
  • Source area changed (packages/harness/src/loop/budgets.ts:1): packages/harness/src/loop/budgets.ts touches a source area and should be reviewed for regression impact.
  • Source area changed (packages/harness/src/loop/hypotheses.ts:1): packages/harness/src/loop/hypotheses.ts touches a source area and should be reviewed for regression impact.
  • Source area changed (packages/harness/src/loop/index.ts:2): packages/harness/src/loop/index.ts touches a source area and should be reviewed for regression impact.
  • Source area changed (packages/harness/src/loop/render.ts:16): packages/harness/src/loop/render.ts touches a source area and should be reviewed for regression impact.
  • Source area changed (packages/harness/src/loop/types.ts:265): packages/harness/src/loop/types.ts touches a source area and should be reviewed for regression impact.
  • Source area changed (scripts/end-user-demo.mjs:373): scripts/end-user-demo.mjs touches a source area and should be reviewed for regression impact.
  • Test area changed (packages/cli/test/loop-e2e.test.ts:34): packages/cli/test/loop-e2e.test.ts touches a test area and should be reviewed for regression impact.
  • Test area changed (packages/cli/test/loop.test.ts:15): packages/cli/test/loop.test.ts touches a test area and should be reviewed for regression impact.
  • Test area changed (packages/harness/test/loop.test.ts:24): packages/harness/test/loop.test.ts touches a test area and should be reviewed for regression impact.

Recommended Checks

  • Add an integration test that reaches packages/cli/src/docs/command-docs/orchestration.ts#ORCHESTRATION_COMMAND_DOCS without mocking the changed boundary.
  • Add an integration test that reaches packages/harness/src/loop/budgets.ts#checkChangedFileBudgets without mocking the changed boundary.
  • Add an integration test that reaches packages/harness/src/loop/budgets.ts#checkModelCallBudget without mocking the changed boundary.
  • Add an integration test that reaches packages/harness/src/loop/budgets.ts#checkWallTimeBudget without mocking the changed boundary.
  • Add an integration test that reaches packages/harness/src/loop/budgets.ts#createLoopBudgetState without mocking the changed boundary.
  • Add an integration test that reaches packages/harness/src/loop/budgets.ts#detectOscillation without mocking the changed boundary.
  • Add an integration test that reaches packages/harness/src/loop/budgets.ts#detectWideningScope without mocking the changed boundary.
  • Add an integration test that reaches packages/harness/src/loop/budgets.ts#LoopBudgetConfig without mocking the changed boundary.
  • Add an integration test that reaches packages/harness/src/loop/hypotheses.ts#mergeHypothesisStatuses without mocking the changed boundary.
  • Add an integration test that reaches packages/harness/src/loop/hypotheses.ts#parseVerifierHypothesisProposals without mocking the changed boundary.
  • Add an integration test that reaches packages/harness/src/loop/types.ts#CodeDecayLoopInput without mocking the changed boundary.
  • Add an integration test that reaches scripts/end-user-demo.mjs without mocking the changed boundary.

Notes

CodeDecay is deterministic and local-first. This report was generated without telemetry, API keys, LLMs, or model calls.

Found by CodeDecay - deterministic, local-first, no telemetry.

@kunaldhongade
kunaldhongade merged commit e069912 into main Aug 6, 2026
10 checks passed
@kunaldhongade
kunaldhongade deleted the feature/680-builder-verifier-loop-close branch August 6, 2026 13:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

area: cli CLI package or command behavior area: docs README, community files, or documentation area: harness Agent and tool harness interfaces documentation Improvements or additions to documentation type: test Test coverage, fixtures, or verification improvements

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

feat(loop): separate builder and verifier roles with bounded repair convergence

1 participant

@kunaldhongade