Skip to content

Reduce repeated Tool execution and enforce CI budget - #222

Merged
KimGLee merged 15 commits into
mainfrom
perf/tool-execution-ci-budget
Sep 11, 2026
Merged

KimGLee merged 15 commits into
mainfrom
perf/tool-execution-ci-budget

Conversation

@KimGLee

@KimGLee KimGLee commented Sep 10, 2026 •

Copy link
Copy Markdown
Owner

Problem

Rapid fixes made the existing MCP lifecycle pay repeatedly for the same contract parsing, evidence interpretation and publication preparation. Some facts were also copied into a second complete receipt before their consumers would accept them. Tests and fixtures kept those obsolete layers alive. More shards could not shorten the long lifecycle method.

Closes #221

Changes and ownership

  • Existing Tool owners share mechanical parsing and exact-input projections inside one observation. Remove superseded helpers and unused intermediate results. Producer admission, lock-time currentness, writes and resulting-state readback remain independent boundaries.
  • At the user's approved Kernel authority locations, distinguish check facts from plan acceptance. For the concrete current substantive-review, changed-scope, rendering and post-Delta member chains, first publication verifies both. Consumers use the native fact; remove the full AuditReceipt publisher, schema, capability, path, CLI/MCP entries and consumer branches together.
  • Preserve the existing page fingerprint semantics unchanged in K12/07's dedicated machine contract, loaded by the existing fingerprint owner. A schema file's retirement does not retire its independent fingerprint guarantees.
  • M results reference the immutable plan rather than copying its definitions. Under the approved SML scope, retain all 34 requirement identities but consume five already-proved requirements directly from their original planned evidence. One explicit failure-non-applicability fact may cover its five same-page dependent requirements; otherwise distinct answers reference the accepted shared condition. Item traceability, differing fingerprints, correction and valid reuse remain.
  • Deliver the existing same-page reviews collection once to the original producer, sharing frozen material and preparation. Each independent fact retains its guarded append and exact readback; later refusal preserves the confirmed prefix. Retire scalar review flags, caller-supplied evidence references and the Runner's duplicate per-item dispatch loop. No new bulk capability, aggregate receipt or all-or-none transaction.
  • Remove the empty global-review wrapper. The real reviewer attestation owns the common plan reconciliation once; the final close aggregate commits its exact evidence set. Real review statements, candidate dispositions, all eight post-Delta checks and full plan closure remain required.
  • Currentness, withdrawal, review rounds, phase handoff, frozen close selection, sealing and Terminal consume the actual evidence identities. Old wrappers and aliases are retired, not readmitted through historical compatibility.
  • Retire unsupported README byte/line limits, fixed wording/headings and hand-maintained required tool lists. Keep general link and owner-based command checks. This is unrelated to the unchanged 360-second CI budget.
  • Regenerate the existing eight checkpoints and interface/catalog projections. Remove obsolete schema and duplicate wrapper assertions from existing owner tests; retain the same real MCP two-batch, correction, revision and Terminal scenario. No additional E2E or full local replay.

No audit requirement, accepted quality threshold or Profile policy is relaxed. The approved Kernel changes concern evidence representation and acceptance placement, not a Tool-created governance exception. The user explicitly brought SML report section 7 into this existing branch before further micro-optimization; this is now included in the same lifecycle change.

Verification status

Protected run 34567059696 on d27fd7dbf4dfb1a57ff1e959779f6a7b565f699b passed all 24 required jobs. Both original full MCP scenarios naturally completed both batches, correction, revision and Terminal. Final required-workflow budget: 357 seconds through the last required job; GitHub final status update: 358 seconds from creation. The 360-second limit passes; the 300-second target is not met and margin remains small.

Earlier run 34565846581 on 26af8936 closed both batches on both Python versions, then failed at an overlooked Terminal projection rule that rejected one common N/A fact covering multiple obligations. Its 252.279 / 230.501 seconds are failed-scenario durations, not complete lifecycle timings. The Terminal projection now excludes covered N/A children from actually-executed dimension coverage, retaining the primary fact and unique real dimension evidence. Frozen-stage owners still reject unauthorized coverage; no new wrapper or weakened Proof contract is introduced. The preceding run 34508730027 passed functional checks but failed the whole-attempt budget.

  • Existing focused owner selections pass, covering native production, plan matching, current/stale/withdrawn evidence, independent L rounds, close members, exact Terminal selection and interface ownership. These overlapping selections are not summed as independent coverage.
  • Real local publication-to-close and Candidate Delta publication seams pass using regenerated checkpoints.
  • make check passes, including source-derived contracts, links, Profile structure, Card/Read Set stamps and catalog consistency.
  • Current catalog: 133 modules, 1,289 test methods, 49 fixtures and 8 bundles; the saved pre-SML state had 133 modules and 1,293 methods. The SML change retires duplicate plan/checklist assertions without adding test files or scenarios. No replacement testing framework is introduced.
  • Local functional checks use Python 3.12 and are not GitHub performance evidence. The existing protected CI supplies Python 3.10 and 3.14 verification and the complete MCP scenarios.

Performance acceptance

The completed pre-change diagnostic 34490686533 on source 6b7ce7c13710e2f9b14021188c87bb7d38046967 took 440.65 seconds on Python 3.10 and 444.26 seconds on Python 3.14 for the scenario command. Both exceeded budget.

The preceding source 604a5b0e237c7859bc2c5e9667088f1cb731dca7 was tested by run 34508730027 (PR merge tree 7a17a02da5373db373873c053fa1f2823f258393). Its original MCP scenario naturally completed successfully on both versions: test-cost report 353.91 / 374.81 seconds; GitHub test step 362 / 383 seconds; full MCP job 374 / 395 seconds. All other required test and contract jobs passed. The required gate correctly rejected the whole attempt at 423.09 seconds; final workflow elapsed was 424 seconds. No interrupted duration is presented as completion, and no nested scope durations are added together.

The saved catalog-observation change 1b7cfec9, SML lifecycle consolidation 26af8936 and Terminal handoff correction d27fd7db are submitted together. The original Terminal owner test now covers shared N/A facts and rejects duplicate actually-executed evidence; 42 related owner tests and 8 Proof tests pass. All eight regenerated checkpoints and make check pass. The original GitHub MCP scenes will measure actual correctness and cost; no new local full replay, reduced scenario, unchanged retry or budget relaxation is used to obtain a pass.

Final run 34567059696: scenario reports 254.759 / 273.501 seconds; full MCP jobs 273 / 294 seconds. Both use the same Ubuntu image, Python versions and dependency lists as the preceding complete run. Same-page review collection spans are 30.710 / 32.042 seconds (4 collections, including correction/revision), compared with 111.128 / 115.241 (5 collections). These spans contain nested MCP calls and are not added to them. The comparison combines saved observation reuse and SML consolidation, not an isolated receipt-count experiment. The Python 3.14 job's 37-second scheduling wait remains included in the workflow budget.

Target 300 seconds, maximum 360 seconds for the whole required workflow attempt, including preparation and final gate. The independent 1,800-second safety bound permits diagnostic completion but does not turn an over-budget run into acceptance. This run passes the limit, not the target; no unchanged retry is used to select a better timing. Report natural completion, stage costs and whole-attempt time separately; do not add nested durations or extrapolate from receipt counts.

@KimGLee KimGLee added bug Something isn't working enhancement New feature or request agent-interface Host-neutral agent interface: contract, projection, MCP server, per-host configs labels Sep 10, 2026
@KimGLee

KimGLee commented Sep 10, 2026

Copy link
Copy Markdown
Owner Author

Complete MCP timing diagnosis — not CI acceptance

Ran only the existing test_required_queue_e2e.py scenario once on each required Python version, on the same Ubuntu runner image and pinned dependency setup. Measured source was fixed at 96a4c36c8dcb74d84927e870b595d68b0d7b74c4; uncommitted performance changes were excluded.

Diagnostic run 34481068588, attempt 1:

Measurement Python 3.10.21 Python 3.14.7
Natural command duration 463.52s 485.36s
Existing E2E module 454.02s 476.94s
Work remaining after 360s 103.52s 125.36s
Complete lifecycle Passed Passed
Performance acceptance Not passed Not passed

Both retain 73 Runner actions, 48 MCP calls, all 34 M atoms, one M withdrawal/reproduction, S natural revision/reproduction, both batch closes and Terminal Proof/Task completion. No unrelated tests ran; no repeat samples were used.

Nonoverlapping wall-clock phases (approximate progress boundaries): initialization 23.52/22.75s; B1 240.10/252.32s; inter-batch checks 1.77/2.05s; B2 177.03/186.43s; Terminal and cleanup 21.11/21.80s. Parent/child inclusive timings are not added together. At 360s, 3.10 was completing revised S review; 3.14 was in revised rendering verification. Both still had evidence reproduction, publishing, merge/close and Terminal work remaining.

The shared Runner/producer path is the main cost boundary. MCP request waiting accounts for the called production chain, not transport overhead alone. Removing even the entire measured M-review cost would still leave the 3.14 command above 360s before CI preparation. Next optimization must address common contract parsing/query, decision projection and repeated mechanical work within existing owners, without weakening separate producer admission, lock currentness, immutable history or resulting-state validation. Internal hotspot proportions are not inferred from these coarse RPC measurements.

The one-off diagnostic controller was added on this same PR branch in dd4dac2 and retired in 322f902; it is not a new permanent test or budget. The required workflow remains unchanged at target 300s / hard 360s. This PR stays draft and not eligible to merge on the diagnostic success. Raw artifacts are attached to the run.

@KimGLee

KimGLee commented Sep 10, 2026

Copy link
Copy Markdown
Owner Author

Remote diagnosis and receipt-necessity review update:

  • CI performance acceptance remains target 300s / limit 360s for the complete required attempt. Execution now uses an independent 1,800s safety deadline; going over 360s must fail acceptance, but must not truncate the measurement.
  • Run 34490686533 measured the fixed source commit 6b7ce7c13710e2f9b14021188c87bb7d38046967, once each on GitHub Python 3.10/3.14. Only the existing complete MCP scenario ran; exhaustive verification was skipped. Both source worktrees remained unchanged.
Result Python 3.10 Python 3.14
Natural command completion 440.65s 444.26s
Entire diagnostic job 458s 456s
Remaining work after 360s 80.65s 84.26s
Functional scenario 1/1 passed 1/1 passed
Performance over budget over budget

Both batches, M withdrawal/replacement, S input changes/reproduction, stage handoff, Terminal Proof and Task completion were preserved. Counts remain 73 Runner actions and 48 MCP requests; lower outer invocation counts do not prove a sensible persisted-object granularity. Parent/child inclusive costs are not added together.

The temporary diagnostic controller/job has been retired in 77c84e5; the execution/acceptance separation remains. The existing CI-owner suite passed 16 tests; the generated Test Catalog remains consistent (134 modules, 1,303 cases). No new tests or full local lifecycle runs were added for this update. Skip-CI commits deliberately avoid rerunning unrelated tests and are not required-check evidence.

Receipt production changes are paused pending the necessity review. The review distinguishes necessary independent facts/history from shared computation and duplicate durable wrappers. Main candidates are plan-bound raw evidence versus AuditReceipt, common context repeated per M atom, and repeated close attestation/global/aggregate payloads. Removing independent AuditReceipt, per-item publication, or global-review correction identities requires explicit review of Kernel evidence, transaction, and withdrawal guarantees; no such guarantee or production object was changed here. Diagnostic report types with no governance acceptance consumer are a separate retirement candidate, not a reason to remove useful error details.

This PR remains a draft. Functional completion of these diagnostics does not meet the 360s requirement and does not authorize merging.

@KimGLee
KimGLee marked this pull request as ready for review September 11, 2026 05:52
@KimGLee
KimGLee merged commit e80e9b9 into main Sep 11, 2026
24 checks passed
@KimGLee
KimGLee deleted the perf/tool-execution-ci-budget branch September 11, 2026 05:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-interface Host-neutral agent interface: contract, projection, MCP server, per-host configs bug Something isn't working enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Restore bounded required CI by reducing repeated Tool execution work

1 participant