Skip to content

docs: sync public docs to post-July harness state and capture value-proof gaps plan - #41

Open
noodlemind wants to merge 2 commits into
mainfrom
docs-post-july-sync
Open

docs: sync public docs to post-July harness state and capture value-proof gaps plan#41
noodlemind wants to merge 2 commits into
mainfrom
docs-post-july-sync

Conversation

@noodlemind

@noodlemind noodlemind commented Aug 3, 2026

Copy link
Copy Markdown
Owner

Why

An external review of the harness evaluated the public docs surface (README, architecture docs, AGENTS.md) as it stood before the July hardening wave (PRs #33#37) and concluded the system was token-heavy, unmeasured, and Copilot-bound — none of which holds at HEAD. The docs lagged reality; this PR closes that lag and captures the four gaps that survived the reconciliation.

What

Commit 1 — docs sync (4 files):

  • README.md: memory-model headline updated to the episodic/semantic/behavioral tiers with a link to docs/MEMORY-MODEL.md; new "Measurement, budgets, and evals" section (CI-enforced 2 KB orient pack + ~900-token engineer budget, harness report host token metering, evals/ suite); knowledge-compounding table gains the consolidated-learnings tier; directory structure adds packages/harness/, evals/, docs/MEMORY-MODEL.md.
  • docs/architecture/engineer-harness.md: new harness report telemetry paragraph; concrete budget number on the VS Code evidence row; freshness marker.
  • docs/architecture/skill-driven-prompt-library.md: "Review Summary" rewritten from proposal voice to shipped-state "Standardization Summary"; @dev-kit/harness command list updated to the full shipped catalog; ## Memory Cards added to the plan-section list (aligns with _plan-template.md); freshness marker.
  • AGENTS.md: budgets/report/evals/memory-model line; directory structure adds packages/, evals/, docs/MEMORY-MODEL.md; freshness marker.

Commit 2 — plan capture:

  • docs/plans/2026-08-02-feat-harness-value-proof-gaps-plan.md: unlocked capture (status: open, phase: 0) of the four adopted gaps — knowledge net-benefit measurement (corpus-gated), session-level compaction, correction-effort metric, verify-tier decide-and-document — plus the reviewer's standing offer to pre-review the release-eval methodology before Terminal-Bench trials run. Each gap locks via its own planning pass before implementation.

Verification

  • harness validate-plan on the new plan: pass, exit 0 (schema v1, criterion mappings, configured checks all green)
  • npm --prefix packages/harness test: 655/655 pass
  • No references to unpublished docs; directory listings match the real tree

Summary by CodeRabbit

  • Documentation
    • Updated project guidance to describe episodic, semantic, and behavioral memory models.
    • Added documentation for context budgets, token-efficiency telemetry, deterministic evaluations, and budget-breach reporting.
    • Expanded architecture references for the harness, evaluation directories, memory cards, and skill-driven workflows.
    • Added a planning document covering knowledge measurement, context compaction, correction-effort tracking, and risk-tiered verification.
    • Updated documentation dates and standardized model-summary guidance.

@noodlemind

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Aug 3, 2026

Copy link
Copy Markdown

Review Change Stack

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 63331372-72ed-4ce9-b5fa-6ab5feb63263

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

The PR updates project and architecture documentation for the July 2026 hardening wave. It documents memory, context budgets, token telemetry, deterministic evaluations, harness commands, and a plan for remaining value-proof gaps.

Changes

Hardening documentation

Layer / File(s) Summary
Project reference updates
AGENTS.md, README.md
Project metadata, directory references, memory-model details, governance, context budgets, token reporting, and deterministic evaluations now reflect the hardening wave.
Harness and skill architecture
docs/architecture/engineer-harness.md, docs/architecture/skill-driven-prompt-library.md
The architecture documents describe harness report, budget checks, token telemetry, deterministic validation, memory cards, and the expanded harness command inventory.
Value-proof gap plan
docs/plans/2026-08-02-feat-harness-value-proof-gaps-plan.md
The new plan defines acceptance criteria, phased implementation work, verification requirements, risk routing, review findings, and activity history for four harness value-proof gaps.
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the documentation updates and the added plan for harness value-proof gaps.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch docs-post-july-sync

Comment @coderabbitai help to get the list of available commands.

@coderabbitai

coderabbitai Bot commented Aug 3, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/plans/2026-08-02-feat-harness-value-proof-gaps-plan.md`:
- Line 53: Update the AC4 baseline wording in the plan to reflect the actual
enforcement mode: use “enforcement: enforce” if the baseline is fail-closed, or
explicitly identify the separate boundary meant by “fail-closed” while retaining
“enforcement: warn.” Keep the documented adopt-or-decline decision and rationale
requirement unchanged.

In `@README.md`:
- Around line 98-101: Update the opening fenced code block surrounding the
directory listing in README.md to use the text language identifier, changing the
bare fence to a text fence while preserving the listing contents and closing
fence.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: e1e3238b-bc68-4ce2-846e-844324f6400a

📥 Commits

Reviewing files that changed from the base of the PR and between 70fa8fd and 08f289d.

📒 Files selected for processing (5)
  • AGENTS.md
  • README.md
  • docs/architecture/engineer-harness.md
  • docs/architecture/skill-driven-prompt-library.md
  • docs/plans/2026-08-02-feat-harness-value-proof-gaps-plan.md

Comment thread docs/plans/2026-08-02-feat-harness-value-proof-gaps-plan.md Outdated
Comment thread README.md
@noodlemind
noodlemind force-pushed the docs-post-july-sync branch from 08f289d to 610b774 Compare August 3, 2026 20:13
@noodlemind
noodlemind force-pushed the docs-post-july-sync branch from 610b774 to 59ca6a3 Compare August 4, 2026 02:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant