From c073996b08ed845ca00a201c7bbd975d55fe4476 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 16 Oct 2026 21:49:50 +0200 Subject: [PATCH 1/2] Fix four code/doc mismatches found while writing the reference upgrade-consumers now runs each consumer's own vendor/damascus installer; the eval adjudicator caps at 25 % like the skills; agent-run no longer hardcodes an Ollama model; test-lock's docstring names the real manifest. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 4 +++ docs/reference.md | 21 +++++++------ evals/run_eval.py | 4 +-- scripts/damascus-agent-run | 10 +++++-- scripts/damascus-test-lock | 2 +- scripts/damascus-upgrade-consumers | 47 +++++++++++++++++++++++++----- tests/dogfood_e2e.sh | 19 ++++++++++-- 7 files changed, 84 insertions(+), 23 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 63afc1c..2c3fad5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,10 @@ ### Fixed +- Make `upgrade-consumers` run each repo's own `vendor/damascus/install.sh`, so its links keep pointing at its pinned submodule instead of the checkout running the command; skip a repo without one unless `--link-to-this-checkout` is given, and exit 1 when an install or verify fails ([#29](https://github.com/forkrul/damascus_development/pull/29)) +- Cap a review round at B+ in the eval's reference adjudicator at under 25 % overlap, as the temper and hone skills specify, instead of 20 % ([#29](https://github.com/forkrul/damascus_development/pull/29)) +- Stop `damascus agent-run` from falling back to a hardcoded Ollama model name; with no model pulled, configured or in `OLLAMA_MODEL`, it now says so and exits 1 ([#29](https://github.com/forkrul/damascus_development/pull/29)) +- Correct the `test-lock` manifest name in its docstring to `.damascus/locks/amber-tests.sha256.json` ([#29](https://github.com/forkrul/damascus_development/pull/29)) - Fix the README's install and upgrade snippets, which committed whatever `master` was instead of the pinned tag: initialise damascus's submodules from inside `vendor/damascus` rather than from the repo root ([#28](https://github.com/forkrul/damascus_development/pull/28)) - Make `./install.sh eval --live-llm` measure something: a seeded defect now counts only when a `[BLOCKING]` finding names it and diagnoses it in words absent from the prompt. Before, any keyword anywhere in the reply counted, and echoing the prompt scored 12/12. The clean control is now scored for false positives, the echo baseline is reported, and every prompt and raw reply is saved under `.damascus/eval-runs/`. The corpus `keywords` field becomes `anchors` + `diagnosis` (corpus 0.3.0) ([#27](https://github.com/forkrul/damascus_development/pull/27)) - Fix the clean control's FR-001, which did not say how to round, so INV-001 failed for sub-cent subtotals; it now rounds down ([#27](https://github.com/forkrul/damascus_development/pull/27)) diff --git a/docs/reference.md b/docs/reference.md index f6541d8..bcf1f09 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -89,7 +89,8 @@ CRITIC_C_MODEL= `LOCAL_OPENAI_URL` in the file sets the server URL when no environment variable does. Without a model name, `local-openai` uses the first model the server lists and `ollama` -the first model Ollama has pulled. +the first model Ollama has pulled, else `OLLAMA_MODEL`; with none of these, an Ollama call +stops with an error rather than guessing a model. ### Environment variables @@ -98,6 +99,7 @@ the first model Ollama has pulled. | `DAMASCUS_LOCAL_URL`, `OPENAI_BASE_URL` | the OpenAI-compatible server's base URL (e.g. `http://127.0.0.1:8080/v1`) | | `DAMASCUS_API_KEY`, `OPENAI_API_KEY` | that server's API key, if it needs one | | `OLLAMA_HOST` | Ollama's URL (default `http://127.0.0.1:11434`) | +| `OLLAMA_MODEL` | the Ollama model to use when none is configured and Ollama lists none | | `DAMASCUS_CUSTOM_CMD` | the command line for `--backend custom`; it receives the prompt on stdin | | `DAMASCUS__BACKEND`, `DAMASCUS__MODEL`, `DAMASCUS_BACKEND`, `DAMASCUS_MODEL` | routing overrides (table above) | @@ -172,20 +174,21 @@ damascus proof-coverage --base-ref HEAD~3 --report specs/001-x/revert-proof.json ## `upgrade-consumers` -Reports on, and re-links, other repos that use damascus: for each one it runs -`install.sh --all-harnesses` and then `--verify`. Read-only checkouts are reported and -skipped. It does not move anyone's `vendor/damascus` pin. +Reports on, and re-links, other repos that use damascus: for each one it runs that repo's +own `vendor/damascus/install.sh --all-harnesses` and then `--verify`, so its links keep +pointing at its own pinned submodule. It does not move anyone's pin — bump that first. +Read-only checkouts are reported and skipped, and so is a repo without +`vendor/damascus` (`SKIPPED_NO_VENDOR`). It exits 1 if any install or verify failed. ```bash damascus upgrade-consumers --scan-dir ~/src --check # report only: harnesses, pinned commit, writability damascus upgrade-consumers --repo ../shop-api # re-link one repo (repeatable) ``` -It runs the `install.sh` of the damascus checkout you invoke it from, and links always -point at that checkout. To keep a repo linked to its own pinned submodule, bump the pin, -then run its own installer (`vendor/damascus/install.sh`) or invoke `upgrade-consumers` -from that submodule with `--repo .`. `--dry-run` passes `--dry-run` to each install, -`--claude-only` links `.claude/` only, and `--json` prints a machine-readable report. +`--link-to-this-checkout` links a repo that has no `vendor/damascus` to the checkout +running the command instead; those links leave the repo, so a fresh clone of it won't +have them. `--dry-run` passes `--dry-run` to each install, `--claude-only` links +`.claude/` only, and `--json` prints a machine-readable report. ## Stage addenda: `.damascus/addenda/.md` diff --git a/evals/run_eval.py b/evals/run_eval.py index 1783c24..9b1b8be 100755 --- a/evals/run_eval.py +++ b/evals/run_eval.py @@ -370,8 +370,8 @@ def adjudicate_stratified_rating(findings: list[dict[str, str]]) -> str: m_b = sum(1 for critics in by_id.values() if len(critics) >= 2) overlap_b = (m_b / d_b) if d_b > 0 else 1.0 - # Low overlap cap applies ONLY when D_B >= 4 and m_B / D_B < 0.20 - low_overlap_cap = d_b >= 4 and overlap_b < 0.20 + # Low overlap cap applies ONLY when D_B >= 4 and m_B / D_B < 25 % (skills/temper, skills/hone) + low_overlap_cap = d_b >= 4 and overlap_b < 0.25 if d_b == 0 and important_count == 0: return "B+" if low_overlap_cap else "A+" if d_b == 0 and important_count <= 2: diff --git a/scripts/damascus-agent-run b/scripts/damascus-agent-run index 2cccce2..5776294 100755 --- a/scripts/damascus-agent-run +++ b/scripts/damascus-agent-run @@ -439,7 +439,7 @@ def select_backend_and_model( model = ( probed["ollama"]["models"][0] if probed["ollama"]["models"] - else os.environ.get("OLLAMA_MODEL", "qwen2.5-coder:14b") + else os.environ.get("OLLAMA_MODEL") ) return backend, model, endpoint_url @@ -546,9 +546,15 @@ def main(argv: list[str] | None = None) -> int: ) elif backend == "ollama": assert endpoint_url is not None + if not model: + sys.stderr.write( + "[ERR] agent-run: no Ollama model to use — none is pulled; pass --model, " + "set OLLAMA_MODEL, or set _MODEL / DEFAULT_MODEL in .damascus/models.conf\n" + ) + return 1 output = run_ollama( endpoint_url, - model or "qwen2.5-coder:14b", + model, system_prompt, user_prompt, args.temperature, diff --git a/scripts/damascus-test-lock b/scripts/damascus-test-lock index 1d45885..b15ee13 100755 --- a/scripts/damascus-test-lock +++ b/scripts/damascus-test-lock @@ -3,7 +3,7 @@ Enforces the `quench` Test Freeze Rule: - At the end of Amber (Step 3), run `damascus-test-lock lock` to record SHA-256 - digests of all test files into `.damascus/locks/amber-tests.sha256`. + digests of all test files into `.damascus/locks/amber-tests.sha256.json`. - At Green (Step 4), run `damascus-test-lock verify` to confirm zero silent modifications, deletions, or weakened assertions occurred in locked tests during implementation. diff --git a/scripts/damascus-upgrade-consumers b/scripts/damascus-upgrade-consumers index b9c3bb9..cb4ef41 100755 --- a/scripts/damascus-upgrade-consumers +++ b/scripts/damascus-upgrade-consumers @@ -4,7 +4,10 @@ Supports: - Scanning a directory of repositories (--scan-dir) or inspecting explicit ones (--repo) - Gracefully detecting read-only bind mounts vs. writable consumer checkouts - - Running `install.sh --all-harnesses` and `install.sh --verify` across writable consumers + - Running each consumer's own `vendor/damascus/install.sh --all-harnesses` and `--verify`, + so its links keep pointing at its pinned submodule. A repo without `vendor/damascus` is + skipped unless `--link-to-this-checkout` is given, which links it to the checkout running + this command instead (links that leave the repo, so a fresh clone of it won't have them). """ from __future__ import annotations @@ -58,7 +61,18 @@ def inspect_consumer(repo: Path) -> dict[str, Any]: } -def upgrade_consumer(repo: Path, all_harnesses: bool = True, dry_run: bool = False) -> dict[str, Any]: +def consumer_installer(repo: Path, link_to_this_checkout: bool) -> Path | None: + """The install.sh to run for a consumer: its own pinned vendor/damascus, else (only when + asked) this checkout's. install.sh links into the checkout it lives in.""" + own = repo / "vendor" / "damascus" / "install.sh" + if own.is_file(): + return own + return INSTALL_SH if link_to_this_checkout else None + + +def upgrade_consumer( + repo: Path, all_harnesses: bool = True, dry_run: bool = False, link_to_this_checkout: bool = False +) -> dict[str, Any]: info = inspect_consumer(repo) if not info["exists"]: info["status"] = "SKIPPED_MISSING" @@ -66,8 +80,13 @@ def upgrade_consumer(repo: Path, all_harnesses: bool = True, dry_run: bool = Fal if not info["writable"] and not dry_run: info["status"] = "SKIPPED_READONLY_MOUNT" return info + installer = consumer_installer(repo, link_to_this_checkout) + if installer is None: + info["status"] = "SKIPPED_NO_VENDOR" + return info + info["installer"] = str(installer) - cmd = ["bash", str(INSTALL_SH)] + cmd = ["bash", str(installer)] if all_harnesses: cmd.append("--all-harnesses") if dry_run: @@ -81,7 +100,7 @@ def upgrade_consumer(repo: Path, all_harnesses: bool = True, dry_run: bool = Fal return info if not dry_run: - ver = subprocess.run(["bash", str(INSTALL_SH), "--verify"], cwd=str(repo), text=True, capture_output=True) + ver = subprocess.run(["bash", str(installer), "--verify"], cwd=str(repo), text=True, capture_output=True) info["verify_returncode"] = ver.returncode info["status"] = "UPGRADED" if ver.returncode == 0 else "VERIFY_FAILED" info["harnesses"] = { @@ -99,7 +118,7 @@ def upgrade_consumer(repo: Path, all_harnesses: bool = True, dry_run: bool = Fal def main(argv: list[str] | None = None) -> int: parser = argparse.ArgumentParser( prog="damascus upgrade-consumers", - description="Inspect and upgrade downstream Damascus consumer repositories to this damascus checkout.", + description="Inspect and re-link downstream Damascus consumer repositories, each with its own vendor/damascus installer.", ) parser.add_argument( "--repo", @@ -126,6 +145,11 @@ def main(argv: list[str] | None = None) -> int: action="store_true", help="Link only .claude/ instead of --all-harnesses (.claude, .agents, .gemini).", ) + parser.add_argument( + "--link-to-this-checkout", + action="store_true", + help="For a repo without vendor/damascus, link it to this checkout instead of skipping it.", + ) parser.add_argument("--json", action="store_true", help="Emit JSON output.") args = parser.parse_args(argv) @@ -148,7 +172,14 @@ def main(argv: list[str] | None = None) -> int: info["status"] = "WRITABLE" if info["writable"] else "READONLY_MOUNT" results.append(info) else: - results.append(upgrade_consumer(repo, all_harnesses=not args.claude_only, dry_run=args.dry_run)) + results.append( + upgrade_consumer( + repo, + all_harnesses=not args.claude_only, + dry_run=args.dry_run, + link_to_this_checkout=args.link_to_this_checkout, + ) + ) if args.json: print(json.dumps({"damascus_root": str(DAMASCUS_ROOT), "consumers": results}, indent=2)) @@ -158,8 +189,10 @@ def main(argv: list[str] | None = None) -> int: harnesses = ",".join(k for k, v in r["harnesses"].items() if v) or "none" hone_str = "hone=yes" if r["has_v02_hone_skill"] else "hone=no" print(f" [{r['status']:22s}] {r['name']:24s} harnesses={harnesses:22s} {hone_str}") + if any(r["status"] == "SKIPPED_NO_VENDOR" for r in results): + print(" SKIPPED_NO_VENDOR: no vendor/damascus/install.sh; pass --link-to-this-checkout to link to this checkout") - return 0 + return 1 if any(r["status"] in ("FAILED", "VERIFY_FAILED") for r in results) else 0 if __name__ == "__main__": diff --git a/tests/dogfood_e2e.sh b/tests/dogfood_e2e.sh index d0cc60f..6e4091d 100755 --- a/tests/dogfood_e2e.sh +++ b/tests/dogfood_e2e.sh @@ -194,8 +194,23 @@ check "proof-coverage confirms 100% of changed production files have RED revert- --fix-ref "HEAD" \ --report "$REPORT_JSON" -check "upgrade-consumers upgrades and verifies writable consumer repo" \ - damascus upgrade-consumers --repo "$TMP_PROJECT" +# A consumer is re-linked with its own vendor/damascus installer (here a symlink to this +# checkout); a repo without one is skipped rather than linked to whoever runs the command. +BARE_PROJECT="$(mktemp -d)" +mkdir -p "$TMP_PROJECT/vendor" +ln -s "$REPO_ROOT" "$TMP_PROJECT/vendor/damascus" +consumer_status() { # repo status [installer]: assert the reported status (and that the repo's own installer ran) + local out; out="$(damascus upgrade-consumers --repo "$1" --json)" || return 1 + grep -q "\"status\": \"$2\"" <<<"$out" || return 1 + [ -z "${3:-}" ] || grep -q "\"installer\": \"$1/vendor/damascus/install.sh\"" <<<"$out" +} +check "upgrade-consumers re-links and verifies a consumer with its own vendor/damascus" \ + consumer_status "$TMP_PROJECT" UPGRADED +check "upgrade-consumers runs the consumer's own vendor/damascus/install.sh" \ + consumer_status "$TMP_PROJECT" UPGRADED installer +check "upgrade-consumers skips a repo without vendor/damascus" \ + consumer_status "$BARE_PROJECT" SKIPPED_NO_VENDOR +rm -rf "$BARE_PROJECT" echo "" echo "=== Dogfood Summary: $pass passed, $fail failed ===" From f61b95b67ceeeac884e6fc45d20d3749461391f2 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 16 Oct 2026 21:51:45 +0200 Subject: [PATCH 2/2] Compare the reported installer path against the resolved repo path (macOS /private/var) Co-Authored-By: Claude Opus 5.5 --- tests/dogfood_e2e.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/dogfood_e2e.sh b/tests/dogfood_e2e.sh index 6e4091d..420ff19 100755 --- a/tests/dogfood_e2e.sh +++ b/tests/dogfood_e2e.sh @@ -202,7 +202,8 @@ ln -s "$REPO_ROOT" "$TMP_PROJECT/vendor/damascus" consumer_status() { # repo status [installer]: assert the reported status (and that the repo's own installer ran) local out; out="$(damascus upgrade-consumers --repo "$1" --json)" || return 1 grep -q "\"status\": \"$2\"" <<<"$out" || return 1 - [ -z "${3:-}" ] || grep -q "\"installer\": \"$1/vendor/damascus/install.sh\"" <<<"$out" + # the tool reports resolved paths (macOS: /var -> /private/var), so compare against one + [ -z "${3:-}" ] || grep -q "\"installer\": \"$(cd "$1" && pwd -P)/vendor/damascus/install.sh\"" <<<"$out" } check "upgrade-consumers re-links and verifies a consumer with its own vendor/damascus" \ consumer_status "$TMP_PROJECT" UPGRADED