Skip to content

Workflow-authoring + CI-gating robustness: 4 lessons from the savant↔scholar migration #170

Description

@Data-Wise

Context

Four reusable lessons surfaced while running the savant↔scholar migration (Epics 1–3) as
multi-agent Workflow orchestrations end-to-end (savant v1.0.0 + scholar v3.0.0 shipped). Each came
from something that actually went wrong and was recovered. Filing so craft's orchestration guidance
and /craft:check can encode them. (Saved locally as feedback memories in the savant project.)

Proposed craft improvements

1. Orchestration guidance: a gate must be control-flow, not advisory

  • Document (and ideally lint for) the rule: a verify/gate agent only blocks if the script
    branches on its result. A schema-less narrator string is advisory — the script never
    reads it.
  • What bit us: a final verify step returned "FAIL: not release-ready" but release ran in the
    same unconditional block, so a public v3.0.0 shipped with cosmetic doc defects.
  • Fix pattern: structured gate schema ({pass, blockers[]}) + if (!verify.pass) return before
    any irreversible step.

2. Orchestration guidance: cross-repo workflows need explicit paths for EVERY agent role

  • When a workflow spans two repos, executor and reviewer prompts must carry absolute paths
    to docs in the other repo, and state which repo each lives in.
  • What bit us: reviewers got a vague plan reference, searched the wrong repo, and (under
    "default BLOCK if uncertain") falsely blocked a correct, CI-green PR — aborting the chain.
  • Fix pattern: pass absolute cross-repo paths to all roles; separate "code might be wrong"
    (block) from "can't find my reference" (fetch, don't block).

3. Docs: the fix-and-resume pattern for false aborts

  • Document Workflow({scriptPath, resumeFromRunId}) for recovering from an
    orchestration/prompt bug: edit only the offending function in the saved script and resume —
    unchanged agents replay from cache, live re-run starts at the first edit. Turned a ~70-min
    false abort into a one-line fix.

4. /craft:check: detect ungated direct-to-dev doc debt

  • In --post-merge (or a new check), flag docs that were committed directly to an integration
    branch when CI only runs on PRs — they're never linted. Recommend gating markdownlint-cli2
    AND mkdocs build --strict on push (or pre-commit), even for local-only sites.
  • What bit us: savant CI is PR-only + local-build-only, so plan-doc lint errors and ~30 dead
    migration links (breaking mkdocs build --strict) accumulated silently until a manual sweep.

Acceptance

Guidance items land in craft's orchestration docs; item 4 lands as a /craft:check capability or a
recommended CI/pre-commit template.


Filed from the savant↔scholar migration session.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions