Openspec is lossy - agents do not fix all the grep'ed findings

[deemyboy]

Why a "fully spec'd" OpenSpec change still drifts at apply-time

Date: 2026-06-22 · Change: service-repo-boundary (#848) · Mode: /opsx:explore (analysis, no implementation)

This doc captures a single exploration thread end-to-end: from the opening question to the
conclusion, with the evidence each step was grounded in. It is a retrospective/analysis — it
narrates the exchange deliberately.

---

The question

If you spec'd up the service-repo-boundary epic with a full spec, then why, when the agent
picks it up with /opsx:apply service-repo-boundary, does it discover that the work is bigger
than the tasks say — e.g. an apply agent finding the remaining offender set includes
SquadService (6 calls) that task 6.1 never named, and having to flag the gap itself?

The spec exists, openspec validate --strict passes — so why is the agent finding holes in it
during implementation?

---

Evidence gathered (read-only, this session)

Live offender count across apps/backend/src/services/ ((tx|prismaClient).<model>. minus
$transaction):

1  ActivityEventWriter.ts
1  MessageService.ts
6  SquadService.ts
── total: 8

Git state — phases 1–5 committed:

0d93e915f refactor(#848): Phase 5 — dev/export/aggregation services
db07b55ff refactor(#848): Phase 4 — GameService
b0e03a2e0 refactor(#848): Phase 3b — finish Phase-1 services
cb986d65b refactor(#848): Phase 3 — invite/request services
186e76b2b refactor(#848): Phase 2 — PlayerService
2816eb955 refactor(#848): Phase 1 — junction repositories

SquadService is not enumerated in the original planning prose — it appears nowhere in
proposal.md; design.md only mentions squad access obliquely (D5: "Squad-model service access …
is owned by SquadRepository — there is no separate 'SquadService phase'"). It was absent from the
per-phase task buckets until corrected mid-flight.

openspec validate --strict on the change:

openspec validate service-repo-boundary --strict --json
→ { "valid": true, "issues": [] }

Passes clean — on the exact change whose task list dropped SquadService's 6 offenders.

---

The reasoning chain (including the wrong turns, corrected)

Wrong framing #1 — "the offender list is a moving target / it drifted"

First answer blamed a "moving target," quoting the proposal's own line "GH #848 measured ~105 in 7
services; six more services have drifted since."

Corrected: the offender set is fixed and fully knowable at spec time — one grep gives it, and
task 1.1 literally says "capture the live offender baseline." From that baseline forward nothing
changes except the offenders you deliberately remove. Fixing is not drift. "Six services drifted
since" was a pre-baseline excuse being parroted to cover a plain gap. Once 1.1 takes the baseline,
every offender is known and must land in a phase. SquadService missing = incomplete
enumeration, not a moving target.

Wrong framing #2 — "it's author prose, not OpenSpec mechanics"

Second answer drew a line: the spec text is the author's prose, separate from OpenSpec the tool.

Corrected (by the user): the author was an agent running the OpenSpec skills —
opsx:explore → propose → apply, end to end. The tool-vs-author split is a dodge. The lossy step
happened inside the OpenSpec workflow, performed by the agent the workflow put in the authoring
seat. It is an OpenSpec-workflow failure, full stop.

---

The actual mechanism of the defect

The baseline grep is complete — a grep cannot miss SquadService. The loss happens in the next
step: the complete machine list is hand-transcribed into thematic phase buckets:

Phase 5 = "dev / export / aggregation"
Phase 6 = "MessageService + policyAcceptance/activityEvent/gameTemplate stragglers"

Each phase gets a clean headline (per design decision D4, "by-model phasing"). SquadService's
offenders are scattered squad/squadPlayer reads + writes that match no phase's headline theme, so
when the full set was paraphrased into named buckets, the offender that fit no bucket fell out.

  complete grep  ──paraphrase──▶  themed phase buckets        result
  (8 offenders)                   ┌─────────────────────┐
                                  │ Phase 5: dev/export │  ✔ assigned
   ActivityEventWriter ×1  ──────▶│ Phase 6: Message +  │  ✔ assigned (6.2)
   MessageService ×1       ──────▶│         stragglers  │  ✔ assigned (6.1)
   SquadService ×6         ──────▶  (no matching theme) │  ✘ DROPPED
                                  └─────────────────────┘

The instant you convert a complete grep into a curated, human-readable phase narrative, you
reintroduce the exact hand-transcription error the grep had eliminated. The buckets read tidy and
complete — every phase has a theme — so nothing visually signals that
Σ(per-phase enumerations) ≠ baseline grep. There is no step that asserts that equality.

Why D5 didn't save it

Design decision D5 ("full-service-clean: a phase takes every service it touches to zero") exists
specifically to catch "off-headline calls get orphaned" — it was added after an audit caught Phase
3b's halfway-house, and Phase 3b is its corrective. It still failed here, because D5 is a sentence
of guidance, not a reconciliation check. A principle in prose doesn't catch what a one-line diff of
two lists would catch.

---

Why OpenSpec didn't catch it either

OpenSpec provides two things, and neither covers completeness:

OpenSpec provides	What it checks	What it does NOT check
Artifact structure (proposal / design / specs / tasks, spec-driven schema)	Shape exists	—
openspec validate --strict	Artifacts are well-formed (requirements have scenarios, delta parses) → issues: []	Whether the tasks cover the baseline offender set

The skills (opsx:explore, propose, apply) hand authoring to an agent and never instruct it to
reconcile the tasks back against the baseline grep the workflow itself started from.

So every stage was OpenSpec-driven and every stage passed the gap straight through:

explore   → surfaced the offenders
propose   → paraphrased them into themed buckets, dropped SquadService
validate  → "structurally valid" (issues: [])           ← reads as "validated"
apply     → trusted the tasks; agent re-greps, finds the hole, flags it

The trap: validate --strict passing reads as "the spec is validated" — but it only validated
the shape. The one thing that mattered — every offender assigned to a phase — is exactly the thing
nothing in the pipeline examines. OpenSpec gave a false sense of rigor: the artifact looks
authoritative while its load-bearing claim (completeness) was left entirely to an agent's paraphrase,
unchecked.

---

Conclusion

Why is the spec wrong? Not drift, not careless mid-apply work, not author-prose-outside-the-tool.

The OpenSpec workflow let an agent convert a complete grep into prose tasks and shipped it as a
"validated" change, with no reconciliation step anywhere between the baseline and the task
list. validate --strict blessed it as valid; the completeness defect — the one thing that
mattered — was never examined by either the skills or the tool. The "validated" stamp covers
shape, not coverage.

The agent-as-author is the weak link, and the OpenSpec workflow wraps no checksum around it. It will
do the same to the next spec unless propose/validate is made to diff the authored enumeration against
its own source.

---

The shape of a fix (noted, not implemented)

The throughline is the user's own principle — derive live, don't hand-type a list
(feedback_centralisation_needs_a_guard). Applied to the planning/verification layer, not just
to the runtime gate:

1. Phase done-condition = the live grep, not a frozen enumeration. Tasks reference
   grep offenders in scope == 0, not "MessageService ×1, SquadService ×6."
2. A conservation invariant the workflow asserts every apply:
   baseline_total == cleared_so_far + remaining_live and every service with a live offender is
   named by an open task — the second clause is the exact check that would have hard-stopped on
   SquadService.
3. Run the services-no-direct-prisma gate in report-only mode from Phase 1, so every phase has
   a live scoreboard; the final phase just flips it to fail-on-nonzero. Completeness becomes
   continuously visible instead of asserted once at the end.

Open, tracked alongside the per-phase spec-conformance question in
~/tmp/openspec-per-phase-spec-conformance-howto.md.

[thlandgraf]

This is one of the cleanest write-ups of the failure I've seen, and the diagnosis is exactly right: the paraphrase step is where the loss happens, and validate --strict can't catch it because it's checking shape, not coverage. I'd push the framing one level up, because I think it generalizes well past this one change.

Any spec claim that's a projection of ground truth - the offender set, a file list, an endpoint inventory, "every service touches zero" - is only ever validated for internal well-formedness unless the claim carries its own re-derivation. The moment a complete machine-derived list gets transcribed into prose tasks, validating that prose can't recover the loss by construction: you're checking that the transcription parses, not that it still equals its source. So "validated" structurally means "internally consistent," never "true against the code." That's less an OpenSpec bug than the ceiling of any static spec validator - the same hole sits in spec-kit, and (full disclosure, I'm the creator) in SPECLAN, where a requirement can sit approved with the identical coverage gap, because approval checks that the acceptance criteria are well-formed, not that they enumerate reality.

The fix-shape is the right instinct - make the done-condition the live grep, assert baseline == cleared + remaining every apply. The one thing I'd flag before leaning on it everywhere: the conservation invariant works beautifully for countable claims where a grep == 0 ground truth exists. Most completeness claims aren't grep-able - "handles all error states," "every mutation is authorized," "no endpoint bypasses the policy layer." Those degrade right back to an agent's paraphrase with no source list to diff against. So the invariant rescues the enumerable subset, but the larger class stays unguarded.

The move that helped me was making validate distinguish two kinds of assertion: machine-verifiable ones that carry an executable predicate it can actually run (your grep-as-done-condition), versus judgment assertions it can only parse. The "validated" stamp then means different things for each, instead of one stamp that silently covers shape and gets read as coverage. Smaller change than re-deriving everything, and it at least kills the false sense of rigor.

Would a first-class "coverage assertion" - a spec element that carries the re-derivation command so validate runs it instead of parsing it - fit OpenSpec's model, or does that belong in a skill rather than the validator?