Skip to content

Compile every generated Screenplay document before returning it - #2402

Merged
woksin merged 1 commit into
mainfrom
feat/screenplay-verify-own-output
Jul 28, 2026
Merged

Compile every generated Screenplay document before returning it#2402
woksin merged 1 commit into
mainfrom
feat/screenplay-verify-own-output

Conversation

@woksin

@woksin woksin commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Added

  • Generated documents are compiled before they are returned; one the Screenplay language rejects is reported as an error naming the first rejected line, so a host exits non-zero instead of writing a document that does not compile

RoundTrip printed a document, compiled it and reprinted it, and its own
comment called it the correctness gate for everything the generator
produces. It lived in the spec project, so it only ever ran against
hand-written fixtures - the generator running against a real application
verified nothing. That is how a command property called Description
shipped a document that did not compile: no fixture happened to use that
name, and 690 green specs said nothing.

Move the capability into the package and read every printed document back
through the Screenplay compiler. A rejected document is the generator
being wrong rather than anything the source declared, so it is reported at
error severity and a host exits non zero. The text is returned regardless
so the rejected line can be read.

The verifier is held concretely rather than injected. A host that could
substitute it could switch it off, and a check nobody runs is the
situation this exists to end. SP0024 suppresses it, because a model
recovered from source the compiler never accepted describes an
application that does not exist.

Reprinting and comparing stays a spec-level gate. Compiling is the cheap,
decisive check; byte-identical reprint is a stronger claim about
information loss and belongs where fixtures are known.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@woksin woksin added the minor label Jul 28, 2026
@woksin

woksin commented Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

Reviewer context — deliberately kept out of the description.

Why this exists. RoundTrip already printed a document, compiled it and reprinted it, and its own comment called it "the correctness gate for everything the generator produces". It lived in Screenplay.Specs, so it only ever ran against hand-written fixtures — the generator running against a real application verified nothing. That is exactly how a command property named Description shipped a document that did not compile: no fixture happened to use that name, and 690 green specs said nothing about it.

Three decisions worth checking

  • The verifier is held concretely, not injected. ScreenplayGenerator has a private ScreenplayVerifier field rather than a constructor parameter. The check was decided always-on, and a host that could substitute it could switch it off. This also leaves the public surface of ScreenplayGenerator unchanged, so the published package is not broken.
  • Runtime does compile only. Reprinting and comparing byte-for-byte stays a spec-level gate. Compiling is the cheap, decisive test of "is this valid"; byte-identical reprint is a stronger claim about information loss and belongs where the fixtures are known.
  • IScreenplayVerifier has no consumer that injects it. Kept for consistency with every other capability in the package (IScreenplayGenerator, IApplicationModelAnalyzer, IScreenplayEmitter), and because adding a public interface later is additive while removing one is breaking. If you would rather not ship an abstraction nothing injects, deleting the file and dropping : IScreenplayVerifier is the whole change.

SP0024 suppresses the new SP0034: a model recovered from source the compiler never accepted describes an application that does not exist, so a poor document made from it is a consequence already reported rather than a second defect. That suppression is covered by a spec which was confirmed to fail when the guard is removed.

What this does not catch. Compiling your own output catches "invalid", never "valid but untrue". The enum defect fixed earlier in #2384 emitted role = 6, which compiles perfectly and is simply wrong. An end-to-end CI gate that generates from a real project remains worth adding for that class.

Verification

  • Screenplay.csproj Release with bin/obj deleted first: 0 errors, 0 warnings.
  • 804 specs passing (783 before).
  • Against a real third-party application (~5300 lines, 121 commands, 172 events): the new check reported nothing — 0 SP0034, 0 error-severity diagnostics, and the document still compiles under the official Screenplay tool with 0 errors. Generation remains deterministic, byte-identical across two runs.
  • TestApps/AspNetCore end to end through the CLI: 52 lines, compiles 0 errors / 0 warnings.

No issue reference in the description because none of the open Screenplay issues (#2385, #2386, #2393#2401) covers this.

@woksin woksin added patch and removed minor labels Jul 28, 2026
@woksin
woksin merged commit 552e7de into main Jul 28, 2026
56 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant