Compile every generated Screenplay document before returning it

Documentation/generating-a-screenplay.md

@@ -50,6 +50,23 @@ transported rather than what it is, so the query was left out (Library.Messaging
 
 Diagnostics come in three severities. **Information** means something is worth knowing but the document is complete. **Warning** means something was left out. **Error** means the document should not be trusted at all — the most common cause being source that did not compile.
 
+## The generator checks its own output
+
+Every diagnostic above names something *your application* declared that the language cannot hold. There is one that names a defect in the generator instead.
+
+After the document is written, the generator hands it straight back to the Screenplay compiler. If the compiler rejects it, `SP0034` is reported as an error — because a `.play` that does not compile is output nobody can use, and there is no way of writing an application that avoids it. This is not a mode you turn on: it runs on every generation, since the only way a rejected document is ever found is by reading each one back.
+
+```text
+Error SP0034: The generated document did not compile - 1 error(s), the first being
+'Invalid description 'description RequestDescription' - expected 'description "<text>"''
+on line 6. That is the generator being wrong rather than anything the source declared,
+and the document is returned as it stands so the line can be read (Library)
+```
+
+The document is still written out, so you can open it at the reported line and see what happened. If you hit this, it is a bug worth [reporting](https://github.com/Cratis/Arc/issues) — include the line, and the C# declaration it came from.
+
+Source that did not compile (`SP0024`) suppresses `SP0034`. A model recovered from symbols the compiler never accepted describes an application that does not exist, so a poor document made from it is a consequence of the broken build rather than a second, separate defect. Fix the build and generate again.
+
 ## Screens
 
 A [vertical slice](./vertical-slices.md) puts the React component that realizes a slice's screen in the same folder as its C#. Roslyn syntax trees carry real file paths, so the generator knows where each slice's source lives and declares a `screen` for every `.tsx` component sitting next to it, named after the file:

Source/DotNET/Screenplay.Specs/for_ScreenplayEmitter/when_emitting/a_library_application.cs

@@ -2,6 +2,7 @@
 // Licensed under the MIT license. See LICENSE file in the project root for full license information.
 
 using Cratis.Arc.Screenplay.Emission;
+using Cratis.Arc.Screenplay.Verification;
 using Cratis.Screenplay.Syntax;
 
 namespace Cratis.Arc.Screenplay.for_ScreenplayEmitter.when_emitting;

Source/DotNET/Screenplay.Specs/for_ScreenplayEmitter/when_emitting/a_projection_using_every_block_kind.cs

@@ -4,6 +4,7 @@
 using Cratis.Arc.Screenplay.Emission;
 using Cratis.Arc.Screenplay.Library;
 using Cratis.Arc.Screenplay.Model;
+using Cratis.Arc.Screenplay.Verification;
 
 namespace Cratis.Arc.Screenplay.for_ScreenplayEmitter.when_emitting;
 

Source/DotNET/Screenplay.Specs/for_ScreenplayEmitter/when_emitting/a_projection_with_an_unmappable_expression.cs

@@ -4,6 +4,7 @@
 using Cratis.Arc.Screenplay.Emission;
 using Cratis.Arc.Screenplay.Library;
 using Cratis.Arc.Screenplay.Model;
+using Cratis.Arc.Screenplay.Verification;
 
 namespace Cratis.Arc.Screenplay.for_ScreenplayEmitter.when_emitting;
 

Source/DotNET/Screenplay.Specs/for_ScreenplayEmitter/when_emitting/a_slice_naming_its_members_after_directives.cs

@@ -4,6 +4,7 @@
 using Cratis.Arc.Screenplay.Emission;
 using Cratis.Arc.Screenplay.Library;
 using Cratis.Arc.Screenplay.Model;
+using Cratis.Arc.Screenplay.Verification;
 
 namespace Cratis.Arc.Screenplay.for_ScreenplayEmitter.when_emitting;
 

Source/DotNET/Screenplay.Specs/for_ScreenplayEmitter/when_emitting/a_slice_that_declares_nothing.cs

@@ -4,6 +4,7 @@
 using Cratis.Arc.Screenplay.Emission;
 using Cratis.Arc.Screenplay.Library;
 using Cratis.Arc.Screenplay.Model;
+using Cratis.Arc.Screenplay.Verification;
 
 namespace Cratis.Arc.Screenplay.for_ScreenplayEmitter.when_emitting;
 

Source/DotNET/Screenplay.Specs/for_ScreenplayEmitter/when_emitting/a_slice_whose_screens_are_all_it_declares.cs

@@ -4,6 +4,7 @@
 using Cratis.Arc.Screenplay.Emission;
 using Cratis.Arc.Screenplay.Library;
 using Cratis.Arc.Screenplay.Model;
+using Cratis.Arc.Screenplay.Verification;
 
 namespace Cratis.Arc.Screenplay.for_ScreenplayEmitter.when_emitting;
 

Source/DotNET/Screenplay.Specs/for_ScreenplayGenerator/given/a_document_the_language_rejects.cs

@@ -0,0 +1,49 @@
+// Copyright (c) Cratis. All rights reserved.
+// Licensed under the MIT license. See LICENSE file in the project root for full license information.
+
+using Cratis.Arc.Screenplay.Emission;
+using Cratis.Arc.Screenplay.Emission.Naming;
+using Cratis.Screenplay.Printing;
+using Cratis.Screenplay.Syntax;
+
+namespace Cratis.Arc.Screenplay.for_ScreenplayGenerator.given;
+
+/// <summary>
+/// Stands in for the printer, writing text that was prepared rather than rendered, so that a document the Screenplay
+/// compiler rejects can be put in front of the generator.
+/// </summary>
+/// <remarks>
+/// Every way of emitting a rejected document that anyone has found has since been fixed, which is exactly why the
+/// check exists - it has to hold for the one nobody has found yet. Replacing the printed text is the smallest seam
+/// that produces one: the analysis, the emitter, the syntax tree the emitter builds and the compiler reading the
+/// text back are all the real ones, and nothing on the path a host takes is substituted. The text is the defect that
+/// shipped, which is a command property called <c>Description</c> written out as the directive of the same name.
+/// </remarks>
+public class a_document_the_language_rejects : Specification
+{
+    /// <summary>
+    /// The text the stand-in printer writes in place of the document.
+    /// </summary>
+    protected const string Rejected = """
+        domain Library
+        module Library
+          feature Authors
+            slice StateChange Registration
+              command RegisterAuthor
+                description RequestDescription
+        """;
+
+    /// <summary>
+    /// The line of the text the Screenplay compiler rejects.
+    /// </summary>
+    protected const int RejectedLine = 6;
+
+    protected IScreenplayEmitter _emitter;
+
+    void Establish()
+    {
+        var printer = Substitute.For<IScreenplayPrinter>();
+        printer.Print(Arg.Any<ApplicationSyntax>()).Returns(Rejected);
+        _emitter = new ScreenplayEmitter(printer, new ScreenplayNaming());
+    }
+}

Source/DotNET/Screenplay.Specs/for_ScreenplayGenerator/when_generating/and_the_document_it_printed_compiles.cs

@@ -0,0 +1,34 @@
+// Copyright (c) Cratis. All rights reserved.
+// Licensed under the MIT license. See LICENSE file in the project root for full license information.
+
+using Cratis.Arc.Screenplay.Emission;
+using Cratis.Arc.Screenplay.Library;
+using Cratis.Screenplay;
+
+namespace Cratis.Arc.Screenplay.for_ScreenplayGenerator.when_generating;
+
+/// <summary>
+/// Reading every generated document back is only worth doing if it stays quiet about the documents that are fine.
+/// A check that cries wolf on a real application is a check people learn to ignore, so this generates the whole
+/// library - every slice kind, every construct the language holds - and expects nothing said about it.
+/// </summary>
+public class and_the_document_it_printed_compiles : given.a_compilation
+{
+    ScreenplayGenerator _generator;
+    ScreenplayGenerationResult _result;
+    CompilationResult<Cratis.Screenplay.Syntax.ApplicationSyntax> _compiled;
+
+    void Establish() => _generator = new(new given.a_recovered_model(LibraryApplication.Build()), new ScreenplayEmitter());
+
+    void Because()
+    {
+        _result = _generator.Generate(_compilation, new ScreenplayOptions());
+        _compiled = new ScreenplayCompiler().Compile(_result.Source);
+    }
+
+    [Fact] void should_be_generating_a_document_that_really_does_compile() => _compiled.Success.ShouldBeTrue();
+    [Fact] void should_be_generating_the_whole_application() => _result.Source.Contains("slice StateChange Registration", StringComparison.Ordinal).ShouldBeTrue();
+    [Fact] void should_not_report_that_the_document_did_not_compile() => _result.Diagnostics.Select(_ => _.Code).ShouldNotContain(ScreenplayDiagnosticCodes.DocumentDidNotCompile);
+    [Fact] void should_report_nothing_at_all() => _result.Diagnostics.ShouldBeEmpty();
+    [Fact] void should_be_successful() => _result.IsSuccess.ShouldBeTrue();
+}

Source/DotNET/Screenplay.Specs/for_ScreenplayGenerator/when_generating/and_the_document_it_printed_does_not_compile.cs

@@ -0,0 +1,43 @@
+// Copyright (c) Cratis. All rights reserved.
+// Licensed under the MIT license. See LICENSE file in the project root for full license information.
+
+using Cratis.Arc.Screenplay.Analysis;
+using Cratis.Screenplay;
+using Microsoft.CodeAnalysis;
+
+namespace Cratis.Arc.Screenplay.for_ScreenplayGenerator.when_generating;
+
+/// <summary>
+/// A document the Screenplay compiler rejects is output nobody can use, and no way of writing an application avoids
+/// it - it is the generator being wrong. Nothing but reading every generated document back finds one, which is how
+/// a property named after a directive shipped with 690 green specifications behind it. So every generation compiles
+/// what it printed, reports a rejected document as an error - which is what makes a host exit non zero - and hands
+/// the text back regardless, because the line that was rejected is the only place to start looking.
+/// </summary>
+public class and_the_document_it_printed_does_not_compile : given.a_document_the_language_rejects
+{
+    Compilation _compilation;
+    ScreenplayGenerator _generator;
+    ScreenplayGenerationResult _result;
+
+    void Establish()
+    {
+        _compilation = Analyzed.Compile(("Library/Authors/Registration/Registration.cs", "namespace Library.Authors.Registration;"));
+        _generator = new(new ApplicationModelAnalyzer(), _emitter);
+    }
+
+    void Because() => _result = _generator.Generate(_compilation, new ScreenplayOptions());
+
+    ScreenplayDiagnostic Reported => _result.Diagnostics.First(_ => _.Code == ScreenplayDiagnosticCodes.DocumentDidNotCompile);
+
+    [Fact] void should_be_printing_a_document_the_language_really_rejects() => new ScreenplayCompiler().Compile(Rejected).Success.ShouldBeFalse();
+    [Fact] void should_report_that_the_document_did_not_compile() => _result.Diagnostics.Select(_ => _.Code).ShouldContain(ScreenplayDiagnosticCodes.DocumentDidNotCompile);
+    [Fact] void should_report_it_as_an_error() => Reported.Severity.ShouldEqual(ScreenplayDiagnosticSeverity.Error);
+    [Fact] void should_report_it_only_once() => _result.Diagnostics.Count(_ => _.Code == ScreenplayDiagnosticCodes.DocumentDidNotCompile).ShouldEqual(1);
+    [Fact] void should_quote_the_first_thing_the_compiler_said() => Reported.Message.Contains("description RequestDescription", StringComparison.Ordinal).ShouldBeTrue();
+    [Fact] void should_say_which_line_was_rejected() => Reported.Message.Contains($"on line {RejectedLine}", StringComparison.Ordinal).ShouldBeTrue();
+    [Fact] void should_say_it_is_the_generator_that_is_wrong() => Reported.Message.Contains("the generator being wrong", StringComparison.Ordinal).ShouldBeTrue();
+    [Fact] void should_still_return_the_document() => _result.Source.ShouldEqual(Rejected);
+    [Fact] void should_not_be_successful() => _result.IsSuccess.ShouldBeFalse();
+    [Fact] void should_not_claim_the_source_it_read_failed_to_compile() => _result.Diagnostics.Select(_ => _.Code).ShouldNotContain(ScreenplayDiagnosticCodes.SourceDidNotCompile);
+}

Source/DotNET/Screenplay.Specs/for_ScreenplayGenerator/when_generating/and_the_source_it_read_did_not_compile.cs

@@ -0,0 +1,45 @@
+// Copyright (c) Cratis. All rights reserved.
+// Licensed under the MIT license. See LICENSE file in the project root for full license information.
+
+using Cratis.Arc.Screenplay.Analysis;
+using Cratis.Screenplay;
+using Microsoft.CodeAnalysis;
+
+namespace Cratis.Arc.Screenplay.for_ScreenplayGenerator.when_generating;
+
+/// <summary>
+/// Source the compiler never accepted still yields symbols, and a model recovered from them describes an application
+/// that does not exist - so a document made from it being poor is the consequence already reported rather than a
+/// second defect. Saying both would send whoever reads the diagnostics hunting for a bug in the generator when the
+/// thing to fix is the build, which is why the same suppression the empty document already gets applies here.
+/// </summary>
+public class and_the_source_it_read_did_not_compile : given.a_document_the_language_rejects
+{
+    const string Source = """
+        namespace Library.Authors.Registration;
+
+        public record RegisterAuthor(string Name)
+        {
+            public string Handle() => ThisDoesNotExist;
+        }
+        """;
+
+    Compilation _compilation;
+    ScreenplayGenerator _generator;
+    ScreenplayGenerationResult _result;
+
+    void Establish()
+    {
+        _compilation = Analyzed.Compile((Analyzed.SlicePath, Source));
+        _generator = new(new ApplicationModelAnalyzer(), _emitter);
+    }
+
+    void Because() => _result = _generator.Generate(_compilation, new ScreenplayOptions());
+
+    [Fact] void should_be_generating_from_source_that_really_does_not_compile() => Analyzed.ErrorsIn((Analyzed.SlicePath, Source)).ShouldNotBeEmpty();
+    [Fact] void should_be_printing_a_document_the_language_really_rejects() => new ScreenplayCompiler().Compile(Rejected).Success.ShouldBeFalse();
+    [Fact] void should_report_that_the_source_did_not_compile() => _result.Diagnostics.Select(_ => _.Code).ShouldContain(ScreenplayDiagnosticCodes.SourceDidNotCompile);
+    [Fact] void should_not_report_the_document_on_top_of_it() => _result.Diagnostics.Select(_ => _.Code).ShouldNotContain(ScreenplayDiagnosticCodes.DocumentDidNotCompile);
+    [Fact] void should_still_return_the_document() => _result.Source.ShouldEqual(Rejected);
+    [Fact] void should_not_be_successful() => _result.IsSuccess.ShouldBeFalse();
+}

Source/DotNET/Screenplay/ScreenplayDiagnosticCodes.cs

@@ -211,4 +211,17 @@ public static class ScreenplayDiagnosticCodes
     /// number is written as it stands and what it names is left unsaid.
     /// </remarks>
     public const string UnnamedEnumerationValue = "SP0033";
+
+    /// <summary>
+    /// The document that was generated does not compile, which is the generator being wrong rather than the source.
+    /// </summary>
+    /// <remarks>
+    /// Every other code here names something the application declared that the language has no counterpart for,
+    /// which is a gap the reader can see and work around. This one names a defect in the generator - a document the
+    /// Screenplay compiler rejects is output nobody can use, and no way of writing an application avoids it. It
+    /// exists because such a document is only ever found by reading every generated one back: a property named
+    /// after a directive shipped once precisely because no fixture happened to use that name. The text is returned
+    /// as it stands so the line that was rejected can be read.
+    /// </remarks>
+    public const string DocumentDidNotCompile = "SP0034";
 }

Source/DotNET/Screenplay/ScreenplayGenerator.cs

@@ -3,6 +3,7 @@
 
 using Cratis.Arc.Screenplay.Analysis;
 using Cratis.Arc.Screenplay.Emission;
+using Cratis.Arc.Screenplay.Verification;
 using Microsoft.CodeAnalysis;
 
 namespace Cratis.Arc.Screenplay;
@@ -15,10 +16,14 @@ namespace Cratis.Arc.Screenplay;
 /// <remarks>
 /// The package ships no dependency injection of its own, so a consumer that just wants to generate a document says
 /// <c>new ScreenplayGenerator()</c> and gets everything wired. The constructor taking collaborators exists for
-/// specifications and for hosts that want to substitute one half.
+/// specifications and for hosts that want to substitute one half. Reading the generated document back is not one of
+/// the halves - a host that could substitute it could switch it off, and a check nobody runs is the situation this
+/// exists to end.
 /// </remarks>
 public class ScreenplayGenerator(IApplicationModelAnalyzer analyzer, IScreenplayEmitter emitter) : IScreenplayGenerator
 {
+    readonly ScreenplayVerifier _verifier = new();
+
     /// <summary>
     /// Initializes a new instance of the <see cref="ScreenplayGenerator"/> class with everything wired up.
     /// </summary>
@@ -38,6 +43,54 @@ public ScreenplayGenerationResult Generate(Compilation compilation, ScreenplayOp
         var analysis = analyzer.Analyze(compilation, resolved);
         var emission = emitter.Emit(analysis.Model, resolved);
 
-        return new(emission.Source, analysis.Model, [.. analysis.Diagnostics, .. emission.Diagnostics]);
+        var diagnostics = new ScreenplayDiagnostics();
+        diagnostics.AddRange(analysis.Diagnostics);
+        diagnostics.AddRange(emission.Diagnostics);
+        ReportDocumentThatDoesNotCompile(emission.Source, analysis.Diagnostics, diagnostics, compilation.AssemblyName);
+
+        return new(emission.Source, analysis.Model, diagnostics.All);
+    }
+
+    /// <summary>
+    /// Reads the printed document back and reports one the Screenplay compiler rejects.
+    /// </summary>
+    /// <param name="source">The printed document.</param>
+    /// <param name="analyzed">What analysis reported, which says whether the source it read compiled.</param>
+    /// <param name="diagnostics">The diagnostics to report to.</param>
+    /// <param name="location">Where to report against.</param>
+    /// <remarks>
+    /// Everything else reported names something the application declared that the language cannot hold. This names
+    /// the generator being wrong, which is why it runs on every generation rather than on request - the only way a
+    /// rejected document is ever found is by reading each one back, and a document nobody happened to look at is
+    /// exactly how one shipped. Reporting it as an error is what makes a host exit non zero, and the text is
+    /// returned regardless so the line that was rejected can be read - the same bargain
+    /// <see cref="ScreenplayDiagnosticCodes.SourceDidNotCompile"/> strikes. That code also suppresses this one, the
+    /// way it suppresses <see cref="ScreenplayDiagnosticCodes.AnalysisUnavailable"/>: a model recovered from symbols
+    /// the compiler never accepted describes an application that does not exist, so a poor document made from it is
+    /// the consequence already reported rather than a second defect.
+    /// </remarks>
+    void ReportDocumentThatDoesNotCompile(
+        string source,
+        IEnumerable<ScreenplayDiagnostic> analyzed,
+        ScreenplayDiagnostics diagnostics,
+        string? location)
+    {
+        if (analyzed.Any(_ => _.Code == ScreenplayDiagnosticCodes.SourceDidNotCompile))
+        {
+            return;
+        }
+
+        var verification = _verifier.Verify(source);
+        if (verification.Compiles)
+        {
+            return;
+        }
+
+        var first = verification.Errors[0];
+
+        diagnostics.Error(
+            ScreenplayDiagnosticCodes.DocumentDidNotCompile,
+            $"The generated document did not compile - {verification.Errors.Count} error(s), the first being '{first.Message}' on line {first.Location.Line}. That is the generator being wrong rather than anything the source declared, and the document is returned as it stands so the line can be read",
+            location);
     }
 }