Report a partly compiling source as a warning when recovery still stands

Documentation/generating-a-screenplay.md

@@ -35,7 +35,7 @@ The generator is used through the Cratis CLI, which ships separately from Arc:
 cratis screenplay generate ./MyApp/MyApp.csproj --file MyApp.play
 ```
 
-Point it at a project or a solution. Anything the generator could not express is reported on the way out, and a run that reports an error exits non-zero — so a broken build never quietly produces a document that looks fine.
+Point it at a project or a solution. Anything the generator could not express is reported on the way out, and a run that reports an error exits non-zero — so a document nothing trustworthy went into never quietly looks fine. What the CLI has to do before it hands the source over is covered in [what the generator expects of its host](#what-the-generator-expects-of-its-host).
 
 Because output is reproducible, a CI step can regenerate and fail when the committed `.play` no longer matches the source.
 
@@ -48,7 +48,49 @@ Warning SP0019: The query 'Raw' returns 'IActionResult', which says how the resu
 transported rather than what it is, so the query was left out (Library.Messaging.Feed)
 ```
 
-Diagnostics come in three severities. **Information** means something is worth knowing but the document is complete. **Warning** means something was left out. **Error** means the document should not be trusted at all — the most common cause being source that did not compile.
+Diagnostics come in three severities. **Information** means something is worth knowing but the document is complete. **Warning** means something was left out. **Error** means the document should not be trusted at all — either because the generator produced something the language rejects, or because nothing at all was recovered from source the compiler accepted.
+
+## What the generator expects of its host
+
+The generator takes a Roslyn `Compilation` and nothing else. It never opens a project file or loads a workspace, which is what lets it be driven from a CLI, from a specification, or from an editor. The other side of that bargain is that **assembling the compilation the way a real build would is the caller's job** — the generator reads what it is handed and cannot tell a missing type from a type that was never written.
+
+The part hosts get wrong is source generators. `MSBuildWorkspace.GetCompilationAsync()` does not run them, and neither does any loading mode that stops at the compile items the project file lists. An Arc application leans on generation heavily — `[LoggerMessage]` partial classes, strongly-typed resource designer classes, the proxy and metrics generators — so a compilation loaded that way is missing every type those emit, and every reference to one becomes an unresolved-symbol error. This is the common case rather than an edge case.
+
+A host should run the project's generators before handing the compilation over:
+
+```csharp
+var driver = CSharpGeneratorDriver.Create(
+    generators: project.AnalyzerReferences
+        .SelectMany(reference => reference.GetGenerators(LanguageNames.CSharp))
+        .Select(GeneratorExtensions.AsSourceGenerator),
+    parseOptions: (CSharpParseOptions)project.ParseOptions!);
+
+driver.RunGeneratorsAndUpdateCompilation(compilation, out var generated, out _);
+```
+
+Then generate from `generated` rather than from `compilation`.
+
+### What happens when it does not
+
+Nothing is hidden, and nothing correct is thrown away. A compilation carrying errors is still analyzed, and `SP0024` says what happened — how many errors there were, the first of them, and how many artifacts came through anyway:
+
+```text
+Warning SP0024: The source did not compile - 607 error(s), the first being 'The name
+'AccountsMessages' does not exist in the current context'. 341 artifact(s) were recovered
+anyway, 341 of them from a declaration no error sits inside, so the document describes those
+exactly as the source states them - a missing type named like 'SomethingMessages' or a
+designer class usually means the compilation was handed over without the compile items a
+build generates (Accounts)
+```
+
+Its **severity is decided rather than fixed**, because "the source did not compile" covers two outcomes that could not be further apart:
+
+- **Warning** when at least one artifact was recovered from a declaration no compilation error sits inside. Those artifacts are described exactly as their source states them whatever failed elsewhere, so the run is successful and the document is worth keeping. A compilation missing its generated symbols lands here — the errors sit in code that declares no artifact, and the model is unaffected.
+- **Error** when none were, either because nothing was recovered at all or because every declaration something came out of is one the compiler could not make sense of. There is then no part of the document a reader could trust, and a host following the contract exits non-zero.
+
+A count is used rather than a proportion deliberately: any threshold would make the same recovery pass for a large application and fail for a small one, and zero is the only number that means recovery was *prevented* rather than merely dented.
+
+Either way the document is written out, so what was recovered can be read.
 
 ## The generator checks its own output
 
@@ -65,7 +107,9 @@ and the document is returned as it stands so the line can be read (Library)
 
 The document is still written out, so you can open it at the reported line and see what happened. If you hit this, it is a bug worth [reporting](https://github.com/Cratis/Arc/issues) — include the line, and the C# declaration it came from.
 
-Source that did not compile (`SP0024`) suppresses `SP0034`. A model recovered from symbols the compiler never accepted describes an application that does not exist, so a poor document made from it is a consequence of the broken build rather than a second, separate defect. Fix the build and generate again.
+Source that did not compile (`SP0024`) suppresses `SP0034` **when it is reported as an error** — a model recovered from symbols the compiler never accepted describes an application that does not exist, so a poor document made from it is a consequence of the broken build rather than a second, separate defect. Fix the build and generate again.
+
+As a warning it suppresses nothing. That severity says the model stands, and a document built from a model that stands is exactly what the check exists for — suppressing it there would hand back a `.play` the language rejects with nothing wrong reported.
 
 ## Screens
 

Source/DotNET/Screenplay.Specs/for_ApplicationModelAnalyzer/when_analyzing/source_that_did_not_compile/and_an_error_sits_in_a_partial_the_build_wrote.cs

@@ -0,0 +1,57 @@
+// Copyright (c) Cratis. All rights reserved.
+// Licensed under the MIT license. See LICENSE file in the project root for full license information.
+
+using Cratis.Arc.Screenplay.Analysis;
+
+namespace Cratis.Arc.Screenplay.for_ApplicationModelAnalyzer.when_analyzing.source_that_did_not_compile;
+
+/// <summary>
+/// A type is written in as many places as it has partial declarations, and the Cratis house style puts one of them in
+/// the intermediate folder of the build. An error there is an error inside the artifact just as much as one in the
+/// file a person wrote, so every place a type is written counts - reading only the first would call a declaration
+/// clean on the strength of the half that happens to be listed first.
+/// </summary>
+public class and_an_error_sits_in_a_partial_the_build_wrote : Specification
+{
+    const string Written = """
+        using System.Threading.Tasks;
+        using Cratis.Chronicle.Events;
+        using Cratis.Chronicle.Reactors;
+
+        namespace Library.Onboarding.Registry;
+
+        [EventType]
+        public record CompanyRegistered(string Name);
+
+        public partial class RegistryNotifier : IReactor
+        {
+            public Task CompanyRegistered(CompanyRegistered @event, EventContext context) => Task.CompletedTask;
+        }
+        """;
+
+    const string Emitted = """
+        namespace Library.Onboarding.Registry;
+
+        public partial class RegistryNotifier
+        {
+            public string Describe() => RegistryMessages.Describe;
+        }
+        """;
+
+    static readonly (string Path, string Text)[] _sources =
+    [
+        ("Library/Onboarding/Registry/Registry.cs", Written),
+        ("Library/obj/Debug/net10.0/Generated/Registry.Logging.g.cs", Emitted)
+    ];
+
+    ApplicationModelAnalysis _analysis;
+
+    void Establish() => _analysis = Analyzed.Source(_sources);
+
+    ScreenplayDiagnostic Reported => _analysis.Diagnostics.First(_ => _.Code == ScreenplayDiagnosticCodes.SourceDidNotCompile);
+
+    [Fact] void should_be_analyzing_source_that_really_does_not_compile() => Analyzed.ErrorsIn(_sources).ShouldNotBeEmpty();
+    [Fact] void should_report_it_as_a_warning() => Reported.Severity.ShouldEqual(ScreenplayDiagnosticSeverity.Warning);
+    [Fact] void should_not_count_the_reactor_the_error_is_written_inside() => Reported.Message.Contains("2 artifact(s) were recovered anyway, 1 of them", StringComparison.Ordinal).ShouldBeTrue();
+    [Fact] void should_still_return_the_reactor() => _analysis.Slice().Reactors.Single().Name.ShouldEqual("RegistryNotifier");
+}

Source/DotNET/Screenplay.Specs/for_ApplicationModelAnalyzer/when_analyzing/source_that_did_not_compile/and_every_artifact_it_recovered_carries_an_error.cs

@@ -0,0 +1,40 @@
+// Copyright (c) Cratis. All rights reserved.
+// Licensed under the MIT license. See LICENSE file in the project root for full license information.
+
+using Cratis.Arc.Screenplay.Analysis;
+
+namespace Cratis.Arc.Screenplay.for_ApplicationModelAnalyzer.when_analyzing.source_that_did_not_compile;
+
+/// <summary>
+/// Something was recovered, but every declaration it came out of is one the compiler could not make sense of - so
+/// what the document says about each of them is read from source that never held together. That is the other way
+/// recovery is genuinely prevented, and it is an error for the same reason as recovering nothing at all: there is
+/// no part of the document a reader could trust.
+/// </summary>
+public class and_every_artifact_it_recovered_carries_an_error : Specification
+{
+    const string Source = """
+        using Cratis.Arc.Commands.ModelBound;
+
+        namespace Library.Authors.Registration;
+
+        [Command]
+        public record RegisterAuthor(string Name)
+        {
+            public void Handle() => ThisDoesNotExist();
+        }
+        """;
+
+    ApplicationModelAnalysis _analysis;
+
+    void Establish() => _analysis = Analyzed.Source(Source);
+
+    ScreenplayDiagnostic Reported => _analysis.Diagnostics.First(_ => _.Code == ScreenplayDiagnosticCodes.SourceDidNotCompile);
+
+    [Fact] void should_be_analyzing_source_that_really_does_not_compile() => Analyzed.ErrorsIn((Analyzed.SlicePath, Source)).ShouldNotBeEmpty();
+    [Fact] void should_report_it_as_an_error() => Reported.Severity.ShouldEqual(ScreenplayDiagnosticSeverity.Error);
+    [Fact] void should_say_how_much_it_recovered() => Reported.Message.Contains("1 artifact(s) were recovered", StringComparison.Ordinal).ShouldBeTrue();
+    [Fact] void should_say_the_errors_sit_inside_what_it_recovered() => Reported.Message.Contains("every declaration they were read from is one an error sits inside", StringComparison.Ordinal).ShouldBeTrue();
+    [Fact] void should_say_the_document_cannot_be_relied_on() => Reported.Message.Contains("describes the application reliably", StringComparison.Ordinal).ShouldBeTrue();
+    [Fact] void should_still_return_whatever_it_recovered() => _analysis.Slice().Commands.Single().Name.ShouldEqual("RegisterAuthor");
+}

Source/DotNET/Screenplay.Specs/for_ApplicationModelAnalyzer/when_analyzing/source_that_did_not_compile.cs → Source/DotNET/Screenplay.Specs/for_ApplicationModelAnalyzer/when_analyzing/source_that_did_not_compile/and_nothing_at_all_was_recovered.cs

@@ -3,29 +3,22 @@
 
 using Cratis.Arc.Screenplay.Analysis;
 
-namespace Cratis.Arc.Screenplay.for_ApplicationModelAnalyzer.when_analyzing;
+namespace Cratis.Arc.Screenplay.for_ApplicationModelAnalyzer.when_analyzing.source_that_did_not_compile;
 
 /// <summary>
-/// Source that does not build still yields symbols, and analyzing them produces a document that looks like an
-/// answer while describing an application that does not exist. A nearly empty document handed back with nothing
-/// wrong reported is the one outcome nobody can act on, so this is an error - which is what makes a host exit non
-/// zero - while whatever was recovered is still returned for whoever wants to look at it.
+/// A build that produced nothing an artifact could be read out of is the case the claim was always true for - a
+/// document describing an empty application handed back with nothing wrong reported is the one outcome nobody can
+/// act on, so this is an error and a host exits non zero on it. It is also the case that must not be mistaken for
+/// an application which simply declares no artifact yet, so "declares nothing" stays suppressed.
 /// </summary>
-public class source_that_did_not_compile : Specification
+public class and_nothing_at_all_was_recovered : Specification
 {
     const string Source = """
-        using Cratis.Arc.Commands.ModelBound;
-        using Cratis.Chronicle.Events;
+        namespace Library.Plumbing;
 
-        namespace Library.Authors.Registration;
-
-        [EventType]
-        public record AuthorRegistered(string Name);
-
-        [Command]
-        public record RegisterAuthor(string Name)
+        public class NotAnArtifact
         {
-            public AuthorRegistered Handle() => new(ThisDoesNotExist);
+            public string Name() => ThisDoesNotExist;
         }
         """;
 
@@ -35,11 +28,13 @@ public record RegisterAuthor(string Name)
 
     ScreenplayDiagnostic Reported => _analysis.Diagnostics.First(_ => _.Code == ScreenplayDiagnosticCodes.SourceDidNotCompile);
 
-    [Fact] void should_be_analyzing_source_that_really_does_not_compile() => Analyzed.ErrorsIn(("Library/Feature/Slice/Slice.cs", Source)).ShouldNotBeEmpty();
+    [Fact] void should_be_analyzing_source_that_really_does_not_compile() => Analyzed.ErrorsIn((Analyzed.SlicePath, Source)).ShouldNotBeEmpty();
     [Fact] void should_report_that_the_source_did_not_compile() => _analysis.Diagnostics.Select(_ => _.Code).ShouldContain(ScreenplayDiagnosticCodes.SourceDidNotCompile);
     [Fact] void should_report_it_as_an_error() => Reported.Severity.ShouldEqual(ScreenplayDiagnosticSeverity.Error);
+    [Fact] void should_say_nothing_at_all_came_out_of_it() => Reported.Message.Contains("Nothing at all was recovered from it", StringComparison.Ordinal).ShouldBeTrue();
     [Fact] void should_say_the_document_cannot_be_relied_on() => Reported.Message.Contains("describes the application reliably", StringComparison.Ordinal).ShouldBeTrue();
     [Fact] void should_quote_the_first_thing_the_compiler_said() => Reported.Message.Contains("ThisDoesNotExist", StringComparison.Ordinal).ShouldBeTrue();
-    [Fact] void should_still_return_whatever_it_recovered() => _analysis.Slice().Commands.Single().Name.ShouldEqual("RegisterAuthor");
+    [Fact] void should_not_also_say_the_application_declares_nothing() => _analysis.Diagnostics.Select(_ => _.Code).ShouldNotContain(ScreenplayDiagnosticCodes.AnalysisUnavailable);
+    [Fact] void should_recover_no_slice() => _analysis.Model.Slices.ShouldBeEmpty();
     [Fact] void should_not_throw_instead() => _analysis.Model.ShouldNotBeNull();
 }

Source/DotNET/Screenplay.Specs/for_ApplicationModelAnalyzer/when_analyzing/source_that_did_not_compile/and_only_some_of_what_it_recovered_carries_an_error.cs

@@ -0,0 +1,44 @@
+// Copyright (c) Cratis. All rights reserved.
+// Licensed under the MIT license. See LICENSE file in the project root for full license information.
+
+using Cratis.Arc.Screenplay.Analysis;
+
+namespace Cratis.Arc.Screenplay.for_ApplicationModelAnalyzer.when_analyzing.source_that_did_not_compile;
+
+/// <summary>
+/// This is the boundary between the two severities. One artifact was read from a declaration an error sits inside
+/// and one was not, and the one that was not is described exactly as its source states it - so "nothing recovered
+/// describes the application reliably" is already false with a single clean declaration. The severity therefore
+/// turns on whether any survived rather than on whether any was hit, and the count says which is which.
+/// </summary>
+public class and_only_some_of_what_it_recovered_carries_an_error : Specification
+{
+    const string Source = """
+        using Cratis.Arc.Commands.ModelBound;
+        using Cratis.Chronicle.Events;
+
+        namespace Library.Authors.Registration;
+
+        [EventType]
+        public record AuthorRegistered(string Name);
+
+        [Command]
+        public record RegisterAuthor(string Name)
+        {
+            public AuthorRegistered Handle() => new(ThisDoesNotExist);
+        }
+        """;
+
+    ApplicationModelAnalysis _analysis;
+
+    void Establish() => _analysis = Analyzed.Source(Source);
+
+    ScreenplayDiagnostic Reported => _analysis.Diagnostics.First(_ => _.Code == ScreenplayDiagnosticCodes.SourceDidNotCompile);
+
+    [Fact] void should_be_analyzing_source_that_really_does_not_compile() => Analyzed.ErrorsIn((Analyzed.SlicePath, Source)).ShouldNotBeEmpty();
+    [Fact] void should_report_it_as_a_warning() => Reported.Severity.ShouldEqual(ScreenplayDiagnosticSeverity.Warning);
+    [Fact] void should_report_nothing_at_all_as_an_error() => _analysis.Diagnostics.Any(_ => _.Severity == ScreenplayDiagnosticSeverity.Error).ShouldBeFalse();
+    [Fact] void should_count_only_the_one_no_error_sits_inside() => Reported.Message.Contains("2 artifact(s) were recovered anyway, 1 of them", StringComparison.Ordinal).ShouldBeTrue();
+    [Fact] void should_quote_the_first_thing_the_compiler_said() => Reported.Message.Contains("ThisDoesNotExist", StringComparison.Ordinal).ShouldBeTrue();
+    [Fact] void should_still_return_whatever_it_recovered() => _analysis.Slice().Commands.Single().Name.ShouldEqual("RegisterAuthor");
+}