Add canonical TESTING.md (v1.2.0, coverlet.collector) - #18
Open
CTristan wants to merge 2 commits into
Open
Conversation
Defines the org-wide C# testing standard that every consumer repo mirrors at its root: framework stack pins, test project naming (.Tests plural), class/method naming (Method_Scenario_Expected), 80/80/80 coverage thresholds, Category 1/2 boundary separation with [ExcludeFromCodeCoverage], three-way InternalsVisibleTo classification (library / application / analyzer-or-generator), I/O abstraction via IFileSystem, and AutoFixture guidance. Drift will be enforced by the forthcoming `dotnet ci check-testing-doc` subcommand in OpenLobotomy.Tooling. Version 1.0.0.
The org-wide testing standard moves off coverlet.msbuild build-time instrumentation and onto the coverlet.collector VSTest XPlat Code Coverage data collector. The old path was incompatible with `dotnet test --no-build` because instrumentation hooks only fire during a build step. Changes in this revision: - Drop the coverlet.msbuild row from the framework version table. - Add a runsettings explainer to the Coverage thresholds section, including the canonical coverlet.runsettings shape that selects OpenCover output for the threshold checker. Coordinated PRs: - open-lobotomy/Tooling#40 — switch CiRunner + CoverageThresholdChecker to the collector path. - Per-consumer migration tracked in: Debug-Panel#2, Harmony-2-for-LMM#1, CTristan/lobotomy-corporation-mods#144, CTristan/LobCorp.ConfigurationManager#7, CTristan/the-silent-orchestrator#2.
There was a problem hiding this comment.
Pull request overview
Adds a canonical docs/TESTING.md to the org-level .github repository to standardize testing, naming, coverage, and IVT policies across Open Lobotomy repos, including updated guidance for collecting coverage via coverlet.collector.
Changes:
- Introduces a shared testing standards document with pinned test/coverage package versions.
- Documents org-wide conventions for test naming, coverage thresholds, and boundary separation (Category 1/2).
- Clarifies IVT policy by project contract type (library vs application vs analyzer/generator) and documents coverage collection via VSTest data collector +
coverlet.runsettings.
Comment on lines
+9
to
+11
| This document defines testing conventions for all Open Lobotomy repositories. It is the canonical source — consumer repos receive a synced copy at `docs/org/TESTING.md` via the `sync-org-docs` workflow in `open-lobotomy/.github`. Do not edit the synced copy directly; changes made here flow out to every repo tagged with the `dotnet-standards` topic. | ||
|
|
||
| When updating this file: bump `Version` in the header, update `Last updated`, and merge to `main`. The sync workflow will open a PR in every tagged consumer repo automatically. |
| | AutoFixture.Xunit3 | 4.19.0 | | ||
| | coverlet.collector | 8.0.0 | | ||
|
|
||
| When a version bump is needed, update `open-lobotomy-tooling`'s `Directory.Packages.props` first, verify its tests pass, then bump this table (minor version) and propagate. |
|
|
||
| --- | ||
|
|
||
| *Version 1.0.0. Changes are proposed via PR to `open-lobotomy/.github`; approved changes bump the version in the header and trigger downstream sync via `dotnet ci sync-testing-doc`.* |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First landing of the canonical
docs/TESTING.mdfor the Open Lobotomy org, plus an in-flight v1.2.0 update that migrates coverage fromcoverlet.msbuildtocoverlet.collector. Two commits stacked on this branch:coverlet.msbuildrow from the framework table, adds a runsettings explainer.The two commits are kept distinct so the v1.2.0 change is reviewable in isolation, but the squash-merge will land them as a single canonical file in
main.Why v1.2.0
The original v1.1.0 row pinned both
coverlet.collectorandcoverlet.msbuildbecause the toolchain used the msbuild build-time instrumentation path. That path is incompatible withdotnet test --no-build: instrumentation hooks only fire during a build step, so--no-buildruns silently produce empty coverage reports. This surfaced onLobotomyCorporation.Mods.ConfigurationManager.IntegrationPR #1 — fix in flight.Coordinated PRs
CiRunnerandCoverageThresholdCheckerto the collector path. Must merge before consumers can drop their msbuild references via the nextOpenLobotomy.Standardsprerelease.open-lobotomy/Debug-Panel#2,open-lobotomy/Harmony-2-for-LMM#1,CTristan/lobotomy-corporation-mods#144,CTristan/LobCorp.ConfigurationManager#7,CTristan/the-silent-orchestrator#2.Test plan
Last updatedfield updatedsync-org-docsworkflow opens PRs in everydotnet-standards-tagged consumer to refreshdocs/org/TESTING.md