Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Project.toml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
name = "SciMLTesting"
uuid = "09d9d899-5365-40a9-917a-5f67fddea283"
authors = ["SciML"]
version = "2.2.0"
version = "2.3.0"

[deps]
Aqua = "4c88cf16-eb10-579e-8560-4a9242c79595"
Expand Down
28 changes: 14 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,8 +50,8 @@ test = ["Test", "SciMLTesting", ...]
| `current_group(; env = "GROUP", default = "All")` | Read the test-group env var, defaulting to `"All"` (empty string also normalizes to the default). |
| `activate_group_env(group_dir; parent, develop, instantiate, develop_sources)` | `Pkg.activate` a per-group `Project.toml`, `develop` the parent package(s) by path, backport `[sources]`, `instantiate`. |
| `develop_sources!(group_dir; parent)` | On Julia < 1.11, `Pkg.develop` the env's `[sources]` path graph (recursively); a no-op on 1.11+. |
| `run_qa(pkg; Aqua, JET, ExplicitImports, aqua, jet, explicit_imports, api_docs, aqua_broken, jet_broken, ei_broken, ...)` | Run the standard Aqua/JET/ExplicitImports QA body, plus the public-API documentation check. Aqua + ExplicitImports come from SciMLTesting's deps (always available; `aqua` on by default, `explicit_imports` opt-in); `using JET` registers JET via its weakdep extension and turns the JET check on; `api_docs` is **on by default** and runs `run_api_docs` (configure via `api_docs_kwargs`, or `api_docs = false` to skip). The `*_broken` kwargs mark known-broken findings as `@test_broken` (see [Known-broken findings](#known-broken-findings-aqua_broken-jet_broken-ei_broken)). |
| `run_api_docs(pkg; docstrings = true, rendered = false, docs_src, ignore, rendered_ignore, docstrings_broken, rendered_broken)` | Assert every exported/`public` name of `pkg` has a docstring (and, opt-in, is rendered in a `@docs` block under `docs/src`). The shared replacement for per-repo `test/QA/public_api_docs.jl` files. |
| `run_qa(pkg; Aqua, JET, ExplicitImports, aqua, jet, explicit_imports, api_docs, aqua_broken, jet_broken, ei_broken, ...)` | Run the standard Aqua/JET/ExplicitImports QA body, plus the public-API documentation check. Aqua and ExplicitImports run by default; `using JET` registers JET via its weakdep extension and turns the JET check on. `api_docs` is **on by default** and runs `run_api_docs`. The `*_broken` kwargs mark known-broken findings as `@test_broken`. |
| `run_api_docs(pkg; docstrings = true, rendered = true, docs_src, ignore, rendered_ignore, docstrings_broken, rendered_broken)` | Assert every exported/`public` name of `pkg` has a docstring and is rendered in a `@docs` block under `docs/src`. The shared replacement for per-repo `test/QA/public_api_docs.jl` files. |
| `public_api_names(pkg)` | The sorted public API of `pkg` (exported names, plus `public` names on Julia ≥ 1.11), with the module's own name dropped. |
| `detect_sublibrary_group(group, lib_dir; default_group = "Core")` | Map a `GROUP` value to a `(sublibrary, test_group)` pair for a monorepo. |

Expand Down Expand Up @@ -185,7 +185,7 @@ and `test/qa/qa.jl`:

```julia
using SciMLTesting, MyPackage
run_qa(MyPackage; explicit_imports = true)
run_qa(MyPackage)
```

Key guarantees:
Expand Down Expand Up @@ -307,7 +307,7 @@ using SciMLTesting, JET, MyPackage
# Aqua + ExplicitImports come from SciMLTesting's deps; `using JET` turns the JET
# check on. The per-repo qa.jl collapses to `explicit_imports = true` plus the
# genuinely-per-repo kwargs (the ExplicitImports per-check ignore-lists).
run_qa(MyPackage; explicit_imports = true,
run_qa(MyPackage;
ei_kwargs = (; all_qualified_accesses_are_public = (; ignore = (:internal_dep_name,))))
```

Expand All @@ -317,31 +317,31 @@ Several SciML repos had grown a hand-copied `test/QA/public_api_docs.jl` asserti
every exported name has a docstring (and is rendered in the manual). `run_api_docs`
replaces those per-repo files with one shared, maintained helper. It runs **by default
inside `run_qa`** (`api_docs = true`), so a plain `run_qa(MyPackage)` already enforces
the docstring check — configure it with `api_docs_kwargs`, or pass `api_docs = false` to
the docstring and rendered-manual checks — configure it with `api_docs_kwargs`, or pass `api_docs = false` to
skip:

```julia
using SciMLTesting, MyPackage

# In the QA body — the docstring check runs by default:
run_qa(MyPackage; explicit_imports = true)
run_qa(MyPackage)

# Also require each public name is rendered in a docs/src @docs block:
run_qa(MyPackage; explicit_imports = true, api_docs_kwargs = (; rendered = true))
# A package without a local manual can explicitly opt out of rendered checks:
run_qa(MyPackage; api_docs_kwargs = (; rendered = false))

# Standalone (outside run_qa), e.g. as its own QA file:
run_api_docs(MyPackage) # every exported/`public` name has a docstring
run_api_docs(MyPackage; rendered = true) # also require each is in a docs/src @docs block
run_api_docs(MyPackage) # every public name has a docstring and is rendered
run_api_docs(MyPackage; rendered = false) # docstrings only
```

* **`docstrings`** (default `true`) — every name in `public_api_names(pkg)` has a
docstring. A re-exported name documented in its defining package counts as
documented (the check follows the binding), so you are not forced to redocument
dependency re-exports.
* **`rendered`** (default `false`, opt-in) — every public name appears in a
* **`rendered`** (default `true`) — every public name appears in a
` ```@docs ` block under `docs_src` (defaults to `<pkgroot>/docs/src`). A
` ```@autodocs ` block satisfies it wholesale. Opt-in because not every repo has a
resolvable local manual (monorepos with shared docs, packages with no manual).
` ```@autodocs ` block satisfies it wholesale. Packages without a resolvable local
manual must explicitly pass `rendered = false`.
* **`ignore` / `rendered_ignore`** — names to exclude (e.g. an un-documentable
re-export), with a comment pointing at the tracking issue.
* **`docstrings_broken` / `rendered_broken`** — mark the check `@test_broken` for a
Expand All @@ -360,7 +360,7 @@ them is exactly the pre-1.6 behavior.

```julia
using SciMLTesting, JET, MyPackage
run_qa(MyPackage; explicit_imports = true,
run_qa(MyPackage;
aqua_broken = (:ambiguities,), # disable + placeholder for a tracked Aqua sub-check
jet_broken = true, # report_package + @test_broken isempty(reports)
ei_broken = (:no_implicit_imports,)) # route this EI check through @test_broken
Expand Down
5 changes: 5 additions & 0 deletions docs/src/api.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# API Reference

```@autodocs
Modules = [SciMLTesting]
```
51 changes: 25 additions & 26 deletions src/SciMLTesting.jl
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,8 @@ single declarative call. The reserved `"All"` group is the curated subset a bare
full suite (including QA and `in_all = false` groups) for agents and long local runs.

The same QA aggregator also owns the public-API *documentation* check: [`run_api_docs`](@ref)
asserts every exported/`public` name has a docstring (and, opt-in, is rendered in the
manual). It runs by default inside [`run_qa`](@ref) (`api_docs = true`), so repos drop
asserts every exported/`public` name has a docstring and is rendered in the manual.
It runs by default inside [`run_qa`](@ref) (`api_docs = true`), so repos drop
their hand-rolled `test/QA/public_api_docs.jl` and get the check for free from a plain
`run_qa(MyPkg)`.

Expand Down Expand Up @@ -609,7 +609,7 @@ end
"""
run_qa(pkg; Aqua = Aqua, JET = ..., ExplicitImports = ExplicitImports,
aqua = Aqua !== nothing, jet = JET !== nothing,
explicit_imports = false, api_docs = true,
explicit_imports = true, api_docs = true,
clean_sources = true,
aqua_kwargs = (;), jet_kwargs = (; target_modules = (pkg,), mode = :typo),
ei_kwargs = (;), api_docs_kwargs = (;),
Expand All @@ -624,7 +624,7 @@ list them as test dependencies. `JET` is the exception: it is compiler-version-p
and so is kept a *weak* dependency, loaded only on demand. `using JET` in your
`test/qa/qa.jl` triggers `SciMLTesting`'s JET extension, which auto-registers the
module; `run_qa` then picks it up. The typical call collapses to just
`run_qa(MyPkg; explicit_imports = true, ei_kwargs = ...)`.
`run_qa(MyPkg; ei_kwargs = ...)`.

Each tool runs if it is both available and enabled:

Expand All @@ -633,8 +633,8 @@ Each tool runs if it is both available and enabled:
* `ExplicitImports` + `explicit_imports` ⇒ ExplicitImports' standard + public-API
checks (see [`run_explicit_imports`](@ref)).
* `api_docs` ⇒ the public-API documentation check (see [`run_api_docs`](@ref)): every
exported/`public` name has a docstring (and, if `api_docs_kwargs` opts in with
`rendered = true`, is rendered in the manual).
exported/`public` name has a docstring and is rendered in the manual unless
`api_docs_kwargs` explicitly sets `rendered = false`.

Enable-flag defaults: `aqua` defaults to `Aqua !== nothing` (on — Aqua is always
available; pass `Aqua = nothing` or `aqua = false` to skip it), and `jet` defaults to
Expand All @@ -644,9 +644,9 @@ QA expectation across the fleet, so it runs for every `run_qa` caller by default
document a repo's public API, or curate exceptions via `api_docs_kwargs` (`ignore`,
`docstrings_broken`), or pass `api_docs = false` to skip it. `api_docs_kwargs` is
forwarded to [`run_api_docs`](@ref) (e.g. `rendered`, `ignore`, `docstrings_broken`).
`explicit_imports` still defaults to **`false`** — its per-repo-curated ignore-lists make
it a deliberate opt-in (`explicit_imports = true`). Setting an enable flag `true` while
its module is unavailable is a configuration error and throws an `ArgumentError`. The
`explicit_imports` defaults to **`true`**. Packages with unavoidable dependency
exceptions provide their per-check ignore-lists through `ei_kwargs`. Setting an enable
flag `true` while its module is unavailable is a configuration error and throws an `ArgumentError`. The
whole thing runs inside a `@testset` named `testset`.

`clean_sources` (default `true`) wraps the `Aqua.test_all` call in
Expand Down Expand Up @@ -712,12 +712,12 @@ pre-1.6 behavior.
# per-repo qa.jl collapses to `explicit_imports = true` plus the genuinely per-repo
# kwargs (the ignore-lists). Aqua + ExplicitImports both run here:
using SciMLTesting, MyPackage
run_qa(MyPackage; explicit_imports = true,
run_qa(MyPackage;
ei_kwargs = (; all_qualified_accesses_are_public = (; ignore = (:internal_dep_name,))))

# Add the JET check by `using JET` (its weakdep extension auto-registers it):
using SciMLTesting, JET, MyPackage
run_qa(MyPackage; explicit_imports = true)
run_qa(MyPackage)

# Aqua only — leave `explicit_imports` at its default and don't load JET:
using SciMLTesting, MyPackage
Expand All @@ -730,7 +730,7 @@ run_qa(MyPackage; Aqua = Aqua, JET = JET, jet = true)
# A repo with a tracked Aqua ambiguities finding, JET known-broken, and one EI check
# known-broken — preserve the suppressions while converting its hand-rolled qa.jl:
using SciMLTesting, JET, MyPackage
run_qa(MyPackage; explicit_imports = true,
run_qa(MyPackage;
aqua_broken = (:ambiguities,), # placeholder: remove when the issue closes
jet_broken = true, # auto-flags Unexpected Pass once JET is clean
ei_broken = (:no_implicit_imports,)) # auto-flags once the check passes
Expand All @@ -743,7 +743,7 @@ function run_qa(
ExplicitImports = ExplicitImports,
aqua::Bool = Aqua !== nothing,
jet::Bool = JET !== nothing,
explicit_imports::Bool = false,
explicit_imports::Bool = true,
api_docs::Bool = true,
clean_sources::Bool = true,
aqua_kwargs = (;),
Expand Down Expand Up @@ -951,13 +951,13 @@ end

# Default docs source dir for a package: <pkgroot>/docs/src, or "" when the package
# root cannot be located (e.g. `pkg` is `Main`). "" is a non-directory, so the
# rendered scan finds nothing and the check fails loudly if it was opted into without
# a resolvable docs tree.
# rendered scan finds nothing and the default check fails loudly without a resolvable
# docs tree.
_default_docs_src(pkg::Module) =
(root = pkgdir(pkg); root === nothing ? "" : joinpath(root, "docs", "src"))

"""
run_api_docs(pkg::Module; docstrings = true, rendered = false,
run_api_docs(pkg::Module; docstrings = true, rendered = true,
docs_src = <pkgroot>/docs/src,
ignore = (), rendered_ignore = (),
docstrings_broken = false, rendered_broken = false,
Expand All @@ -968,19 +968,19 @@ Assert that `pkg`'s public API (see [`public_api_names`](@ref)) is documented.
This is the shared, per-repo-free replacement for the hand-rolled
`test/QA/public_api_docs.jl` files that were copied into individual SciML repos.
[`run_qa`](@ref) runs it by default (`api_docs = true`), so a plain `run_qa(MyPkg)`
already covers the docstring check; call `run_api_docs` directly only to run it outside
`run_qa` or to opt into the `rendered` check.
already covers both checks; call `run_api_docs` directly only to run it outside
`run_qa` or to configure either check independently.

Two checks, each its own nested `@testset`:

* **docstrings** (`docstrings = true`, on by default): every public API name has a
docstring. A re-exported name documented in its defining package counts as
documented (the check follows the binding, not a local docstring), so a repo is
not forced to redocument names it re-exports from a dependency.
* **rendered** (`rendered = false`, opt-in): every public API name appears inside a
* **rendered** (`rendered = true`, on by default): every public API name appears inside a
```` ```@docs ```` block somewhere under `docs_src`, so it is rendered in the
manual. This is opt-in because it needs a resolvable docs tree and does not fit
every repo (monorepos with shared docs, packages with no manual). If any
manual. Packages without a resolvable local manual must explicitly set
`rendered = false`. If any
```` ```@autodocs ```` block is present the check passes wholesale — `@autodocs`
renders whole modules, so anything with a docstring is already rendered.

Expand All @@ -1006,21 +1006,20 @@ version — no per-repo `if VERSION` guards needed.
```julia
# In test/qa/qa.jl — the whole per-repo public-API-docs check:
using SciMLTesting, MyPackage
run_api_docs(MyPackage) # docstrings only
run_api_docs(MyPackage; rendered = true) # also require rendering in docs/src
run_api_docs(MyPackage) # docstrings and rendering in docs/src
run_api_docs(MyPackage; rendered = false) # docstrings only

# It runs by default inside run_qa; use api_docs_kwargs to configure it, or
# api_docs = false to skip it:
run_qa(MyPackage; explicit_imports = true,
api_docs_kwargs = (; rendered = true, ignore = (:reexported_from_dep,)))
run_qa(MyPackage; api_docs_kwargs = (; ignore = (:reexported_from_dep,)))
```

See also [`run_qa`](@ref), whose `api_docs`/`api_docs_kwargs` keywords call this.
"""
function run_api_docs(
pkg::Module;
docstrings::Bool = true,
rendered::Bool = false,
rendered::Bool = true,
docs_src::AbstractString = _default_docs_src(pkg),
ignore = (),
rendered_ignore = (),
Expand Down
Loading
Loading