development.md

Development guide

This guide is for contributors. User-facing installation and command examples live in the project README.

Documentation and local work records

Keep docs/ focused on maintained user guides, architecture decisions and reusable development procedures. Update the authoritative page in place; do not add per-session review replies, commit-by-commit acceptance histories, machine-specific timings, raw logs or temporary measurement JSON here.

Keep those work records in an untracked local review directory or the existing ignored target/ output directories. Report verification results in the change discussion and CI; retain release evidence with the release workflow artifacts. Extract enduring design constraints into the appropriate guide. Regression fixtures, protocol snapshots and intentional audit ledgers remain versioned at their established locations: they are executable contracts, not work logs.

For performance comparisons, retain already-built baseline binaries, use fixed source identities, warm both versions, and alternate runs without concurrent builds or tests. Measure native loading separately from rendering/serialization; report timing and peak memory together. Keep raw samples outside docs/ and do not rebuild historical trees in temporary directories on a space-constrained host. Use the repository's existing build directory and leave isolated package compilation to CI when local constraints prohibit it.

For document geometry work, the local layout probe separates native parsing, index construction, immutable view construction and resize at 40/80/120 cells:

cargo run --locked --release -p mant-ui --example layout_profile -- /path/to/manual.1.gz 'label to locate'

The same probe accepts an explicit .md path and reports outline references none/summary/all and independent explanation time between loading and TUI construction. Record occurrence/target precision, charged steps/bytes and retained records alongside elapsed time; a fast truncated scan is not a full inventory. For process peak memory, run the already-built example under the platform's resource measurement tool. Do not compare debug and release timings or attribute baseline IR allocation to the new reference index.

The probe also times application construction with the current document already present in its caller-owned scope. Append repeatable --scope=PATH arguments to include additional documents; ordinary token arguments still locate text in the primary document, and -- makes all remaining arguments literal tokens. For example:

cargo run --locked --release -p mant-ui --example layout_profile -- /path/to/gcc.1.gz --scope=/path/to/git.1.gz --scope=docs/manuals/mant.md '--help'

The current document is moved into the scope without a preparation-time clone; additional loading is reported separately as scope_prepare_ms, before app_current_in_scope_ms starts. Both report the complete scope document count. Application timing includes its internally constructed view and ownership/copy costs, but excludes source parsing, caller scope preparation and App teardown. It is not a pure clone measurement and should not be subtracted from the independent view timing to infer exact copy costs. Process peak RSS covers the whole probe, including earlier queries and all preloaded scope documents, not only application construction. Record every scope input hash alongside the primary input when comparing runs.

Add --shared to measure app_shared_scope_ms using caller-owned Arc snapshots instead of the convenience API's owned-document copying. Snapshot preparation occurs before application timing, and each invocation constructs only one application. Compare separate, alternating runs with identical scope inputs; whole-process RSS still includes all earlier loading and queries.

It reports source-token columns when a token fits on one row; a wrapped/absent token is not a failed search. Reuse one DocumentView for width changes, record the input hash and producer commit, and compare source-relative columns after removing page/UI margins. Deterministic source probes in the engine and UI tests pin rules independently of the installed host corpus. Native layout research uses the explicitly recorded mandoc CVS revisions and a groff cross-check; this does not upgrade the vendored 1.14.6 parser or import upstream test licenses.

Prerequisites

• Linux with glibc, macOS, or Windows
• Rust 1.88 or newer with cargo, clippy, and rustfmt
• For native-manual work: GCC on Linux, Clang on macOS, or MSVC on Windows; Unix also needs zlib development headers

The workspace vendors libmandoc and maintains its own manual index, so system man and mandoc executables are not prerequisites. Markdown parsing and the deterministic fixture suite do not require installed manual sources either.

Start from a fresh clone

Build and run the unified command directly through Cargo:

cargo run --locked -p mant -- git
cargo run --locked -p mant -- --input README.md

Run the complete local verification boundary before handing off a change:

bash scripts/check.sh

On Windows, run the native product boundary from PowerShell:

.\scripts\check-windows.ps1

It tests libmandoc-rs, mant-ir, mant-protocol, mant-sources, mant-codec, mant-loader, mant-query, mant-render, mant-engine, mant-ui, and mant, including the shared roff fixture suites.

The product crates are workspace default-members, so a bare cargo build, cargo test, or cargo clippy works on Windows. Both platform verification scripts include the standalone libmandoc-rs package and native parser tests. They also run that package with --all-features, as does native macOS CI, so its default-off reference renderers and Serde contract execute on every supported target without enabling unrelated workspace maintenance features.

The default mant-codec feature set is independently checked by scripts/check-codec-consumer.sh. Its separate consumer workspace exercises the public Markdown/tldr/encoding APIs and rejects enabled native, protocol, engine or frontend dependencies in its normal/build graph. This is distinct from workspace tests, where the product enables the codec's roff feature. Both packaged codec feature surfaces are tested as well. All consumer build products use the repository target/ directory, not a temporary build directory.

scripts/check-loader-consumer.sh independently exercises read-only Markdown file loading with mant-loader's default features. Its normal/build dependency graph excludes native parsers, compression, source acquisition, engine queries, and frontends. The default and roff loader surfaces also run as separate test and packaged-source checks; workspace feature unification is not evidence for the default-only boundary.

scripts/check-query-consumer.sh builds an independent consumer that supplies its own semantic IR and graph. It exercises single-document queries and borrowed collection queries, and rejects enabled loader, renderer, frontend, native and update dependencies. This guards against accidentally restoring host authority through a query convenience API.

mant-render is tested independently and in the packaged-source workspace. scripts/check-render-consumer.sh exercises authored IR/DTO rendering from an independent consumer and rejects query, loader, native and terminal dependencies. It renders caller-supplied IR and protocol results without executing queries or loading sources. Its codec dependency is Markdown-only; workspace feature unification must not be mistaken for a native dependency of the renderer. UI integration tests separately assert real terminal cells, because unchanged concatenated text alone cannot prove correct grapheme rendering or hit maps.

scripts/check-ui-consumer.sh embeds the reader with authored IR and explicit host services. Its independent normal/build graph rejects loaders, queries, native parsers, source updates and the static pager. Crossterm's own transitive event dependencies do not grant this component terminal acquisition authority.

scripts/check-cli-features.py runs eight isolated product builds: no default features, each of roff, tui, pager, mcp, and update alone, the default product, and all features. It inspects normal/build dependency graphs, exercises the actual help and structured input/output boundary, and checks that schemas remain complete even when execution capabilities are absent. Native catalog metadata remains discoverable without a native parser; reading native content must then report the unavailable capability rather than a missing document. The script runs in Linux, macOS and Windows verification, using Python 3's standard library (python3 on Unix, python on Windows). It does not replace MCP session tests or real terminal recovery tests. Build products and disposable probe inputs stay under repository target/; user data paths are isolated for process probes. Do not run this matrix concurrently with another Cargo gate. The packaged-source gate also runs cargo test -p mant --no-default-features --lib, so the minimal surface cannot depend on files omitted from its published archive.

The script checks formatting and installer syntax, runs every workspace test, runs clippy with all targets and features, builds the optimized executable, and smoke-tests its human and JSON surfaces. The result is target/release/mant.

Unix process tests also use Python 3's standard-library PTY support to verify display selection, wrapped overflow, short-output pass-through, and terminal restoration after paging or TUI exit, including real CLI SIGINT/SIGTERM exits. The executable host's private terminal harness also triggers a Rust host-callback panic and a broken-output initialization error after raw-mode acquisition, checking exact termios restoration and alternate-screen cleanup. These are disposable POSIX children; they do not imply recovery from SIGKILL or machine failure. Platform-neutral policy matrices and redirected-process tests run on Windows as well; PTY checks do not substitute for Windows Terminal interactive testing.

CLI help embeds generated TLDR examples from the leading preface in docs/manuals/mant.md. After editing that preface, regenerate them with:

cargo run --locked -p mant --example generate_help_tldr

Commit the generated crates/mant/src/arguments/help_tldr_generated.rs with the manual change. The normal mant integration tests compare it with the production TLDR parser's output, so Unix and Windows verification catch drift. The generated constants are included in the published crate; installed users need neither the repository manual nor a generator to build or display help.

The repository's CLI manual-coverage test also compares clap's public option metadata (including generated help/version flags and visible aliases) with the self-manual's semantic entries. Every spelling must identify one actual case-sensitive option definition, and clap aliases must belong to its explicitly declared alias group. Other explanation evidence is allowed; the complete response is not required to contain only that owner. Obsolete options must not remain in the manual. This is a checkout-only corpus check because published crates omit repository manuals; the coverage checker's synthetic regressions still run against packaged sources. Add or change the authoritative entry whenever the public CLI changes, rather than extending a separate option allowlist.

For semantic-entry changes, follow the content and explanation design constraints. Preserve independent expected names and source content; neither more entries nor more successful explain calls proves correctness. Annotation-only Markdown work needs original-event comparisons of visible punctuation, lists, paragraphs, links and hard breaks, plus IR binding/serde tests. These are acceptance requirements enforced by the ordinary-owner and metadata regressions, not a reason to remove comments and reparse a differently structured document. Keep unique-navigation assertions on select_excerpt; explain has an independent name/form/content/relationship oracle, bounded pagination, no-evidence and partial-source tests. Semantic profilers count selectable names separately from explicit alias groups/relations. Zero diagnostics or candidate counts alone do not establish complete recall.

CI uses --build-profile debug for the final smoke test because its test and Clippy steps have already populated that profile. Local checks keep release as the default, and tagged publication performs a separate optimized build. When an exact commit has already completed every full CI job on dev, a fast-forward push of that commit to main verifies and reuses the recorded check suite instead of executing it twice. Direct pushes to main, pull requests, and manual runs still execute the full suite. macOS adds native compile and test coverage without repeating the Linux lint pass. Windows retains its full native verification boundary for Windows-only path, shell, packaging, and parser behavior.

After pushing dev, monitor the run for that exact commit with gh run watch. CI monitoring ends only when the full aggregate gate is green, not when a failure has merely been reported. Inspect failed job logs, fix and test the cause, add focused follow-up commits, push, and follow the new run. Preserve already-pushed history; do not rewrite commits to hide failed iterations.

Project verification and release builds set LIBMANDOC_RS_DENY_WARNINGS=1, which promotes warnings from libmandoc's C build to errors on GCC, Apple Clang, and MSVC. Pinned upstream 1.14.6 retains a documented MSVC baseline for C4100, C4146, C4200, C4244, and C4267. C4200 is confined to four C99 flexible-array members that MSVC diagnoses as an extension even in C11 mode. ManT-owned C shim and compatibility sources do not inherit that baseline. The crate leaves the policy disabled for ordinary downstream builds so a new compiler warning cannot break an otherwise compatible published dependency; maintainers can opt into the same strict boundary for a focused command by setting the variable explicitly.

Each native job caches downloaded crates and compiled third-party dependencies using a key derived from its Rust compiler, Cargo manifests, lockfiles, and compiler environment. Workspace crates and incremental artifacts are excluded to keep restore time and repository cache use bounded. Pull requests may restore an existing cache but only branch pushes persist new entries.

Focused commands are useful while iterating:

cargo fmt --all --check
cargo test --locked --workspace
cargo test --locked -p mant-ui
cargo clippy --locked --workspace --all-targets --all-features -- -D warnings
cargo build --locked --release -p mant

Dependency policy is declared in deny.toml. CI runs cargo-deny across all features and every supported target family to reject known vulnerabilities, yanked packages, unapproved licenses, wildcard requirements, and dependencies from untrusted registries or Git repositories. With cargo-deny 0.20.2 installed, run the same audit locally:

cargo deny check

Native distribution notices are generated from the same locked, multi-target graph with cargo-about 0.9.2. After changing dependencies, regenerate the checked-in report and review its package and license mapping:

scripts/generate-rust-licenses.sh
scripts/generate-rust-licenses.sh --check

The first command refreshes the checked-in report; --check regenerates it in a temporary file and fails without modifying the worktree when the report is stale. CI uses the same check mode.

Pull requests also use GitHub dependency review. Dependabot supplies weekly version updates, while security updates, secret scanning with push protection, and private vulnerability reporting are repository settings rather than files in the checkout. OpenSSF Scorecard publishes a weekly, externally visible assessment and uploads its findings to GitHub code scanning.

Repository map

Cargo.toml                    Root Rust workspace and shared dependency policy
CHANGELOG.md                  Independent crate compatibility and migration history
deny.toml                     Dependency license, advisory, source, and ban policy
about.toml                    Distributable Rust dependency notice policy
LICENSE                       Apache-2.0 terms for ManT-authored work
THIRD_PARTY_NOTICES.md        Repository-wide third-party distribution map
THIRD_PARTY_LICENSES.html     Generated Rust dependency license report
SECURITY.md                   Supported versions and private reporting policy
crates/mant-ir/               Semantic IR, ResolvedContent, paths, visitors, validation, indexes
crates/mant-protocol/         Versioned request/response DTOs and JSON Schema
crates/mant-sources/          Local Markdown registry and transactional source updates
crates/mant-codec/            In-memory Markdown/tldr, optional roff lowering, document Markdown
crates/mant-loader/           Read-only catalogs, native roots, bounded inputs and linked scopes
crates/mant-query/            Pure semantic queries over existing content and borrowed collections
crates/mant-render/           Pure IR/DTO text and report formatting, roles and grapheme primitives
crates/mant-engine/           Request validation and loader/query composition
crates/mant-ui/               Ratatui reader, navigation, search, and terminal styling
crates/mant/                  CLI/MCP, terminal delivery and optional source/tldr maintenance
crates/libmandoc-rs/          Owned libmandoc parse/render API, private C shim, vendored source
fuzz/                        Standalone cargo-fuzz workspace
tests/contracts/             Stable JSON contract fixtures consumed by Rust tests
tests/fixtures/              Fixed Markdown and real roff integration sources
tests/consumers/codec-markdown/  Independent default-codec consumer and feature resolver
scripts/check.sh             Canonical local and CI verification sequence
scripts/check-windows.ps1    Native Windows verification sequence
scripts/check-libmandoc-symbols.sh  Reject downstream-visible unprefixed C symbols
scripts/check-packaged-crates.sh  Build and test exact published crate source sets
scripts/check-codec-consumer.sh  Verify the standalone codec's pure-Rust dependency boundary
scripts/check-loader-consumer.sh  Verify read-only, default-feature source loading
scripts/check-query-consumer.sh  Verify pure queries over authored IR and borrowed scopes
scripts/check-render-consumer.sh  Verify pure IR/DTO reports without query or host authority
scripts/check-ui-consumer.sh  Verify reader embedding without loaders or terminal delivery
scripts/check-cli-features.py  Eight isolated capability builds and process contract probes
scripts/build-and-smoke.sh   Unix debug/release product build and smoke test
scripts/build-and-smoke.ps1 Windows debug/release product build and smoke test
scripts/find-successful-ci.sh  Exact-commit full CI verification for automation
scripts/generate-rust-licenses.sh  Rebuild the locked Rust license report
scripts/finalize-cyclonedx.mjs  Normalize generated release SBOMs reproducibly
scripts/install-ci-native-dependencies.sh  Install native Linux CI prerequisites
scripts/install.sh           Latest-release installer for Linux and macOS
scripts/install.ps1          Latest-release installer for Windows x64
scripts/fuzz.sh              Run selected cargo-fuzz targets for a bounded time
scripts/package-release.sh   Reproducible Linux release archive assembly
scripts/package-release.ps1 Windows x64 ZIP assembly
scripts/package-manuals.sh   Reproducible platform-independent manual archive
scripts/publish-crates.sh    Ordered independent-version crates.io publication
scripts/update-protocol-schema-snapshot.sh  Regenerate a deliberate protocol snapshot
scripts/update-reader-screenshot.sh  Host-stable Linux README screenshot capture
scripts/audit-roff-fidelity.py  Visible-content differential audit
scripts/roff_reference.py      Bounded POSIX reference-renderer execution
scripts/audit-roff-structure.py  Native AST-to-IR topology audit
scripts/audit-roff-projection.py  CommonMark round-trip topology audit
scripts/audit-roff-layout.py  Source-gated renderer layout audit
scripts/audit-roff-targets.py  Native zero-width target-conservation audit
scripts/audit-roff-semantics.py  Semantic-entry precision audit
scripts/roff_query_gold.py      Source-bound explanation gold comparisons
scripts/check-roff-audit-coverage.py  Cross-ledger corpus coverage verification
scripts/roff_audit_common.py  Shared roff audit identities and helpers
crates/mant-engine/examples/support/  Shared profiler framing, not semantic oracles
docs/architecture/           Design decisions and stable-boundary documentation
docs/installation.md         User installation methods and platform requirements
docs/sources.md              Markdown source configuration and update behavior
docs/manuals/mant.md          User command, discovery, TUI, and MCP overview
docs/manuals/mant-ir.md       In-process normalized model reference
docs/manuals/mant-protocol.md Process contracts and compact MCP interface reference
docs/manuals/mant-markdown.md Supported Markdown and semantic extensions
docs/manuals/mant-roff.md     Native manual compatibility and lowering levels
docs/releasing.md            Maintainer release procedure
docs/manuals/manifest.txt     Exact self-hosted manual set shipped in releases
docs/assets/                 README screenshots and documentation assets

Documentation ownership

Each topic has one authoritative home so examples and compatibility promises do not drift between overview pages:

Question	Authoritative document
What is ManT and how do I install it?	Root README.md and docs/installation.md
Which command, selector, or key should I use?	mant(1) in docs/manuals/mant.md
Which Markdown or roff constructs are retained?	mant-markdown(7) and mant-roff(7)
What does an in-process node mean?	mant-ir(7) and the mant-ir rustdoc
Which identities and projections cross host, JSON, or MCP boundaries?	mant-protocol(5); generated schemas cover the structured JSON boundaries
How are document collections configured?	docs/sources.md
Which crate owns a behavior?	docs/architecture/native-engine.md and that crate's README/rustdoc
How is a release produced and attested?	docs/releasing.md

Crate READMEs are included directly as crate-level rustdoc. Keep Rust examples valid doctests and use repository-absolute HTTPS links for material that is not packaged with the crate. User manuals remain ordinary Markdown so ManT can parse and ship them as its own documentation library.

Generated paths are excluded from version control:

• target/ and fuzz/target/ — Cargo build output
• dist/ — locally assembled release archives

Testing boundaries

Rust is authoritative for parser correctness, IR and protocol contracts, semantic option extraction, terminal presentation, process behavior, and output rendering. Fixed real roff sources in tests/fixtures/roff/real/ are covered by native integration tests; their provenance and licenses are documented in that directory.

The downstream native-symbol namespace audit uses GNU nm --defined-only and runs in Linux CI. macOS and Windows compile the same generated prefix map and retain strict native warning, link, parser, and renderer checks, but those jobs do not provide an equivalent exported-symbol inventory. The mixed-language TSAN and ASan commands below are deliberate maintainer/release checks rather than per-push gates; ordinary tests cannot substitute for their race and memory instrumentation.

The file docs/manuals/mant.md is executable documentation. Tests parse it through the supported Markdown subset, require its embedded quick reference and semantic entries, and reject lossy fallback diagnostics.

Fuzzing

The standalone fuzz/ workspace keeps fuzz-only dependencies out of the shipped crates. CI compiles every target on stable Rust; randomized execution is a local or scheduled activity because a short nondeterministic CI run is not a reliable security boundary.

The maintained targets follow externally supplied data rather than crate boundaries:

• markdown_parse is the high-throughput Markdown parser target.
• markdown_pipeline exercises Markdown parsing, semantic selectors, search, outlines, excerpts, and every output renderer.
• tldr_page covers the TLDR subset and command-token parser independently.
• roff_pipeline crosses the native libmandoc boundary, exercises its bounded ASCII/UTF-8/HTML reference renderers, and then exercises the same semantic projections and renderers as Markdown.
• catalog_query covers bounded literal/regex discovery, hierarchical paths, relevance ordering, filters, and pagination without touching the host file system.

Curated seeds under fuzz/corpus/ reach semantic comments, links, TLDR placeholders, man and mdoc macros, and hierarchical catalog names quickly. All targets cap individual inputs at 64 KiB so time is spent exploring syntax rather than repeatedly rendering oversized documents. File discovery, archive transactions, network downloads, and terminal event handling remain in deterministic unit and integration tests because byte mutation cannot model their state transitions faithfully.

Install cargo-fuzz and a nightly toolchain, then run every target serially. The first argument is the number of seconds per target; additional arguments select individual targets:

cargo install cargo-fuzz --locked
rustup toolchain install nightly --profile minimal
scripts/fuzz.sh 60
scripts/fuzz.sh 300 roff_pipeline markdown_pipeline

Minimize any artifact with cargo fuzz tmin, turn the minimized input into a named regression test, and only then discard or archive the generated corpus.

On Linux, regenerate the README reader image with:

scripts/update-reader-screenshot.sh

The script builds the release executable, registers the repository's ManT manual in an isolated XDG hierarchy, opens it in a fixed Xvfb/xterm surface, activates View → Expand All, and captures the result. It requires Xvfb, xterm, xdotool, Fontconfig, and ImageMagick; the pinned JetBrains Mono files and their OFL-1.1 license live under docs/assets/fonts/. The script pins its font, geometry, terminal settings, and interaction sequence. The rendering tools come from the host, however, so byte-identical captures are expected only with the same host toolchain. Always inspect the resulting image before committing it.

libmandoc-rs also has a self-contained package boundary: its parser, compression, include-policy, virtual source, diagnostics, optional serde, and optional reference-renderer tests must pass from Cargo's staged package directory without requiring fixtures from sibling crates.

Source installation activation has deterministic fault-injection tests over every rename/parent-sync step, including failed rollback and recovery retries. They use the real transaction ordering and on-disk old/new trees. A sync failure after activation can leave the complete new tree installed; it is not a promise of rollback or durability after power loss. These tests do not simulate storage hardware failures or every document-writing/metadata-preparation failure. Its heavier concurrency check stays outside routine CI and must instrument both languages across the FFI boundary:

rustup toolchain install nightly --profile minimal
rustup component add rust-src --toolchain nightly
crates/libmandoc-rs/scripts/check-thread-safety
crates/libmandoc-rs/scripts/check-address-safety

The runner recompiles std with TSAN, explicitly instruments the vendored C objects, uses an isolated target directory, and exercises in-memory parsing, source-relative .so, virtual bundles, and all reference renderers. Use --rounds N for a longer local soak. The ASan companion checks exact caller-owned input tails, owned-AST access after releasing the native parser, output-limit boundaries, and isolated deep-tree parse/render rejection plus error cleanup in both languages. The runners support x86_64 and aarch64 Linux/glibc and macOS; Windows retains ordinary concurrent and boundary regressions in CI but is outside these local sanitizer runners. The runners, patch series, and upstream checksum are repository maintenance inputs and are intentionally absent from the published crate, which ships only the resulting buildable vendor tree.

For ownership-transfer performance work, run the dependency-free generated benchmark before and after the change on the same host and release profile:

cargo bench --locked --package libmandoc-rs --bench ast_transfer

It reports complete parse-to-owned-AST latency, input size, node count, and estimated owned Rust payload for small, medium, and large generated manuals. Treat its timing as comparative local evidence rather than a stable CI threshold; sanitizer and semantic regression tests remain the correctness gates.

When changing a versioned IR projection or protocol type, update the Rust contract, generated-schema, process, and projection tests in the same change. External stdio remains a closed boundary: unknown request fields and incompatible response shapes fail before application code receives them. The in-process UI consumes mant_ir::ResolvedContent directly and never serializes it first.

Native fixtures

For independent behavior matrices, write minimal inputs and expected results from the documented semantic contract in the existing Rust tests. Assert aliases/forms, source ownership and exact line boundaries as appropriate, not only global token presence. Reuse existing licensed real fixtures; do not copy an external formatter's implementation, test harness or sample text into a new regression merely by changing names. Reference rendering stays an explicitly enabled local check, with output and environment details kept in its review bundle rather than imported as unquestioned golden output. This does not remove any existing attribution or license obligations.

Do not replace a real distribution fixture with a hand-written approximation when fixing parser or lowering behavior. Add the smallest redistributable real source that reproduces the problem, record its origin and license under tests/fixtures/roff/real/, and assert the normalized structure rather than a terminal screenshot. Renderer tests then verify the same IR independently.

Roff fidelity audit

The existing real-fixture catalogue also feeds an optional differential audit against the host man(1) and groff renderer:

cargo build --package mant
cargo build --package libmandoc-rs --example roff_ast_profile
python3 scripts/audit-roff-fidelity.py --fixtures --json /tmp/mant-fidelity.json
python3 scripts/audit-roff-fidelity.py --manpath /usr/share/man --max-pages 100
python3 scripts/audit-roff-fidelity.py --manpath /usr/share/man --max-pages-per-section 25 --findings-only
python3 scripts/audit-roff-fidelity.py --manpath /usr/share/man \
  --source-pattern '^[.]Dd' --recheck-recorded --findings-only
python3 scripts/audit-roff-fidelity.py --manpath /usr/share/man \
  --max-pages-per-section 25 --syntax-priority \
  --syntax-cache /tmp/mant-roff-syntax.json.gz \
  --syntax-report /tmp/mant-roff-syntax-report.json --findings-only
python3 scripts/audit-roff-fidelity.py --manpath /usr/share/man \
  --max-pages-per-section 12 --syntax-priority \
  --review-dir /tmp/mant-roff-review
python3 scripts/audit-roff-fidelity.py --manpath /tmp/debian-man \
  --max-pages 200 --syntax-priority --dedupe-across-corpora \
  --syntax-cache /tmp/mant-debian-syntax.json.gz \
  --audit-db tests/fixtures/roff/FIDELITY_AUDIT.csv \
  --corpus debian-sid-amd64 --findings-only
python3 scripts/audit-roff-fidelity.py --manpath /usr/share/man --max-pages-per-section 25 \
  --audit-db tests/fixtures/roff/FIDELITY_AUDIT.csv --corpus archlinux-host
python3 scripts/audit-roff-fidelity.py --manpath /usr/share/man --recorded-only \
  --audit-db tests/fixtures/roff/FIDELITY_AUDIT.csv --corpus archlinux-host
python3 scripts/audit-roff-fidelity.py --manpath /usr/share/man --retry-skipped \
  --audit-db tests/fixtures/roff/FIDELITY_AUDIT.csv --corpus archlinux-host
python3 scripts/audit-roff-fidelity.py --manpath /usr/share/man --pending-only \
  --audit-db tests/fixtures/roff/FIDELITY_AUDIT.csv --corpus archlinux-host

The audit compares normalized visible tokens and contiguous token phrases, plus narrowly source-conditioned whitespace and punctuation that carry mdoc semantics. It deliberately ignores general line wrapping, indentation, blank lines, typography, headers, footers, and ManT-specific visible link targets. A REVIEW result is a candidate for human inspection rather than a test failure; reference renderers and ManT intentionally differ in several presentation details. Stable path sampling uses the path and --seed, so repeating a bounded local scan selects the same pages. Use --max-pages-per-section when broad coverage matters more than matching the host corpus distribution, repeat --man-section to focus on exact suffixes such as 1, 3ssl, or 8, and repeat --source-pattern to require multiple multiline source shapes before sampling. Source-directed sweeps are the appropriate complement to random and AST-priority sampling when a rare macro family needs complete review.

Roff structure audit

Content comparison intentionally normalizes away line wrapping and layout, so it cannot detect every AST-to-IR topology error. The companion audit owns a narrower internal oracle: the same libmandoc AST that ManT lowers. It compares source-aware IR against observable no-fill lines, list and definition items, table rows and spans, relative indentation, explicit breaks, and typed links; terminal geometry remains out of scope.

cargo build --package mant-engine --example roff_structure_profile
python3 scripts/audit-roff-structure.py --fixtures --json /tmp/mant-structure.json
python3 scripts/audit-roff-structure.py --manpath /usr/share/man \
  --corpus archlinux-host --replay-fidelity-records --findings-only
python3 scripts/audit-roff-structure.py --manpath /usr/share/man \
  --source-pattern '^[.]nf$' --recheck-recorded --findings-only

STRUCTURE_AUDIT.csv is an independent incremental ledger, while its guide defines the review lifecycle. The batch profilers and audit scripts are development tools: no host corpus, renderer, or batch profile is part of normal CI. CI runs only their dependency-free self-checks and the focused Rust regressions that follow a confirmed finding.

The separate CommonMark projection oracle catches valid-but-wrong Markdown topology after lowering has already succeeded:

cargo build --package mant-engine --example roff_projection_profile
python3 scripts/audit-roff-projection.py --fixtures \
  --json /tmp/mant-projection.json
python3 scripts/audit-roff-projection.py --fixtures --recheck-recorded \
  --verify --findings-only
python3 scripts/audit-roff-projection.py --manpath /usr/share/man \
  --corpus archlinux-host --replay-fidelity-records --findings-only

It reparses full renders and a deterministic first/middle/last section sample, then compares section trees, list nesting, fenced-block nesting, and literal entity spellings that CommonMark would otherwise decode. Its independent PROJECTION_AUDIT.csv and guide prevent a projection finding from being mistaken for an AST-to-IR lowering failure. The --verify form is read-only and gates every checked-in roff fixture in the Unix CI boundary; local host corpora remain incremental and non-gating.

Roff renderer-layout audit

The optional renderer-layout audit is separate from both the content and AST-to-IR ledgers. It uses the same local man(1)/groff reference rendering as the fidelity auditor, but only compares source-gated line boundaries, spacing, and authored relative indentation within the documented lowering policy; formatter-owned display gutters and body margins are out of scope. In particular, RS nesting currently uses a normalized four-column increment rather than reproducing its numeric distance. Differences caused solely by that policy need an explained review outcome, not a product-fidelity claim; the structure audit checks retained nesting or semantic-container ownership. Exact HP hanging geometry is likewise outside the current contract. It does not re-run, modify, or invalidate completed FIDELITY_AUDIT.csv or STRUCTURE_AUDIT.csv rows.

cargo build --package mant
python3 scripts/audit-roff-layout.py --manpath /tmp/new-release/share/man \
  --corpus new-release-amd64 --max-pages-per-section 20 \
  --json /tmp/mant-layout.json --findings-only

For a new layout pass over the exact bytes already completed by the existing content audit, use --replay-fidelity-records with that corpus. This reads FIDELITY_AUDIT.csv only as a source-identity index, accepting its comparable clean and review rows but not historical skipped or hard-failure records. The prior audit results remain immutable while the new observations enter LAYOUT_AUDIT.csv.

LAYOUT_AUDIT.csv is evidence for newly selected renderer-layout sweeps, not a claim that older content/structure samples were retroactively checked. It is independent of both older ledgers: a completed layout row never rewrites or upgrades their historical conclusions. Its guide defines the narrow signal and review lifecycle. Do not add it to daily CI; only its self-check and focused regressions derived from confirmed findings belong there.

Roff target-conservation audit

Visible text and renderer comparisons cannot detect a zero-width destination that disappears while all surrounding words remain intact. The independent target audit therefore lowers the same owned libmandoc parse used as its oracle, classifies every deep-link owner as retained, deliberately excluded, or unclassified, and compares each retained owner occurrence with a compatible canonical IR identity or exact authored fragment alias. The comparison keeps AST child paths, source lines, owner macro/kind, section, required IR role, and container class separate. Lowered anchors also retain their addressable owner source span, and the profiler records their concrete block/item/definition/cell path. A target moved onto a neighbouring owner therefore cannot satisfy the original obligation merely because its name, role, section, and broad container still agree:

cargo build --package mant-engine --example roff_target_profile
python3 scripts/audit-roff-targets.py --fixtures --recheck-recorded \
  --verify --findings-only
python3 scripts/audit-roff-targets.py --manpath /usr/share/man \
  --corpus archlinux-host --recheck-recorded --findings-only \
  --json /tmp/mant-roff-targets.json

The fixture form is part of the Unix verification boundary. Distribution trees remain local release-time evidence and are recorded incrementally in TARGET_AUDIT.csv. Its guide defines the oracle, review lifecycle, and latest complete sweep. Automatic section tags are not compared as literal strings because ManT deliberately derives a section ID from the complete visible title, while an explicit Tg destination remains exact. Unknown owner classes, invalid or empty target identities/fragments, genuine unexpected targets, role collisions, duplicates, and dangling links are independent findings. Every candidate or native-parser failure requires manual review; confirmed loss becomes a licensed real fixture and a focused Rust assertion.

Roff semantic-entry precision audit

Target conservation cannot detect lowering that preserves every destination but incorrectly promotes numbered prose or a target-only placement artifact to a semantic entry. The independent semantic profile inventories the final SemanticIndex, its source context and value-domain origins, and scans the IR for high-confidence classification errors. It also inventories punctuated ordinal candidates directly from the owned mdoc AST, retains each authored definition-list subtype and complete term sequence, and requires the final IR block at that source line to have the intended retained or recovered shape. This independent source-side ledger detects over-conversion after the original definition term has already disappeared from the IR:

cargo build --package mant-engine --example roff_semantic_profile
python3 scripts/audit-roff-semantics.py --fixtures --recheck-recorded \
  --verify --findings-only
python3 scripts/audit-roff-semantics.py --manpath /usr/share/man \
  --corpus archlinux-host --recheck-recorded --findings-only \
  --json /tmp/mant-roff-semantics.json

Punctuated integer definitions left outside an ordered list, corresponding term or value entries, entries with neither an alias nor a visible form, and non-value children inside a Choices domain are review findings. So are qualifying Bl -tag sequences that remain definitions and ordinal -diag, -hang, -inset, or -ohang definitions that become ordered lists. Aliasless generic terms and entries in NOTES, FOOTNOTES, or REFERENCES are reported as bounded census data, not failures: those categories can be valid and require source-aware human judgment. The fixture ledger is SEMANTIC_AUDIT.csv; its guide records the exact contract and latest broad sweep.

The same profiler accepts an optional queries array (at most 16 literals) and calls the production explanation collector. It adds queryProfiles without changing the ordinary census. Native AST paths at the returned source coordinates are candidates for human inspection, not proof that the head/body belongs to that owner. Reviewed query gold additionally fixes source SHA, full head, source coordinates, exact kind/names, body witnesses, independent empty owners and forbidden direct evidence. Counts alone are not a correctness oracle.

# Network-free checked-in sources only; part of scripts/check.sh.
python3 scripts/audit-roff-semantics.py --fixtures \
  --query-gold tests/fixtures/roff/ENTRY_QUERY_GOLD.json
# Optional local panel; missing/drifted sources remain unresolved, never clean.
python3 scripts/audit-roff-semantics.py \
  --query-gold tests/fixtures/roff/ENTRY_QUERY_GOLD.json \
  --query-root corpus=/path/to/fixed-manual-corpus \
  --json target/entry-query-audit.json

The manifest contains selection/provenance notes, not third-party manual bodies. An expectation may describe a reviewed conservative Term without claiming a runtime grammar or complete value domain. Do not mechanically bless new output as gold: review source, AST candidates and the specific direct body first. An unreviewed query or omitted evidence is not a pass. A developer-only profiler request with snapshot (a saved QueryBundle) and queries can replay old IR through the current collector; label this separately from running an old binary. It has no native AST witnesses and cannot establish historical timing or parser behavior. Keep large raw snapshots and run logs outside docs/.

Mandoc reference replay

Optional fidelity/layout reference execution uses a dedicated process group, the requested wall-clock timeout, a CPU limit of ceil(timeout) + 1 seconds, 1 GiB address space, a combined 64 MiB stdout/stderr budget and 16 MiB stdin budget. Output-limit and timeout failures discard partial results instead of comparing them. Child resource limits survive exec; output is drained while input is written. The reference environment fixes LC_ALL/LANG=C.UTF-8, TZ=UTC, width and pager/formatter settings, retaining only PATH and TMPDIR from the caller; resolved manual roots are supplied explicitly. Run on a POSIX audit host with that locale available. Pure profiler/ledger checks do not need an external renderer.

This is a resource boundary, not a filesystem/network sandbox. A hostile renderer can access host files or detach descendants; use trusted corpus input or run the audit inside an externally restricted container/VM. Resource-limit checks do not certify arbitrary third-party formatters as safe. The bounded runner's self-tests are included in fidelity --self-check.

The supplementary mandoc route reuses the same exact historical source identities rather than selecting a second sample. It is useful for native mdoc punctuation and formatter behavior, but it does not replace groff as an independent parser-family oracle.

python3 scripts/audit-roff-fidelity.py --manpath /tmp/exact/share/man \
  --corpus exact-corpus --replay-source-records \
  --reference-kind mandoc --reference mandoc \
  --reference-id mandoc-1.14.6-1 \
  --audit-db tests/fixtures/roff/MANDOC_FIDELITY_AUDIT.csv --findings-only
python3 scripts/audit-roff-layout.py --manpath /tmp/exact/share/man \
  --corpus exact-corpus --replay-fidelity-records \
  --reference-kind mandoc --reference mandoc \
  --reference-id mandoc-1.14.6-1 \
  --audit-db tests/fixtures/roff/MANDOC_LAYOUT_AUDIT.csv --findings-only

The content route sends decompressed bytes directly to mandoc and records its package identity. It expands only pure redirect pages within the selected manual hierarchy; embedded include execution remains deliberately out of scope. Reconstruct missing corpus files from the exact official artifacts and verify their decompressed hashes before replay. Never commit downloaded manual trees or /tmp review bundles.

The independent mandoc content guide and mandoc layout guide record the current scope, renderer identity, commands, and human conclusions. Replay corpora serially because atomic CSV checkpoints protect interruption, not concurrent writers.

--syntax-priority replaces path-only ranking with deterministic greedy coverage over the actual owned libmandoc AST. The development-only roff_ast_profile example reports macro names, node roles and parent/child shapes, normalized list/display/font state, tables, equations, parser diagnostic classes, and rendering-relevant node flags without copying document text. It also reports bounded interaction features: a node or parent/child context paired with its flags and normalized attributes, plus attribute pairs on the same node. The sampler weights these combinations ahead of isolated features, first preferring shapes absent from the completed CSV ledger and then underrepresented and rarer forms. This distinguishes merely having seen .SY, a no-fill node, and a font from having exercised their exact combination.

--syntax-report records per-feature corpus, ledger, reused-source, and selected-page counts plus representative paths; it measures exercised parser structure, not semantic correctness. --syntax-cache avoids reparsing unchanged (corpus, path, source hash) identities and supports compact .json.gz files. Both the profiler response and cache carry a feature-schema identity, so changing the set of observed AST flags or shapes invalidates and rebuilds an older cache instead of silently treating stale profiles as complete. Profiling uses bounded subprocess batches and isolates an abnormal native-parser exit down to the exact page instead of losing the whole scan. Unreadable host paths remain visible as report errors but do not masquerade as syntax features.

--review-dir writes a local, untracked review bundle for every selected page: the decompressed source, normalized reference text, ManT text, its exact finding, and a path-safe manifest. Use a bounded syntax-prioritized selection to build a deliberate manual review batch, and never commit the bundle because it can contain third-party manuals and renderer-specific output. The regular JSON report and CSV ledger remain compact metadata rather than a copy of the reviewed corpus.

Treat fidelity loss as three related but distinct classes: an unhandled construct, a recognized construct with an unmapped operand, or an incorrectly lowered recognized construct. Structured parser/lowering diagnostics can expose the first two classes, but a zero diagnostic count never proves fidelity because the third class requires an external oracle or a focused semantic regression. The local differential audit is therefore the discovery surface; reproducible fixtures and exact Rust assertions remain the CI gate.

The optional CSV audit database makes successive local runs incremental. A page is omitted from a normal run only when its corpus name, relative path, and decompressed-source SHA-256 match a row whose automated scan completed; an upgraded page and a historical scan_status=skipped row are scanned again. For a newly extracted distribution or release corpus, --dedupe-across-corpora additionally reuses a completed record only when decompressed bytes, topic, and exact manual section all match. Sources containing .so or .mso are never reused this way because their result depends on the owning hierarchy. Reused pages remain visible in the syntax report with their originating corpus and path; they do not create misleading duplicate CSV scan rows.

Pages containing .so requests use the indexed-manual path with an exact derived MANT_MANPATH, so aliases exercise the product resolver and localized hierarchies do not fall through to the default language. Renderer failures and empty comparison surfaces are hard audit failures rather than silently accepted coverage. Automated candidates enter as review_status=pending; a later clean recheck clears that automated state. After inspecting the source and both renderings, mark durable conclusions false-positive, confirmed-open, or confirmed-fixed and explain the decision in note. --retry-skipped migrates historical gaps, --pending-only revisits unresolved signals, --recheck-recorded deliberately ignores the completion index, and --recorded-only re-runs every unchanged page represented by that corpus. For multiple roots, paths are relative to their common parent so same-named man/ directories remain distinguishable. The CSV records corpus exploration; fixed fixtures and focused Rust tests, rather than the host database, remain the CI regression boundary.

The full oracle is a local and release-time discovery tool, not a per-push CI dependency. Ordinary CI runs only its dependency-free self-check plus the focused Rust regressions derived from confirmed findings. When the audit exposes real semantic loss, add the smallest licensed page to the existing source catalogue, document its provenance, and encode the confirmed behavior in the corresponding crates/mant-engine/tests/<source>/ module or a shared assertion. Do not add an allowlist merely to silence an unexplained candidate.

Every new corpus expansion also has a manual review budget. Inspect every REVIEW and HARD result, then inspect representative clean pages that collectively cover the corpus's table forms, no-fill or display content, font changes, links or includes, and its dominant macro dialect (man or mdoc). Compare the source, the reference renderer, ManT text, and the structured result when a layout difference could hide an IR error. Record each candidate's durable conclusion in FIDELITY_AUDIT.csv; record the corpus-level scope and any confirmed fixes in tests/fixtures/roff/FIDELITY_AUDIT.md. When that review establishes that ManT preserves the source semantics more usefully than the observed terminal reference, add the exact source-hash-specific evidence to tests/fixtures/roff/REFERENCE_RENDERER_DEVIATIONS.csv instead of loosely calling the host formatter “wrong”. This makes the ledger evidence of both automated breadth and deliberate human inspection without turning host-specific presentation into a CI gate.

Prefer complete, immutable release artifacts for permanent corpus rows. Rolling current, snapshot, and ports trees are useful bounded exploration, but keep their CSVs and review bundles under /tmp because their identities rotate. Record the verified artifact/build identity and aggregate result in the audit guide. Promote only a confirmed defect's exact licensed source and focused regression, or replay the whole tree under a stable release identity, into the repository ledgers.

For the aligned mandoc route, use the exact reference_id plus -T utf8 -O width=200 in the deviation row. The coverage check then requires that current-renderer evidence to match either a reviewed false-positive comparison or a confirmed-fixed source conclusion in MANDOC_FIDELITY_AUDIT.csv; ordinary wrapping, equivalent tokenization, repeated instances of an existing class, and deliberate ManT layout policy do not qualify.

The audit routes share an explicit source-identity baseline rather than merely similar row counts. Structure and CommonMark projection cover every recorded fidelity identity; renderer-layout covers the fidelity rows with a completed two-renderer comparison. The mandoc content profile exactly replays that historical baseline plus checked-in fixtures, and its layout profile covers every comparable mandoc row. Target-conservation and semantic-entry precision each cover every checked-in fixture while their broader distribution sweeps remain independent. Other targeted groff/structure sweeps may remain supersets. See the coverage contract and run python3 scripts/check-roff-audit-coverage.py before concluding a local audit expansion.

ManT intentionally does not expose this comparison as a user-facing mant --verify fidelity certificate. A reference renderer is unavailable on some supported platforms, installed macro packages and pages are host state, and groff/mandoc presentation differences require human interpretation. A successful comparison therefore cannot certify an arbitrary excerpt. For a specific statement that will be quoted or published, use a source-directed audit over the owning page, inspect the source and both renderings, and turn any durable semantic invariant into a repository fixture.