Deep expansion: test suite, CI, docs, batch/Markdown companion, and roadmap

[cognis-digital]

Motivation

packpeek is a solid, single-purpose triage tool, but before this PR the repository had no automated test suite (only inline CI smoke commands), the README's "Example output" block showed a generic placeholder unrelated to the tool (a packs/weight/contents schema that packpeek never emits), and the Python companion could only produce SARIF and YARA for one report at a time.

This PR raises the repo to production grade across four axes — tests/CI, docs, features, and internal cleanup — while remaining strictly additive: nothing is removed, and every existing flag, entry point, and JSON field keeps its meaning and output.

What changed

Tests & CI

• New tests/ suite (48 tests).
  • tests/test_sarif.py — 36 Python unit tests for the companion: report normalisation (single object / JSON array / NDJSON / blank lines / garbage), SARIF rendering (packed→error, likely→warning, clean→no results, rule declaration, batch aggregation), YARA (family→marker mapping, dedup, sorted family names, batch union, worst-verdict meta), Markdown, worst_verdict, and full main() behavior over stdin/files including the --fail-on gate and every error path.
  • tests/test_packpeek_c.py — 12 end-to-end tests that compile packpeek.c and drive the real binary: packed/clean/likely verdicts, exit codes, marker offset accuracy, family de-duplication, --threshold behavior, missing-file / no-input / --help handling, JSON schema completeness, and a pipe into the companion. These skip automatically when no C compiler is available (via the packpeek_bin session fixture in tests/conftest.py).
• CI workflow (.github/workflows/ci.yml) now has three jobs: the existing build-test (C build + verdict smoke tests, plus new Markdown and --fail-on gate checks), a pytest job (installs requirements-dev.txt and runs the full suite), and a lint job (flake8 syntax/undefined-name checks). PYTHONUTF8=1 is set workflow-wide.
• Packaging: pyproject.toml (setuptools build, pytest config, and a packpeek-sarif console entry point) and requirements-dev.txt.

Docs

• README overhaul: overview, an ASCII architecture diagram, install/build, usage (single + batch), an accurate Output sample, a companion reference table, a configuration reference, an exit-code table, and a FAQ. The broken placeholder "Example output" is replaced with a real packpeek report (markers preserved for any tooling that manages that block).
• docs/ARCHITECTURE.md — the JSON contract, verdict truth-table, entropy formula, signature-table design, and companion internals.
• docs/USAGE.md — worked examples for SARIF, YARA, Markdown, batching, CI gating, and programmatic use.
• ROADMAP.md — near/mid/long-term direction and explicit non-goals.

Features (companion, additive)

• Markdown output — python sarif.py --md / --markdown renders a summary table (one row per file) with a worst-verdict footer. Great for PR comments.
• Batch / aggregate input — the companion now accepts a single report, a JSON array, or newline-delimited JSON, so for f in *; do packpeek "$f"; done | python sarif.py works. SARIF gains one result per non-clean file, YARA unions every detected marker, Markdown renders every row.
• CI gate — --fail-on {clean,likely-packed,packed} exits 2 when the worst verdict across all inputs meets the level, without changing the emitted document (so you can tee the report and still gate).

Refactor (no behavior change)

• Type hints and docstrings throughout sarif.py; extracted shared helpers (load_reports, worst_verdict, _result_for, _MARKERS, _parse_args).
• to_sarif / to_yara accept a single dict or a list; the single-dict path is byte-for-byte identical to the previous output (covered by a test).

Test results

Locally (Windows, no C toolchain): 36 passed, 12 skipped (C tests skip without a compiler). On CI (ubuntu, gcc present) all 48 run. flake8 selection E9,F63,F7,F82 passes clean.

Run it yourself:

python -m pip install -r requirements-dev.txt
PYTHONUTF8=1 python -m pytest -ra

New usage examples

# Markdown triage summary
packpeek suspicious.exe | python sarif.py --md

# Scan a directory, one aggregated SARIF document
for f in samples/*; do packpeek "$f"; done | python sarif.py > all.sarif

# Gate CI: non-zero exit if anything is confidently packed
for f in build/artifacts/*; do packpeek "$f"; done | python sarif.py --fail-on packed

Backward compatibility

Additive only. No files or public functionality removed. Existing packpeek flags/exit codes and the emitted JSON contract are unchanged; the companion's default (no-flag) SARIF and --yara outputs are unchanged for a single report. All new capabilities are opt-in.

Checklist

• New meaningful tests with real assertions (48 total)
• CI installs deps and runs tests + lint on push/PR
• README + docs accurate to the actual code
• New feature wired in, documented, and tested
• No files removed; existing behavior preserved
• Full suite green before opening this PR