An evidence-first Compliance Intelligence Engine that transforms security scanner findings into standardized, evidence-backed compliance artifacts — an evidence-backed PR summary comment, OSCAL, SARIF, and coverage reports.
Detection is delegated entirely to best-in-class OSS engines (Checkov, Semgrep, CodeQL, Trivy, and more). The core engine is scanner-agnostic: any tool that emits SARIF can feed it. What audit-packs adds is the normalization → compliance mapping → evidence generation → output layer: reviewers see not just "S3 bucket unencrypted" but:
NIST 800-53 / SC-13 — Cryptographic Protection Severity:
high| Engine:checkov(CKV_AWS_19) Evidence:server_side_encryption_configuration is not set
| Scanner | Status |
|---|---|
| Checkov | Supported |
| Semgrep | Supported |
| CodeQL | Supported (SARIF dir input) |
| Trivy | Supported |
| tfsec | Supported |
| gitleaks | Supported |
Checkov and Semgrep are excellent at finding IaC misconfigurations. They are not designed to answer the question auditors and GRC teams actually ask: which compliance controls are affected, and where is the evidence? audit-packs bridges that gap by wrapping detection output in a compliance control mapping layer, confidence scoring, and audit-grade evidence packaging — without replacing or re-implementing any detection engine.
Refer to the complete Setup & Integration Guide for detailed CLI, VS Code extension, and notification configuration.
# .github/workflows/audit.yml
name: Audit Packs
on:
pull_request:
jobs:
audit:
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write # required to post the PR summary comment
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0 # required for diff-only scanning
- uses: prakharsingh/audit-packs@v0
with:
frameworks: nist-800-53,soc2
fail-on: highThe action posts an evidence-backed PR summary comment (diff-filtered, updated in place on re-runs), writes an OSCAL assessment-results JSON, a control coverage matrix, and an aggregate SARIF file, then exits non-zero if any finding meets or exceeds fail-on.
| Input | Default | Description |
|---|---|---|
frameworks |
required | Comma- or newline-separated pack IDs to evaluate. See Framework coverage. |
fail-on |
high |
Minimum severity that fails the check. One of low, medium, high, critical. |
base-ref |
origin/main |
Base git ref to diff against. Change for non-standard default branch names. |
scan-mode |
both |
diff — PR comments + gate only. full — posture outputs only. both — all paths (recommended). |
emit-oscal |
true |
Write OSCAL assessment-results JSON to oscal.json. |
emit-coverage |
true |
Write a control coverage matrix to coverage.md / coverage.html and append to the job summary. |
seo-title |
Audit Packs Control Coverage Matrix |
HTML <title>, Open Graph title, and JSON-LD name for coverage.html. |
seo-description |
Compliance control coverage report generated by audit-packs. |
Meta description, Open Graph description, and JSON-LD description for coverage.html. |
seo-canonical-url |
"" |
Optional canonical URL for coverage.html when publishing the report. |
emit-sarif |
true |
Write an aggregate SARIF file to audit-packs.sarif. |
adjudication-mode |
off |
LLM adjudication: off (disabled), advisory (score and log, no filtering), enforce (suppress findings below min-confidence). |
min-confidence |
0.70 |
Composite confidence threshold (0.0–1.0). Findings below this are suppressed in enforce mode. |
models-config |
audit-models.yaml |
Repo-relative path to a model routing YAML that maps roles to providers. Falls back to built-in defaults if absent. |
detector-model |
"" |
Override the detector role's model (sets DETECTOR_MODEL env). |
verifier-model |
"" |
Override the verifier role's model (sets VERIFIER_MODEL env). |
adversarial-model |
"" |
Override the challenger role's model (sets CHALLENGER_MODEL env). |
judge-model |
"" |
Override the consensus role's model (sets CONSENSUS_MODEL env). |
codeql-sarif |
"" |
Repo-relative path to directory of CodeQL SARIF files. Gracefully skipped if absent. |
ast-rules |
ast-rules |
Path to AST rule scripts directory. Scripts here run against all Python files in the workspace. For safety, rules committed inside the scanned repo are ignored unless AUDIT_ALLOW_REPO_CONFIG=true; a path outside the workspace is always trusted. |
trivy-enabled |
true |
Enable Trivy filesystem + image scanning. Requires trivy binary ≥ v0.69.2 on the runner. |
trivy-image |
"" |
Docker image reference for trivy image scan. Skipped when empty. Only used when trivy-enabled is true. |
tfsec-enabled |
true |
Enable tfsec Terraform security checks. |
gitleaks-enabled |
true |
Enable gitleaks secret detection. |
allow-repo-config |
false |
Trust repo-supplied configuration (ast-rules/, scanner plugins, audit-models.yaml base_url/api_key_env). The scanned repository is untrusted by default — only enable for repositories you fully control. |
redact-engines |
"" |
Comma-separated list of scanner engines whose raw output is fully masked before reaching any output sink (PR comments, Slack, Jira, OSCAL). Note: masking is applied post-adjudication; the LLM still sees raw evidence for these engines. |
baseline-file |
"" |
Repo-relative path to a baseline JSON file created by audit-packs baseline. When provided, findings already in the baseline are excluded from the severity gate and appear in a "Baselined (pre-existing)" PR comment section. |
baseline-gate-severity |
"" |
Safety valve: baseline entries at or above this severity still gate the PR. One of low / medium / high / critical. Empty = excuse all baselined findings regardless of severity. |
github-token |
${{ github.token }} |
GitHub token used to post the PR summary comment. Defaults to the workflow's automatic token. |
pr-number |
${{ github.event.pull_request.number }} |
Pull request number to comment on. Defaults to the number of the triggering pull_request event (empty on non-PR events). |
By default, audit-packs treats the scanned repository as untrusted. Rules and configuration files committed inside the workspace cannot affect scan behaviour without an explicit opt-in — this prevents a compromised repo from redirecting scans to attacker-controlled endpoints or injecting malicious AST rules.
Set the environment variable AUDIT_ALLOW_REPO_CONFIG=true on the action step to opt in:
- uses: prakharsingh/audit-packs@v0
env:
AUDIT_ALLOW_REPO_CONFIG: "true" # trust repo-supplied AST rules and model configAUDIT_ALLOW_REPO_CONFIG governs three things:
| What | Behaviour when false (default) |
Behaviour when true |
|---|---|---|
ast-rules dir inside the repo |
Silently ignored | Loaded and executed |
| Auto-discovered scanner plugin dirs inside the repo | Silently ignored | Loaded |
base_url / api_key_env in a repo audit-models.yaml |
Silently ignored | Honoured |
A path passed via the ast-rules input that points outside the scanned workspace is always trusted regardless of this flag.
The scanned repository is untrusted by default. Three kinds of repo-supplied configuration can cause code execution or credential redirection, and are ignored unless you explicitly opt in:
| Surface | What it could do |
|---|---|
ast-rules/*.py |
arbitrary code execution at import time |
scanner plugins (scanners/*.yaml) |
arbitrary subprocess execution |
audit-models.yaml base_url / api_key_env |
send your API credentials to an attacker endpoint |
Opt in only for repositories you fully control:
- CLI:
audit-packs --allow-repo-config ... - Action:
with: { allow-repo-config: "true" } - Env:
AUDIT_ALLOW_REPO_CONFIG=true
Every trust decision (honored or ignored, with the reason) is printed in the run output and the GitHub job summary, so a compliance run documents exactly what repo-supplied content it obeyed.
Evidence redaction is always on (core/redact.py): gitleaks evidence is fully masked, and all
other engines' evidence is scrubbed against built-in secret patterns before
reaching any sink (PR comments, Slack, Jira, OSCAL, SARIF, console). There is
no off switch.
When you first add audit-packs to an existing project, pre-existing findings will trip the gate. Baseline suppression lets you snapshot the current state so future runs only gate on new findings while pre-existing debt stays visible as evidence.
Step 1 — Generate the baseline. Run the baseline subcommand locally (or in a one-off CI job):
audit-packs baseline \
--frameworks nist-800-53,soc2 \
--packs-dir packs \
--baseline-file .audit-baseline.json
This writes a .audit-baseline.json file containing a fingerprint for every current finding.
Step 2 — Commit the baseline file.
git add .audit-baseline.json
git commit -m "chore: add audit-packs baseline snapshot"
Step 3 — Reference the baseline in your workflow.
- uses: prakharsingh/audit-packs@v0
with:
frameworks: nist-800-53,soc2
baseline-file: .audit-baseline.json
- Pre-existing findings matched by the baseline appear in a "Baselined / pre-existing" section in the PR comment. They are not counted against the
fail-onthreshold. - Genuinely new findings still fail the gate as usual.
- Stale entries — baseline fingerprints with no matching current finding — are reported as "resolved" with a hint to regenerate the baseline.
Use baseline-gate-severity to never excuse findings at or above a given severity,
even when they appear in the baseline:
- uses: prakharsingh/audit-packs@v0
with:
frameworks: nist-800-53
baseline-file: .audit-baseline.json
baseline-gate-severity: critical # critical findings always gate
After fixing pre-existing issues or when the fingerprint set drifts, regenerate:
audit-packs baseline --frameworks nist-800-53,soc2 --packs-dir packs \
--baseline-file .audit-baseline.json
git add .audit-baseline.json
git commit -m "chore: refresh audit-packs baseline"
The .audit-baseline.json file is human-readable JSON committed to the repo:
{
"schema_version": "1",
"generated_at": "2024-01-15T10:30:00Z",
"frameworks": ["nist-800-53"],
"tool_version": "1.2.0",
"entries": [
{
"fingerprint": "a3f8c2d1...",
"fingerprint_method": "primary",
"engine": "checkov",
"check_id": "CKV_AWS_19",
"file": "main.tf",
"line": 42,
"severity": "high",
"message": "Ensure S3 bucket has server side encryption enabled"
}
]
}
| Output | Path | Description |
|---|---|---|
oscal-path |
oscal.json |
OSCAL assessment-results document for audit evidence packages. |
coverage-md-path |
coverage.md |
Markdown control coverage matrix. |
coverage-html-path |
coverage.html |
HTML control coverage matrix. |
sarif-path |
audit-packs.sarif |
Aggregate SARIF file for upload to GitHub Code Scanning. |
The action posts one PR summary comment per run, identified by a hidden marker so re-runs update it in place rather than appending a new comment (upsert_summary_comment, report.py:430). The comment lists every finding that touches a changed line — diff-filtered so unchanged-line findings are silently dropped — with per-finding framework/control tags, confidence score, and evidence:
audit-packs compliance report
Severity Control Engine Finding Confidence highnist-800-53/ SC-13 — Cryptographic Protectioncheckov(CKV_AWS_19)Ensure S3 bucket has encryption enabled 0.92 Evidence
server_side_encryption_configuration is not set
Because the comment is updated in place, reviewers always see the current state of the scan — no stale comment threads accumulate across pushes.
When emit-oscal: true, the action writes an OSCAL assessment-results document to oscal.json. This is the machine-readable format GRC tools and FedRAMP / NIST 800-53 evidence packages expect.
- uses: prakharsingh/audit-packs@v0
id: audit
- name: Upload OSCAL evidence
uses: actions/upload-artifact@v4
with:
name: oscal-assessment-results
path: ${{ steps.audit.outputs.oscal-path }}When emit-coverage: true, the action writes coverage.md and coverage.html and appends the matrix to the Actions job summary. The matrix lists every control in the selected frameworks, whether it is automatically assessable via IaC checks, and its current pass / fail / not-applicable status.
coverage.html is a complete SEO-ready document with description, robots, Open Graph, Twitter card, optional canonical URL, and JSON-LD metadata. Set seo-title, seo-description, and seo-canonical-url when publishing the report as a static page.
When emit-sarif: true, findings across all engines are merged into a single SARIF file. Upload it to GitHub Code Scanning for a unified security overview:
- uses: prakharsingh/audit-packs@v0
- uses: github/codeql-action/upload-sarif@v3
with:
sarif_file: audit-packs.sarif| Framework | Pack ID | Type | Automated controls |
|---|---|---|---|
| NIST SP 800-53 Rev 5 | nist-800-53 |
Canonical | 20 |
| SOC 2 Type II (AICPA 2017) | soc2 |
Crosswalk → NIST 800-53 | 17 of 39 (22 are governance-only) |
| ISO/IEC 27001:2022 | iso27001 |
Crosswalk → NIST 800-53 | 10 |
| PCI-DSS v4.0 | pci-dss |
Crosswalk → NIST 800-53 | 8 |
| FedRAMP Moderate | fedramp |
Crosswalk → NIST 800-53 | 8 |
| HIPAA Security Rule | hipaa |
Crosswalk → NIST 800-53 | 6 |
| GDPR (technical controls) | gdpr |
Crosswalk → NIST 800-53 | 5 |
| Org-policy (custom) | org-policy |
Crosswalk → NIST 800-53 | 6 (configurable) |
NIST 800-53 is the canonical pack. Every other framework is a crosswalk pack: each control maps to one or more NIST controls, which resolve to engine check IDs. Adding a new framework never requires touching detection logic — you add a YAML pack.
For the full per-control matrix — every automated control across all frameworks resolved to its underlying engine rules — see docs/CONTROL_MAPPING_MATRIX.md.
| Mode | What runs | Use case |
|---|---|---|
diff |
PR summary comment + severity gate | Fast PR feedback; no posture outputs |
full |
Coverage matrix, OSCAL, aggregate SARIF | Scheduled compliance snapshots; no PR gate |
both |
All of the above (default) | Recommended for PRs — gate on every push, posture on every merge |
git diff ──────────────────────────────────────────────────────────────────────┐
│ diff-filter
Checkov ──────────► SARIF ─┐ │ (PR-changed
Semgrep ──────────► SARIF ─┤ │ lines only)
CodeQL (optional) ► SARIF ─┤ │
Detection agents ► SARIF ─┴──► normalize ──► Finding[] │
(GDPR, HIPAA, │ │
SOC2, FedRAMP, enrich (evidence + │
OrgPolicy, doc context) │
DataFlow) │ │
data-flow analysis │
│ │
└──── diff-filtered ─────────┤
│
┌────────────────────────────────────────┘
▼
map to framework controls
│
adjudicate (AI ensemble,
if enabled)
│
confidence gate
│
┌─────────────────┼──────────────────────┐
▼ ▼ ▼
PR summary comment severity gate posture outputs
(control-tagged, (exit 1 if ≥ (OSCAL, coverage
evidence-backed) fail-on threshold) matrix, SARIF)
Detection is never re-implemented. Checkov, Semgrep, and CodeQL run as subprocesses and emit SARIF. Framework-specific detection agents (GDPRAgent, HIPAAAgent, SOC2Agent, FedRAMPAgent, OrgPolicyAgent, DataFlowAgent) apply heuristics for controls that engines cannot observe directly — they also emit SARIF. normalize.py converts all SARIF to a common Finding model. Pack YAML files map (engine, check_id) pairs to control IDs.
Seven rules ship alongside the action to cover gaps not detectable by Checkov:
| Rule ID | What it catches |
|---|---|
weak-cipher |
DES / RC4 / MD5 usage in Python |
hardcoded-credential |
Secrets assigned to variables |
no-tls-verify |
TLS verification disabled |
overpermissive-iam |
Wildcard IAM actions or resources |
missing-audit-log |
Logging / audit trail not configured |
insecure-config |
Insecure configuration flags (debug mode, plaintext storage) |
pii-fields |
PII field names in data models and API schemas |
When adjudication-mode is advisory or enforce, each finding passes through a four-role LLM ensemble before the confidence gate:
- Detector — establishes an initial confidence assessment, acting as a compliance auditor.
- Verifier — argues why the finding is a genuine compliance violation.
- Adversarial — argues why the finding is a false positive.
- Judge — weighs both arguments and produces the final consensus score.
The final composite score is a weighted average of six signals:
| Signal | Weight | Source |
|---|---|---|
| Rule confidence | 20% | Emitted by the engine or agent in SARIF |
| Data-flow confidence | 20% | Source-to-sink flow analysis (dataflow.py) |
| Model consensus | 25% | Judge's agreement score from the AI ensemble |
| Evidence confidence | 15% | Richness of code snippets and PR / commit file context |
| Control severity | 10% | Criticality rank of the mapped control |
| Historical precision | 10% | Long-term true-positive rate tracked per check ID |
A finding whose composite score falls below min-confidence (default 0.70) is suppressed when adjudication-mode: enforce. In advisory mode the score is logged but no finding is filtered. In off mode (default) no LLM calls are made.
Create audit-models.yaml in your repo root to map each role to a provider and model. The action falls back to built-in defaults if the file is absent.
# audit-models.yaml
models:
detector:
provider: openai
model: gpt-4o
api_key_env: OPENAI_API_KEY
verifier:
provider: anthropic
model: claude-opus-4-5
api_key_env: ANTHROPIC_API_KEY
adversarial:
provider: google
model: gemini-1.5-pro
api_key_env: GOOGLE_API_KEY
judge:
provider: openai
model: gpt-4o
api_key_env: OPENAI_API_KEYSupported providers: openai, anthropic, google, ollama, openai-compatible. Supply the corresponding API key secrets as environment variables on the step.
You can also override individual roles without a config file using per-role inputs:
- uses: prakharsingh/audit-packs@v0
with:
frameworks: nist-800-53
adjudication-mode: enforce
judge-model: gpt-4o-mini # cheaper judge for high-volume repos
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}Edit packs/org-policy/controls.yaml to define internal controls and map them to NIST 800-53 controls:
framework: org-policy
schema_version: '2'
title: Acme Corp Security Policy
crosswalk: nist-800-53
controls:
- id: ACME-ENC-1
title: All data stores must be encrypted at rest
maps_to:
- SC-13
- SC-28
- id: ACME-NET-1
title: No public S3 buckets permitted
maps_to:
- SC-7
- id: ACME-LOG-1
title: Enable audit logging for all services
maps_to:
- AU-2Any check ID already mapped in packs/nist-800-53/controls.yaml is automatically surfaced under your org control ID with no other changes required.
audit-packs can consume CodeQL SARIF artifacts to combine SAST findings with IaC findings in a single compliance view. Run codeql-action/analyze with upload: false, then pass the output directory to audit-packs:
- name: Initialize CodeQL
uses: github/codeql-action/init@v3
with:
languages: python,javascript
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v3
with:
output: codeql-results # write SARIF to this directory
upload: false # prevent double-upload; audit-packs handles it
- uses: prakharsingh/audit-packs@v0
with:
frameworks: nist-800-53,soc2
codeql-sarif: codeql-resultsIf codeql-sarif is absent or the directory is empty, CodeQL findings are silently skipped — the rest of the scan runs normally.
For complete setup and configuration details, see the Setup & Integration Guide.
Prerequisites: Python 3.11.4+, git, uv (recommended for the workspace install)
For running the CLI against your own repos:
pipx install audit-packs
pipx inject audit-packs checkov semgrep # optional scannersFor contributing / running tests:
# Clone the repo
git clone https://github.com/prakharsingh/audit-packs.git
cd audit-packs
# Install all workspace packages editably + dev deps via uv
uv sync
# Or install editably via pipx from source
pipx install ./packages/action --force
pipx inject audit-packs \
./packages/core ./packages/mapping ./packages/evidence ./packages/ai --force# Run all tests
pytest -v
# Run a single test file
pytest tests/test_packs.py -v
# Run a single test
pytest tests/test_packs.py::test_map_findings_crosswalk_soc2 -v# Reinstall only changed packages
pipx inject audit-packs ./packages/action ./packages/mapping --force
# Test from any git repo — uses bundled default rules for Semgrep if rules-path is omitted
audit-packs --frameworks nist-800-53,soc2 \
--packs-dir ~/projects/audit-packs/packsFor full CLI documentation see the Setup & Integration Guide. Key commands:
audit-packs --init— interactive wizard that bootstraps youraudit-models.yamland workflow config.audit-packs pack init <id>— scaffold a new framework pack directory.audit-packs pack validate <path>— validate a pack'scontrols.yamlschema and crosswalk references.audit-packs pack test <path> --fixture <dir>— run pack mappings against scan fixtures.audit-packs pack publish <path>— package a pack as a distributable tarball.audit-packs pack install <source>— install a pack from a URL, GitHubowner/repo@version, or local tarball.audit-packs --validate-policy— syntax-check custom compliance pack YAMLs without running a scan.--slack-webhook <url>— post a Slack notification when the severity gate trips.--jira-url / --jira-project / --jira-email / --jira-token— create a Jira issue on gate failure.
Build the Docker action image:
docker build -t audit-packs:dev .Run the Docker smoke test:
pytest tests/test_docker_smoke.py -v
# or directly:
./tests/docker_smoke.shThe Python source is organized as a uv workspace of five packages under packages/. Each package is independently installable and declares its inter-package dependencies in its own pyproject.toml.
packages/
core/src/audit_packs_core/ # pure-Python primitives, no network/subprocess
models.py # Finding, ControlFinding, ControlStatus, AdjudicationResult dataclasses
diff.py # parse_unified_diff() → {file: set[line]}
normalize.py # sarif_to_findings(); extract_rule_confidences()
dataflow.py # extract_data_flows() (Python / HCL / YAML), flow_confidence()
redact.py # redact_evidence(), apply_engine_config() — single chokepoint before every output sink; always on, no off switch
trust.py # TrustPolicy, repo_config_allowed() — trust gate for repo-supplied executable content
mapping/src/audit_packs_mapping/ # depends on: core
packs.py # load_pack(), iter_controls(), map_findings() — control mapping + NIST crosswalk
coverage.py # compute_coverage() → list[ControlStatus]
oscal.py # to_assessment_results() — NIST OSCAL assessment-results JSON
evidence/src/audit_packs_evidence/ # depends on: core
evidence.py # enrich(), fetch_pr_context() [GitHub API], evidence_confidence()
agents.py # GDPRAgent, HIPAAAgent, SOC2Agent, FedRAMPAgent, OrgPolicyAgent, DataFlowAgent, Nist80053Agent
ai/src/audit_packs_ai/ # depends on: core, mapping; optional LLM SDKs via [ai] extra
adjudicate.py # AI ensemble (detector → verifier → adversarial → judge) [LLM HTTP]
confidence.py # score_finding(), apply_confidence_gate(), DEFAULT_WEIGHTS
action/src/audit_packs_action/ # depends on: core, mapping, evidence, ai — top-level entrypoint
cli.py # analyze() (diff path) + assess() (full path) + main()
engines.py # CheckovEngine, SemgrepEngine, CodeQLEngine, ASTEngine, TrivyEngine, TfsecEngine, GitleaksEngine, DeclarativeEngine
report.py # build_summary_comment(), upsert_summary_comment(), build_coverage_matrix(), build_sarif(), post_slack_message(), create_jira_issue(), write_job_summary()
trust.py # deprecated re-export shim → audit_packs_core.trust
packs/ # Framework YAML packs (data only — no detection logic)
nist-800-53/controls.yaml # canonical: (engine, check_id) → control
soc2/controls.yaml, gdpr/controls.yaml, hipaa/controls.yaml,
iso27001/controls.yaml, pci-dss/controls.yaml, fedramp/controls.yaml,
org-policy/controls.yaml # all crosswalk → nist-800-53
rules/ # Authored Semgrep rules bundled with the action
weak-cipher.yaml no-tls-verify.yaml pii-fields.yaml
insecure-config.yaml hardcoded-credential.yaml
overpermissive-iam.yaml missing-audit-log.yaml
The dependency graph is acyclic: core → mapping → ai and core → evidence, with action depending on all four. Only ai pulls optional LLM SDKs (via its [ai] extra).
Key design constraints:
- Detection is never re-implemented. Engines run as subprocesses; findings arrive as SARIF.
- Packs are data, not code. A framework pack is pure YAML mapping check IDs to controls.
- Network and subprocess I/O is confined to four modules:
engines.py,evidence.py,adjudicate.py,report.py. Everything else is pure Python and testable without network access or installed tools.
Contributions are welcome! Please refer to CONTRIBUTING.md for local development setup, guidelines on adding framework packs or custom rules, and pull request requirements.
