AgenticAuditor AI

Status: v0.1 — Beta / Early Development This project is under active development. Core architecture and questionnaire logic are scaffolded; framework control data is being populated iteratively. Not yet recommended for production governance decisions — see Roadmap below.

A guided, walkthrough-based governance assessment tool for AI agents and agentic skills. AgenticAuditor AI cross-references user-provided answers about an agent's design, permissions, and behavior simultaneously against five major AI security and governance frameworks, then produces a single, deduplicated remediation report.

The Problem

AI security governance today is fragmented across overlapping frameworks:

• NIST AI Risk Management Framework (AI RMF)
• MITRE ATLAS
• OWASP Machine Learning Security Top 10
• OWASP LLM Top 10
• OWASP Agentic Skills Top 10 (AST10)

Each framework independently identifies real risks, but they overlap significantly — the same underlying gap (e.g., missing least-privilege permission scoping) may appear as a distinct control in three or four of these frameworks simultaneously. Manually cross-referencing them is slow, repetitive, and error-prone. Existing checklist-based tools (including OWASP's own AST10 Security Assessment Checklist) require a human to work through a flat list of items framework-by-framework, with no automated synthesis across frameworks and no deduplication of overlapping controls.

What AgenticAuditor AI Does

AgenticAuditor AI is not a detection or scanning tool — it does not ingest logs, telemetry, or live system data. Instead, it works the way a building inspector works: the user is the data source. AgenticAuditor AI asks structured, conditional questions about a system (e.g., "Does this agent have outbound network access?"), and only surfaces relevant follow-up questions based on prior answers (e.g., egress-control questions only appear if outbound access was declared).

Each answer is cross-referenced against control entries in all five frameworks. Where multiple frameworks flag the same underlying gap, AgenticAuditor AI collapses them into a single, unified remedy — rather than presenting five redundant line items for one real problem — along with severity ratings and citations back to each source framework.

Output: a single governance report showing identified gaps, grouped by severity, each with full traceability to every framework and control it maps to.

Why This Exists

This project was built to directly address a gap OWASP's own AST10 project identifies: no single tool or checklist covers the full governance picture, and effective assessment requires combining multiple analysis approaches. AgenticAuditor AI is a proposed piece of that pipeline — the synthesis layer that ties a human's structured self-assessment back to specific, citable controls across the broader AI governance ecosystem, not just one framework in isolation.

Architecture

agentic-auditor/
├── app.py                      # Streamlit entry point / walkthrough UI
├── data/                       # Framework control mappings (JSON)
│   ├── nist_ai_rmf.json
│   ├── mitre_atlas.json
│   ├── owasp_ml_top10.json
│   ├── owasp_llm_top10.json
│   └── owasp_ast10.json
├── questionnaire/
│   ├── walkthrough.yaml        # Sectioned questions with conditional follow-ups
│   └── loader.py                # YAML loader + show_when visibility logic
├── engine/
│   ├── cross_reference.py      # Maps answers -> control IDs across all 5 frameworks
│   └── deduplication.py        # Collapses overlapping controls into unified remedies
└── tests/                      # Unit tests (pytest)

How it works

1. Walkthrough — User answers a sequence of questions, organized by category (permissions, data handling, network egress, supply chain, identity, monitoring). Follow-up questions appear conditionally based on prior answers.
2. Cross-reference — Each answer indicating a gap triggers a lookup against control entries in all five framework data files.
3. Deduplication — Controls from different frameworks that address the same underlying issue (linked via a shared remedy_group) are merged into one remedy, retaining the highest applicable severity.
4. Report — A final results view presents deduplicated remedies grouped by severity, each expandable to show every source framework and control ID it maps to.

Installation

git clone https://github.com/csperera/agentic-auditor.git
cd agentic-auditor
pip install -r requirements.txt

Usage

streamlit run app.py

Run tests:

python -m pytest tests/ -v

Current Status (v0.1)

• Project scaffolding complete
• Questionnaire structure with conditional logic (show_when)
• Cross-reference and deduplication engine (logic complete, awaiting data)
• Streamlit scaffold verified, test suite passing (12/12)
• Framework control data populated (NIST AI RMF, MITRE ATLAS, OWASP ML/LLM/AST10)
• Full questionnaire content mapped to control IDs
• Complete walkthrough UI with results/report page
• LLM-generated natural-language remedy explanations
• Exportable report output (PDF/JSON)

Roadmap

• v0.2 — Populate all five framework data files with real control mappings; wire questionnaire questions to control IDs
• v0.3 — Full walkthrough UI, results/report page with severity grouping and citations
• v0.4 — LLM-generated natural-language remedy explanations (translating raw control citations into plain-language guidance), exportable reports (PDF/JSON), session persistence
• Future — Expanded framework coverage, custom framework import, team/enterprise multi-assessment support, additional AI-assisted features (e.g., dynamic follow-up question generation, free-text answer interpretation)

Framework Attribution

This project references and cross-links publicly available framework content, including:

• NIST AI Risk Management Framework — U.S. National Institute of Standards and Technology
• MITRE ATLAS — © MITRE Corporation
• OWASP ML Security Top 10, OWASP LLM Top 10, OWASP Agentic Skills Top 10 (AST10) — © OWASP Foundation, licensed under CC BY-SA 4.0

AgenticAuditor AI does not redistribute framework source text verbatim; control summaries are original synthesis with citation back to each framework's official documentation. AgenticAuditor AI is an independent project and is not officially affiliated with or endorsed by NIST, MITRE, or the OWASP Foundation.

Contributing

This project is in early beta. Issues, discussion, and contributions are welcome once v0.2 framework data is in place. Please open an issue to discuss significant changes before submitting a pull request.

License

This project is licensed under the MIT License

Author

Built by Christian Perera as part of an ongoing AI security governance tooling initiative.