From 14d46a39d0c7ff3eee0bc2b880c7effd1ee89236 Mon Sep 17 00:00:00 2001 From: Gagan Deep Date: Thu, 30 Jul 2026 15:31:44 +0530 Subject: [PATCH 1/2] docs: honesty pass on the default filter, and fix changelog attributions The README described the bare @shield_tool as blocking "common secrets (API keys, passwords)". It does not: SensitiveDataFilter substring-matches the words secret/password/api_key, so a real sk-... or AKIA... credential passes straight through while benign prose containing "password" is blocked. The Quickstart note now says what the filter is, and a Limitations section states that it is not a secret-detection engine. Also documents behaviour shipped in 0.3.2 but never written down: the decorator handles sync and async functions, blocks log a structured warning to stderr, and URLAllowList rejects disallowed schemes. The 0.2.x changelog sections were one release out of step with the tags. Verified against the actual ranges: v0.2.0..v0.2.1 confused-deputy payloads (was filed under 0.2.0) v0.2.1..v0.2.2 eight doc commits (three were filed under 0.2.1, five under 0.3.0) v0.2.2..v0.3.0 the adaptive fuzzer only 0.2.2 had no section at all despite being on PyPI. Added, and the link footer now covers 0.2.2 through 0.3.3. --- CHANGELOG.md | 26 +++++++++++++++++++++----- README.md | 10 +++++++--- pyproject.toml | 2 +- uv.lock | 2 +- 4 files changed, 30 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a8869b4..4c039e8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,14 @@ All notable changes to this project are documented here. ## [Unreleased] +## [0.3.3] - 2026-07-30 + +- docs: state plainly what the default `SensitiveDataFilter` does. It matches keywords; it is not a secret-detection engine, and the Quickstart no longer implies otherwise +- docs: add a `Limitations` section +- docs: note that `@shield_tool` handles sync and async functions and logs a structured warning to stderr on a block +- docs: record `file://`-style disallowed schemes among the Default Deny cases +- docs: correct the 0.2.x changelog attributions, which were one release out of step with the tags + ## [0.3.2] - 2026-07-30 - fix: `@shield_tool` now wraps coroutine functions and async generators in kind. Previously it always produced a sync wrapper, so `inspect.iscoroutinefunction()` returned `False` on a shielded `async def` and frameworks (LangChain, MCP) never awaited it — the model received a coroutine `repr` instead of the tool's result @@ -25,23 +33,26 @@ All notable changes to this project are documented here. - feat: `modelfuzz scan` is now an adaptive fuzzer — it evolves refused attacks into more deceptive variants and retries within a `--budget-s` time budget, instead of sending a fixed list of static prompts - test: add coverage for the scan mutation loop, budget handling, and error paths - ci: run tests with all extras installed so the `scan` path is exercised + +## [0.2.2] - 2026-07-23 + - fix: point PyPI `Homepage` at https://www.modelfuzz.com instead of the GitHub repo - docs: add a zero-clone "Try It Now" snippet, verified against the published PyPI package - docs: add CHANGELOG.md and SECURITY.md - docs: link website, LinkedIn, and PyPI from the README - docs: show real `demo.py` and `modelfuzz scan` output with runnable commands - -## [0.2.1] - 2026-07-22 - - docs: clarify default `PolicyEngine` behavior in the Quickstart - docs: document the `modelfuzz scan` CLI - docs: add Scarf analytics pixel to README +## [0.2.1] - 2026-07-22 + +- feat: add confused-deputy and authority-escalation scan payloads + ## [0.2.0] - 2026-07-22 - feat: add the offensive scanner (`modelfuzz scan`) — red-teams an OpenAI-compatible endpoint with prompt-injection payloads and reports which ones trick it into an unsafe tool call - feat: report `⚠️ INCONCLUSIVE` instead of a false-safe result when every prompt errors out -- feat: add confused-deputy and authority-escalation scan payloads ## [0.1.3] - 2026-07-21 @@ -58,7 +69,12 @@ All notable changes to this project are documented here. - CI workflow (lint + tests), MIT license, unit/integration test suite - Package renamed from `agentshield` to `modelfuzz` -[Unreleased]: https://github.com/higagan/modelfuzz/compare/v0.2.1...HEAD +[Unreleased]: https://github.com/higagan/modelfuzz/compare/v0.3.3...HEAD +[0.3.3]: https://github.com/higagan/modelfuzz/compare/v0.3.2...v0.3.3 +[0.3.2]: https://github.com/higagan/modelfuzz/compare/v0.3.1...v0.3.2 +[0.3.1]: https://github.com/higagan/modelfuzz/compare/v0.3.0...v0.3.1 +[0.3.0]: https://github.com/higagan/modelfuzz/compare/v0.2.2...v0.3.0 +[0.2.2]: https://github.com/higagan/modelfuzz/compare/v0.2.1...v0.2.2 [0.2.1]: https://github.com/higagan/modelfuzz/compare/v0.2.0...v0.2.1 [0.2.0]: https://github.com/higagan/modelfuzz/compare/v0.1.3...v0.2.0 [0.1.3]: https://github.com/higagan/modelfuzz/compare/v0.1.2...v0.1.3 diff --git a/README.md b/README.md index b93f16e..f401a25 100644 --- a/README.md +++ b/README.md @@ -29,7 +29,7 @@ def send_email(to_address: str, subject: str, body: str) -> None: smtp.send(to_address, subject, body) ``` -> **Note:** When used bare, `@shield_tool` applies a default `PolicyEngine` that blocks common secrets (API keys, passwords). To define custom rules (like `URLAllowList`), simply pass your own engine: `@shield_tool(engine=my_engine)`. +> **Note:** When used bare, `@shield_tool` applies a default `PolicyEngine` with a basic `SensitiveDataFilter`. For production use, define your own rules (like `URLAllowList` or custom secret scanners) and pass your own engine: `@shield_tool(engine=my_engine)`. Works bare (`@shield_tool`) or called (`@shield_tool()`) — both wrap `send_email` identically. Any argument that trips a policy raises `ModelFuzzBlockError` before the function body runs. @@ -99,8 +99,12 @@ Output: ## How It Works - **`PolicyEngine`** — runs an ordered list of policies against every tool-call argument and short-circuits on the first violation. Policies are plain callables (`(value) -> Violation | None`), so writing your own is a one-function job. -- **`@shield_tool` decorator** — wraps any function so every positional and keyword argument passes through the engine before the function body runs. A violation raises `ModelFuzzBlockError`; the tool never executes. -- **Default Deny** — allowlist rules like `URLAllowList` block anything not explicitly permitted: unknown domains, userinfo tricks (`http://api.internal.com@evil.com`), and unparseable URLs are all treated as violations. When in doubt, the call doesn't run. +- **`@shield_tool` decorator** — wraps any function (sync or async) so every positional and keyword argument passes through the engine before the function body runs. A violation raises `ModelFuzzBlockError` and logs a structured warning to stderr; the tool never executes. +- **Default Deny** — allowlist rules like `URLAllowList` block anything not explicitly permitted: unknown domains, userinfo tricks (`http://api.internal.com@evil.com`), disallowed schemes (`file://`), and unparseable URLs are all treated as violations. When in doubt, the call doesn't run. + +## Limitations + +ModelFuzz v0.3.3 provides the interception point, the policy protocol, and an adaptive fuzzer. The default `SensitiveDataFilter` uses basic keyword matching and is not a substitute for a dedicated secret-scanning engine. The hosted SaaS dashboard (coming soon) will provide centralized audit logs, policy versioning, and enterprise-grade secret detection. ## Red-Team Scanner diff --git a/pyproject.toml b/pyproject.toml index c32c31f..8c12d5d 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "modelfuzz" -version = "0.3.2" +version = "0.3.3" description = "Runtime guardrails for AI agents." readme = "README.md" license = { text = "MIT" } diff --git a/uv.lock b/uv.lock index 01b8939..99b4a4f 100644 --- a/uv.lock +++ b/uv.lock @@ -286,7 +286,7 @@ wheels = [ [[package]] name = "modelfuzz" -version = "0.3.2" +version = "0.3.3" source = { editable = "." } dependencies = [ { name = "typer" }, From 20bd5e5d9be48e024ea2af9af0f150cf76093a11 Mon Sep 17 00:00:00 2001 From: Gagan Deep Date: Thu, 30 Jul 2026 15:36:41 +0530 Subject: [PATCH 2/2] docs: state the filter's actual gap, and split the roadmap out The first Limitations wording said the default filter was "basic keyword matching" and "not a substitute for a dedicated secret-scanning engine" -- true but abstract. It never told the reader the fact that matters: a real sk-... or AKIA... credential passes straight through. Vagueness in the one section whose job is disclosure reads as hedging, so it now names the three literal strings it matches and says plainly what it misses. Every claim in the section was verified against the code: sk- / AKIA keys pass through dict keys, bytes, not inspected sets, objects policies see one argument at a time, never the whole call The hosted dashboard moved to its own Roadmap section. A product pitch inside a Limitations block invites the reading that the gap is a business decision, which is the standard criticism of open-core security tooling. --- CHANGELOG.md | 2 +- README.md | 6 +++++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4c039e8..95fdce4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,7 +7,7 @@ All notable changes to this project are documented here. ## [0.3.3] - 2026-07-30 - docs: state plainly what the default `SensitiveDataFilter` does. It matches keywords; it is not a secret-detection engine, and the Quickstart no longer implies otherwise -- docs: add a `Limitations` section +- docs: add a `Limitations` section naming what the default filter does and does not catch, and a separate `Roadmap` section for the hosted dashboard - docs: note that `@shield_tool` handles sync and async functions and logs a structured warning to stderr on a block - docs: record `file://`-style disallowed schemes among the Default Deny cases - docs: correct the 0.2.x changelog attributions, which were one release out of step with the tags diff --git a/README.md b/README.md index f401a25..352d0d5 100644 --- a/README.md +++ b/README.md @@ -104,7 +104,11 @@ Output: ## Limitations -ModelFuzz v0.3.3 provides the interception point, the policy protocol, and an adaptive fuzzer. The default `SensitiveDataFilter` uses basic keyword matching and is not a substitute for a dedicated secret-scanning engine. The hosted SaaS dashboard (coming soon) will provide centralized audit logs, policy versioning, and enterprise-grade secret detection. +ModelFuzz provides the interception point, the policy protocol, and an adaptive fuzzer. The default `SensitiveDataFilter` matches the literal strings `secret`, `password`, and `api_key` — it does not recognise credential formats, so a real `sk-…` or `AKIA…` key will pass through it. Treat it as a demo default and write policies for your own threat model. Also: policies see each argument in isolation, not the whole call, and only `str`, `list`, `tuple`, and `dict` values are inspected. + +## Roadmap + +A hosted dashboard is in development, providing centralized audit logs, policy versioning, and managed secret detection. ## Red-Team Scanner