Skip to content

feat(auto): zero-config evaluation of vanilla pydantic models#176

Open
sromoam wants to merge 6 commits into
devfrom
sr/pydantic_comparison
Open

feat(auto): zero-config evaluation of vanilla pydantic models#176
sromoam wants to merge 6 commits into
devfrom
sr/pydantic_comparison

Conversation

@sromoam

@sromoam sromoam commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

What

Add a zero-config evaluation path so a vanilla Pydantic model (for example, a Strands agent response_model) can be scored with a single call, no StructuredModel subclass, comparators, thresholds, or schema required:

import stickler
result = stickler.evaluate(ground_truth, prediction)
result.overall_score, result.f1, result.field_scores
result.explain()   # per-field comparator/threshold/weight + why

Why

Configuring an evaluation is the main adoption barrier: users want the benefits of a tuned eval without making (and later defending) per-field decisions. The two pre-existing entry points also fail on real models: from_json_schema(Model.model_json_schema()) crashes on Optional/Union and degrades enums/dates, and unannotated fields fall back to string edit-distance even for float/bool/date. This infers a sensible comparator per field from the live Python type plus field-name hints, so "unconfigured" already means "reasonably configured," and .explain() makes every choice auditable.

How

New additive src/stickler/auto/ package (nothing existing changed except top-level exports):

  • inference.py: per-field comparator/threshold/weight from the type (bool/Enum/Literal->Exact, int->Numeric exact, float->Numeric tolerant, date->Date, str->Levenshtein) plus name-token refinement (*_id->Exact, *amount->Numeric, notes->Fuzzy). Optional unwrapping, an availability gate (Semantic/BERT/LLM are never auto-selected; missing optional deps degrade and are recorded).
  • builder.py: walks live cls.model_fields (never the lossy/crashing model_json_schema path), recurses nested BaseModel / List[BaseModel] with cycle detection, delegates to ModelFactory.create_model_from_fields, caches shadow classes in a WeakKeyDictionary.
  • facade.py: evaluate / eval_for / EvalSpec / EvalResult. Normalizes instances via model_dump(mode="json") and reads both field_scores and derived precision/recall/f1/accuracy from a single compare_with call.

Weights default to uniform 1.0 (business-criticality is not encoded in a plain model); weight_hints=True opts into name-based weighting. Per-field overrides were prototyped but removed for now: the top-level-only version silently dropped nested keys, so it is deferred rather than shipped half-working. The InferredSpec config dict remains the single contract a future override or agentic-config layer would target.

Docs and examples

  • Getting-Started: ultra-quick-start.md (in nav).
  • Guides/Use-Cases: evaluate-strands-agent.md (in nav).
  • examples/notebooks/Ultra_Quick_Start.ipynb (clean, verified to run end-to-end).
  • examples/scripts/strands_agent_eval_demo.py (runs the real agent with [llm] + creds, else falls back to a fabricated prediction so it is always runnable).
  • Root README.md blurb linking to the Ultra Quick Start.

Testing

  • 22 new tests in tests/auto/; full suite green (1450 passed, 2 skipped).
  • ruff check clean on all new files (including the notebook).
  • mkdocs build renders both new pages; internal links resolve.
  • New exports covered by the existing __all__-consistency test.

sromoam added 6 commits July 22, 2026 17:09
@sromoam
feat(auto): zero-config evaluation of vanilla pydantic models
b89fb74 
Add stickler.evaluate(gt, pred) / stickler.eval_for(cls): turn any plain
pydantic BaseModel (e.g. a Strands agent response_model) into a scored
stickler evaluation with no StructuredModel subclass, JSON schema, or
x-aws-stickler-* annotation.

New src/stickler/auto/ package:
- inference.py: per-field comparator/threshold/weight inference from the
  live python type + field-name tokens (Optional-unwrap, enum/Literal/date
  handling, availability gate). Weights default to 1.0; name-token weight
  bumps are opt-in via weight_hints. Semantic/BERT/LLM never auto-selected.
- builder.py: walks cls.model_fields (never the lossy/crashing
  model_json_schema path), recurses nested BaseModel / List[BaseModel] with
  cycle detection, delegates to ModelFactory.create_model_from_fields, caches
  shadow classes in a WeakKeyDictionary.
- facade.py: evaluate / eval_for / EvalSpec / EvalResult. Normalizes instances
  via model_dump(mode="json") and reads metrics from a single compare_with
  call (field_scores + derived precision/recall/f1/accuracy). .explain()
  surfaces per-field provenance.

Overrides ({field: ComparableField}) are honored verbatim at highest
precedence. Exported from top-level stickler. 23 new tests; full suite green.
@sromoam
refactor(auto): drop per-field overrides (top-level-only was half-wor…
b5a65ec 
…king)

The overrides={} param only matched top-level field names and silently
dropped nested / list-element keys (e.g. "lines.price"), which is a footgun.
Rather than ship a partial API, remove it entirely: evaluate/eval_for now
infer every field, with weight_hints as the only tuning knob.

A proper nested override API (dotted paths, model-list weight-only guard,
loud errors on unknown keys) can be added later; the InferredSpec config
dict remains the single contract a future producer would target.

Removes overrides from evaluate/eval_for/EvalSpec/structured_model_for,
the override provenance branch, and the override test. 22 auto tests pass;
full suite green.
@sromoam
docs(auto): ultra quick start page, notebook, and Strands example
a04ea53 
For the "I don't want to configure anything or read docs" user: show that
stickler.evaluate(gt, pred) scores a vanilla Pydantic model with zero setup,
and that .explain() makes every inferred decision defensible.

- docs: Getting-Started/ultra-quick-start.md (+ nav) — the one-line pitch,
  a full worked Invoice example (enum/date/optional/nested list), the
  inference cheat-sheet, and how to defend/override.
- docs: Guides/Use-Cases/evaluate-strands-agent.md (+ nav) — the whole
  integration for a Strands agent's structured_output, single-pair and
  dataset loops, and the [llm] extra note.
- examples/notebooks/Ultra_Quick_Start.ipynb — clean (output-free, per repo
  convention); every code cell verified to run end-to-end.
- examples/scripts/strands_agent_eval_demo.py — runs the real agent when
  [llm] + creds are present, else falls back to a fabricated prediction so
  it's always runnable.
- examples/README.md — surface both new artifacts.
@sromoam
docs(auto): soften ultra-quick-start tone, remove em dashes
a8feae4 
Drop the "I'm too lazy..." quote framing (reads as passive-aggressive to a
reader who didn't say it) in favor of a plain "evaluate as-is, no config
required" opener, in both the doc page and the notebook.

Remove all em dashes from the auto docs, notebook, examples, and package
docstrings/README per user preference; replace with commas, colons,
parentheses, or separate sentences.
@sromoam
docs(auto): remove em dashes hidden as unicode escapes in notebook
348a4d0 
The prior "remove em dashes" pass missed the notebook: JSON serializes
non-ascii as backslash-u escapes, so a literal-character grep reported it
clean when 13 em dashes were still present. Rewrote the affected markdown
cells with plain punctuation (also dropping the leftover "defend"/"stay
lazy" tone that was already softened in the doc page), and fixed the
cell-4 import ordering ruff flagged. Notebook re-verified: valid,
lint-clean, executes end-to-end.
@sromoam
docs: surface zero-config stickler.evaluate in root README
f9fa57d 
Add a short section after Installation so the vanilla-Pydantic evaluation
path is discoverable from the front door, linking to the Ultra Quick Start.
@sromoam
sromoam requested a review from adiadd July 23, 2026 16:26
@github-actions

github-actions Bot commented Jul 23, 2026

Copy link
Copy Markdown

Security Scan Results

ASH Security Scan Report

  • Report generated: 2026-07-23T16:26:18+00:00
  • Time since scan: 2 minutes

Scan Metadata

  • Project: ASH
  • Scan executed: 2026-07-23T16:23:51+00:00
  • ASH version: 3.1.2

Summary

Scanner Results

The table below shows findings by scanner, with status based on severity thresholds and dependencies:

  • Severity levels:
    • Suppressed (S): Findings that have been explicitly suppressed and don't affect scanner status
    • Critical (C): Highest severity findings that require immediate attention
    • High (H): Serious findings that should be addressed soon
    • Medium (M): Moderate risk findings
    • Low (L): Lower risk findings
    • Info (I): Informational findings with minimal risk
  • Duration (Time): Time taken by the scanner to complete its execution
  • Actionable: Number of findings at or above the threshold severity level that require attention
  • Result:
    • PASSED = No findings at or above threshold
    • FAILED = Findings at or above threshold
    • MISSING = Required dependencies not available
    • SKIPPED = Scanner explicitly disabled
    • ERROR = Scanner execution error
  • Threshold: The minimum severity level that will cause a scanner to fail
    • Thresholds: ALL, LOW, MEDIUM, HIGH, CRITICAL
    • Source: Values in parentheses indicate where the threshold is set:
      • global (global_settings section in the ASH_CONFIG used)
      • config (scanner config section in the ASH_CONFIG used)
      • scanner (default configuration in the plugin, if explicitly set)
  • Statistics calculation:
    • All statistics are calculated from the final aggregated SARIF report
    • Suppressed findings are counted separately and do not contribute to actionable findings
    • Scanner status is determined by comparing actionable findings to the threshold
Scanner Suppressed Critical High Medium Low Info Actionable Result Threshold
bandit 0 0 0 0 4165 0 0 PASSED MEDIUM (global)
cdk-nag 0 0 0 0 0 0 0 PASSED MEDIUM (global)
cfn-nag 0 0 0 0 0 0 0 MISSING MEDIUM (global)
checkov 0 0 0 0 0 0 0 PASSED MEDIUM (global)
detect-secrets 0 0 0 0 0 0 0 PASSED MEDIUM (global)
grype 0 0 0 0 0 0 0 MISSING MEDIUM (global)
npm-audit 0 0 0 0 0 0 0 PASSED MEDIUM (global)
opengrep 0 0 0 0 0 0 0 MISSING MEDIUM (global)
semgrep 0 19 0 0 0 0 19 FAILED MEDIUM (global)
syft 0 0 0 0 0 0 0 MISSING MEDIUM (global)

Top 7 Hotspots

Files with the highest number of security findings:

Finding Count File Location
7 .github/workflows/workflow.yml
3 .github/workflows/security.yaml
2 .github/workflows/docs.yml
2 .github/workflows/lint.yaml
2 .github/workflows/run_pytest.yaml
2 .github/workflows/security-pr-comment.yaml
1 pyproject.toml

Detailed Findings

Show 19 actionable findings

Finding 1: yaml.github-actions.security.github-actions-mutable-action-tag.github-actions-mutable-action-tag

  • Severity: HIGH
  • Scanner: semgrep
  • Rule ID: yaml.github-actions.security.github-actions-mutable-action-tag.github-actions-mutable-action-tag
  • Location: .github/workflows/docs.yml:19

Description:
GitHub Actions step uses a mutable tag or branch reference. Tags and branch names can be silently repointed by the action owner, enabling supply-chain attacks — as seen in the trivy-action and kics-github-action compromises. Pin the reference to a full 40-character commit SHA instead, e.g. uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608.

Code Snippet:

- uses: actions/checkout@v4

Finding 2: yaml.github-actions.security.github-actions-mutable-action-tag.github-actions-mutable-action-tag

  • Severity: HIGH
  • Scanner: semgrep
  • Rule ID: yaml.github-actions.security.github-actions-mutable-action-tag.github-actions-mutable-action-tag
  • Location: .github/workflows/docs.yml:22

Description:
GitHub Actions step uses a mutable tag or branch reference. Tags and branch names can be silently repointed by the action owner, enabling supply-chain attacks — as seen in the trivy-action and kics-github-action compromises. Pin the reference to a full 40-character commit SHA instead, e.g. uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608.

Code Snippet:

uses: astral-sh/setup-uv@v5

Finding 3: yaml.github-actions.security.github-actions-mutable-action-tag.github-actions-mutable-action-tag

  • Severity: HIGH
  • Scanner: semgrep
  • Rule ID: yaml.github-actions.security.github-actions-mutable-action-tag.github-actions-mutable-action-tag
  • Location: .github/workflows/lint.yaml:16

Description:
GitHub Actions step uses a mutable tag or branch reference. Tags and branch names can be silently repointed by the action owner, enabling supply-chain attacks — as seen in the trivy-action and kics-github-action compromises. Pin the reference to a full 40-character commit SHA instead, e.g. uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608.

Code Snippet:

- uses: actions/checkout@v4

Finding 4: yaml.github-actions.security.github-actions-mutable-action-tag.github-actions-mutable-action-tag

  • Severity: HIGH
  • Scanner: semgrep
  • Rule ID: yaml.github-actions.security.github-actions-mutable-action-tag.github-actions-mutable-action-tag
  • Location: .github/workflows/lint.yaml:18

Description:
GitHub Actions step uses a mutable tag or branch reference. Tags and branch names can be silently repointed by the action owner, enabling supply-chain attacks — as seen in the trivy-action and kics-github-action compromises. Pin the reference to a full 40-character commit SHA instead, e.g. uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608.

Code Snippet:

uses: astral-sh/setup-uv@v5

Finding 5: yaml.github-actions.security.github-actions-mutable-action-tag.github-actions-mutable-action-tag

  • Severity: HIGH
  • Scanner: semgrep
  • Rule ID: yaml.github-actions.security.github-actions-mutable-action-tag.github-actions-mutable-action-tag
  • Location: .github/workflows/run_pytest.yaml:16

Description:
GitHub Actions step uses a mutable tag or branch reference. Tags and branch names can be silently repointed by the action owner, enabling supply-chain attacks — as seen in the trivy-action and kics-github-action compromises. Pin the reference to a full 40-character commit SHA instead, e.g. uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608.

Code Snippet:

- uses: actions/checkout@v4

Finding 6: yaml.github-actions.security.github-actions-mutable-action-tag.github-actions-mutable-action-tag

  • Severity: HIGH
  • Scanner: semgrep
  • Rule ID: yaml.github-actions.security.github-actions-mutable-action-tag.github-actions-mutable-action-tag
  • Location: .github/workflows/run_pytest.yaml:18

Description:
GitHub Actions step uses a mutable tag or branch reference. Tags and branch names can be silently repointed by the action owner, enabling supply-chain attacks — as seen in the trivy-action and kics-github-action compromises. Pin the reference to a full 40-character commit SHA instead, e.g. uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608.

Code Snippet:

uses: astral-sh/setup-uv@v5

Finding 7: yaml.github-actions.security.github-actions-mutable-action-tag.github-actions-mutable-action-tag

  • Severity: HIGH
  • Scanner: semgrep
  • Rule ID: yaml.github-actions.security.github-actions-mutable-action-tag.github-actions-mutable-action-tag
  • Location: .github/workflows/security-pr-comment.yaml:20

Description:
GitHub Actions step uses a mutable tag or branch reference. Tags and branch names can be silently repointed by the action owner, enabling supply-chain attacks — as seen in the trivy-action and kics-github-action compromises. Pin the reference to a full 40-character commit SHA instead, e.g. uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608.

Code Snippet:

uses: actions/download-artifact@v4

Finding 8: yaml.github-actions.security.github-actions-mutable-action-tag.github-actions-mutable-action-tag

  • Severity: HIGH
  • Scanner: semgrep
  • Rule ID: yaml.github-actions.security.github-actions-mutable-action-tag.github-actions-mutable-action-tag
  • Location: .github/workflows/security-pr-comment.yaml:27

Description:
GitHub Actions step uses a mutable tag or branch reference. Tags and branch names can be silently repointed by the action owner, enabling supply-chain attacks — as seen in the trivy-action and kics-github-action compromises. Pin the reference to a full 40-character commit SHA instead, e.g. uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608.

Code Snippet:

uses: actions/github-script@v7

Finding 9: yaml.github-actions.security.github-actions-mutable-action-tag.github-actions-mutable-action-tag

  • Severity: HIGH
  • Scanner: semgrep
  • Rule ID: yaml.github-actions.security.github-actions-mutable-action-tag.github-actions-mutable-action-tag
  • Location: .github/workflows/security.yaml:18

Description:
GitHub Actions step uses a mutable tag or branch reference. Tags and branch names can be silently repointed by the action owner, enabling supply-chain attacks — as seen in the trivy-action and kics-github-action compromises. Pin the reference to a full 40-character commit SHA instead, e.g. uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608.

Code Snippet:

- uses: actions/checkout@v4

Finding 10: yaml.github-actions.security.github-actions-mutable-action-tag.github-actions-mutable-action-tag

  • Severity: HIGH
  • Scanner: semgrep
  • Rule ID: yaml.github-actions.security.github-actions-mutable-action-tag.github-actions-mutable-action-tag
  • Location: .github/workflows/security.yaml:20

Description:
GitHub Actions step uses a mutable tag or branch reference. Tags and branch names can be silently repointed by the action owner, enabling supply-chain attacks — as seen in the trivy-action and kics-github-action compromises. Pin the reference to a full 40-character commit SHA instead, e.g. uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608.

Code Snippet:

uses: astral-sh/setup-uv@v5

Finding 11: yaml.github-actions.security.github-actions-mutable-action-tag.github-actions-mutable-action-tag

  • Severity: HIGH
  • Scanner: semgrep
  • Rule ID: yaml.github-actions.security.github-actions-mutable-action-tag.github-actions-mutable-action-tag
  • Location: .github/workflows/security.yaml:42

Description:
GitHub Actions step uses a mutable tag or branch reference. Tags and branch names can be silently repointed by the action owner, enabling supply-chain attacks — as seen in the trivy-action and kics-github-action compromises. Pin the reference to a full 40-character commit SHA instead, e.g. uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608.

Code Snippet:

uses: actions/upload-artifact@v4

Finding 12: yaml.github-actions.security.github-actions-mutable-action-tag.github-actions-mutable-action-tag

  • Severity: HIGH
  • Scanner: semgrep
  • Rule ID: yaml.github-actions.security.github-actions-mutable-action-tag.github-actions-mutable-action-tag
  • Location: .github/workflows/workflow.yml:18

Description:
GitHub Actions step uses a mutable tag or branch reference. Tags and branch names can be silently repointed by the action owner, enabling supply-chain attacks — as seen in the trivy-action and kics-github-action compromises. Pin the reference to a full 40-character commit SHA instead, e.g. uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608.

Code Snippet:

- uses: actions/checkout@v4

Finding 13: yaml.github-actions.security.github-actions-mutable-action-tag.github-actions-mutable-action-tag

  • Severity: HIGH
  • Scanner: semgrep
  • Rule ID: yaml.github-actions.security.github-actions-mutable-action-tag.github-actions-mutable-action-tag
  • Location: .github/workflows/workflow.yml:22

Description:
GitHub Actions step uses a mutable tag or branch reference. Tags and branch names can be silently repointed by the action owner, enabling supply-chain attacks — as seen in the trivy-action and kics-github-action compromises. Pin the reference to a full 40-character commit SHA instead, e.g. uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608.

Code Snippet:

uses: astral-sh/setup-uv@v5

Finding 14: yaml.github-actions.security.github-actions-mutable-action-tag.github-actions-mutable-action-tag

  • Severity: HIGH
  • Scanner: semgrep
  • Rule ID: yaml.github-actions.security.github-actions-mutable-action-tag.github-actions-mutable-action-tag
  • Location: .github/workflows/workflow.yml:28

Description:
GitHub Actions step uses a mutable tag or branch reference. Tags and branch names can be silently repointed by the action owner, enabling supply-chain attacks — as seen in the trivy-action and kics-github-action compromises. Pin the reference to a full 40-character commit SHA instead, e.g. uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608.

Code Snippet:

uses: actions/upload-artifact@v4

Finding 15: yaml.github-actions.security.github-actions-mutable-action-tag.github-actions-mutable-action-tag

  • Severity: HIGH
  • Scanner: semgrep
  • Rule ID: yaml.github-actions.security.github-actions-mutable-action-tag.github-actions-mutable-action-tag
  • Location: .github/workflows/workflow.yml:45

Description:
GitHub Actions step uses a mutable tag or branch reference. Tags and branch names can be silently repointed by the action owner, enabling supply-chain attacks — as seen in the trivy-action and kics-github-action compromises. Pin the reference to a full 40-character commit SHA instead, e.g. uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608.

Code Snippet:

uses: actions/download-artifact@v4

Finding 16: yaml.github-actions.security.github-actions-mutable-action-tag.github-actions-mutable-action-tag

  • Severity: HIGH
  • Scanner: semgrep
  • Rule ID: yaml.github-actions.security.github-actions-mutable-action-tag.github-actions-mutable-action-tag
  • Location: .github/workflows/workflow.yml:50

Description:
GitHub Actions step uses a mutable tag or branch reference. Tags and branch names can be silently repointed by the action owner, enabling supply-chain attacks — as seen in the trivy-action and kics-github-action compromises. Pin the reference to a full 40-character commit SHA instead, e.g. uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608.

Code Snippet:

uses: pypa/gh-action-pypi-publish@release/v1

Finding 17: yaml.github-actions.security.github-actions-mutable-action-tag.github-actions-mutable-action-tag

  • Severity: HIGH
  • Scanner: semgrep
  • Rule ID: yaml.github-actions.security.github-actions-mutable-action-tag.github-actions-mutable-action-tag
  • Location: .github/workflows/workflow.yml:64

Description:
GitHub Actions step uses a mutable tag or branch reference. Tags and branch names can be silently repointed by the action owner, enabling supply-chain attacks — as seen in the trivy-action and kics-github-action compromises. Pin the reference to a full 40-character commit SHA instead, e.g. uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608.

Code Snippet:

uses: actions/download-artifact@v4

Finding 18: yaml.github-actions.security.github-actions-mutable-action-tag.github-actions-mutable-action-tag

  • Severity: HIGH
  • Scanner: semgrep
  • Rule ID: yaml.github-actions.security.github-actions-mutable-action-tag.github-actions-mutable-action-tag
  • Location: .github/workflows/workflow.yml:69

Description:
GitHub Actions step uses a mutable tag or branch reference. Tags and branch names can be silently repointed by the action owner, enabling supply-chain attacks — as seen in the trivy-action and kics-github-action compromises. Pin the reference to a full 40-character commit SHA instead, e.g. uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608.

Code Snippet:

uses: pypa/gh-action-pypi-publish@release/v1

Finding 19: package_managers.uv.uv-missing-dependency-cooldown.uv-missing-dependency-cooldown

  • Severity: HIGH
  • Scanner: semgrep
  • Rule ID: package_managers.uv.uv-missing-dependency-cooldown.uv-missing-dependency-cooldown
  • Location: pyproject.toml:116-117

Description:
This pyproject.toml configures uv but does not set a dependency cooldown. Newly published packages can be malicious or unstable. Add exclude-newer = "7 days" under [tool.uv] to wait 7 days before resolving newly published package versions. Added in: 0.9.17 Reference: https://docs.astral.sh/uv/concepts/resolution/#dependency-cooldowns

Code Snippet:

[tool.uv]
default-groups = ["dev"]

Report generated by Automated Security Helper (ASH) at 2026-07-23T16:26:19+00:00

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@adiadd adiadd Awaiting requested review from adiadd

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@sromoam