fix(json-schema): annotate optional fields as Optional[T] so from_json accepts None

[valter-silva-au]

Issue #, if available: #149

Supersedes #150 (same work; branch renamed to follow a type/slug convention, so the
PR had to be reopened — GitHub can't repoint an open PR's head branch). Carries the
rebase onto current dev plus the changes from @adiadd's review on #150.

Problem

from_json(process_rich_values=True) rejects None for optional fields that plain ModelClass(**data) accepts. A model built from a JSON Schema raises pydantic ValidationError ("Input should be a valid string ... input_value=None") when an optional field is omitted and rich-value processing is on.

Root cause

When a JSON-Schema field is optional (absent from required), from_json_schema() builds a Pydantic field with default=None but a non-nullable annotation (bare str, not Optional[str]). Plain construction tolerates the omitted field, but the rich-value path round-trips nested objects through from_json(...).model_dump() (ConfigurationHelper._process_nested_structured_data). model_dump() materializes the None default, and re-validation checks it against the bare annotation — which fails. The default-vs-annotation mismatch is the bug.

Fix

The core fix widens an optional field's annotation to Optional[T] so the None default is valid. Following @adiadd's review on #150, the widening is made consistent across every producer and consumer of the annotation:

• Producer (single site). The three widening blocks in json_schema_field_converter.py collapse into the sole dispatcher convert_property_to_field — the convergence point with fix: support JSON Schema list-form nullable types #127. The condition is not is_required or default is None, so a required field that declares an explicit "default": null is also widened (the invariant is "a None default needs a nullable annotation").
• Comparison path. ConfigurationHelper.is_structured_field_type now recognizes Optional[StructuredModel] (reusing the existing _is_optional_structured_model). Without this, an optional nested object inside a list-of-objects item was routed down the non-hierarchical path and silently lost its nested metric breakdown in compare_recursive — exactly the population [BUG] from_json(process_rich_values=True) rejects None for optional fields that plain ModelClass(**data) accepts #149 targets.
• Stickler-config path. field_converter.py (used by model_from_json) widens too, closing a round-trip hole: schema → to_stickler_config → model_from_json previously rebuilt a fixed model back into a broken one.
• HTML report. data_extractors.py threshold extraction unwraps Optional[...] so nested field thresholds no longer vanish from HTML reports for schema-built optional models.

Required fields keep bare annotations.

Contract note (intentional, documented)

Widening optional fields to Optional[T] means an explicit null is now accepted for an optional typed field (e.g. {"note": null} for an optional string), where it previously raised, and model_json_schema() emits anyOf: [T, null] for such fields. This is deliberate — model predictions emit explicit nulls and should score as a value rather than crash evaluation. It is documented in docs/docs/Advanced/dynamic-models.md and pinned by tests.

Testing

• TestOptionalFieldNoneFromJson — nested/primitive/array optional fields round-trip through both M(**data) and M.from_json(data, process_rich_values=True). The primitive/array cases are nested one level down so they exercise the real re-validation path (the previous top-level versions passed even without the fix; per @adiadd they did not pin the bug). Includes a required-field-with-"default": null case.
• TestOptionalNestedBreakdownPreserved — compare_recursive keeps the nested fields breakdown for an optional nested object inside a list of objects.
• TestOptionalFieldExplicitNullContract — explicit-null acceptance through both construction paths + the anyOf:[T,null] schema shape.
• New test_configuration_helper_optional_structured.py, test_field_converter_optional_roundtrip.py, test_data_extractors_optional_thresholds.py for the comparison, stickler-config round-trip, and HTML-threshold fixes respectively.
• Tidied the review nits (redundant assert, obscure __import__ lambda, stale round-trip comment, helper pointer).

Full suite: 1443 passed, 2 skipped (pre-existing skips), 0 failures. ruff check clean.

Risk

Low–medium. The changes are additive — they teach consumers to recognize a shape (Optional[T]) that was already half-handled elsewhere, and bring annotations into agreement with the None defaults that were already being assigned. The single-site refactor is behavior-identical (verified by the existing converter tests).

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[github-actions]

Security Scan Results

ASH Security Scan Report

• Report generated: 2026-06-27T08:26:59+00:00
• Time since scan: 1 minute

Scan Metadata

• Project: ASH
• Scan executed: 2026-06-27T08:25:13+00:00
• ASH version: 3.1.2

Summary

Scanner Results

The table below shows findings by scanner, with status based on severity thresholds and dependencies:

• Severity levels:
  • Suppressed (S): Findings that have been explicitly suppressed and don't affect scanner status
  • Critical (C): Highest severity findings that require immediate attention
  • High (H): Serious findings that should be addressed soon
  • Medium (M): Moderate risk findings
  • Low (L): Lower risk findings
  • Info (I): Informational findings with minimal risk
• Duration (Time): Time taken by the scanner to complete its execution
• Actionable: Number of findings at or above the threshold severity level that require attention
• Result:
  • PASSED = No findings at or above threshold
  • FAILED = Findings at or above threshold
  • MISSING = Required dependencies not available
  • SKIPPED = Scanner explicitly disabled
  • ERROR = Scanner execution error
• Threshold: The minimum severity level that will cause a scanner to fail
  • Thresholds: ALL, LOW, MEDIUM, HIGH, CRITICAL
  • Source: Values in parentheses indicate where the threshold is set:
    • global (global_settings section in the ASH_CONFIG used)
    • config (scanner config section in the ASH_CONFIG used)
    • scanner (default configuration in the plugin, if explicitly set)
• Statistics calculation:
  • All statistics are calculated from the final aggregated SARIF report
  • Suppressed findings are counted separately and do not contribute to actionable findings
  • Scanner status is determined by comparing actionable findings to the threshold

Scanner	Suppressed	Critical	High	Medium	Low	Info	Actionable	Result	Threshold
bandit	0	1	0	0	4182	0	1	FAILED	MEDIUM (global)
cdk-nag	0	0	0	0	0	0	0	PASSED	MEDIUM (global)
cfn-nag	0	0	0	0	0	0	0	MISSING	MEDIUM (global)
checkov	0	1	0	0	0	0	1	SKIPPED	MEDIUM (global)
detect-secrets	0	0	0	0	0	0	0	PASSED	MEDIUM (global)
grype	0	0	0	0	0	0	0	MISSING	MEDIUM (global)
npm-audit	0	0	0	0	0	0	0	PASSED	MEDIUM (global)
opengrep	0	0	0	0	0	0	0	MISSING	MEDIUM (global)
semgrep	0	0	0	0	0	0	0	PASSED	MEDIUM (global)
syft	0	0	0	0	0	0	0	MISSING	MEDIUM (global)

Top 2 Hotspots

Files with the highest number of security findings:

Finding Count	File Location
1	.github/workflows/workflow.yml
1	tests/test_top_level_exports.py

Detailed Findings

Show 2 actionable findings

Finding 1: CKV2_GHA_1

• Severity: HIGH
• Scanner: checkov
• Rule ID: CKV2_GHA_1
• Location: .github/workflows/workflow.yml:11-12

Description:
Ensure top-level permissions are not set to write-all

---

Finding 2: B102

• Severity: HIGH
• Scanner: bandit
• Rule ID: B102
• Location: tests/test_top_level_exports.py:71-73

Description:
Use of exec detected.

Code Snippet:

ns = {}
        exec("from stickler import *", ns)  # noqa: S102
        exported_names = set(ns) - {"__builtins__"}

---

Report generated by Automated Security Helper (ASH) at 2026-06-27T08:27:00+00:00