Skip to content

compliance: scope edge-agentic checker (rename to edge_agentic.py, required ruleset/model args) #397

Description

@Palanivelg

Follow-up to PR #346 (thread #346 (comment)), flagged by @viraatc as fine to address in a follow-up.

compliance/checker.py is edge-agentic-specific (BFCL metric keys, qwen3.6-27b / mlperf-edge-current defaults, hardcoded attestation note) under a generic top-level compliance/ name.

Proposed:

  • Rename to compliance/edge_agentic.py (or move next to the edge ruleset).
  • Make ruleset_name/model_name required args instead of defaults.
  • Name the model's golden_accuracy keys to match the scorer's breakdown keys (overall_accuracy, normalized_single_turn_score) so the _ACCURACY_METRIC_KEYS mapping disappears.
  • Generalizing into a real framework can wait until a second benchmark needs one; coordinate with feat(compliance): MLPerf TEST04 caching audit — extensible AuditTest framework #332.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions