Skip to content

Add a LocalModelEngine conformance harness #4

Description

@AIyahFun

Create a runtime-neutral conformance harness for LocalModelEngine adapters.

The harness should measure skill selection, tool selection, argument validity, clarification behavior, cancellation, and unsafe-call rate without executing tools automatically.

Acceptance criteria:

  • Uses only repository evaluation fixtures.
  • Reports each metric separately.
  • Rejects invalid structured output instead of repairing it into an action.
  • Stores no prompt or response content in default logs.
  • Includes a small fake-engine implementation for tests.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:contractsPublic contracts and generated SDK typesevaluationReproducible model or hardware evaluationhelp wantedExtra attention is neededsafetySecurity, privacy, or medical safety boundary

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions