Skip to content

Release 0.2.0 with schema-enforced judge responses - #6

Merged
dpaluy merged 6 commits into
masterfrom
cdx/release-schema-enforced-judge-responses
Jul 11, 2026
Merged

Release 0.2.0 with schema-enforced judge responses#6
dpaluy merged 6 commits into
masterfrom
cdx/release-schema-enforced-judge-responses

Conversation

@dpaluy

@dpaluy dpaluy commented Apr 30, 2026

Copy link
Copy Markdown
Owner

Summary

  • Bumps RubricLLM::VERSION from 0.1.2 to 0.2.0.
  • Adds 0.2.0 changelog notes for schema-backed judge responses and explicit judge contract failures.
  • Applies RubyLLM structured output schema enforcement when available.
  • Raises RubricLLM::JudgeError for malformed, missing-score, non-numeric, or out-of-range judge responses.
  • Adds offline regression coverage for schema usage and judge response contract failures.

Why

Judge scoring should be governed by an API/schema contract, not prompt compliance alone. This release makes invalid judge responses fail explicitly instead of becoming nil scores or clamped values.

Changes

release: surface judge contract failures in 0.2.0

  • What: Adds a shared metric response schema, applies it through chat.with_schema, validates parsed judge responses, updates tests, bumps the gem version, and documents the release in CHANGELOG.md.
  • Why: GitHub issue Schema-enforce judge responses #3 requires schema-backed judge responses and explicit failures for invalid judge contracts.

Testing

  • bundle exec rake test
  • bundle exec rubocop
  • bundle exec rake
  • gem build rubric_llm.gemspec
  • git diff --check

Risk & Rollout

  • Risk: Medium - this intentionally changes invalid judge outputs from nil/clamped results into RubricLLM::JudgeError failures.
  • Rollback: Revert e404b72 to restore the previous parsing and version/changelog state.
  • Monitoring: Watch downstream evaluation/test integrations for newly surfaced JudgeError failures from providers that return malformed or out-of-range payloads.

Closes #3

dpaluy added 5 commits April 30, 2026 00:56
Judge outputs now use RubyLLM structured response schemas when available, with local validation rejecting malformed payloads and invalid score contracts. The version and changelog move this into 0.2.0 because callers can now receive JudgeError where bad model output previously became nil or was clamped.

Constraint: GitHub issue #3 requires schema-backed judge responses and explicit failures for malformed, missing-score, and out-of-range responses
Rejected: Keep clamping out-of-range scores | hides judge contract violations that should fail loudly
Confidence: high
Scope-risk: moderate
Directive: Do not reintroduce nil-on-malformed judge output without updating the issue #3 contract tests
Tested: bundle exec rake; gem build rubric_llm.gemspec; git diff --check
Not-tested: Live provider structured-output API call
Related: GH-3
The named schema wrapper passed to chat.with_schema is only
normalized correctly by ruby_llm >= 1.13; older 1.x versions embed
the wrapper verbatim as the schema definition. Also corrects the
0.2.0 changelog date.

CodeFactory builder
ruby_llm >= 1.13 auto-parses response content into a Hash when a
schema is set, so parse_json crashed on every valid schema-enforced
response. Pass Hash content directly to validation and make the
test stub emulate ruby_llm's schema response parsing.

CodeFactory builder
Judge#call now guarantees a validated Hash or raises JudgeError, so
the metric-level nil guards and score clamping were unreachable.
Narrow the evaluator rescue to JudgeError: judge failures still
record nil scores with error details so batch runs continue, but
programming errors now propagate instead of hiding as nil scores.

CodeFactory builder
@dpaluy
dpaluy force-pushed the cdx/release-schema-enforced-judge-responses branch from a18af76 to 2cae472 Compare July 11, 2026 21:04
Newer rubocop on CI flags the any?/include? pattern. Use
Set#intersect? with relevant as receiver since Array#intersect?
rejects the Set argument.

CodeFactory builder
@dpaluy
dpaluy force-pushed the cdx/release-schema-enforced-judge-responses branch from 2cae472 to 355c713 Compare July 11, 2026 21:05
@dpaluy
dpaluy merged commit a96e4bf into master Jul 11, 2026
2 checks passed
@dpaluy
dpaluy deleted the cdx/release-schema-enforced-judge-responses branch July 11, 2026 21:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Schema-enforce judge responses

1 participant