Release 0.2.0 with schema-enforced judge responses - #6
Merged
Conversation
Judge outputs now use RubyLLM structured response schemas when available, with local validation rejecting malformed payloads and invalid score contracts. The version and changelog move this into 0.2.0 because callers can now receive JudgeError where bad model output previously became nil or was clamped. Constraint: GitHub issue #3 requires schema-backed judge responses and explicit failures for malformed, missing-score, and out-of-range responses Rejected: Keep clamping out-of-range scores | hides judge contract violations that should fail loudly Confidence: high Scope-risk: moderate Directive: Do not reintroduce nil-on-malformed judge output without updating the issue #3 contract tests Tested: bundle exec rake; gem build rubric_llm.gemspec; git diff --check Not-tested: Live provider structured-output API call Related: GH-3
The named schema wrapper passed to chat.with_schema is only normalized correctly by ruby_llm >= 1.13; older 1.x versions embed the wrapper verbatim as the schema definition. Also corrects the 0.2.0 changelog date. CodeFactory builder
ruby_llm >= 1.13 auto-parses response content into a Hash when a schema is set, so parse_json crashed on every valid schema-enforced response. Pass Hash content directly to validation and make the test stub emulate ruby_llm's schema response parsing. CodeFactory builder
CodeFactory builder
Judge#call now guarantees a validated Hash or raises JudgeError, so the metric-level nil guards and score clamping were unreachable. Narrow the evaluator rescue to JudgeError: judge failures still record nil scores with error details so batch runs continue, but programming errors now propagate instead of hiding as nil scores. CodeFactory builder
dpaluy
force-pushed
the
cdx/release-schema-enforced-judge-responses
branch
from
July 11, 2026 21:04
a18af76 to
2cae472
Compare
Newer rubocop on CI flags the any?/include? pattern. Use Set#intersect? with relevant as receiver since Array#intersect? rejects the Set argument. CodeFactory builder
dpaluy
force-pushed
the
cdx/release-schema-enforced-judge-responses
branch
from
July 11, 2026 21:05
2cae472 to
355c713
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
RubricLLM::VERSIONfrom0.1.2to0.2.0.0.2.0changelog notes for schema-backed judge responses and explicit judge contract failures.RubricLLM::JudgeErrorfor malformed, missing-score, non-numeric, or out-of-range judge responses.Why
Judge scoring should be governed by an API/schema contract, not prompt compliance alone. This release makes invalid judge responses fail explicitly instead of becoming nil scores or clamped values.
Changes
release: surface judge contract failures in 0.2.0
chat.with_schema, validates parsed judge responses, updates tests, bumps the gem version, and documents the release inCHANGELOG.md.Testing
bundle exec rake testbundle exec rubocopbundle exec rakegem build rubric_llm.gemspecgit diff --checkRisk & Rollout
RubricLLM::JudgeErrorfailures.e404b72to restore the previous parsing and version/changelog state.JudgeErrorfailures from providers that return malformed or out-of-range payloads.Closes #3