fix: remove score validation bounds to support raw logits from cross-encoder#17
Conversation
…encoder - Remove ge=-1.0, le=1.0 constraints from RerankResult.score field - Allows torch CrossEncoder raw logits (e.g., 7.45, -2.3) to pass validation - Router layer applies sigmoid normalization when OPENAI_RERANK_AUTO_SIGMOID=true - Fixes bug where valid rerank scores were rejected before normalization - Restores documented behavior of auto-sigmoid feature
There was a problem hiding this comment.
Pull request overview
This PR fixes a validation error that prevented legitimate cross-encoder reranking models from functioning correctly. The previous code enforced artificial score bounds (-1.0 to 1.0) on the RerankResult.score field, which caused validation errors when cross-encoder models (like cross-encoder/ms-marco-MiniLM-L-6-v2) returned raw logits outside this range (e.g., 7.45 or -2.3).
Key Changes:
- Removes the
ge=-1.0, le=1.0constraints from theRerankResult.scorefield to allow raw logits - Updates the field description to clarify that scores can be raw logits or normalized values
- Minor formatting improvements to
ConfigDictdefinitions (multi-line format)
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
@fparrav Thanks for the PR! I reviewed the response model changes—removing the ge/le constraint on RerankResult.score makes sense now that scores can be raw logits or normalized differently, and it should prevent Pydantic validation errors when values fall outside [-1, 1]. Since this is limited to the response schema/description, the risk looks low. LGTM — approved. If it’s not already covered, a quick note in the API docs about whether scores are normalized would be helpful too.
Summary
ms-marco-MiniLM-L-6-v2)-1.0to1.0constraint on rerank scores that was blocking legitimate model outputsProblem
The current code enforces score bounds (
ge=-1.0, le=1.0) inRerankResult, which causes validation errors with standard cross-encoder models that output raw logits (e.g., scores like7.45or-2.3). This breaks reranking functionality with industry-standard models.Solution
ge=-1.0, le=1.0constraints fromRerankResult.scorefieldOPENAI_RERANK_AUTO_SIGMOID=trueTesting
Tested with
cross-encoder/ms-marco-MiniLM-L-6-v2on macOS Apple Silicon:Impact