Skip to content

Conversation

@fparrav
Copy link
Contributor

@fparrav fparrav commented Dec 15, 2025

Summary

  • Fixes validation error when using cross-encoder models that return raw logits (e.g., ms-marco-MiniLM-L-6-v2)
  • Removes artificial -1.0 to 1.0 constraint on rerank scores that was blocking legitimate model outputs

Problem

The current code enforces score bounds (ge=-1.0, le=1.0) in RerankResult, which causes validation errors with standard cross-encoder models that output raw logits (e.g., scores like 7.45 or -2.3). This breaks reranking functionality with industry-standard models.

Solution

  • Remove ge=-1.0, le=1.0 constraints from RerankResult.score field
  • Allow models to return their natural output format (raw logits)
  • Normalization (sigmoid) is already handled at the router layer when OPENAI_RERANK_AUTO_SIGMOID=true

Testing

Tested with cross-encoder/ms-marco-MiniLM-L-6-v2 on macOS Apple Silicon:

  • Before: Validation error on scores > 1.0
  • After: Scores pass validation, normalization applied correctly at router level

Impact

  • Unblocks compatibility with most sentence-transformers cross-encoder models
  • Restores documented behavior of auto-sigmoid feature
  • No breaking changes (only removes overly restrictive validation)

…encoder

- Remove ge=-1.0, le=1.0 constraints from RerankResult.score field
- Allows torch CrossEncoder raw logits (e.g., 7.45, -2.3) to pass validation
- Router layer applies sigmoid normalization when OPENAI_RERANK_AUTO_SIGMOID=true
- Fixes bug where valid rerank scores were rejected before normalization
- Restores documented behavior of auto-sigmoid feature
Copilot AI review requested due to automatic review settings December 15, 2025 16:24
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a validation error that prevented legitimate cross-encoder reranking models from functioning correctly. The previous code enforced artificial score bounds (-1.0 to 1.0) on the RerankResult.score field, which caused validation errors when cross-encoder models (like cross-encoder/ms-marco-MiniLM-L-6-v2) returned raw logits outside this range (e.g., 7.45 or -2.3).

Key Changes:

  • Removes the ge=-1.0, le=1.0 constraints from the RerankResult.score field to allow raw logits
  • Updates the field description to clarify that scores can be raw logits or normalized values
  • Minor formatting improvements to ConfigDict definitions (multi-line format)

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copy link
Owner

@joonsoome joonsoome left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fparrav Thanks for the PR! I reviewed the response model changes—removing the ge/le constraint on RerankResult.score makes sense now that scores can be raw logits or normalized differently, and it should prevent Pydantic validation errors when values fall outside [-1, 1]. Since this is limited to the response schema/description, the risk looks low. LGTM — approved. If it’s not already covered, a quick note in the API docs about whether scores are normalized would be helpful too.

@joonsoome joonsoome merged commit 18a792d into joonsoome:main Dec 23, 2025
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants