feat(rerank): portable score_threshold via opt-in score normalization

[chethanuk]

Summary

RerankConfig.threshold defaults to 0.1 and is applied as score > threshold. That assumes reranker scores live in [0, 1] (VikingDB/Cohere) — but cross-encoder rerankers (BGE/MiniLM served via OpenAI-compatible / LiteLLM endpoints) return raw, unbounded logits, for which a fixed 0.1 threshold is meaningless (it filters almost everything or nothing depending on the model). The threshold is not portable across providers.

Adds an opt-in RerankConfig.normalize_scores (default False). When enabled, each reranker score is mapped through an overflow-safe logistic (sigmoid) into [0, 1] before thresholding and blending, so a threshold in [0, 1] means the same thing everywhere.

How it works

flowchart TD
    A["rerank_batch → raw score per doc"] --> B{"normalize_scores?"}
    B -- "False (default) → parity" --> C["use raw score<br/>logit space, may be unbounded"]
    B -- "True (opt-in)" --> D["sigmoid(score) → [0, 1]<br/>overflow-safe, monotonic"]
    C --> E{"score &gt; threshold"}
    D --> E
    E -- keep --> F["directory retained + blended"]
    E -- drop --> G["filtered out"]

Loading

With normalize_scores=True the threshold is portable in [0, 1] across providers; with it off, threshold lives in the provider's native score space (bounded for VikingDB/Cohere, unbounded logits for logit-based cross-encoders).

Why opt-in (not auto-detected)

Whether a reranker returns a logit or a probability is model-dependent, not provider-dependent — the same OpenAI-compatible endpoint can serve either. So there is no safe way to auto-detect it; normalization is an explicit switch. (Leave it off for providers that already return normalized [0, 1] scores, or double-normalization distorts them.)

Correctness

• default=False = byte-identical parity with the prior _finite_score(score, fallback) loop across every branch (valid / NaN / non-finite / unparseable scores).
• Sigmoid is strictly monotonic, so it only recalibrates the threshold/blend — it never reorders results.
• The transform applies to genuine reranker scores only; a per-doc fallback substitution (invalid rerank score → vector score) is already in score space and is left untouched.
• _sigmoid is overflow-safe for extreme logits (±1000) — the exponent is always ≤ 0. Output is [0, 1]: endpoints are reachable for extreme logits via floating-point saturation.
• Field(default=False, strict=True); extra="forbid" keeps typos fatal.

Tests / docs

Covers parity-off, sigmoid mapping, overflow safety, fallback-not-transformed, and config validation. en + zh rerank-table rows added, threshold-range semantics documented for both normalization states.

Review

Eight inline review-bot comments were triaged and independently verified. Valid ones fixed (sigmoid range wording (0,1)→[0,1], normalize_scores added to the full-schema example, threshold-range docs, VikingDB double-normalize caveat); two over-reaching suggestions declined with rationale — a per-provider raise (contradicts the model-dependent design) and a hot-path sentinel refactor for an exact-0.0 sigmoid output that requires a logit ≲ −745 (unreachable for real rerankers). No executable code changed by the review round.

Independent of the max_chars_per_doc ladder — this fixes a pre-existing threshold-portability bug and branches off main. Part of #2880.

[chethanuk]

Please review cc: @qin-ctx @ZaynJarvis @zhoujh01 @yufeng201 @chenjw - Please let me know if you require any changes, or if the project is currently accepting contributions