Surface rerankers and modernize Hugging Face integration pages

[Tom Aarsen (tomaarsen)]

Overview

• Modernize the Sentence Transformers and Hugging Face embedding and cross-encoder reranker integration pages, fixing deprecated langchain_community imports and stale 2023-era model defaults
• Surface HuggingFace local models where they're currently under-represented: add HuggingFaceEmbeddings to the Python "Top integrations" table, add a new shared reranker-tabs snippet, and add a brand-new document_transformers/index.mdx with a reranker comparison table (previously that entire directory had no index page and was completely absent from the sidebar navigation)
• Add a new "Choose an embedding model" conceptual guide and a reranking section in the flagship RAG tutorial

Type of change

Type: New and updated documentation page

Related issues/PRs

N/A

Checklist

• I have read the contributing guidelines, including the language policy
• I have tested my changes locally using docs dev
• All code examples have been tested and work correctly
• I have used root relative paths for internal links
• I have updated navigation in src/docs.json if needed

I couldn't run make broken-links locally, but did run make lint_prose.

Additional notes

I'm the maintainer of Sentence Transformers, and I'm pitching this PR with two main goals.

The first is to highlight rerankers more across the docs. The flagship RAG tutorial never mentioned them, even though reranking is probably the cheapest retrieval-quality win available on top of vector search. The entire document_transformers/ directory had no index page and was completely absent from the sidebar navigation, so pages like cross_encoder_reranker.mdx were only reachable via direct URL or cross-link. And cross_encoder_reranker.mdx itself was using a 2023-era model default.

The second is to update the Sentence Transformers / Hugging Face integration pages. Several still imported from deprecated langchain_community paths (including HuggingFaceInferenceAPIEmbeddings, which has been explicitly deprecated since langchain-community==0.2.2), the default model IDs were a couple of years stale, and the main sentence_transformers.mdx page opened with a red "experienced users only" warning that the current pip install langchain-huggingface has made obsolete.

For context: the prose and code changes here were drafted with AI assistance and then critically edited by me before opening the PR. Every claim about the current state of the code, every recommended model, and every cross-link was checked against the repo and against the live PyPI packages (langchain-huggingface, langchain-community, langchain-classic), and the model recommendations are based on my own experience with the Sentence Transformers ecosystem and MTEB (e.g. the newly released lightonai/DenseOn model).

Concrete changes:

• New src/oss/python/integrations/document_transformers/index.mdx with a reranker comparison table (HuggingFace first, with a "local" column), plus a content-transformation section and a card grid of all 21 pages. Registered in the "Integrations by component" sidebar group.
• New "Improve retrieval with reranking" section in rag.mdx, backed by a new reranker-tabs-py.mdx snippet covering HuggingFace (local, no API key), Cohere, Jina, Voyage AI, and FlashRank.
• cross_encoder_reranker.mdx: default bumped to BAAI/bge-reranker-v2-m3, plus a model-selection table covering mixedbread-ai/mxbai-rerank-large-v2, Alibaba-NLP/gte-multilingual-reranker-base, Qwen/Qwen3-Reranker-0.6B, and the classic cross-encoder/ms-marco-* family.
• sentence_transformers.mdx: warning removed; now covers device/throughput (with ST's auto-selection noted), normalization, batch sizes, query/document prompts for instruction-aware models, and Text Embeddings Inference for production.
• bge_huggingface.mdx restructured by model generation: BAAI/bge-m3 via HuggingFaceEmbeddings as the recommended path, bge-*-en-v1.5 via HuggingFaceBgeEmbeddings as a zero-config alternative. Also caught that the legacy class defaults to "Represent this question..." while the v1.5 model card actually recommends "Represent this sentence..."; the generic-class example uses the model-card wording.
• Migrated deprecated imports on providers/huggingface.mdx, instruct_embeddings.mdx, and huggingfacehub.mdx, with short deprecation notes (including the explicit HuggingFaceInferenceAPIEmbeddings deprecation).
• New choosing-embeddings.mdx: provider-neutral conceptual guide covering deployment patterns, MTEB, cost/latency/dimensionality/context-length trade-offs, multilingual support, prompts, licensing, and rerankers / hybrid / late-interaction (including lightonai/DenseOn).
• Hygiene: added HuggingFaceEmbeddings to the Python embeddings/index.mdx "Top integrations" table; bumped infinity_rerank.mdx and volcengine_rerank.mdx embedding examples from all-MiniLM-L6-v2 to BAAI/bge-m3.

I also ran the dev server and browsed each changed page end-to-end before opening this. Happy to pare back or split if you'd prefer. Let me know. I think it'll be nice to display the rerankers a tad more prominently in the docs via this.

• Tom Aarsen

[github-actions]

Thanks for opening a docs PR, Tom Aarsen (@tomaarsen)! When it's ready for review, please add the relevant reviewers:

• @npentrel or @lnhsingh (LangChain)
• @mdrxy (Python integrations)

[Tom Aarsen (tomaarsen)]

I'm glad to leave these changes be, but I get lots of questions about choosing embedding models personally, so I think this kind of page can be very helpful.

[Tom Aarsen (tomaarsen)]

The crux of the PR: showing rerankers more prominently, they're very strong, but a bit underutilized/underdocumented right now.