feat: add trust_remote_code option to AutoDocumentEncoder/AutoQueryEncoder - #2645
Open
St4r4x wants to merge 1 commit into
Open
feat: add trust_remote_code option to AutoDocumentEncoder/AutoQueryEncoder#2645St4r4x wants to merge 1 commit into
St4r4x wants to merge 1 commit into
Conversation
…coder Some HuggingFace models (e.g. nomic-ai/nomic-embed-text-v1) require trust_remote_code=True to load, which previously required patching pyserini or accepting a blocking Y/N terminal prompt every run. Threads an optional trust_remote_code=False parameter through to both AutoModel.from_pretrained() and load_auto_tokenizer() (which already forwards **kwargs to AutoTokenizer.from_pretrained(), so no change needed there). Defaults to False, matching current behavior.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2207
What
Adds an optional
trust_remote_code: bool = Falseparameter toAutoDocumentEncoderandAutoQueryEncoder, threaded through to bothAutoModel.from_pretrained()and the tokenizer loader. This lets models that require custom code on the Hub (e.g.nomic-ai/nomic-embed-text-v1, named in the issue) load without a blocking Y/N terminal prompt or needing to patch pyserini locally.Scope
Scoped to the two
Auto*Encoderclasses named in the issue — these are the genericAutoModel.from_pretrained(...)-based wrappers. Model-specific encoders elsewhere inpyserini/encode/(uniCOIL, SPLADE, DPR, etc.) load fixed, known architectures and don't need this knob.load_auto_tokenizer()in_base.pyalready forwards**kwargstoAutoTokenizer.from_pretrained(), so no change was needed there beyond passingtrust_remote_codethrough as a kwarg.Default is
False, so existing behavior is unchanged for everyone not opting in.Not included (happy to add if wanted)
Didn't wire this into the CLI scripts (
scripts/encode_queries.pyetc.) — kept the diff to the encoder classes themselves per the issue's literal ask. Let me know if CLI flag support would be useful too.Validation
Checked every call site of
AutoDocumentEncoder/AutoQueryEncoderin the repo (tests,pyserini/search/faiss/_searcher.py,pyserini/encode/query.py,pyserini/demo/miracl.py,scripts/encode_queries.py) — all use keyword arguments after the first positional param, so this is fully backward compatible. Didn't run the full test suite (several of these tests download real models), but reviewed the changed logic againstload_auto_tokenizer's existing kwarg-forwarding behavior to confirm the tokenizer side works without further changes.