fix(embedding): bound vectorizer text input - #3015
Closed
r266-tech wants to merge 1 commit into
Closed
Conversation
Collaborator
|
fixed by #3024 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #3010.
Collection vectorization currently forwards the full text field to the provider. If a file is larger than the embedding model accepts, the provider rejects the request and the file never gets a vector.
This PR applies the same configured input-bound idea at the
VectorizerAdapterboundary:max_input_tokensfrom the vectorizer config when presentvectorize_documentOne tradeoff: dense and sparse provider generation share one
textfield in the current vectorizer API, so a configured cap bounds that shared provider input. The raw content field is still persisted unchanged.Tests
ruff check openviking/storage/vectordb/vectorize/vectorizer.py tests/unit/test_vectorize_file_strategy.pyruff format --check openviking/storage/vectordb/vectorize/vectorizer.py tests/unit/test_vectorize_file_strategy.pypython3 -m py_compile openviking/storage/vectordb/vectorize/vectorizer.py tests/unit/test_vectorize_file_strategy.pyI could not run
PYTHONPATH=. pytest tests/unit/test_vectorize_file_strategy.py -q -o addopts=''in this unsynced local environment because collection imports require declared project dependencies such asjson_repairthat are not installed here.Codex review: final pass approved after reworking config handling, fail-closed truncation paths, vectorize_one coverage, and media-field preservation.