Skip to content

fix(embedding): bound vectorizer text input - #3015

Closed
r266-tech wants to merge 1 commit into
volcengine:mainfrom
r266-tech:r266/bound-vectorizer-text
Closed

fix(embedding): bound vectorizer text input#3015
r266-tech wants to merge 1 commit into
volcengine:mainfrom
r266-tech:r266/bound-vectorizer-text

Conversation

@r266-tech

Copy link
Copy Markdown
Contributor

Summary

Fixes #3010.

Collection vectorization currently forwards the full text field to the provider. If a file is larger than the embedding model accepts, the provider rejects the request and the file never gets a vector.

This PR applies the same configured input-bound idea at the VectorizerAdapter boundary:

  • reads max_input_tokens from the vectorizer config when present
  • bounds only provider-facing text before vectorize_document
  • leaves the raw record data untouched, so stored fields/full-text indexing still keep the original content
  • preserves image/video payloads
  • treats missing/disabled/malformed limits as no limit, with warnings for malformed values
  • fails closed before the provider call if the configured truncation helper is unavailable or raises

One tradeoff: dense and sparse provider generation share one text field in the current vectorizer API, so a configured cap bounds that shared provider input. The raw content field is still persisted unchanged.

Tests

  • ruff check openviking/storage/vectordb/vectorize/vectorizer.py tests/unit/test_vectorize_file_strategy.py
  • ruff format --check openviking/storage/vectordb/vectorize/vectorizer.py tests/unit/test_vectorize_file_strategy.py
  • python3 -m py_compile openviking/storage/vectordb/vectorize/vectorizer.py tests/unit/test_vectorize_file_strategy.py
  • direct adapter smoke check for configured cap behavior

I could not run PYTHONPATH=. pytest tests/unit/test_vectorize_file_strategy.py -q -o addopts='' in this unsynced local environment because collection imports require declared project dependencies such as json_repair that are not installed here.

Codex review: final pass approved after reworking config handling, fail-closed truncation paths, vectorize_one coverage, and media-field preservation.

@qin-ctx

qin-ctx commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

fixed by #3024

@qin-ctx qin-ctx closed this Jul 6, 2026
@github-project-automation github-project-automation Bot moved this from Backlog to Done in OpenViking project Jul 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

[Bug] Embedding pipeline does not chunk oversized files — files exceeding embedding model ctx-size fail silently

2 participants