Add neural search ingest processors and neural_sparse query schemas - #1191
Open
Hailong-am wants to merge 5 commits into
Open
Add neural search ingest processors and neural_sparse query schemas#1191Hailong-am wants to merge 5 commits into
Hailong-am wants to merge 5 commits into
Conversation
Hailong-am
requested review from
Tokesh,
VachaShah,
Xtansia,
aabeshov,
harshavamsi,
karenyrx,
lucy66hw,
sachetalva and
sean-
as code owners
August 12, 2026 14:55
Hailong-am
force-pushed
the
feat/ingest-neural-processors
branch
from
August 12, 2026 15:23
5d881cb to
fab6307
Compare
…g processor schemas Add neural search ingest processor schemas to ingest._common.yaml: - SparseEncodingProcessor: sparse vector (token-weight) generation for neural sparse search. Fields: model_id (required), field_map (required), batch_size. - TextImageEmbeddingProcessor: multimodal (text+image) embeddings. Fields: model_id (required), field_map (required). - TextChunkingProcessor: splits long text into chunks before embedding. Fields: algorithm (required, ChunkingAlgorithm oneOf), field_map (required), max_chunk_limit. Supporting schemas: - ChunkingAlgorithm: oneOf FixedTokenLengthChunkingAlgorithm | DelimiterChunkingAlgorithm - FixedTokenLengthChunkingAlgorithm: token_limit, overlap_rate, tokenizer, max_chunk_limit - DelimiterChunkingAlgorithm: delimiter, max_chunk_limit All three processors are registered in ProcessorContainer alongside the existing text_embedding processor. Signed-off-by: Hailong Cui <ihailong@amazon.com>
Hailong-am
force-pushed
the
feat/ingest-neural-processors
branch
from
August 12, 2026 15:33
b7b6b72 to
98756b6
Compare
Signed-off-by: Hailong Cui <ihailong@amazon.com>
The sparse encoding model (opensearch-neural-sparse-encoding-doc-v1) takes too long to register in CI, causing timeout failures. The existing neural_search.yaml test already validates the ingest pipeline PUT schema with a text_embedding processor. The sparse_encoding processor schema is validated structurally via lint and unit tests. Signed-off-by: Hailong Cui <ihailong@amazon.com>
Signed-off-by: Hailong Cui <ihailong@amazon.com>
Hailong-am
changed the title
Contributor
Author
|
@harshavamsi can you help to review it? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds missing neural search schemas to the OpenAPI specification:
Ingest Processors (added to
ProcessorContainer)sparse_encodingmodel_id,field_map,batch_sizetext_image_embeddingmodel_id,field_maptext_chunkingalgorithm,field_map,max_chunk_limitChunking Algorithm Schemas
ChunkingAlgorithm— wrapper withfixed_token_lengthordelimiterkeyFixedTokenLengthChunkingAlgorithm—token_limit,overlap_rate,tokenizer,max_chunk_limitDelimiterChunkingAlgorithm—delimiter,max_chunk_limitQuery DSL:
neural_sparseAdded
NeuralSparseQuerytoQueryContainer(field-keyed pattern, same asneuralandknn):query_text— text to encode into sparse tokensmodel_id— sparse encoding model IDquery_tokens— pre-computed token weights (alternative to model inference)max_token_score— (deprecated since 2.12)analyzer— tokenizer to use instead of ML modelIntegration Test
text_chunking.yaml— validatestext_chunkingprocessor pipeline creation + document ingestion (no ML model required, version >= 2.13)Validation
npm run lint:spec— 0 errorsnpm run merge— resolves cleanlynpm run test:unit— 198 tests pass (39 suites)References