Feature elasticsearch#603
Conversation
|
Noted @lauritssn and great work! FYI for readers - we're discussing progress on this on our Discord (https://discord.gg/duxWqxKk6J). AFAIK there's currently some extra testing to do before this can be merged, right? |
I think it works ok - tests are working and I have been running it on my own machine without errors. There is some stuff around the caching that requires a change in server.py I am unsure if is the good way to do it. |
|
You can use this sign off: Signed-off-by: Laurits Søgaard Nielsen lsn@werktoej.dk |
|
@lauritssn we can't modify your commit messages ourselves, they have to by you - unfortunately. Fortunately - it's easy to fix. You can ask Claude/GPT for help with doing this but the process is basically:
|
lauritssn
force-pushed
the
feature-elasticsearch
branch
from
629a357 to
43aa790
Compare
lauritssn
force-pushed
the
feature-elasticsearch
branch
2 times, most recently
from
e88da71 to
6c756ab
Compare
- Add ElasticsearchDocumentDatabase for document storage - Add ElasticsearchVectorDatabase for vector similarity search - Add SDK options to configure elasticsearch stores - Add configuration documentation and examples - Add comprehensive test coverage Signed-off-by: Laurits Søgaard Nielsen <lsn@werktoej.dk>
- Add SortDirection enum and Cursor dataclass to persistence/common.py - Add encode_cursor/decode_cursor utilities in app_modules/common.py - Implement pagination in ElasticsearchDocumentCollection.find() - Update capabilities.py to use FindResult interface - Add pagination tests for Elasticsearch adapter Signed-off-by: Laurits Søgaard Nielsen <lsn@werktoej.dk>
…h tests - Fix delete_collection pattern to include _vecdb_ in index name matching - Add unique test prefixes per test to prevent index conflicts - Update tests for NullEmbedder rename (was NoOpEmbedder) - Fix test_document_upgrade to create proper index mappings before inserting - Add VectorDatabase type annotation and import to SDK Signed-off-by: Laurits Søgaard Nielsen <lsn@werktoej.dk>
lauritssn
force-pushed
the
feature-elasticsearch
branch
2 times, most recently
from
d43a0c2 to
83c26fa
Compare
…es store The association schema hasn't changed since 0.5.0, but the document loader needs to handle the newer version strings. Signed-off-by: Laurits Søgaard Nielsen <lsn@werktoej.dk>
lauritssn
force-pushed
the
feature-elasticsearch
branch
from
83c26fa to
9b2358a
Compare
- Move elasticsearch_document_database.md to docs/adapters/persistence/ - Move elasticsearch_configuration.md to docs/adapters/vector_db/ - Update headers to match existing adapter documentation style - Remove broken internal reference link Signed-off-by: Laurits Søgaard Nielsen <lsn@werktoej.dk>
|
|
||
| if doc["version"] == "0.5.0": | ||
| # 0.5.0+ all use the same schema - no migration needed | ||
| if doc["version"] in ("0.5.0", "0.6.0", "0.7.0"): |
There was a problem hiding this comment.
Good observation of this bug! But actually, this is not a clean fix.
We want to update every document to the latest version. For 0.5.0 and 0.6.0, we want to update the document to 0.7.0.
There was a problem hiding this comment.
ok - should be fixed now
| session_store: Literal["transient", "local"] | str | SessionStore = "transient", | ||
| customer_store: Literal["transient", "local"] | str | CustomerStore = "transient", | ||
| variable_store: Literal["transient", "local"] | str | ContextVariableStore = "transient", | ||
| session_store: Literal["transient", "local", "elasticsearch"] |
There was a problem hiding this comment.
Since very few users are using Elasticsearch for persistence, we don't want to include it in the SDK.
With MongoDB, even though we didn't include it in the options, we implemented it, so you need to set an environment variable to enable it.
What we did with the other adapters is add an explanation to the Elasticsearch.md file on how to enable this adapter using configure_container.
Please take a look at the docs/adapters/persistence/snowflake.md for an example.
There was a problem hiding this comment.
Should be removed now
There was a problem hiding this comment.
I still see Elasticsearch as an option for stores. The SDK should remain as it was.
- Remove 'elasticsearch' from session_store, customer_store, variable_store, and cache_store Literal types in SDK Server constructor - Keep 'elasticsearch' as a valid option for vector_store - Update persistence elasticsearch.md with SDK/Module Setup section showing how to use configure_container for Elasticsearch persistence (like Snowflake) - Update vector_db elasticsearch.md with Quick Start showing both direct SDK usage (vector_store='elasticsearch') and advanced configure_container approach - Add new SDK exports: AgentStore, SessionDocumentStore, CustomerDocumentStore, ContextVariableDocumentStore, IdGenerator, EventEmitterFactory, VectorDatabase - Update installation examples to use uv instead of pip Signed-off-by: Laurits Søgaard Nielsen <lsn@werktoej.dk>
Update _association_document_loader to migrate all older document versions (0.3.0, 0.4.0, 0.5.0, 0.6.0) to the latest version 0.7.0 instead of just passing them through or migrating to an intermediate version. Signed-off-by: Laurits Søgaard Nielsen <lsn@werktoej.dk>
Remove the remaining "elasticsearch" option from vector_store Literal type and its implementation code. Users who want to use Elasticsearch for vector storage should use configure_container to set it up, similar to other adapters like Snowflake and Qdrant. This keeps the SDK unchanged from its original state before the Elasticsearch PR, as requested by the maintainers. Signed-off-by: Laurits Søgaard Nielsen <lsn@werktoej.dk>
The SDK should remain unchanged. Users who want to use Elasticsearch should use configure_container to set it up, as documented in docs/adapters/persistence/elasticsearch.md and docs/adapters/vector_db/elasticsearch.md. Signed-off-by: Laurits Søgaard Nielsen <lsn@werktoej.dk>
|
@mc-dorzo Can you please check if everything is as expected now? |
3 participants
Add Elasticsearch as a database, vector database and cache file storage. Initial version - needs a little more testing to ensure everything works as expected - especially on the retrieval part. Some files may need to be removed before merge.