fix(milvus): make get_milvus_client directory-create loop-safe for milvus-lite - #146
Open
sahilkanger wants to merge 2 commits into
Open
Conversation
The repo-wide `ruff check` / `ruff format --check` gates were red on every PR because scrapegraph_task.py had un-sorted imports (I001) and trailing whitespace (W291). Auto-fixed with ruff (no behavior change) so this PR's CI can go green. Unrelated to the pinecone change in this PR.
sahilkanger
force-pushed
the
fix/113-milvus-get-milvus-client-async-loop
branch
from
July 23, 2026 20:35
6e81d26 to
0c41ea7
Compare
Contributor
Author
|
Two notes on CI: (1) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(milvus): make get_milvus_client directory-create loop-safe for milvus-lite
Refs #113
Summary
MilvusAdapter.get_milvus_client()crashes when used with a local (milvus-lite) file URIfrom inside cognee's running event loop. It tried to ensure the parent directory exists by
driving cognee's async file-storage helper from a sync method:
loop.run_until_complete(...)raises
RuntimeErroron the already-running loop, and theexcept RuntimeErrorfallback callsasyncio.run(...), which also raises inside a running loop. This PR replaces that block with aplain, loop-safe
os.makedirs(db_dir, exist_ok=True).Root cause
get_milvus_client()is synchronous but is called fromasyncadapter methods(
create_collection, etc.) that run inside cognee's event loop. Both branches of the olddirectory-ensure logic touch the event loop:
So any local milvus-lite run fails before a collection can be created. Server URIs (
http...)skip the block and were unaffected.
Solution
A local milvus-lite URI is always a real filesystem path, so ensuring its parent directory needs
no async storage abstraction:
oswas already imported; the now-unusedget_file_storageimport is removed. Version bumped0.1.2 → 0.1.3per the repo's fix-bump convention.Scope note (the other half of #113)
The issue also surfaces a
KeyError: 'milvus'underENABLE_BACKEND_ACCESS_CONTROL=True, becauseMilvus ships no
DatasetDatabaseHandler. I deliberately left that out of this PR: whileinvestigating I found the adapter doesn't currently scope its connection to
database_name(theMilvusClientis created withoutdb_name) andprune()drops every collection on theconnection — so simply mirroring the Qdrant handler (dataset →
vector_database_name, delete →prune()) would make deleting one dataset wipe all of them. Doing multi-tenant Milvus correctlyneeds a real per-dataset isolation model (server
db_namecreate/drop vs. per-file milvus-lite)that should be validated against a live instance. I've left a separate comment on #113 to align
with maintainers on the intended isolation model before building it, rather than ship a data-loss
risk here. Happy to follow up with that PR once the design is agreed.
Changes
Testing performed
python -m py_compileon the changed module — OK.ruff check/ruff format --checkon the package — clean.examples/example.py,tests/test_milvus.py) exercise thispath with a local URI, but need LLM/embedding keys and (for the example) a reachable Milvus;
those secrets aren't available to a fork PR locally. The change is a self-contained,
loop-safe directory create.
Checklist
lintpasses locally (ruff checkon the package)formatpasses locally (ruff format --checkon the package)