not saving on disk

[arita37]

It seems after insertion, data is not saved on disk,
any idea ?

[1yefuwang1]

It is documented in the README. You can save data in a file by specifying index_file_path.

-- Required fields: table_name, vector_name, dimension, max_elements
-- Optional fields:
-- 1. distance_type: defaults to l2
-- 2. ef_construction: defaults to 200
-- 3. M: defaults to 16
-- 4. random_seed: defaults to 100
-- 5. allow_replace_deleted: defaults to true
-- 6. index_file_path: no default value. If not provided, the table will be memory-only. If provided, vectorlite will try to load index from the file and save to it when db connection is closed.
create virtual table {table_name} using vectorlite({vector_name} float32[{dimension}] {distance_type}, hnsw(max_elements={max_elements}, {ef_construction=200}, {M=16}, {random_seed=100}, {allow_replace_deleted=true}), {index_file_path});

Here's an example

Please note, I do plan to revisit how data is stored. It could be changed in the future.

[arita37]

Thanks for reply. Do you think it’s possiblento implement this query select rowid, json_vector(emb) from table_emb in order to get full dump of the table ? (validation purpose).

…

On Nov 27, 2024, at 18:09, WANG YEFU ***@***.***> wrote:  It is documented in the README -- Required fields: table_name, vector_name, dimension, max_elements -- Optional fields: -- 1. distance_type: defaults to l2 -- 2. ef_construction: defaults to 200 -- 3. M: defaults to 16 -- 4. random_seed: defaults to 100 -- 5. allow_replace_deleted: defaults to true -- 6. index_file_path: no default value. If not provided, the table will be memory-only. If provided, vectorlite will try to load index from the file and save to it when db connection is closed. create virtual table {table_name} using vectorlite({vector_name} float32[{dimension}] {distance_type}, hnsw(max_elements={max_elements}, {ef_construction=200}, {M=16}, {random_seed=100}, {allow_replace_deleted=true}), {index_file_path}); Here's an example — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.

[1yefuwang1]

Thanks for the interest in this project.

Currently, a full table scan is deliberately disabled because I want to discourage slow queries. So for now, your query will return an empty set (or be rejected?).
Of cource it is possible to implement full table scans. It can be useful when a user loses track of what data have been inserted and wants to inspect the table. I'll try to implement it in version 0.3.0 or 0.4.0.

If you know the rowid of vectors inserted, which needs to be manually specified not auto-generated by design, validation can also done by looping over the rowids

select rowid, vector_to_json(emb) from table_emb where rowid = {each_rowid}

[arita37]

Thank you. Use cases: 1) to check who rowid have inserted. 2) to check rowid, vector values which have been inserted. Now I create a fake zero vector and KNN on this to get the rowid inserted….. (Not best way). Also checking embedding size,embedding name (metadata…) —> single db is self-contained 3) Have some utils function and put into the examples folder

…

On Nov 28, 2024, at 14:55, WANG YEFU ***@***.***> wrote:  Currently, a full table scan is deliberately disabled because I want to discourage slow queries. So for now, your query will return an empty set (or be rejected?). Of cource it is possible to implement full table scans. It can be useful when a user loses track of what data have been inserted and wants to inspect the table. I'll try to implement it in version 0.3.0 or 0.4.0. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.