Why choose SQLite for DenoKV over alternatives like LevelDB?

[thelovekesh]

DenoKV utilizes SQLite for creating key-value stores and queues. Although SQLite is efficient, why not opt for a more native solution like LevelDB, which is specifically designed as a persistent key-value database?

Although I’m new to this area, I executed a script to read and write records using both DenoKV and LevelDB. Here are the results: I performed one million reads and writes with LevelDB, while I limited DenoKV to 100,000 due to significant slowdowns after 500,000 records.

LevelDB script - 1 million read and and 1 million writes

import { randomUUID } from 'node:crypto';

import { Level } from 'level';

const db = new Level('example', { valueEncoding: 'json' });

for (let i = 0; i < 1000000; i++) {
  const key = `key_${randomUUID()}`;

  await db.put(key, { message: 'Hello, World!' });
  const val = await db.get(key);
  console.log(val);
}

deno run -A script.ts  17.60s user 3.11s system 110% cpu 18.808 total

DenoKV script - 100, 000 reads and 100, 000 writes

import { randomUUID } from 'node:crypto';

const db = await Deno.openKv();

for (let i = 0; i < 100000; i++) {
  const key = `key_${randomUUID()}`;

  await db.set([key], { message: `Hello, World!` });
  const { value } = await db.get([key]);

  console.log(value);
}

deno run -A --unstable-kv script.ts  17.91s user 7.44s system 34% cpu 1:12.75 total

The scripts were executed on a machine equipped with 64 GB of DDR5 memory and the following configuration:

Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          39 bits physical, 48 bits virtual
  Byte Order:             Little Endian
CPU(s):                   20
  On-line CPU(s) list:    0-19
Vendor ID:                GenuineIntel
  Model name:             12th Gen Intel(R) Core(TM) i7-12700H
    CPU family:           6
    Model:                154
    Thread(s) per core:   2
    Core(s) per socket:   14
    Socket(s):            1
    Stepping:             3
    CPU max MHz:          4700.0000
    CPU min MHz:          400.0000

I understand there can be various reasons for these results, but from a high-level perspective, LevelDB is significantly faster at its tasks.

[Zizaco]

I believe there's more to consider than just raw performance. The LevelDB ecosystem isn't as robust, user-friendly, and established as SQLite. This makes SQLite a safer choice to avoid a lot of headaches.

Finally, if you are really settled on the idea of using LevelDB (and you need to handle 50k writes/sec in a single box), you can implement a Deno KV backend using LevelDB.

BTW: Deno KV uses a FoundationDB backend in Deno deployment.

[thelovekesh]

I understand there are many factors involved, such as ACID, partitioning, concurrency etc. Since DenoKV in Deno deployment utilizes FoundationDB, which is also an embeddable key-value database(if I am not wrong), my question revolves around why they chose SQLite over an embeddable KV database for their local APIs.

While SQLite is well-established so does the LevelDB and I agree the key differences is user-friendliness and community support. From a high-level perspective, SQLite is just an implementation detail here; users and developers interact with it through Deno bindings, so it doesn't make much difference what database is used under the hood.

I believe it would be beneficial for the developers to provide documentation or guidance on extending DenoKV to support other key-value databases. Currently, DenoKV with SQLite doesn't perform well as needed in production, although it works great in local environments.

[bartlomieju]

We chose SQLite because it was already a dependency in Deno used in other places (eg. Web Cache API) and didn't want to bring yet another dependency. After considerations it came out that SQLite is sufficient for local version of Deno KV.