You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: src/content/posts/2025-02-16-sqlite-on-the-server-is-misunderstood/page.mdx
+51-35Lines changed: 51 additions & 35 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,7 +5,7 @@ published: "2025-02-16"
5
5
category: technical
6
6
keywords: ['sqlite', 'durable-object', 'turso']
7
7
title: "SQLite-on-the-Server Is Misunderstood: Better At Hyper-Scale Than Micro-Scale"
8
-
description: "We're Rivet, a new open-source, self-hostable serverless platform. We've been in the weeds with SQLite-on-the-server recently and – boy – do we have a lot of thoughts to share. Give us a star on GitHub, we'll be sharing a lot more about SQLite soon!"
8
+
description: "SQLite is a powerful foundation for distributed databases, not just single-node local deployments."
9
9
seoTitle: "Why SQLite Works Better at Hyper-Scale"
10
10
seoDescription: "Learn why SQLite-on-the-server fits hyper-scale systems when each tenant gets an isolated database with local reads, durable storage, and simple operations."
We're Rivet, a new open-source, self-hostable serverless platform. We've been in the weeds with SQLite-on-the-server recently and – boy – do we have a lot of thoughts to share. [Give us a star on GitHub](https://github.com/rivet-dev/engine), we'll be sharing a lot more about SQLite soon!
19
-
</Lead>
17
+
<Tip>
18
+
**Update (August 2026):** Rivet launched [SQLite for Rivet Actors](/docs/actors/sqlite) at the beginning of 2026. It's open-source, self-hostable infrastructure for server-side SQLite at scale, including support for S3 tiered storage.
19
+
</Tip>
20
20
21
21
There's been [a lot of discussion](https://lobste.rs/s/t1enph/siren_call_sqlite_on_server) recently about the pros and cons of SQLite on the server. After reading many of these conversations, I realized that my perspective on the power of SQLite-on-the-server is lopsided from popular opinion: SQLite's strengths really shine at scale, instead of with small hobbyist deployments that it's frequently referenced in.
22
22
@@ -41,7 +41,7 @@ These characteristics make SQLite an attractive option for personal projects, li
41
41
42
42
Tools like [LiteFS](https://github.com/superfly/litefs), [Litestream](https://litestream.io/), [rqlite](https://github.com/rqlite/rqlite), [Dqlite](https://dqlite.io/), and [Bedrock](https://bedrockdb.com/) enhance SQLite with replication and high availability for micro-scale deployments.
43
43
44
-
However, this post focuses on [Cloudflare Durable Objects](https://developers.cloudflare.com/durable-objects/get-started/tutorial-with-sql-api/) and [Turso](https://turso.tech/) to highlight the often-overlooked advantages of SQLite at scale.
44
+
However, this post focuses on [Rivet Actors](/docs/actors/sqlite), [Cloudflare Durable Objects](https://developers.cloudflare.com/durable-objects/get-started/tutorial-with-sql-api/), and [Turso](https://turso.tech/) to highlight the often-overlooked advantages of SQLite at scale.
45
45
46
46
## Databases at Hyper-Scale Today: Sharded Databases & Partitioning Keys
47
47
@@ -81,9 +81,7 @@ Sharded databases power almost every large tech company because they provide:
81
81
- Horizontal scalability by partitioning data across nodes.
82
82
- Optimized writes for high-ingestion workloads
83
83
84
-
<Tip>
85
-
Apple runs the world's largest known number of Cassandra/ScyllaDB instances [with roughly 300,000 nodes as of 2022](https://x.com/erickramirezau/status/1578063811495477248). Almost all large companies have shared that they also run massive clusters of partitioned databases in the thousands of nodes.
86
-
</Tip>
84
+
Apple runs the world's largest known number of Cassandra/ScyllaDB instances [with roughly 300,000 nodes as of 2022](https://x.com/erickramirezau/status/1578063811495477248).
87
85
88
86
### Challenges with Current Partitioning Solutions
89
87
@@ -94,9 +92,11 @@ While partitioning strategies improve scalability, they introduce significant ch
94
92
-**Complex cross-partition operations**: Enforcing ACID properties across partitions is difficult. Companies often resort to complicated [two-phase commits](https://en.wikipedia.org/wiki/Two-phase_commit_protocol) or design with an acceptable level of data inconsistency.
95
93
-**Data inconsistency**: Without strong constraints between tables & partitions, data frequently becomes "dirty" because of interrupted transactions or failure to propagate changes.
96
94
97
-
## Enter SQLite at Scale: Cloudflare Durable Objects & Turso
## Enter SQLite at Scale: Rivet Actors, Cloudflare Durable Objects & Turso
98
98
99
-
[Cloudflare Durable Objects](https://developers.cloudflare.com/durable-objects/get-started/tutorial-with-sql-api/) and [Turso](https://turso.tech/) demonstrate how SQLite will change how hyper-scale applications may be architected in the future.
99
+
[Rivet Actors](/docs/actors/sqlite), [Cloudflare Durable Objects](https://developers.cloudflare.com/durable-objects/get-started/tutorial-with-sql-api/), and [Turso](https://turso.tech/) demonstrate how SQLite will change how hyper-scale applications may be architected in the future.
100
100
101
101
These databases provide:
102
102
@@ -105,20 +105,19 @@ These databases provide:
105
105
-**Global distribution**: Databases are placed closer to users, improving query performance.
106
106
-**Built-in replication and durability**: Unlike traditional SQLite, these services replicate data across multiple regions for high availability.
107
107
108
-
### A Better Hyper-Scale Database: SQLite Database Per Partition
108
+
### A Better Hyper-Scale Database: SQLite Databases Per Entity
109
109
110
110
<imgsrc={SqliteHyperscale.src}alt="A cluster of independent SQLite databases serves queries through an HTTP server" />
111
111
112
-
Using SQLite with Cloudflare Durable Objects & Turso allows defining databases per entity, effectively replacing partitioning keys.
112
+
Using SQLite with Rivet Actors, Cloudflare Durable Objects, or Turso allows defining databases per entity, effectively replacing partitioning keys.
113
113
114
114
Instead of storing chat logs in a single partition, each chat channel can have its own SQLite database that also includes more tables, like participants and reactions. A sample schema could look like this:
115
115
116
116
```sql {{ "title": "schema.sql" }}
117
117
-- Notice there is no channel_id in this table, since all of this data is local
118
118
-- to a single SQLite database per channel.
119
119
--
120
-
-- The channel ID will be specified when defining which Durable Object to
121
-
-- connect to below (see )`CHAT_DO.idFromName(channelId)` below.
120
+
-- The channel ID identifies which per-channel database to connect to below.
122
121
CREATETABLEmessages (
123
122
id INTEGERPRIMARY KEY,
124
123
sent_at TIMESTAMP,
@@ -134,13 +133,35 @@ CREATE TABLE participants (
134
133
);
135
134
```
136
135
137
-
From Cloudflare Durable Objects or Turso, this SQLite partition database could be queried like this:
136
+
The example above can be implemented with Rivet Actors, Cloudflare Durable Objects, and Turso:
@@ -191,24 +216,15 @@ function getMessages(channelId: string) {
191
216
-**Full Power of SQL migrations**: SQLite provides the full power of SQL migrations & leveraging existing migrations tools such as Drizzle & Prisma.
192
217
-**Lazy schema migrations**: Changing schema is tricky at scale. Assuming your migrations are lightweight, they can be executed on demand after the SQLite database is opened at the cost of a slightly higher p99 after deploys.
193
218
194
-
<Tip>
195
-
**Who are the Turso founders?** It's no coincidence that the founders of Turso used to work at ScyllaDB, where they saw firsthand the complexities of large-scale partitioned databases on a daily basis.
196
-
</Tip>
197
-
198
219
## Where SQLite Still Falls Short
199
220
200
221
Despite its benefits, SQLite at scale presents a few challenges:
201
222
202
-
- Lack of an open-source, self-hosted solution.
203
-
- No built-in cross-database querying, making complex analytics difficult without a dedicated data lake.
204
-
- Limited database tooling, such as SQL browsers, ETL pipelines, monitoring, and backups. [StarbaseDB](https://starbasedb.com/) is addressing this for Cloudflare Durable Objects with SQLite.
223
+
-~~Lack of an open-source, self-hosted solution for SQLite-per-entity (Litestream only provides a single database, while Turso Cloud's multi-database orchestration is closed source).~~ (Update: Rivet now provides an open-source, self-hostable solution with [SQLite for Rivet Actors](/docs/actors/sqlite).)
224
+
- No built-in cross-database querying, making complex analytics difficult without a dedicated data lake. (Update: There are experiments using DuckDB for cross-SQLite queries, but they are not production-ready yet.)
225
+
-~~Limited database tooling, such as SQL browsers, ETL pipelines, monitoring, and backups. [StarbaseDB](https://starbasedb.com/) is addressing this for Cloudflare Durable Objects with SQLite.~~ (Update: Rivet and Cloudflare now both provide visual SQLite editors.)
205
226
- Non-standard protocols for communicating with SQLite-on-the-server. In contrast, PostgreSQL, MySQL, and Cassandra all have a well standardized wire protocols across all cloud providers that has led to a rich community of tools.
206
-
- There are no case studies like Cassandra & DynamoDB have of using SQLite with this architecture at hyper-scale. This will change with time.
207
-
208
-
<Tip>
209
-
We're shipping an open-source & self-hostable alternative to Cloudflare Durable Objects with SQLite in the near future. Follow us on [X](https://x.com/rivet_dev), [Bluesky](https://bsky.app/profile/rivet.dev), or [Discord](https://rivet.dev/discord) to get notified on updates.
210
-
</Tip>
211
227
212
228
## Conclusion
213
229
214
-
SQLite on the server is more than a lightweight solution for small deployments – it's an increasingly viable alternative to traditional partitioned databases. By leveraging SQLite-per-partition solutions like Turso and Durable Objects, developers gain rich SQL capabilities, ACID compliance, and significant operational advantages.
230
+
SQLite on the server is useful for more than small-scale deployments. It's a compelling option for sharded deployments at scale. SQLite-per-partition solutions like Rivet Actors, Turso, and Durable Objects give developers rich SQL capabilities, ACID compliance, and significant operational advantages.
0 commit comments