Skip to content

Commit 61a6aac

Browse files
ampagentNathaneil Flurry
andcommitted
docs(blog): update SQLite at scale post
Amp-Thread-ID: https://ampcode.com/threads/T-01a04a21-e26a-71e9-ad22-f8e3ccc40a45 Co-authored-by: Nathaneil Flurry <nathan@rivet.gg>
1 parent b45062e commit 61a6aac

1 file changed

Lines changed: 51 additions & 35 deletions

File tree

  • src/content/posts/2025-02-16-sqlite-on-the-server-is-misunderstood

src/content/posts/2025-02-16-sqlite-on-the-server-is-misunderstood/page.mdx

Lines changed: 51 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ published: "2025-02-16"
55
category: technical
66
keywords: ['sqlite', 'durable-object', 'turso']
77
title: "SQLite-on-the-Server Is Misunderstood: Better At Hyper-Scale Than Micro-Scale"
8-
description: "We're Rivet, a new open-source, self-hostable serverless platform. We've been in the weeds with SQLite-on-the-server recently and – boy – do we have a lot of thoughts to share. Give us a star on GitHub, we'll be sharing a lot more about SQLite soon!"
8+
description: "SQLite is a powerful foundation for distributed databases, not just single-node local deployments."
99
seoTitle: "Why SQLite Works Better at Hyper-Scale"
1010
seoDescription: "Learn why SQLite-on-the-server fits hyper-scale systems when each tenant gets an isolated database with local reads, durable storage, and simple operations."
1111
---
@@ -14,9 +14,9 @@ export const SqliteMicroscale = { src: "https://assets.rivet.dev/website/blog/20
1414
export const Cassandra = { src: "https://assets.rivet.dev/website/blog/2025-02-16-sqlite-on-the-server-is-misunderstood/cassandra.png", width: 1934, height: 2178, format: "png" };
1515
export const SqliteHyperscale = { src: "https://assets.rivet.dev/website/blog/2025-02-16-sqlite-on-the-server-is-misunderstood/sqlite-hyperscale.png", width: 1674, height: 2172, format: "png" };
1616

17-
<Lead>
18-
We're Rivet, a new open-source, self-hostable serverless platform. We've been in the weeds with SQLite-on-the-server recently and – boy – do we have a lot of thoughts to share. [Give us a star on GitHub](https://github.com/rivet-dev/engine), we'll be sharing a lot more about SQLite soon!
19-
</Lead>
17+
<Tip>
18+
**Update (August 2026):** Rivet launched [SQLite for Rivet Actors](/docs/actors/sqlite) at the beginning of 2026. It's open-source, self-hostable infrastructure for server-side SQLite at scale, including support for S3 tiered storage.
19+
</Tip>
2020

2121
There's been [a lot of discussion](https://lobste.rs/s/t1enph/siren_call_sqlite_on_server) recently about the pros and cons of SQLite on the server. After reading many of these conversations, I realized that my perspective on the power of SQLite-on-the-server is lopsided from popular opinion: SQLite's strengths really shine at scale, instead of with small hobbyist deployments that it's frequently referenced in.
2222

@@ -41,7 +41,7 @@ These characteristics make SQLite an attractive option for personal projects, li
4141

4242
Tools like [LiteFS](https://github.com/superfly/litefs), [Litestream](https://litestream.io/), [rqlite](https://github.com/rqlite/rqlite), [Dqlite](https://dqlite.io/), and [Bedrock](https://bedrockdb.com/) enhance SQLite with replication and high availability for micro-scale deployments.
4343

44-
However, this post focuses on [Cloudflare Durable Objects](https://developers.cloudflare.com/durable-objects/get-started/tutorial-with-sql-api/) and [Turso](https://turso.tech/) to highlight the often-overlooked advantages of SQLite at scale.
44+
However, this post focuses on [Rivet Actors](/docs/actors/sqlite), [Cloudflare Durable Objects](https://developers.cloudflare.com/durable-objects/get-started/tutorial-with-sql-api/), and [Turso](https://turso.tech/) to highlight the often-overlooked advantages of SQLite at scale.
4545

4646
## Databases at Hyper-Scale Today: Sharded Databases & Partitioning Keys
4747

@@ -81,9 +81,7 @@ Sharded databases power almost every large tech company because they provide:
8181
- Horizontal scalability by partitioning data across nodes.
8282
- Optimized writes for high-ingestion workloads
8383

84-
<Tip>
85-
Apple runs the world's largest known number of Cassandra/ScyllaDB instances [with roughly 300,000 nodes as of 2022](https://x.com/erickramirezau/status/1578063811495477248). Almost all large companies have shared that they also run massive clusters of partitioned databases in the thousands of nodes.
86-
</Tip>
84+
Apple runs the world's largest known number of Cassandra/ScyllaDB instances [with roughly 300,000 nodes as of 2022](https://x.com/erickramirezau/status/1578063811495477248).
8785

8886
### Challenges with Current Partitioning Solutions
8987

@@ -94,9 +92,11 @@ While partitioning strategies improve scalability, they introduce significant ch
9492
- **Complex cross-partition operations**: Enforcing ACID properties across partitions is difficult. Companies often resort to complicated [two-phase commits](https://en.wikipedia.org/wiki/Two-phase_commit_protocol) or design with an acceptable level of data inconsistency.
9593
- **Data inconsistency**: Without strong constraints between tables & partitions, data frequently becomes "dirty" because of interrupted transactions or failure to propagate changes.
9694

97-
## Enter SQLite at Scale: Cloudflare Durable Objects & Turso
95+
<span id="enter-sq-lite-at-scale-cloudflare-durable-objects-and-turso"></span>
96+
97+
## Enter SQLite at Scale: Rivet Actors, Cloudflare Durable Objects & Turso
9898

99-
[Cloudflare Durable Objects](https://developers.cloudflare.com/durable-objects/get-started/tutorial-with-sql-api/) and [Turso](https://turso.tech/) demonstrate how SQLite will change how hyper-scale applications may be architected in the future.
99+
[Rivet Actors](/docs/actors/sqlite), [Cloudflare Durable Objects](https://developers.cloudflare.com/durable-objects/get-started/tutorial-with-sql-api/), and [Turso](https://turso.tech/) demonstrate how SQLite will change how hyper-scale applications may be architected in the future.
100100

101101
These databases provide:
102102

@@ -105,20 +105,19 @@ These databases provide:
105105
- **Global distribution**: Databases are placed closer to users, improving query performance.
106106
- **Built-in replication and durability**: Unlike traditional SQLite, these services replicate data across multiple regions for high availability.
107107

108-
### A Better Hyper-Scale Database: SQLite Database Per Partition
108+
### A Better Hyper-Scale Database: SQLite Databases Per Entity
109109

110110
<img src={SqliteHyperscale.src} alt="A cluster of independent SQLite databases serves queries through an HTTP server" />
111111

112-
Using SQLite with Cloudflare Durable Objects & Turso allows defining databases per entity, effectively replacing partitioning keys.
112+
Using SQLite with Rivet Actors, Cloudflare Durable Objects, or Turso allows defining databases per entity, effectively replacing partitioning keys.
113113

114114
Instead of storing chat logs in a single partition, each chat channel can have its own SQLite database that also includes more tables, like participants and reactions. A sample schema could look like this:
115115

116116
```sql {{ "title": "schema.sql" }}
117117
-- Notice there is no channel_id in this table, since all of this data is local
118118
-- to a single SQLite database per channel.
119119
--
120-
-- The channel ID will be specified when defining which Durable Object to
121-
-- connect to below (see )`CHAT_DO.idFromName(channelId)` below.
120+
-- The channel ID identifies which per-channel database to connect to below.
122121
CREATE TABLE messages (
123122
id INTEGER PRIMARY KEY,
124123
sent_at TIMESTAMP,
@@ -134,13 +133,35 @@ CREATE TABLE participants (
134133
);
135134
```
136135

137-
From Cloudflare Durable Objects or Turso, this SQLite partition database could be queried like this:
136+
The example above can be implemented with Rivet Actors, Cloudflare Durable Objects, and Turso:
138137

139138
<CodeGroup>
140-
```javascript {{ "title": "cloudflare_worker.js" }}
141-
// This Durable Object represents one partition
142-
//
143-
// It contains a completely isolated SQLite database
139+
```typescript {{ "title": "Rivet Actors" }}
140+
// === server.ts ===
141+
142+
import { actor } from "rivetkit";
143+
import { db } from "rivetkit/db";
144+
145+
// Each channel has its own actor.
146+
export const chatChannel = actor({
147+
db: db(),
148+
actions: {
149+
getMessages: (c) =>
150+
c.db.execute("SELECT * FROM messages ORDER BY sent_at ASC"),
151+
},
152+
});
153+
154+
// === client.ts ===
155+
156+
function getMessages(channelId: string) {
157+
return client.chatChannel.getOrCreate([channelId]).getMessages();
158+
}
159+
```
160+
161+
```typescript {{ "title": "Cloudflare Durable Objects" }}
162+
// === server.ts ===
163+
164+
// Each channel has its own Durable Object.
144165
export class ChatChannel extends DurableObject {
145166
async getMessages() {
146167
// Execute query
@@ -149,8 +170,10 @@ export class ChatChannel extends DurableObject {
149170
}
150171
}
151172

173+
// === client.ts ===
174+
152175
function getMessages(channelId: string) {
153-
// Here, the channel ID corresponds to the database partition
176+
// The channel ID corresponds to the database shard.
154177
const id = env.CHAT_CHANNEL_DURABLE_OBJECT.idFromName(channelId);
155178
const channelDurableObject = c.env.CHAT_CHANNEL_DURABLE_OBJECT.get(id);
156179

@@ -159,14 +182,16 @@ function getMessages(channelId: string) {
159182
}
160183
```
161184

162-
```javascript {{ "title": "turso.js" }}
185+
```typescript {{ "title": "Turso" }}
186+
// === client.ts ===
187+
163188
import { createLibsqlClient } from "@libsql/client";
164189
import { createApiClient } from "@tursodatabase/api";
165190

166191
const api = createApiClient(/* ... */);
167192

168193
function getMessages(channelId: string) {
169-
// Each database represents one partition
194+
// Each channel has its own database.
170195
//
171196
// This gets an existing channel and connects to it
172197
const { hostname } = await api.databases.retrieve(`channel-${channelId}`);
@@ -191,24 +216,15 @@ function getMessages(channelId: string) {
191216
- **Full Power of SQL migrations**: SQLite provides the full power of SQL migrations & leveraging existing migrations tools such as Drizzle & Prisma.
192217
- **Lazy schema migrations**: Changing schema is tricky at scale. Assuming your migrations are lightweight, they can be executed on demand after the SQLite database is opened at the cost of a slightly higher p99 after deploys.
193218

194-
<Tip>
195-
**Who are the Turso founders?** It's no coincidence that the founders of Turso used to work at ScyllaDB, where they saw firsthand the complexities of large-scale partitioned databases on a daily basis.
196-
</Tip>
197-
198219
## Where SQLite Still Falls Short
199220

200221
Despite its benefits, SQLite at scale presents a few challenges:
201222

202-
- Lack of an open-source, self-hosted solution.
203-
- No built-in cross-database querying, making complex analytics difficult without a dedicated data lake.
204-
- Limited database tooling, such as SQL browsers, ETL pipelines, monitoring, and backups. [StarbaseDB](https://starbasedb.com/) is addressing this for Cloudflare Durable Objects with SQLite.
223+
- ~~Lack of an open-source, self-hosted solution for SQLite-per-entity (Litestream only provides a single database, while Turso Cloud's multi-database orchestration is closed source).~~ (Update: Rivet now provides an open-source, self-hostable solution with [SQLite for Rivet Actors](/docs/actors/sqlite).)
224+
- No built-in cross-database querying, making complex analytics difficult without a dedicated data lake. (Update: There are experiments using DuckDB for cross-SQLite queries, but they are not production-ready yet.)
225+
- ~~Limited database tooling, such as SQL browsers, ETL pipelines, monitoring, and backups. [StarbaseDB](https://starbasedb.com/) is addressing this for Cloudflare Durable Objects with SQLite.~~ (Update: Rivet and Cloudflare now both provide visual SQLite editors.)
205226
- Non-standard protocols for communicating with SQLite-on-the-server. In contrast, PostgreSQL, MySQL, and Cassandra all have a well standardized wire protocols across all cloud providers that has led to a rich community of tools.
206-
- There are no case studies like Cassandra & DynamoDB have of using SQLite with this architecture at hyper-scale. This will change with time.
207-
208-
<Tip>
209-
We're shipping an open-source & self-hostable alternative to Cloudflare Durable Objects with SQLite in the near future. Follow us on [X](https://x.com/rivet_dev), [Bluesky](https://bsky.app/profile/rivet.dev), or [Discord](https://rivet.dev/discord) to get notified on updates.
210-
</Tip>
211227

212228
## Conclusion
213229

214-
SQLite on the server is more than a lightweight solution for small deployments – it's an increasingly viable alternative to traditional partitioned databases. By leveraging SQLite-per-partition solutions like Turso and Durable Objects, developers gain rich SQL capabilities, ACID compliance, and significant operational advantages.
230+
SQLite on the server is useful for more than small-scale deployments. It's a compelling option for sharded deployments at scale. SQLite-per-partition solutions like Rivet Actors, Turso, and Durable Objects give developers rich SQL capabilities, ACID compliance, and significant operational advantages.

0 commit comments

Comments
 (0)