Skip to content

overview.md

Latest commit

 

History

History
298 lines (219 loc) · 10.3 KB

overview.md

File metadata and controls

298 lines (219 loc) · 10.3 KB

Prisma Streams Overview

Prisma Streams is a Bun + TypeScript implementation of the Durable Streams HTTP protocol.

Use index.md for the full documentation map.

Every stream has a profile.

  • If no profile is declared when the stream is created, it is treated as generic.
  • evlog is the built-in request-log profile for canonical wide events, redaction, requestId/traceId routing-key defaults, and automatic default schema/search/rollup installation.
  • generic means a plain durable stream with optional user-managed schema validation.
  • metrics is the built-in metrics profile for canonical interval summaries, default search/rollups, and object-store-native metrics companions.
  • state-protocol is the built-in live/touch profile for JSON State Protocol streams.
  • Profiles define stream semantics; schemas define payload shape.

See stream-profiles.md.

This repository currently contains two server modes:

  • full mode: a self-hosted server with SQLite WAL storage, segmenting, upload, and index maintenance
  • local mode: an embedded single-SQLite server intended for trusted local development workflows, especially npx prisma dev

Current Durability Model

Full mode today has two different durability points:

  • append ACK means the write is durable in local SQLite
  • object-store durability happens only after segment upload plus manifest publication

--bootstrap-from-r2 rebuilds published stream history and metadata from object storage. It does not restore transient local SQLite state such as the unuploaded WAL tail, producer dedupe state, or runtime live/template state.

A stream becomes recoverable from object storage after its first manifest is published.

Not implemented today:

  • an object-store-acked mode that would only ACK after persistence to R2
  • a cluster quorum mode that would only ACK after a storage quorum accepts the write

Status

  • The publishable npm surfaces are intentionally split:
    • @prisma/streams-local exports startLocalDurableStreamsServer and its server types
    • @prisma/streams-local/internal/daemon exists for Prisma CLI integration and is intentionally internal
    • @prisma/streams-server is the Bun-only full server package, CLI, and package Compute entrypoint

See conformance.md for current compatibility status, verification commands, and known gaps.

Security

Full server startup requires an explicit auth mode.

  • Use --auth-strategy api-key with API_KEY for built-in authentication on every request.
  • Use --no-auth only behind a trusted reverse proxy, API gateway, VPN boundary, or other authenticated perimeter.
  • Treat the local development server as a loopback-only tool for trusted local workflows.

See security.md and auth.md.

Prerequisites

  • Bun >=1.3.6 for the full self-hosted server and repository workflows
  • Bun >=1.2.0 or Node.js >=22 for the published @prisma/streams-local package

Quick Start

bun install

# Full server (self-hosted pipeline)
bun run src/server.ts --object-store local --no-auth

# Local development server
bun run src/local/cli.ts start --name default --port 8080

# Status / stop / reset
bun run src/local/cli.ts status --name default
bun run src/local/cli.ts stop --name default
bun run src/local/cli.ts reset --name default

Notes:

  • Full server startup requires --object-store local|r2 and exactly one auth mode: --no-auth or --auth-strategy api-key.
  • Prisma Compute deployments from npm should follow the package-based flow in the top-level README.md.
  • Repository Compute bundle deployments can use src/compute/entry.ts, which injects --object-store r2 for the server entrypoint and --auto-tune when DS_MEMORY_LIMIT_MB is set. Compute deployments still need an explicit auth mode.
  • If deploying from this repository with a prebuilt artifact, use bun run build:compute-bundle and deploy that artifact with --skip-build so worker-thread entrypoints such as segmenter_worker.ts and processor_worker.ts are included.
  • Full mode binds to 127.0.0.1 by default. Set DS_HOST=0.0.0.0 if you intentionally want a non-loopback bind inside a trusted network boundary.
  • Local mode is designed for development and Prisma CLI integration, not hostile-network deployment.
  • The default local data root remains under envPaths("prisma-dev").data/durable-streams/ for compatibility with the Prisma development workflow.

Local Integration API

The supported package import for local development integration is:

import { startLocalDurableStreamsServer } from "@prisma/streams-local";

const server = await startLocalDurableStreamsServer({
  name: "default",
  hostname: "127.0.0.1",
  port: 0,
});

console.log(server.exports.http.url);
console.log(server.exports.sqlite.path);

await server.close();

The published @prisma/streams-local surface is built to run on Bun >=1.2.0 and Node >=22. The full self-hosted server remains Bun-only.

The local embedded runtime always applies the built-in 1024 MB auto-tune preset for cache and concurrency budgets. It does not enable the process-level memory pressure guard, because embedded callers may share a larger host Node/Bun process whose RSS is not owned by Streams.

The package smoke tests cover the local Live path under both host runtimes:

  • bun run test:node-local-package
  • bun run test:bun-local-package

@prisma/streams-local/internal/daemon is exported for Prisma CLI integration, but it is intentionally internal and does not carry the same compatibility guarantee as @prisma/streams-local.

More detail is in local-dev.md.

Full Server

The full server is started via:

bun run src/server.ts --object-store local --no-auth

Published CLI package:

bunx --package @prisma/streams-server prisma-streams-server --object-store local --no-auth

Bind control:

DS_HOST=127.0.0.1 PORT=8080 bun run src/server.ts --object-store local --no-auth

API key auth:

API_KEY=replace-with-at-least-10-characters \
  bun run src/server.ts --object-store local --auth-strategy api-key

Optional flags:

  • --stats
  • --hist
  • --bootstrap-from-r2
  • --auto-tune[=MB]

Object Store Configuration

Local MockR2:

bun run src/server.ts --object-store local --no-auth

Real R2:

DURABLE_STREAMS_R2_BUCKET=your-bucket \
DURABLE_STREAMS_R2_ACCOUNT_ID=your-account-id \
DURABLE_STREAMS_R2_ACCESS_KEY_ID=your-access-key \
DURABLE_STREAMS_R2_SECRET_ACCESS_KEY=your-secret \
API_KEY=replace-with-at-least-10-characters \
  bun run src/server.ts --object-store r2 --auth-strategy api-key

Prisma Compute deployment from the published npm package is documented in the top-level README.md.

Repository prebuild for Prisma Compute deployments:

bun run build:compute-bundle

PRISMA_API_TOKEN=... \
  bunx @prisma/compute-cli compute deploy \
    --service your-service-id \
    --path .compute-build/bundle \
    --entrypoint compute/entry.js \
    --skip-build

src/compute/entry.ts has the same Compute argv behavior as the package Compute entrypoint, but is intended for repository-built artifacts.

Compute demo deployment with Studio and the evlog generator:

bun run build:compute-demo-bundle

PRISMA_API_TOKEN=... \
  bunx @prisma/compute-cli deploy \
    --service your-service-id \
    --skip-build \
    --path .compute-demo-build/bundle \
    --entrypoint compute/demo_entry.js

That artifact starts the normal Streams server plus:

  • /studio for the streams-only Prisma Studio UI
  • /generate for a bulk evlog ingest page with a stream-name field defaulting to demo-app and 1k, 10k, and 100k actions

See compute-demo.md.

Compute verification demo:

bun run demo:compute-verify --url https://your-service.cdg.prisma.build

The Compute verification workload uses large mixed-entropy binary rows so segmenting still cuts by DS_SEGMENT_MAX_BYTES under the compression-aware seal heuristic. That avoids needing a tiny DS_SEGMENT_TARGET_ROWS override just to force cuts for an overly compressible demo payload.

Development Commands

bun run typecheck
bun run check:result-policy
bun test
bun run test:conformance:local
bun run test:conformance

Documentation

Open Source Baseline

This repository now includes: