Jetstream

Full-network archive and streaming service for atproto.

The original jetstream codebase is available here.

Docker images are available here.

NOTE: This project is not yet deployed to production, and there will be backwards-incompatible changes to the on-disk format in the next few days. Do not run this yourself and expect it to remain stable until we tag a 1.0 release (coming soon!).

See CONTRIBUTING.md for guidelines.

Getting started

Jetstream development uses Nix for a pinned Go and toolchain environment. Install Nix, then enter the dev shell with either:

./dev.sh
# or
just dev

Running Locally

For development purposes, to run against the real production network in a setup that doesn't require a whole-network backfill:

# backfill 20 random repos, then cut over to the live tail
just run-prod serve --max-backfill-repos=20

# backfill a small number of chosen DIDs (csv), then cut over to the live tail
just run-prod serve --backfill-repos=did:plc:4uz2445cjiw7w4nobfgnu35f

This repo also ships with an extremely minimal atproto simulator (PLC, PDS, and the Relay). To run the local environment against it, use two terminals like:

# terminal 1: starts the simulator on :7777 with 10,000 mock accounts
# (this takes a minute to start up)
just simulator serve

# terminal 2: jetstream points at the simulator
just run serve

Simulator and prod data are always isolated, so you can swap between them without worry (they each get a unique data directory).

To fully reset your local environment (warning: destructive action!):

just clean  # removes all built binaries and all data directories

Testing and Linting

To run the linter and tests, you can do things like:

just       # run the linter and all -short tests
just lint  # run the linter

just test                     # everything, -short mode
just test ./internal/foo/...  # one package
just test-race                # full suite with -race
just test-long                # full suite without -short

just oracle                   # heavier simulator oracle (stress mode)

Inspecting segment files

# inspect all segment files
just run inspect-all

# inspect a single file
just run inspect-segment ./data-prod/segments/seg_0000000000.jss

Dumps the header, footer, per-block stats, and collection event counts for a sealed segment.

Web Dashboard

The jetstream server also offers a web dashboard with some basic read-only features.

Start the server locally in your desired configuration, then open http://localhost:8080/status in your browser.