Private Data Availability Proxy

A Celestia Data Availability (DA) proxy, enabling use of the canonical JSON RPC but intercepting and verifiably encrypting sensitive data before submission on the public DA network, and enable decryption on retrieval. Non-sensitive calls are unmodified.

Verifiable encryption is enabled via an SP1 Zero Knowledge Proof (ZKP).

Jump to a section:

• Send requests this service: Interact
• Spin up an instance of the service: Operate
• Build & troubleshoot: Develop

Known Limitations

Presently all HTTP requests to the proxy are transparently proxied to an upstream Celestia node, interception logic handles these JSON RPC methods:

• (complete) blob.Submit encrypts before proxy submission
• (not implemented)blob.Get proxy result is decrypted before responding

At time of writing, as it should be possible to change these limitations internally:

• #11
• #12

It's possible to change these, but requires upstream involvement:

• Max blob size on Celestia is presently ~2MB
• Upstream jsonrpsee en/decryption middleware feature into lumina Rust client?

Please open an issue if you have any requests!

Architecture

---
config:
  look: handDrawn
  theme: dark
  layout: elk
---
flowchart TD
 subgraph VE_Container["zkVM or TEE"]
        Encrypt["Encryption<br>(ChaCha, AES, ...)"]
  end
 subgraph VE_Out_Seal["Verifiable Encrypted Data"]
        VE_Out["Encrypted Data"]
        VE_Hash["Input Data Anchor<br>(Hash)"]
        VE_Seal["Seal/Attestation"]
  end
 subgraph P_Container["zkVM or TEE"]
        Program["Program<br>(DEX, Sequencer)"]
  end
 subgraph P_Out_Seal["Verifiable Program Output"]
        P_Out["Program Output"]
        P_Hash["Input Data Anchor<br>(Hash)"]
        P_Seal["Seal/Attestation"]
  end
 subgraph Chain["Blockchain<br>{Ethereum}"]
        App_Contracts["App Contract"]
        Blobstream["Blobstream"]
  end
    In["{Confidential}<br>Input Data"] --> Encrypt & Program
    Encrypt --> VE_Out_Seal
    Program --> P_Out_Seal
    VE_Hash == Identical === P_Hash
    Key(["{Confidential}<br>Encryption Key"]) -- Provided as input<br>or embedded ---> Encrypt
    VE_Out_Seal --> DA["DA<br>{Celestia}"]
    DA --> Blobstream
    P_Out_Seal --> App_Contracts
    Blobstream -. "Confirm DA for<br>Encrypted Data" .-> App_Contracts
    Chain -. Chain State .-> Program
    Key_Recover["Key Recovery Process<br>(MPC, Threshold, Etc.)"] -.-> Key
    App_Contracts@{ shape: doc}
    Blobstream@{ shape: doc}
    In@{ shape: docs}
    DA@{ shape: cyl}
    Key_Recover@{ shape: h-cyl}
    Key@{ shape: dbl-circ}

Loading

The PDA proxy depends on a connection to:

1. Celestia Data Availability (DA) Node to:
   • Submit and retrieve (verifiable encrypted) blob data.
2. (Optional) Succinct prover network as a provider to generate Zero-Knowledge Proofs (ZKPs) of data existing on Celestia. See the ZKP program for details on what is proven.

Interact

First you need to configure your environment and nodes.

Then any HTTP1 client works to send Celestial JSON RPC calls to the proxy:

# Proxy running on 127.0.0.1:26657
# See: <https://mocha.celenium.io/blob?commitment=S2iIifIPdAjQ33KPeyfAga26FSF3IL11WsCGtJKSOTA=&hash=AAAAAAAAAAAAAAAAAAAAAAAAAFHMGnPWX5X2veY=&height=4499999>

source .env
# blob.Get
curl -H "Content-Type: application/json" -H "Authorization: Bearer $CELESTIA_NODE_WRITE_TOKEN" -X POST \
     --data '{ "id": 1, "jsonrpc": "2.0", "method": "blob.Get", "params": [ 4499999, "AAAAAAAAAAAAAAAAAAAAAAAAAFHMGnPWX5X2veY=", "S2iIifIPdAjQ33KPeyfAga26FSF3IL11WsCGtJKSOTA="] }' \
     $PDA_SOCKET
# blob.GetAll
curl -H "Content-Type: application/json" -H "Authorization: Bearer $CELESTIA_NODE_WRITE_TOKEN" -X POST \
     --data '{ "id": 1, "jsonrpc": "2.0", "method": "blob.GetAll", "params": [ 4499999, [ "AAAAAAAAAAAAAAAAAAAAAAAAAFHMGnPWX5X2veY=" ] ] }' \
     $PDA_SOCKET
# blob.Submit (dummy data)
# Note: send "{}" as empty `tx_config` object, so the node uses it's default key to sign & submit to Celestia
curl -H "Content-Type: application/json" -H "Authorization: Bearer $CELESTIA_NODE_WRITE_TOKEN" -X POST \
     --data '{ "id": 1, "jsonrpc": "2.0", "method": "blob.Submit", "params": [ [ { "namespace": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAMJ/xGlNMdE=", "data": "DEADB33F", "share_version": 0, "commitment": "aHlbp+J9yub6hw/uhK6dP8hBLR2mFy78XNRRdLf2794=", "index": -1 } ], { } ] }' \
     https://$PDA_SOCKET \
     --verbose \
     --insecure
    # ^^^^ DO NOT use insecure TLS in real scenarios!
# blob.Submit (example input ~1.5MB)
cd scripts
./test_example_data_file_via_curl.sh

Celestia has many API client libraries to build around a PDA proxy.

Operate

TODO: notice on single job at a time

• single GPU 100% used per job
• presently no way to scale on multi-GPU

Most users will want to pull and run this service using Docker or Podman via container registry, see running containers.

To build and run, see developing instructions

Requirements

TODO

1. A machine to run with a minimum of:

   • L4 NVIDIA GPU
   • GB RAM
   • Ports accessible (by default):
     • service listening at TODO
     • Light client (local or remote) over 26658
     • (Optional) Succinct prover network over 443

   Example AWS instance: g6.xlarge (single L4 GPU + 8 vCPU)

2. A Celestia Light Node installed & running accessible on localhost, or elsewhere. Alternatively, use an RPC provider you trust.

   • Configure and fund a Celestia Wallet for the node to sign and send transactions with.
   • Generate and set a node JWT with write permissions and set in .env for the proxy to use.

Configure

Required and optional settings are best configured via a .env file. See example.env for configurable items.

cp example.env .env

# edit .env

Using containers

The images are available:

# ghcr:
docker pull ghcr.io/celestiaorg/pda-proxy

# Docker hub:
docker pull celestiaorg/pda-proxy

Don't forget you need to configure your environment.

Setup Host

As we don't want to embed huge files, secrets, and dev only example static files, you will need to place them on the host machine in the following paths:

1. Setup a DNS to point to your instance with email and domain.
2. Create and update an .env (see config.
3. Run ./scripts/setup_remote_host.sh or otherwise see the scripts to manually configure similarly.
   1. ONLY for development & testing! copy the unsafe example TLS files from ./service/static to app/static on the host
      • You should use:

      TLS_CERTS_PATH=/app/static/sample.pem
      TLS_KEY_PATH=/app/static/sample.rsa

Note that scripts run on the host update the /app/.env file with specific settings for the Celestia node. Logs will print very important information please read those carefully.

Running containers

With a correct setup of the host, you can startup both the proxy and local celestia node with:

docker compose --env-file /app/.env up -d

Or manually just the proxy itself:

# if you are developing from this repo:
just docker-run

# If you are only running:
source .env
mkdir -p $PDA_DB_PATH
# Note socket assumes running "normally" with docker managed by root
docker run --rm -it \
  --user $(id -u):$(id -g) \
  -v /var/run/docker.sock:/var/run/docker.sock \
 -v $PDA_DB_PATH:$PDA_DB_PATH \
  --env-file {{ env-settings }} \
  --env RUST_LOG=pda_proxy=debug \
  --network=host \
  -p $PDA_PORT:$PDA_PORT \
  "$DOCKER_CONTAINER_NAME"

Develop

First, some tooling is required:

1. Rust & Cargo - install instructions
2. SP1 zkVM Toolchain - install instructions
3. Protocol Buffers (Protobuf) compiler - official examples contain install instructions
4. (Optional) Just - a modern alternative to make installed
5. NVIDIA compiler & container toolkit https://docs.succinct.xyz/docs/sp1/generating-proofs/hardware-acceleration#software-requirements

Then:

1. Clone the repo

   git clone https://github.com/your-repo-name/pda-proxy.git
   cd pda-proxy

2. Choose a Celestia Node

   • See the How-to-guides on nodes to run one yourself, or choose a provider & set in env.
   • NOTE: You must have the node synced back to the oldest possible height you may encounter in calling this service for it to fulfill that request.

3. Configure required env variables

4. Build and run the service

   # NOT optimized, default includes debug logs printed
   just run-debug
   # Optimized build, to test realistic performance w/ INFO logs
   just run-release

There are many other helper scripts exposed in the justfile, get a list with:

# Print just recipes
just

Containerization

Docker and Podman are configured in Dockerfile to build an image with that includes a few caching layers to minimize development time & final image size -> publish where possible. To build and run in a container:

# Using just
just docker-build
just docker-run

# Manually

## Build
[docker|podman] build -t eq_service .

## Setup
source .env
mkdir -p $PDA_DB_PATH

## Run (example)
[docker|podman] run --rm -it -v $PDA_DB_PATH:$PDA_DB_PATH --env-file .env --env RUST_LOG=eq_service=debug --network=host -p $PDA_PORT:$PDA_PORT pda_proxy

Importantly, the DB should persist, and the container must have access to connect to the DA light client (likely port 26658) and Succinct network ports (HTTPS over 443).

The images are built and published for releases - see running containers for how to pull them.

License

MIT License

Acknowledgments

Based heavily on:

• https://github.com/eigerco/lumina/tree/main/rpc
• https://github.com/celestiaorg/eq-service