Extending Smelt

This guide covers how to customize and extend Smelt for your particular development needs. The environment is designed to be modified—that's rather the point of having it run locally.

Adding a New Service

Every service in Smelt lives in its own directory under systems/. This keeps concerns separated and makes it possible to run services individually when the full stack feels excessive.

Creating the System Directory

Create a directory structure following the established pattern:

systems/my-service/
├── compose.yml        # Docker Compose service definition
├── config/            # Configuration files (mounted into container)
├── entrypoint.sh      # (optional) Initialization script
└── README.md          # Documentation for this system

The config/ directory is optional but recommended. Configuration files mounted from here can be edited without rebuilding images—a convenience you'll appreciate during development.

Compose File Template

Here's a working template that follows the conventions used throughout Smelt:

# My Service System - Brief description
#
# Longer description of what this service does and why it exists.
# Used by: list services that depend on this one

services:
  my-service:
    image: myorg/my-service:dev
    ports:
      - "15XXX:80"  # Host:Container - pick the next free slot in smelt's 15XXX range
    volumes:
      - my-service-data:/data
      - ../../generated/keys/my-service.pem:/keys/my-service.pem:ro
      - ./config:/config:ro
    environment:
      - LOG_LEVEL=info
      - MY_SERVICE_DID=did:web:my-service
    healthcheck:
      test: ["CMD", "curl", "-sf", "http://localhost:80/health"]
      # start_interval (docker engine 25+) polls every 1s during start_period
      # so compose's depends_on gate detects ready services quickly. Drops to
      # `interval` once the first probe succeeds. Required for smelt to keep
      # its sub-15s snapshot-restored boot time.
      start_interval: 1s
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 10s
    depends_on:
      some-dependency:
        condition: service_healthy
    restart: unless-stopped
    networks:
      - storacha-network

volumes:
  my-service-data:

A few notes on conventions:

Internal port 80: Services that need did:web resolution should listen on port 80 internally. The did:web specification defaults to port 80, so did:web:my-service resolves to http://my-service:80/.well-known/did.json within the Docker network.
Read-only mounts: Use :ro for keys and config files. This prevents accidental writes and makes your intentions clear.
Health checks: Required for any service that others depend on. The depends_on condition service_healthy doesn't work without them.
start_period: Services that need initialization time (piri takes nearly three minutes) should set this generously. The health check won't mark the container unhealthy during this period.

Including in Root Compose

Add your system to the root compose.yml:

include:
  - path: systems/blockchain/compose.yml
  - path: systems/common/compose.yml
  - path: systems/signing-service/compose.yml
  - path: systems/delegator/compose.yml
  - path: systems/indexing/compose.yml
  - path: systems/piri/compose.yml
  - path: systems/upload/compose.yml
  - path: systems/guppy/compose.yml
  - path: systems/my-service/compose.yml  # Add your service here

Order matters only insofar as it affects readability. Docker Compose resolves dependencies from the depends_on declarations, not from include order.

Network Connectivity

All services connect to storacha-network, an external Docker network created during make init. This provides:

DNS resolution: Service names are resolvable as hostnames. From any container, http://piri:3000 reaches piri.
Isolation: Only services on this network can communicate. Your host machine accesses services through published ports.

Example cross-service communication:

environment:
  - PIRI_ENDPOINT=http://piri-0:3000
  - INDEXER_ENDPOINT=http://indexer:80
  - BLOCKCHAIN_RPC=ws://blockchain:8545

If your service needs to be reached by others, ensure it binds to 0.0.0.0 (not localhost or 127.0.0.1).

Generating Service Keys

Service keys are generated by pkg/generate/keys.go as part of make generate (which runs the Go CLI at cmd/smelt/). To add a new service that needs an Ed25519 identity key, add its name to the nonPiriServiceKeys slice in pkg/generate/keys.go:

// pkg/generate/keys.go
var nonPiriServiceKeys = []string{
    "delegator",
    "indexer",
    "upload",
    // ...
    "my-service",
}

Then regenerate and restart:

make regen
make clean && make up

This produces generated/keys/my-service.pem (private) and generated/keys/my-service.pub (public). Mount the .pem file into your container via the compose file.

Key generation is idempotent — existing keys are preserved across runs. Use make regen or go run ./cmd/smelt generate --force to overwrite.

Customizing Configuration

Environment Variables

The .env file in the project root overrides default values. Copy the example:

cp .env.example .env

Available variables:

Variable	Default	Description
`ANVIL_BLOCK_TIME`	3	Seconds between blockchain blocks
`UPLOAD_LOG_LEVEL`	info	Upload service verbosity
`INDEXER_LOG_LEVEL`	info	Indexer verbosity
`PIRI_LOG_LEVEL`	info	Piri verbosity

Host ports are not configurable via environment variables — they live directly in the per-service compose.yml files, all within smelt's dedicated 15XXX range. Edit the compose.yml file if you need to change one.

Variables are referenced in compose files using ${VARIABLE:-default} syntax.

Service-Specific Configuration

Each service has configuration files in systems/<service>/config/:

File	Service	Format
`piri-base-config.toml`	piri	TOML
`delegator.yaml`	delegator	YAML
`guppy-config.toml`	guppy	TOML
`signer.yaml`	signing-service	YAML

These files are mounted read-only into containers. Edit them, then restart the affected service:

docker compose restart piri

For changes to take effect without restart, some services support configuration reload signals—check individual service documentation.

Contract Addresses

Smart contract addresses are baked into systems/blockchain/state/deployed-addresses.json and must match values in service configs. If you deploy new contracts, update:

systems/piri/config/piri-base-config.toml — The [pdp.contracts] section
systems/delegator/config/delegator.yaml — The contract section
systems/signing-service/compose.yml — The --service-contract-address argument

The default addresses work with the pre-deployed state in systems/blockchain/state/anvil-state.json.

Adding Network Simulation

Network problems are inevitable in distributed systems. Simulating them locally is faster than waiting for production to surface issues.

Using Toxiproxy

Toxiproxy sits between services and introduces configurable failures: latency, packet loss, bandwidth throttling, connection resets.

Create the system directory:

systems/network-chaos/
├── compose.yml
└── toxiproxy.json

compose.yml:

# Network Chaos System - Toxiproxy for network simulation
#
# Introduces controllable network failures between services.
# Useful for testing retry logic, timeouts, and resilience.

services:
  toxiproxy:
    image: ghcr.io/shopify/toxiproxy:2.7.0
    ports:
      - "15200:8474"  # Toxiproxy API (host 15200, within smelt's 15XXX range)
      - "15201:3334"  # Proxy to piri
      - "15202:9001"  # Proxy to indexer
    volumes:
      - ./toxiproxy.json:/config/toxiproxy.json:ro
    command: ["-config", "/config/toxiproxy.json"]
    networks:
      - storacha-network

toxiproxy.json:

[
  {
    "name": "piri-proxy",
    "listen": "0.0.0.0:3334",
    "upstream": "piri-0:3000"
  },
  {
    "name": "indexer-proxy",
    "listen": "0.0.0.0:9001",
    "upstream": "indexer:80"
  }
]

Applying Toxics

With Toxiproxy running, apply network conditions via its API:

# Add 200ms latency to piri requests
curl -X POST http://localhost:15200/proxies/piri-proxy/toxics \
  -H "Content-Type: application/json" \
  -d '{"name":"latency","type":"latency","attributes":{"latency":200}}'

# Add 10% packet loss (connections dropped)
curl -X POST http://localhost:15200/proxies/piri-proxy/toxics \
  -H "Content-Type: application/json" \
  -d '{"name":"timeout","type":"timeout","attributes":{"timeout":0},"toxicity":0.1}'

# Limit bandwidth to 1KB/s (simulates slow networks)
curl -X POST http://localhost:15200/proxies/piri-proxy/toxics \
  -H "Content-Type: application/json" \
  -d '{"name":"bandwidth","type":"bandwidth","attributes":{"rate":1}}'

# Remove a toxic
curl -X DELETE http://localhost:15200/proxies/piri-proxy/toxics/latency

# Reset all toxics on a proxy
curl -X POST http://localhost:15200/proxies/piri-proxy/toxics/populate -d '[]'

Routing Traffic Through Toxiproxy

Configure clients to connect through the proxy instead of directly. For example, modify guppy-config.toml to use the proxy port:

[network]
# Instead of http://piri:3000, route through toxiproxy
upload_url = "http://toxiproxy:3334"

Connecting to External Services

Smelt defaults to entirely local services. Sometimes you need to test against real infrastructure.

Using a Real Blockchain

To connect piri and signing-service to an external EVM chain:

Update signing-service in systems/signing-service/compose.yml:

command: [
  "--host", "0.0.0.0",
  "--port", "7446",
  "--rpc-url", "wss://your-chain-endpoint.example.com",  # External RPC
  "--service-contract-address", "0x...",  # Deployed contract address
  "--signing-key-path", "/keys/payer-key.hex",
  "--service-key-file", "/keys/signing-service.pem",
  "--service-did", "did:web:signing-service"
]

Update piri config in systems/piri/config/piri-base-config.toml:

[pdp]
chain_id = "ACTUAL_CHAIN_ID"  # e.g., "314159" for Filecoin Calibration
payer_address = "0x..."

[pdp.contracts]
verifier = "0x..."
provider_registry = "0x..."
service = "0x..."
service_view = "0x..."
payments = "0x..."
usdfc_token = "0x..."

Update delegator config in systems/delegator/config/delegator.yaml:

contract:
  chain_client_endpoint: "wss://your-chain-endpoint.example.com"
  payments_contract_address: "0x..."
  service_contract_address: "0x..."
  registry_contract_address: "0x..."
  transactor:
    chain_id: ACTUAL_CHAIN_ID
    key: "0x..."  # Funded wallet private key

Ensure your wallet is funded on the target chain. PDP operations cost gas.
Stop local blockchain by commenting it out of compose.yml or removing its depends_on references.

Using External IPNI

To query production IPNI for content discovery:

Update systems/indexing/indexer/compose.yml:

environment:
  - IPNI_ENDPOINT=https://cid.contact

Note that you cannot announce to production IPNI from a local piri—production IPNI won't accept advertisements from localhost. This configuration is for reading only.

Building and Pushing Custom Images

Building Locally

# Build all images (only those with build contexts)
make build

# Build specific service
docker compose build upload

# Build with no cache (when you suspect caching issues)
docker compose build --no-cache upload

# Build with build arguments
docker compose build --build-arg VERSION=1.2.3 piri

Using Custom Image Tags

Parameterize image tags in compose files:

services:
  my-service:
    image: myorg/my-service:${MY_SERVICE_TAG:-dev}

Then run with a specific version:

MY_SERVICE_TAG=v1.2.3 make up

Or set it in your .env file for persistence.

Pushing to a Registry

# Tag with your registry
docker tag smelt-upload:latest ghcr.io/yourorg/upload:dev

# Push (ensure you're logged in: docker login ghcr.io)
docker push ghcr.io/yourorg/upload:dev

Using Local Builds of Service Repositories

If you're developing a service (piri, guppy, indexer) and want to test local changes:

Build the image locally in that repository
Tag it with the name Smelt expects:

# In piri repository
docker build -t forreststoracha/piri:dev .

# Or use docker compose build if piri's compose.yml has a build context

Restart Smelt to pick up the new image:

make down && make up

Running Individual Systems Standalone

Each system can run independently if its dependencies are available. This is useful for focused testing.

# Just blockchain (no dependencies)
cd systems/blockchain && docker compose up -d

# Just common services (DynamoDB)
cd systems/common && docker compose up -d

# Indexing stack (needs redis, which is included)
cd systems/indexing && docker compose up -d

Note that most systems declare external: true for storacha-network, so you must create it first:

docker network create storacha-network

Or run make init once to handle all setup.

Dependency Map

For standalone operation, ensure these dependencies are running:

System	Requires
blockchain	—
common (dynamodb-local)	—
signing-service	blockchain
delegator	blockchain, dynamodb-local
indexing (ipni + indexer)	redis (included)
piri	blockchain, signing-service, delegator, indexer, dynamodb-local
upload	piri, indexer, dynamodb-local
guppy	upload, piri

Adding More Piri Nodes

Multi-piri is declarative. To run additional piri nodes — for replication tests, provider-failover scenarios, or mixed-backend configurations — edit smelt.yml at the repo root:

version: 1
piri:
  nodes:
    - storage: { db: sqlite,   blob: filesystem }  # piri-0
    - storage: { db: postgres, blob: filesystem }  # piri-1
    - storage: { db: sqlite,   blob: s3 }          # piri-2

Then run make up (or make generate first, then make up). The generator does the following for each new node:

Allocates an Anvil wallet (up to 9 nodes total, limited by pre-funded accounts)
Creates keys: generated/keys/piri-{N}.pem and generated/keys/piri-{N}-wallet.hex
Emits a service definition into generated/compose/piri.yml exposed on host port 15100 + N
Includes shared piri-postgres and/or piri-minio services if any node uses those backends
Creates per-node postgres databases (piri_{N}) via the piri-postgres-init sidecar
Sets per-node S3 bucket prefix (piri-{N}-) via environment variable

Hot-add and hot-remove work the same way: edit smelt.yml, run make up. Existing containers and data volumes are preserved; only the added or removed services change. Removed containers' data volumes and per-node postgres databases are left behind; use make clean to wipe them.

See docs/MULTI_PIRI.md for the full manifest schema, Anvil wallet mapping, and design trade-offs.

Debugging Tips

Verbose Logging

Most services respect LOG_LEVEL or equivalent environment variables:

environment:
  - LOG_LEVEL=debug
  - RUST_LOG=debug  # For Rust services

Restart the service after changing:

docker compose restart piri

Inspecting UCAN Invocations

At debug log level, services log UCAN invocations and receipts. Search for:

invocation — incoming UCAN requests
receipt — responses to invocations
capability — specific permissions being invoked
delegation — capability delegations being used

docker compose logs -f piri 2>&1 | grep -E "(invocation|receipt|capability)"

Checking DynamoDB State

DynamoDB Local provides a web shell at http://localhost:15010/shell/

To list tables:

var dynamodb = new AWS.DynamoDB({
  endpoint: 'http://localhost:15010',
  region: 'us-west-1'
});
dynamodb.listTables({}, function(err, data) {
  console.log(data.TableNames);
});

Or use the AWS CLI:

aws dynamodb list-tables --endpoint-url http://localhost:15010 --region us-west-1

aws dynamodb scan --table-name delegator-allow-list \
  --endpoint-url http://localhost:15010 --region us-west-1

Inspecting Blockchain State

Query the local Anvil chain via JSON-RPC:

# Get latest block number
curl -s -X POST http://localhost:15000 \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' | jq

# Get chain ID
curl -s -X POST http://localhost:15000 \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_chainId","params":[],"id":1}' | jq

# Get account balance (replace address)
curl -s -X POST http://localhost:15000 \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_getBalance","params":["0x70997970C51812dc3A010C7d01b50e0d17dc79C8","latest"],"id":1}' | jq

# Get contract storage slot
curl -s -X POST http://localhost:15000 \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_getStorageAt","params":["0x0DCd1Bf9A1b36cE34237eEaFef220932846BCD82","0x0","latest"],"id":1}' | jq

Inspecting Container State

# Shell into a running container
docker compose exec piri bash

# View container environment
docker compose exec piri env

# Check container filesystem
docker compose exec piri ls -la /data/piri

# View container logs with timestamps
docker compose logs -t piri | tail -100

Network Debugging

# Test connectivity between containers
docker compose exec guppy curl -v http://piri-0:3000/

# Check DNS resolution
docker compose exec guppy nslookup piri-0

# List containers on the network
docker network inspect storacha-network --format '{{range .Containers}}{{.Name}} {{end}}'

Resetting Individual Services

Sometimes a service gets into a bad state. Reset it without affecting others:

# Stop and remove container + volume
docker compose rm -sf piri-0
docker volume rm smelt_piri-0-data

# Restart just that service
docker compose up -d piri-0

For a complete reset, make fresh removes everything and rebuilds from scratch.

Writing System Documentation

Each system should have a README.md following this structure:

# System Name

Brief description of what this system does.

## Services

- **service-name** - What this service does

## Ports

| Host Port | Container Port | Service | Description |
|-----------|----------------|---------|-------------|
| 15XXX | ... | service | What this port exposes |

## Configuration

- `config/file.toml` - What this configures

## Keys

- `../../generated/keys/service.pem` - What this key is for

## Volumes

- `volume-name` - What data this persists

## Dependencies

- dependency-name (service_healthy)

## Used By

- downstream-service

This consistency helps developers understand unfamiliar systems quickly.

Summary

Smelt is designed to be modified. The conventions described here—system directories, compose patterns, network topology, key management—exist to make modifications predictable. When in doubt, examine how existing systems are structured and follow the established patterns.

The goal is a development environment that stays out of your way while you work on the interesting problems.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extending Smelt

Adding a New Service

Creating the System Directory

Compose File Template

Including in Root Compose

Network Connectivity

Generating Service Keys

Customizing Configuration

Environment Variables

Service-Specific Configuration

Contract Addresses

Adding Network Simulation

Using Toxiproxy

Applying Toxics

Routing Traffic Through Toxiproxy

Connecting to External Services

Using a Real Blockchain

Using External IPNI

Building and Pushing Custom Images

Building Locally

Using Custom Image Tags

Pushing to a Registry

Using Local Builds of Service Repositories

Running Individual Systems Standalone

Dependency Map

Adding More Piri Nodes

Debugging Tips

Verbose Logging

Inspecting UCAN Invocations

Checking DynamoDB State

Inspecting Blockchain State

Inspecting Container State

Network Debugging

Resetting Individual Services

Writing System Documentation

Summary

FilesExpand file tree

EXTENDING.md

Latest commit

History

EXTENDING.md

File metadata and controls

Extending Smelt

Adding a New Service

Creating the System Directory

Compose File Template

Including in Root Compose

Network Connectivity

Generating Service Keys

Customizing Configuration

Environment Variables

Service-Specific Configuration

Contract Addresses

Adding Network Simulation

Using Toxiproxy

Applying Toxics

Routing Traffic Through Toxiproxy

Connecting to External Services

Using a Real Blockchain

Using External IPNI

Building and Pushing Custom Images

Building Locally

Using Custom Image Tags

Pushing to a Registry

Using Local Builds of Service Repositories

Running Individual Systems Standalone

Dependency Map

Adding More Piri Nodes

Debugging Tips

Verbose Logging

Inspecting UCAN Invocations

Checking DynamoDB State

Inspecting Blockchain State

Inspecting Container State

Network Debugging

Resetting Individual Services

Writing System Documentation

Summary