Rainbow

A to-be-released production-grade IPFS HTTP Gateway written in Go (using Boxo).

---

About

Rainbow is an implementation of the IPFS HTTP Gateway API, based on boxo which is the tooling that powers Kubo IPFS implementation. It uses the same Go code as the HTTP gateway in Kubo, but is fully specialized to just be a gateway:

• Rainbow acts as Amino DHT and Bitswap client only.
• Rainbow does not pin, or permanently store any content. It is just meant to act as gateway to content present in the network.
• Rainbow settings are optimized for production deployments and streamlined for specific choices (flatfs datastore, writethrough uncached blockstore etc.)
• Denylist and denylist subscription support is included.
• And more to come...

Building

go build

Running

rainbow

Use rainbow --help for documentation.

Docker

Automated Docker container releases are available from the Github container registry:

• 🟢 Releases
  • latest always points at the latest stable release
  • vN.N.N point at a specific release tag
• 🟠 Unreleased developer builds
  • main-latest always points at the HEAD of the main branch
  • main-YYYY-DD-MM-GITSHA points at a specific commit from the main branch
• ⚠️ Experimental, unstable builds
  • staging-latest always points at the HEAD of the staging branch
  • staging-YYYY-DD-MM-GITSHA points at a specific commit from the staging branch
  • This tag is used by developers for internal testing, not intended for end users

When using Docker, make sure to pass necessary config via -e:

$ docker pull ghcr.io/ipfs/rainbow:main-latest
$ docker run --rm -it --net=host -e RAINBOW_SUBDOMAIN_GATEWAY_DOMAINS=dweb.link ghcr.io/ipfs/rainbow:main-latest

See /docs/environment-variables.md.

Configuration

CLI and Environment Variables

Rainbow can be configured via command-line arguments or environment variables.

See rainbow --help and /docs/environment-variables.md for information on the available options.

Rainbow uses a --datadir (or RAINBOW_DATADIR environment variable) as location for persisted data. It defaults to the folder in which rainbow is run.

Peer Identity

Using a key file: By default generates a libp2p.key in its data folder if none exist yet. This file stores the libp2p peer identity.

Using a seed + index: Alternatively, random can be initialized with a 32-byte, b58 encoded seed and a derivation index. This allows to use the same seed for multiple instances of rainbow, and only change the derivation index.

The seed and index can be provided as command line arguments or environment vars (--seed , --seed-index). The seed can also be provided as a seed file in the datadir folder. A new random seed can be generated with:

rainbow gen-seed > seed

To facilitate the use of rainbow with systemd LoadCredential= directive, we look for both libp2p.key and seed in $CREDENTIALS_DIRECTORY first.

AutoConf

Rainbow supports automatic configuration of bootstrap peers, DNS resolvers, and HTTP routing endpoints through the autoconf feature. When enabled (default), Rainbow will fetch configuration from a remote URL and automatically expand the special auto placeholder value with network-appropriate defaults.

This feature can be configured via:

• --autoconf / RAINBOW_AUTOCONF: Enable/disable autoconf (default: true)
• --autoconf-url / RAINBOW_AUTOCONF_URL: URL to fetch configuration from (default: https://conf.ipfs-mainnet.org/autoconf.json)
• --autoconf-refresh / RAINBOW_AUTOCONF_REFRESH: How often to refresh configuration (default: 24h)

The auto placeholder can be used in:

• --bootstrap / RAINBOW_BOOTSTRAP: Bootstrap peer multiaddrs
• --http-routers / RAINBOW_HTTP_ROUTERS: HTTP routing endpoints
• --dnslink-resolvers / RAINBOW_DNSLINK_RESOLVERS: DNS-over-HTTPS resolvers

Note: When autoconf is disabled (--autoconf=false), using the auto placeholder will cause an error. You must provide explicit values for these configurations when autoconf is disabled.

Denylists

Rainbow can subscribe to append-only denylists using the --denylists flag. The value is a comma-separated list of URLs to subscribe to, for example: https://denyli.st/badbits.deny. This will download and update the denylist automatically when it is updated with new entries.

Denylists can be manually placed in the $RAINBOW_DATADIR/denylists folder too.

See NoPFS for an explanation of the denylist format. Note that denylists should only be appended to while Rainbow is running. Editing differently, or adding new denylist files, should be done with Rainbow stopped.

Blockstores

Rainbow ships with a number of possible blockstores for the purposes of caching data locally. Because Rainbow, as a gateway-only IPFS implementation, is not designed for long-term data storage there are no long term guarantees of support for any particular backing data storage.

See Blockstores for more details.

Garbage Collection

Over time, the datastore can fill up with previously fetched blocks. To free up this used disk space, garbage collection can be run.

Automatic Garbage Collection

Automatic GC is enabled by default, and configurable with RAINBOW_GC_INTERVAL and RAINBOW_GC_THRESHOLD.

Manual Garbage Collection

Garbage collection can also be manually triggered. This process can be automated by using a cron job.

The API route to trigger GC is http://$RAINBOW_CTL_LISTEN_ADDRESS/mgr/gc. The BytesToFree parameter must be passed as JSON in the POST request body to specify the upper limit of how much disk space should be cleared. GC will try to clear as much space as needed, up to BytesToFree, to create RAINBOW_GC_THRESHOLD of free space. Setting this parameter to a very high value will GC the entire datastore.

Example cURL commmand to run GC:

curl -v --data '{"BytesToFree": 1099511627776}' http://127.0.0.1:8091/mgr/gc

Logging

While the logging can be controlled via environment variable it is also possible to dynamically modify the logging at runtime.

• http://$RAINBOW_CTL_LISTEN_ADDRESS/mgr/log/level?subsystem=<system name or * for all system>&level=<level> will set the logging level for a subsystem
• http://$RAINBOW_CTL_LISTEN_ADDRESS/mgr/log/ls will return a comma separated list of available logging subsystems

Purging Peer Connections

Connections to a specific peer, or to all peers, can be closed and the peer information removed from the peer store. This can be useful to help determine if the presence/absence of a connection to a peer is affecting behavior. Be aware that purging a connection is inherently racey as it is possible for the peer to reestablish a connection at any time following a purge.

If RAINBOW_DHT_SHARED_HOST=false this endpoint will not show peers connected to DHT host, and only list ones used for Bitswap.

• http://$RAINBOW_CTL_LISTEN_ADDRESS/mgr/purge?peer=<peer_id> purges connection and info for peer identifid by peer_id
• http://$RAINBOW_CTL_LISTEN_ADDRESS/mgr/purge?peer=all purges connections and info for all peers
• http://$RAINBOW_CTL_LISTEN_ADDRESS/mgr/peers returns a list of currently connected peers

Example cURL commmand to show connected peers and purge peer connection:

curl http://127.0.0.1:8091/mgr/peers
curl http://127.0.0.1:8091/mgr/purge?peer=QmQzqxhK82kAmKvARFZSkUVS6fo9sySaiogAnx5EnZ6ZmC

Tracing

See docs/tracing.md.

Deployment

Suggested method for self-hosting is to run a prebuilt Docker image.

An ansible role to deploy Rainbow is available within the ipfs.ipfs collection in Ansible Galaxy (https://github.com/ipfs-shipyard/ansible). It includes a systemd service unit file.

Release

1. Create a PR from branch release-vX.Y.Z against main that:
   1. Tidies the CHANGELOG.md with the changes for the current release
   2. Updates the version.json file
2. Once the release checker creates a draft release, copy-paste the changelog into the draft
3. Merge the PR, the release will be automatically created once the PR is merged

License

Dual-licensed under MIT + Apache 2.0