Important

This project is no longer maintained and the repository is archived.

The code is here and unchanged, but nobody is keeping it current with Kubo or the Google Cloud SDK; go.mod still targets Go 1.20. Expect to do some work before it builds against a recent Kubo.

One thing to know before you try: the ipfs init --profile gcsds step below depends on ipfs/kubo#9889, which was closed without being merged, so that profile does not exist in upstream Kubo.

The repository can be unarchived, but only for a named maintainer who commits to keeping it up. If that is you, reach out to the IPFS Foundation.

GCS Datastore Implementation for IPFS

An implementation of the go-datastore interface backed by Google Cloud Storage, GCS. The implementation is based on go-ds-s3

Building binary

Build this plugin as an included plugin with kubo.

Clone kubo.

git clone https://github.com/ipfs/kubo

Get the GCS datastore plugin and add it to the preload list.

(cd kubo; go get github.com/ipfs-shipyard/go-ds-gcs/plugin)
echo "gcsds github.com/ipfs-shipyard/go-ds-gcs/plugin 0" >> kubo/plugin/loader/preload_list

Build kubo with the plugin

(cd kubo; make build)

(Optionally) install kubo onto the local system.

(cd kubo; make install)

Building Docker container

Follow the "Building binary" section first. Clone the GCS datastore plugin and build the Docker entrypoint.

git clone https://github.com/ipfs-shipyard/go-ds-gcs
(cd go-ds-gcs; go build docker/entrypoint.go)

Build docker and container.

docker build -f go-ds-gcs/docker/Dockerfile -t ipfs .

(Optional) tag and push image to repository. Replace repository with your GCR repository.

docker tag ipfs gcr.io/repository/ipfs
docker push gcr.io/repository/ipfs

Deployment: Docker

ipfs is the image name from the previous section, "Building docker container"

docker run -d -p 4001:4001 -p 5001:5001 -p 8080:8080 --name=ipfs ipfs

Or you can use the tagged image from the previous section.

docker run -d -p 4001:4001 -p 5001:5001 -p 8080:8080 --name=ipfs gcr.io/repository/ipfs

Deployment: Kubernetes (GKE)

1. Create a GKE cluster. The node pool must have write permission to the GCS bucket.
2. Replace repository with your repository name in kubernetes/ipfs.yml
3. Replace mynamespace with your namespace name in kubernetes/ipfs.yml
4. Create namespace in GKE.

kubectl create namespace mynamespace

5. Deploy to GKE.

kubectl apply -f kubernetes/ipfs.yml

6. IPFS can now be accessed at ipfs.mynamespace from other pods in the GKE cluster:

curl http://ipfs.mynamespace:8080/...

Deployment: Local

For a brand new ipfs instance (no data stored yet):

1. Install kubo/cmd/ipfs/ipfs
2. Initialize IPFS. Replace mybucket with your bucket name. Note that this depends on this Pull Request.

KUBO_GCS_BUCKET=mybucket ipfs init --profile gcsds

4. To run as a server, also run

ipfs config profile apply server
ipfs config Addresses.Gateway /ip4/0.0.0.0/tcp/8080
ipfs daemon

Configuration

The config file should include the following. Replace mybucket with your bucket name.

{
	"Datastore": {
		"Spec": {
			"mounts": [{
				"child": {
					"bucket": "mybucket",
					"cachesize": 40000,
					"prefix": "ipfs",
					"type": "gcsds",
					"workers": 100
				},
				"mountpoint": "/blocks",
				"prefix": "flatfs.datastore",
				"type": "measure"
			}]
		}
	}
}

Google Cloud credentials

Google Cloud credentials should automatically be provided when running in Google Compute Engine (GCE) or Google Kubernetes Engine (GKE). Note that for both GCE and GKE, the (node) VM needs to have write permission (scope) to GCS. For GKE, this is achieved by creating the node pool with the "Storage read/write" scope, which is "devstorage.read_write".

In other environments, you may have to provide credentials. One way is to use the GOOGLE_APPLICATION_CREDENTIALS environment variable. See this document for more details.

Contribute

This repository is archived and is not accepting contributions. Drive-by patches will not reopen it. Unarchiving requires someone willing to be named as maintainer and to keep the plugin working against current Kubo. If that is you, contact the IPFS Foundation. Forking is fine too.

This repository falls under the IPFS Code of Conduct and the Google Code of Conduct

License

Apache 2.0