You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+58-57Lines changed: 58 additions & 57 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,8 +1,10 @@
1
-
# ⇧📁 shifted-storage
1
+
# ⇧📁 shiftedstorage
2
2
3
-
*shifted-storage* is a tailored configuration of [Docker], [IPFS Cluster] and [Tailscale] that allows a trusted network of archives to cooperatively back up each other's data. This work is part of [Shift Collective]'s [Modeling Sustainable Futures: Exploring Decentralized Digital Storage for Community Based Archives] project, which was funded by the [Filecoin Foundation for the Decentralized Web]. For more details you can read reports linked from the project's homepage.
In a nutshell, the goal of *shifted-storage* is to provide an alternative to "big-tech" storage services, that is:
5
+
*shiftedstorage* is a tailored configuration of [Docker], [IPFS Cluster] and [Tailscale] that allows a trusted network of archives to cooperatively back up each other's data. This work is part of [Shift Collective]'s [Modeling Sustainable Futures: Exploring Decentralized Digital Storage for Community Based Archives] project, which was funded by the [Filecoin Foundation for the Decentralized Web]. For more details you can read reports linked from the project's homepage.
6
+
7
+
In a nutshell, the goal of *shiftedstorage* is to provide an alternative to "big-tech" storage services, that is:
6
8
7
9
-*Decentralized* instead of *Centralized*: the software is open source and can
8
10
be deployed on infrastructure that is operated by the members in their data centers,
@@ -15,12 +17,12 @@ In a nutshell, the goal of *shifted-storage* is to provide an alternative to "bi
15
17
any time.
16
18
-*Private* instead of *Public*: many peer-to-peer and distributed web systems
17
19
are built around the idea of data being globally available, and easy to
18
-
replicate. Data in *shifted-storage* is not made available to the
20
+
replicate. Data in *shiftedstorage* is not made available to the
19
21
wider IPFS network. The use of Tailscale allows peers to communicate
20
22
directly with each other using a virtual private mesh network, that only they
21
23
can see.
22
24
23
-
*shifted-storage* is really just a Docker Compose configuration for reliably bringing up Docker services that allows a network of shifted-storage instances to talk to each other. The containers are:
25
+
*shiftedstorage* is really just a Docker Compose configuration for reliably bringing up Docker services that allows a network of shiftedstorage instances to talk to each other. The containers are:
24
26
25
27
**tailscale*: a Tailscale client that establishes your node's connection to other trusted nodes in the mesh network.
26
28
**ipfs*: an IPFS daemon running on the Tailscale network.
@@ -35,63 +37,60 @@ Of course it's not all rainbows and unicorns, there are tradeoffs to this approa
35
37
* Tailscale makes establishing a virtual private mesh network easy using the open source Wireguard software and some of their own open source code and infrastructure. Howevver Tailscale are a company and could decide to change how they do things at any time.
36
38
* Tailscale doesn't have access to any of the stored data, but they do know the network topology of the IPFS cluster, and could be issued a subpoena in some jurisdictions that forces them to share who is a member of the network. Read more about this [here](https://tailscale.com/blog/tailscale-privacy-anonymity).
37
39
38
-
## Setup Bootstrap Node
39
-
40
-
*Note: if you are creating a shifted-storage node in an existing network jump down to the [Let Others Join](#let-others-join) section below.*
41
-
42
-
The first node in a *shifted-storage* network is known as the bootstrap node. It requires a bit more setup than subsequent nodes because the Tailscale mesh network needs to be created and configured, and a couple secret keys need to be defined. This bootstrap node will be used by subsequent nodes to find the rest of the network when they join.
40
+
## Install
43
41
44
-
### Tailscale
45
-
46
-
Fill this in! The end goal is to get a `TS_AUTHKEY`.
42
+
First, install the `ipfs` and `ipfs-cluster-ctl` command line utilities, and make sure they are in your system path. You don't need to run the cluster on your workstation, however having these utilities available makes it easy to talk to running nodes.
The security and privacy of your shifted-storage network is provided by two keys:
47
+
You will need to install the `shiftedstorage` utility which helps create the Docker compose file for the bootstrap node and then clone that for subsequent nodes in your network.
51
48
52
-
* IPFS Swarm Key
53
-
* IPFS Cluster Key
49
+
You can install it with:
54
50
55
-
You will want to keep these in a secure place and only share them with others using a secure, end-to-end encrypted communication channel like Signal or WhatsApp.
51
+
```
52
+
pip install shiftedstorage
53
+
```
56
54
57
-
To create the keys you can:
55
+
or if you have [uv]() installed you can just run it without installing it with `uvx`:
58
56
59
57
```
60
-
$ openssl rand -hex 32 > swarm.key
61
-
$ openssl rand -hex 32 > ipfs-cluster.key
58
+
uvx shiftedstorage
62
59
```
63
60
64
-
### Cluster Peer Name
61
+
##Setup Bootstrap Node
65
62
66
-
You will need to create a name for your node. This name will be used to create a Tailscale host name in your virtual private network. You could use an abbreviated form of your organization name, or your personal name.
63
+
*Note: if you are creating a shiftedstorage nodein an existing network jump down to the [Let Others Join](#let-others-join) section below.*
67
64
68
-
### Get the Docker Configuration
65
+
The first node in a *shiftedstorage* network is known as the bootstrap node. It requires a bit more setup than subsequent nodes because the Tailscale mesh network needs to be created and configured, and a couple secret keys need to be defined. This bootstrap node will be used by subsequent nodes to find the rest of the network when they join.
Be sure to also mention that any admins should get invited to the Tailscale so they can see it from their workstation.
72
+
73
+
### Create Compose File
74
+
75
+
Use your Tailscale token to create your bootstrap node, which here is named `bootstrap` but can be whatever you like. This will be the hostname of the bootstrap node in your Tailscale network.
This should write a Docker Compose configuration to `compose.yaml`.
82
+
83
+
**DANGER: Be careful to not make this compose file public since it contains secret keys!**
84
+
85
+
### Start Bootstrap
87
86
88
-
Now we are ready to run!
87
+
You can now start up your bootstrap node with:
89
88
90
89
```bash
91
90
$ docker compose up -d
92
91
```
93
92
94
-
If you want to stop the service at any time you can execute this command, as long as you are in the `shifted-storage` directory:
93
+
If you want to stop the service at any time you can execute this command, as long as you are in the `shiftedstorage` directory:
95
94
96
95
```
97
96
$ docker compose stop
@@ -101,51 +100,53 @@ $ docker compose stop
101
100
102
101
In order to let others join the network you will need to share a modified version of the compose file with them via a secure channel (e.g. WhatsApp or Signal).
103
102
104
-
The provided `bootstrap.py` utility will read your existing `compose.yml` and `.env` file and execute some commands in your running docker containers to determine additional information for new nodes to use when bootstrapping into the network:
105
-
106
-
- Tailscale IP
107
-
- IPFS Peer ID
108
-
- IPFS Cluster Peer ID
109
-
110
-
You need to supply a "node name" for the new node in your cluster. It's good to use a name without spaces or punctuation that will help you identify the node later since this is the hostname that it will appear under in the Tailscale network. For example, if you have are adding a node for "Warrior Women Project" you could:
You can then share the compose.yml file with the new member for them to use in their own Docker environment.
117
-
118
-
**DANGER: Be careful to share this file using a private channel with end-to-end encryption like Signal or WhatsApp. It contains secret keys for the Tailscale network as well as your IPFS Cluster.**
107
+
This will write out a `acme-compose.yml` file which you can share via a secure channel with someone running a machine at that organization.
119
108
120
-
## Joining a Network
109
+
You should be able to run this using `docker compose`, but for the original Shift-FFDW project we have been standardizing on partners using a [QNAP NAS](https://www.qnap.com/en-us/product/tbs-h574tx) which makes is easy to set up. You can use whatever device you want as long as you can run Docker on it and it has at least 4GB of memory (more couldn't hurt).
121
110
122
-
In order to join an existing *shifted-storage* network you will need to be given a `compose.yml` file by one of the other members.
123
-
124
-
You should be able to run this using Docker, but for our project we have been standardizing on QNAP devices that are running the Container Station application.
111
+
For people with a QNAP you can:
125
112
126
113
1. Install Container Station from Apps if it's not already available.
127
114
2. Open Container Station.
128
115
3. Click `Applications` option in the menu on the left.
129
116
4. Click the `Create` button.
130
-
5. In the Application Name box enter `shifted-storage`
117
+
5. In the Application Name box enter `shiftedstorage`
131
118
6. Paste the contents of the supplied `compose.yml` file into the text box.
132
119
7. Click the `create` button.
133
120
8. Click the `Containers` option in the menu on the left.
134
121
9. Verify that you see three containers running.
135
122
136
123
## Working With Storage
137
124
125
+
The shiftedstorage utility offers some functionality to add and remove content from storage. These are really just wrappers around the `ipfs-cluster-ctl` command, which you can choose to use directly of course.
0 commit comments