|
1 | 1 | # Local end-to-end testing |
2 | 2 |
|
3 | | -A step-by-step recipe for running the full migration flow on a single machine against a real Nextcloud, before the Cloudery is wired up. The unit tests in `test/` mock every network call; this guide is for the higher-confidence check that the service actually boots, consumes RabbitMQ messages, asks a Cloudery for a token, and drives the Stack through a real transfer. |
| 3 | +A step-by-step recipe for running the full migration flow on a single machine against a real Nextcloud, before the Cloudery is wired up. The [unit tests](development.md#testing) mock every network call; this guide is for the higher-confidence check that the service actually boots, consumes RabbitMQ messages, asks a Cloudery for a token, and drives the Stack through a real transfer. Environment variables referenced below are described in [Configuration](configuration.md). |
4 | 4 |
|
5 | 5 | ## What you need |
6 | 6 |
|
@@ -63,7 +63,7 @@ You should see `['migration']`. |
63 | 63 |
|
64 | 64 | ## 3. Mock the Cloudery |
65 | 65 |
|
66 | | -The consumer fetches its Stack token from the Cloudery, which is not available locally. A tiny Node HTTP server that returns a pre-minted Stack token on every call is enough. Save this somewhere outside the repo, for example `/tmp/cloudery-mock.js`: |
| 66 | +The consumer fetches its Stack token from the Cloudery, which is not available locally. A tiny Node HTTP server that returns a pre-minted Stack token on every call is enough. Unlike the real Cloudery, this mock ignores the request body (the consumer still sends the expected `audience` and `scope` fields; the real endpoint validates them). Save this somewhere outside the repo, for example `/tmp/cloudery-mock.js`: |
67 | 67 |
|
68 | 68 | ```js |
69 | 69 | const http = require('http') |
@@ -96,20 +96,20 @@ server.listen(PORT, '127.0.0.1', () => { |
96 | 96 | }) |
97 | 97 | ``` |
98 | 98 |
|
99 | | -Mint a long-lived CLI token that covers every doctype the consumer touches, and save it to a file: |
| 99 | +Mint a long-lived CLI token that covers every doctype the consumer touches. The same token is used for the trigger curl in step 5, so keep it in a variable or a file: |
100 | 100 |
|
101 | 101 | ```bash |
102 | | -cozy-stack instances token-cli cozy.localhost:8080 \ |
| 102 | +export STACK_TOKEN=$(cozy-stack instances token-cli cozy.localhost:8080 \ |
103 | 103 | io.cozy.nextcloud.migrations \ |
104 | 104 | io.cozy.remote.nextcloud.files \ |
105 | 105 | io.cozy.files \ |
106 | | - io.cozy.settings > /tmp/stack_token |
| 106 | + io.cozy.settings) |
107 | 107 | ``` |
108 | 108 |
|
109 | | -Start the mock: |
| 109 | +Start the mock in a new shell, passing the token through: |
110 | 110 |
|
111 | 111 | ```bash |
112 | | -STACK_TOKEN=$(cat /tmp/stack_token) node /tmp/cloudery-mock.js |
| 112 | +STACK_TOKEN="$STACK_TOKEN" node /tmp/cloudery-mock.js |
113 | 113 | ``` |
114 | 114 |
|
115 | 115 | ## 4. Build and run the consumer |
@@ -137,48 +137,48 @@ The consumer does not auto-load `.env`. Either pass the variables inline (as abo |
137 | 137 | When it is up you should see: |
138 | 138 |
|
139 | 139 | ```json |
140 | | -{"event":"rabbitmq.subscribed","exchange":"migration","queue":"migration.nextcloud.commands"} |
| 140 | +{"event":"rabbitmq.subscribed","exchange":"migration","queue":"migration.nextcloud.commands","routing_key":"nextcloud.migration.requested"} |
141 | 141 | ``` |
142 | 142 |
|
143 | 143 | ## 5. Trigger a migration |
144 | 144 |
|
145 | | -Mint a token for the caller (the Settings UI's scope in production) and POST to the Stack. Replace the URL, login, and password with your real Nextcloud credentials: |
| 145 | +POST to the Stack using the same `STACK_TOKEN` from step 3. Its scope is a superset of what the trigger endpoint needs. Replace the URL, login, and password with your real Nextcloud credentials: |
146 | 146 |
|
147 | 147 | ```bash |
148 | | -TOKEN=$(cozy-stack instances token-cli cozy.localhost:8080 io.cozy.nextcloud.migrations) |
149 | | - |
150 | | -curl -i -X POST http://cozy.localhost:8080/remote/nextcloud/migration \ |
151 | | - -H "Authorization: Bearer $TOKEN" \ |
| 148 | +MIGRATION_ID=$(curl -s -X POST http://cozy.localhost:8080/remote/nextcloud/migration \ |
| 149 | + -H "Authorization: Bearer $STACK_TOKEN" \ |
152 | 150 | -H "Content-Type: application/json" \ |
153 | 151 | -H "Accept: application/vnd.api+json" \ |
154 | 152 | -d '{ |
155 | 153 | "nextcloud_url": "https://your-nextcloud.example/", |
156 | 154 | "nextcloud_login": "you@example.org", |
157 | 155 | "nextcloud_app_password": "xxxxx-xxxxx-xxxxx-xxxxx-xxxxx", |
158 | 156 | "source_path": "/" |
159 | | - }' |
| 157 | + }' | python3 -c "import sys,json; print(json.load(sys.stdin)['data']['id'])") |
| 158 | + |
| 159 | +echo "migration: $MIGRATION_ID" |
160 | 160 | ``` |
161 | 161 |
|
162 | | -On success the Stack returns `201 Created` with the tracking document id. The consumer receives the RabbitMQ message within a second, calls the mock to get a Stack token, and starts walking the Nextcloud tree. |
| 162 | +On success the Stack returns `201 Created` with the tracking document id (captured above). The consumer receives the RabbitMQ message within a second, calls the mock to get a Stack token, and starts walking the Nextcloud tree. |
163 | 163 |
|
164 | 164 | ## 6. Verify |
165 | 165 |
|
166 | | -Watch the consumer logs for `migration.started`, one `migration.file_transferred` per file, and a final `migration.completed`. The tracking document can also be read straight from CouchDB: |
| 166 | +Watch the consumer logs for `migration.started`, one `migration.file_transferred` per file, and a final `migration.completed`. The tracking document can also be read straight from CouchDB, using the `$MIGRATION_ID` captured in step 5. Replace `admin:password` with whatever admin credentials your local CouchDB was installed with; they are independent from the RabbitMQ ones: |
167 | 167 |
|
168 | 168 | ```bash |
169 | 169 | PFX=$(cozy-stack instances show cozy.localhost:8080 \ |
170 | 170 | | python3 -c "import sys,json; print(json.load(sys.stdin)['attributes']['prefix'])") |
171 | 171 |
|
172 | 172 | curl -s -u admin:password \ |
173 | | - "http://localhost:5984/${PFX}%2Fio-cozy-nextcloud-migrations/<migration_id>" \ |
| 173 | + "http://localhost:5984/${PFX}%2Fio-cozy-nextcloud-migrations/${MIGRATION_ID}" \ |
174 | 174 | | python3 -m json.tool |
175 | 175 | ``` |
176 | 176 |
|
177 | 177 | Migrated files land under `/Nextcloud/...` in the target Cozy. You can also browse them from the Drive app at `http://cozy.localhost:8080/`. |
178 | 178 |
|
179 | 179 | ## Known gotchas |
180 | 180 |
|
181 | | -- **Stack CLI tokens do not expire** in cozy-stack's default configuration, so the mock can serve the same static token for an entire session. If you destroy and recreate the instance, re-mint the token and restart the mock. |
| 181 | +- **Stack CLI tokens are tied to the instance.** If you destroy and recreate `cozy.localhost:8080`, re-mint `STACK_TOKEN` and restart the mock; the previous token will be rejected by the new instance. |
182 | 182 | - **The Stack probe failure message is generic.** If the Stack returns `401 nextcloud credentials are invalid` but you know the password is right, the Nextcloud OCS probe endpoint likely returned a non-200 for reasons other than auth. Recent stacks probe OCS Core (`/ocs/v2.php/cloud/user`), which cannot be disabled. |
183 | 183 | - **Conflict handling is name-only.** If the target Cozy directory already contains a file with the same name as a Nextcloud file, the migration silently keeps the Cozy version, even when the bytes differ. To retry a clean migration, destroy the instance or delete the `/Nextcloud` tree in Drive before firing the next request. |
184 | 184 | - **Stuck tracking documents.** Only one `pending` or `running` migration is allowed per instance. If the consumer crashes mid-run, mark the stuck doc as `failed` in CouchDB (edit the `status` field) before triggering again, otherwise you will get a `409 Conflict`. |
|
0 commit comments