feat(storage): support operator-approved provider replacement

Author: strahe

cmd/synaps3/main.go

@@ -319,6 +319,10 @@ func runServe(ctx context.Context, src config.Source) error {
 			Receipts:      walletReceiptClient,
 			Readiness:     filecoinReadiness,
 			Observability: observabilityChecker,
+			Terminator:    storageClient,
+			// The epoch comes from the same node that reports wallet receipts,
+			// so replacement adds no new RPC connection.
+			Epochs: synapse.NewChainEpochReader(walletReceiptClient),
 		},
 		ProviderIdentity: admin.NewProviderIdentityResolver(client.SPRegistry(), cfg.Filecoin.RPCURL, logger),
 		Logger:           logger,

docs/en/concepts/filecoin-storage-flow.md

@@ -52,14 +52,24 @@ Retry after restoring RPC connectivity, storage provider reachability, wallet fu
 
 Health checks record storage provider and local data set status. The dashboard uses those results to show copies that are `unavailable`, `degraded`, or `unknown`.
 
-If an established provider becomes temporarily unavailable while the initial copies are still being stored, SynapS3 keeps using the other assigned writable copies. The unfinished copy waits without consuming retries and resumes automatically when the original provider becomes reachable again. SynapS3 does not automatically select a replacement provider. Repairing copies that became unavailable after storage completed remains part of the planned replica repair feature below.
+If an established provider becomes temporarily unavailable while the initial copies are still being stored, SynapS3 keeps using the other assigned writable copies. The unfinished copy waits without consuming retries and resumes automatically when the original provider becomes reachable again. SynapS3 does not automatically select a replacement provider; see [Replace a Storage Provider](#replace-a-storage-provider) for the operator-approved path. Repairing copies that became unavailable after storage completed remains part of the planned replica repair feature below.
 
 ## Target and Minimum Replicas
 
 The target replica count is frozen when an upload starts. By default, **Release cache after** is **All replicas (strict)**: every target replica frozen for that upload must be readable and committed. A bucket can instead set an explicit count from 1 through the current target. An explicit count stays if Replicas later increases; lowering Replicas below that count is rejected until Release cache after is also lowered. **All replicas (strict)** follows each upload's frozen target. Once that threshold is met, the version becomes stored and its cache follows the configured eviction policy, while remaining replicas continue until the upload's target is reached. The dashboard keeps showing replica sync progress until every frozen target replica is done.
 
 Changing the target affects new uploads. Changing the minimum also re-evaluates retained cache for current uploads. Increasing the minimum does not move versions that are already stored back to an earlier state and cannot restore cache that has already been deleted.
 
+## Replace a Storage Provider
+
+When a provider becomes permanently unavailable, or you plan to move away from one, open the bucket in the dashboard, choose **Details**, then **Storage** → **Data Sets**, and replace the provider. This is always an explicit decision: SynapS3 never swaps a provider on its own, because doing so creates a new paid service and changes where your data lives.
+
+One confirmation covers the whole move. SynapS3 creates the new storage service, switches new uploads to it once it is ready, copies existing data across, and only then shuts down the old provider. Objects copy from another replica or from local cache. An object with neither cannot be copied, and the old provider is not shut down. Data that can still be read from the old provider stays readable until every retained version is readable on the new one.
+
+While replacements run, the replica shows every move that still needs progress or operator attention, including parallel moves on other replicas. Progress is counted in stored items, since content shared by several versions is copied once. Once copying is finished, the dashboard separates content that was transferred from content that was deleted before it needed to move.
+
+Some steps wait rather than fail. The dashboard distinguishes creating the new service, waiting for it to become writable, an unreachable provider, wallet funds, and missing readable content. Most waits resume on their own. If an object has no other replica and no local cache, replacement stays waiting until one is available. Retry from the same Data Sets list when work has run out of attempts, or when shutting down the old provider needs a payment settled first. A target already in use cannot be retried; choose another provider.
+
 ## What Users See
 
 - S3 upload can succeed before Filecoin storage finishes.

docs/en/operations/troubleshooting.md

@@ -132,6 +132,8 @@ Retry only after RPC connectivity, storage provider availability, wallet funds,
 synaps3 admin task retry 42
 ```
 
+Provider replacement work is the exception: do not retry it from Tasks. Finished or stopped replacement tasks provide **Open Data Sets**, which opens the affected bucket directly at **Details** → **Storage** → **Data Sets**. Use **Retry replacement** only when that action is shown. If the selected provider already stores this bucket, choose a different provider instead.
+
 ## Provider or RPC Issues
 
 Check provider health and Filecoin readiness in the dashboard, or inspect the Admin API:

docs/en/operations/upgrade-recovery.md

@@ -61,7 +61,8 @@ Receive write -> save object -> record metadata -> return success -> continue ba
 | Private provider URL blocked | Keep blocked by default; enable `filecoin.allow_private_networks` only for trusted private deployments. |
 | Database full | Free space or scale the database. |
 | Cache disk full | Increase disk, raise `cache.max_size_gb`, or restore upload and eviction progress. |
-| Process crash | Restart the service, then verify health and task statistics; unfinished tasks become eligible to continue. |
+| Provider is permanently unavailable, or must be evacuated | Open the bucket, choose **Details**, then **Storage** → **Data Sets**, and replace the provider. New uploads move to the new provider once it is ready. Existing objects copy from another replica or from local cache; an object with neither cannot be copied, and the old provider is not shut down. If the selected target is already in use, choose another provider rather than retrying it. |
+| Process crash | Restart the service, then verify health and task statistics; unfinished tasks become eligible to continue. If shutting down an unused replacement service was already submitted, SynapS3 continues checking that request instead of submitting it again. |
 
 A provider becoming unavailable after a copy has already been stored does not necessarily create a retryable task. Use storage-health views to identify affected copies. Restoring the target copy count is part of [Planned Replica Repair](../concepts/filecoin-storage-flow.md#planned-replica-repair).
 

docs/en/reference/admin-api.md

@@ -93,6 +93,7 @@ Treat these endpoints as change-window operations. They can change data, credent
 | S3 users | `POST /api/v1/s3-users`, `PUT /api/v1/s3-users/{accessKey}`, `POST /api/v1/s3-users/{accessKey}/secret`, `DELETE /api/v1/s3-users/{accessKey}` | Changes client access or invalidates credentials. |
 | Buckets and objects | bucket create, owner/copy-policy updates, object upload/download/delete/restore/permanent-delete | Changes or exposes user-visible S3 data and metadata. |
 | Tasks and storage health | task retry, diagnostic refresh, storage provider and data set refresh | Requeues work or refreshes operational status. |
+| Provider replacement | `POST /api/v1/buckets/{name}/data-sets/{id}/replacement`, `POST /api/v1/storage-replacements/{id}/retry` | Creates a new paid storage service, moves a replica to it, and ends the old service. |
 
 ## Health and Metrics
 
@@ -130,6 +131,9 @@ Treat these endpoints as change-window operations. They can change data, credent
 | `POST` | `/api/v1/buckets/{name}/objects/permanent-delete` | Permanently delete an object version. |
 | `POST` | `/api/v1/buckets/{name}/objects/deleted/permanent-delete` | Permanently delete a deleted object version. |
 | `GET` | `/api/v1/buckets/{name}/storage-health/affected-versions` | List versions affected by storage health issues. |
+| `GET` | `/api/v1/buckets/{name}/data-sets/{id}/replacement/providers` | List the providers this replica can move to, and why the others cannot take it. |
+| `POST` | `/api/v1/buckets/{name}/data-sets/{id}/replacement` | Authorize replacing the storage provider behind a replica. |
+| `POST` | `/api/v1/storage-replacements/{id}/retry` | Resume a failed or attention-holding provider replacement. |
 
 For object upload, the HTTP `Content-Type` is the uploaded object's content type. It is not a JSON request marker.
 
@@ -189,6 +193,68 @@ A missing or permanently deleted source returns `404 Not Found`; insufficient ca
 
 The restore streams synchronously for up to one hour and requires enough cache capacity for the new destination version.
 
+### Replace a Storage Provider
+
+`POST /api/v1/buckets/{name}/data-sets/{id}/replacement` is the only way to replace the storage provider behind a replica. One confirmation authorizes all of it: a new paid storage service, moving new uploads to it, copying existing data across, and ending the old service once every retained version is readable on the new provider. Objects copy from another replica or from local cache. An object with neither cannot be copied, and the old provider is not shut down.
+
+Choose the new provider automatically:
+
+```json
+{ "mode": "automatic", "client_request_id": "019d2e22-8c36-7d5b-a6be-5f7fa6d6f584" }
+```
+
+Automatic selection excludes every provider the bucket has ever used, including retired ones. Or name the provider yourself:
+
+```json
+{ "mode": "manual", "provider_id": "202", "client_request_id": "019d2e22-8c36-7d5b-a6be-5f7fa6d6f584" }
+```
+
+A named provider must appear in the complete available, active, PDP-capable provider inventory. It may be one the bucket used before, provided that earlier service has already been retired. The provider being replaced, and any provider still holding a live generation of this bucket, are rejected.
+
+`client_request_id` is required after trimming and must contain 1–128 characters. The first successful request returns `201 Created`. Replaying the same bucket, source, mode, and manual provider with the same ID returns the original record and `200 OK`, even after the replica has switched. Reusing the ID with different parameters returns `409 Conflict` with `replacement_idempotency_conflict`. An automatic replay is resolved before reading the provider inventory, so a later inventory change cannot choose a different provider.
+
+Confirming again for the same replica supersedes the earlier request and returns `201 Created`; it is not a conflict. The unused provider from the earlier request is shut down. `replacement_active` means something else: the replica is the target of another unfinished replacement, which has to be resolved first.
+
+Only the replica that currently receives writes can be replaced; a historical generation is reported with `"replaceable": false` in `GET /api/v1/buckets/{name}`.
+
+A successful first confirmation returns `201 Created` with the replacement record; an exact replay returns `200 OK`. `GET /api/v1/buckets/{name}` returns the bucket's recent replacements in `replacements`, newest first, up to the 50 most recent.
+
+Replacement moves through these states:
+
+| Status | Meaning |
+| --- | --- |
+| `preparing_target` | The new service is being created. Writes still go to the current provider. |
+| `migrating` | The new provider receives new uploads while existing data is copied across. |
+| `waiting` | Paused. `wait_reason` and `wait_message` distinguish service creation (`target_creating`), writable confirmation (`target_writable`), an unreachable provider (`target`), funding, source availability, and safe retirement waits. Most waits resume without action. |
+| `retiring` | Everything is copied and the old service is being ended. |
+| `cleanup_attention` | Ending the old service needs an operator decision, such as settling payment debt. |
+| `failed` | Work ran out of attempts and needs to be retried. |
+| `completed` | The old service is ended and the replica now lives on the new provider. |
+| `superseded` | A later confirmation replaced this request. |
+
+`last_error` is set only for `failed` and `cleanup_attention`, and is cleared by a retry. Waiting never sets it, because waiting is not a failure. A failed response may also include `failure_reason`. `target_in_use` is permanent for that approved target: choose a different provider; the retry endpoint returns a conflict.
+
+`items_total` and `items_copied` count unique stored content, not object versions: content shared by many versions is copied once. Content deleted while migration is in progress is no longer needed and is not counted as copied. After copying finishes, the response reports how much content was copied and how much no longer needed to move; `items_copied/items_total` is not a completion percentage. The confirmation dialog instead counts referenced versions and total size.
+
+`POST /api/v1/storage-replacements/{id}/retry` resumes a `failed` or `cleanup_attention` replacement on the same approved provider.
+
+Choosing a different provider requires a new confirmation, and is only available while the retiring provider still holds the replica. Once the new provider has taken the replica over, each generation holds data the other does not, so confirming again on either one is refused (`replacement_source_not_current` on the old, `replacement_active` on the new) and the approved copy has to be finished with retry.
+
+Conflicts return `409 Conflict` with a stable code:
+
+```json
+{
+  "error": "that provider already stores a replica of this bucket",
+  "code": "replacement_target_in_use"
+}
+```
+
+The codes are `replacement_active`, `replacement_target_in_use`, `replacement_target_unavailable`, `replacement_no_eligible_provider`, `replacement_source_not_current`, `replacement_superseded`, `replacement_not_retryable`, `replacement_task_running`, and `replacement_idempotency_conflict`. An invalid provider choice returns `400 Bad Request` with `replacement_target_invalid`; a currently unavailable manual target returns `400` with `replacement_target_unavailable`; an unknown bucket, data set, or replacement returns `404 Not Found`; an unavailable storage service returns `503 Service Unavailable`; internal failures return `500 Internal Server Error`.
+
+`GET /api/v1/buckets/{name}/data-sets/{id}/replacement/providers` lists every provider currently reported available with `eligible`, an `ineligible_reason` of `current_source` or `already_serves_bucket`, and `previously_used` for a provider this bucket has used and fully retired. Providers that cannot take the replica are listed rather than omitted, so an operator can see why one they expected is unavailable. It is the same inventory the storage topology reports under the `Available` filter, so a provider listed there is offered here and an unreachable one is offered in neither. Eligibility is the same rule the confirmation enforces. Automatic selection is stricter still: it never returns to a provider this bucket has used, which a manual choice may.
+
+Confirmation only checks what SynapS3 has recorded. A provider that still runs a storage service for this bucket on chain is detected when the replacement prepares its target: the replacement stops at `failed` with the provider and data set named, and the operator confirms again on a different provider. The replica has not moved at that point, so nothing is at risk. A provider whose earlier service for this bucket was retired normally can be chosen again.
+
 ## Tasks
 
 | Method | Path | Purpose |
@@ -202,6 +268,8 @@ The restore streams synchronously for up to one hour and requires enough cache c
 | `GET` | `/admin/exhausted-tasks` | List exhausted tasks. Supports `limit` up to `1000`. |
 | `POST` | `/admin/exhausted-tasks/{id}/retry` | Retry an exhausted task (legacy path). |
 
+Replacement and retirement task responses that refer to a bucket include `bucket_name` in list and reference-detail responses. Retrying replacement work from the task queue returns `409 Conflict` with `"code": "replacement_task_retry_unsupported"`. When a replacement task has completed or stopped, use **Open Data Sets**, or open the bucket and go to Details → Storage → Data Sets. A `target_in_use` failure has no Retry action because it requires a different provider.
+
 ## Wallet and Filecoin
 
 | Method | Path | Purpose |

docs/en/reference/cli-api.md

@@ -97,6 +97,8 @@ Admin global flags must appear after `admin` and before the subcommand:
 
 Task listing supports `--type`, `--stage`, `--status`, `--limit`, and `--offset`. `--stage` requires `--type`.
 
+`synaps3 admin task retry` does not retry provider replacement work. Use **Open Data Sets** from a finished or stopped replacement task, or open the bucket and go to **Details** → **Storage** → **Data Sets**. If the selected provider already stores this bucket, choose a different provider instead of retrying.
+
 Cache eviction policy accepts `lru`, `after_upload`, or `none`. LRU watermarks must satisfy `0 <= low < high <= 100`; these settings are retained but inactive under the other policies.
 
 ## Settings Safety