Skip to content

troubleshooting.md

Latest commit

 

History

History
162 lines (107 loc) · 5.61 KB

troubleshooting.md

File metadata and controls

162 lines (107 loc) · 5.61 KB
Raw
title Troubleshooting
description Diagnose common SynapS3 setup, health, wallet, cache, task, and provider issues.

Troubleshooting

Start here when an upload, download, login, or background storage operation fails. Check health first, then narrow the problem by wallet, cache, task, or storage provider signals.

First Checks

curl http://127.0.0.1:9090/healthz
synaps3 admin status
synaps3 admin task stats

Expected healthy baseline:

{"status":"ok"}

If health is not ok, use the error text as the next branch.

Setup Mode

Health may return:

{"status":"setup"}

This means SynapS3 is missing required settings. Review the configuration validation details before changing a value.

Check the reported fields:

synaps3 admin status
synaps3 admin settings get

If the wallet private key is missing, generate one:

synaps3 wallet generate

If the missing value is the wallet key, set the generated private key in SYNAPS3_FILECOIN_PRIVATE_KEY or filecoin.private_key in the config file. Restart SynapS3, check /healthz, and verify the effective settings.

Expected result: health changes from setup to ok after restart.

Unhealthy Background Tasks

Example:

{"status":"unhealthy","errors":["worker/uploader: not responding"]}

Check task pressure:

synaps3 admin task stats
synaps3 admin task list --status running --limit 20

After a restart, unfinished tasks become eligible to continue. If background task processing stays unhealthy, record the current task state, inspect logs, and then restart the service.

Wallet Funding or Deposit Fails

Check wallet status:

synaps3 admin status

For Calibration, fund the wallet address again:

synaps3 wallet fund-testnet 0x...

Then retry deposit and FWSS approval:

synaps3 wallet deposit 2 # 2 USDFC
synaps3 wallet approve

If faucet funding fails, claim manually from ChainSafe or Plumbline, then rerun synaps3 admin status.

Successful faucet claims print CalibnetUSDFC: <hash> and CalibnetFIL: <hash>. A confirmed deposit or approval prints Transaction: <hash> and Status: confirmed; an existing approval prints FWSS approval: already approved. If these results do not appear, verify RPC connectivity, wallet funds, and the reported error before retrying.

Cache Full

Upload endpoints can fail when cache capacity is exhausted. Check usage:

synaps3 admin status
synaps3 admin settings get cache.max_size_gb
synaps3 admin settings get cache.eviction_policy
synaps3 admin settings get cache.lru_high_watermark_percent
synaps3 admin settings get cache.lru_low_watermark_percent

Recovery options:

  • Confirm the host has free disk space, then increase cache.max_size_gb if capacity allows.
  • Restore storage provider connectivity and background task progress so queued uploads can complete and cache eviction can run.
  • Use the default lru policy for capacity-based cleanup. Lower the high watermark to leave more write headroom, and keep 0 <= low < high <= 100.
  • Use after_upload only when each version should be removed on the next Evictor poll after its bucket's minimum durable copies commit.
  • Use none when automatic removal must be disabled.

LRU cannot remove multipart staging data, versions below their bucket's minimum durable copies, or versions without a readable committed remote copy. A write does not synchronously run eviction, so 507 Insufficient Storage can continue until the Evictor catches up or safe candidates become available.

Failed LRU deletion tasks remain visible as exhausted work and become eligible again after a one-hour cooldown. Fix the reported filesystem or database problem first; use synaps3 admin task retry <id> to retry sooner.

After changing a cache setting, restart SynapS3, check /healthz, and verify the effective cache values with synaps3 admin settings get.

Exhausted Tasks

List exhausted work:

synaps3 admin task list --status exhausted --limit 100

Retry only after RPC connectivity, storage provider availability, wallet funds, FWSS approval, and cache disk capacity are ready.

synaps3 admin task retry 42

Provider replacement work is the exception: do not retry it from Tasks. Finished or stopped replacement tasks provide Open Data Sets, which opens the affected bucket directly at Details → Storage → Data Sets. Use Retry replacement only when that action is shown. If the selected provider already stores this bucket, choose a different provider instead.

Provider or RPC Issues

Check provider health and Filecoin readiness in the dashboard, or inspect the Admin API:

curl -u admin http://127.0.0.1:9090/api/v1/filecoin/readiness
curl -u admin http://127.0.0.1:9090/api/v1/observability/providers

Enter the Admin password at curl's no-echo prompt.

Recovery:

  • Restore the configured filecoin.rpc_url.
  • Confirm provider URLs are reachable from the SynapS3 host.
  • Keep filecoin.allow_private_networks = false unless private provider URLs are expected and trusted.

S3 Client Cannot Upload

Check these in order:

  1. S3 client uses path-style addressing.
  2. Access key and secret came from synaps3 admin s3-user create.
  3. Endpoint is http://localhost:8080 for local evaluation or the correct HTTPS address for production.
  4. Object size is between 127 and 1,065,353,216 bytes, and the object key meets the S3 compatibility limits.
  5. Dashboard task view shows whether Filecoin storage is queued, running, or exhausted.