feat: add native Storj support via storj.io/uplink

[solracsf]

Fix #1655

ToDo

• CN documentation

---

JuiceFS could only reach Storj through the S3-compatible MT or ST gateway. Both gateways have a critical limitation: ListObjects does not return results in lexicographic order. This breaks several JuiceFS operations that rely on sorted object iteration.

Add a new backend (--storage storj) that communicates directly with the Storj satellite via the native storj.io/uplink library, bypassing the gateway entirely.

Usage:

juicefs format \
  --storage storj \
  --bucket <bucket-name> \
  --access-key <access-grant> \
  --block-size 64M \  <-- Recommended by Storj (prints a Warning if not set) -->
  <meta-url> <volume-name>

The --access-key flag takes a Storj serialised Access Grant string. --secret-key is unused with this backend.

The native uplink API also does not guarantee lexicographic order. Both List and ListAll collect the full result set from the iterator, sort it with sort.Slice, then return/stream in order. This makes gc, fsck, sync, and destroy work correctly without workarounds.

Three env vars control the two connection pools:

STORJ_CONN_CAPACITY Total cached connections to storage nodes (default to 100)

STORJ_CONN_KEY_CAPACITY Cached connections per storage node (default to 5)

STORJ_SAT_CONN_CAPACITY Total cached connections to the satellite (default to 10)

All three accept a non-negative integer; set to 0 to disable pooling. Invalid or unset values silently fall back to the defaults. The idle expiration on both pools is 2 minutes, matching what the uplink library's own default pool uses.

Example: tighten connections to reduce open-file descriptors on a busy/low-CPU host:

STORJ_CONN_CAPACITY=40 STORJ_CONN_KEY_CAPACITY=2 juicefs mount ...

OBJbench tests

# ./juicefs objbench \
  --storage storj \
  --access-key <key> \
  --block-size 64M \
  storj://<bucket> \
  -p 12

Start Functional Testing ...
+----------+---------------------+-------------+
| CATEGORY |         TEST        |    RESULT   |
+----------+---------------------+-------------+
|    basic |     create a bucket |        pass |
|    basic |       put an object |        pass |
|    basic |       get an object |        pass |
|    basic |       get non-exist |        pass |
|    basic |  get partial object |        pass |
|    basic |      head an object |        pass |
|    basic |    delete an object |        pass |
|    basic |    delete non-exist |        pass |
|    basic |        list objects |        pass |
|     sync |         special key |        pass |
|     sync |    put a big object |        pass |
|     sync | put an empty object |        pass |
|     sync |    multipart upload |        pass |
|     sync |  change owner/group | not support |
|     sync |   change permission | not support |
|     sync |        change mtime | not support |
+----------+---------------------+-------------+

Start Performance Testing ...
put small objects: 100/100 [==============================================================]  35.4/s    used: 2.828582747s
get small objects: 100/100 [==============================================================]  74.3/s    used: 1.345257091s
   upload objects: 16/16 [==============================================================]  1.7/s     used: 9.188273387s
 download objects: 16/16 [==============================================================]  3.6/s     used: 4.398856162s
     list objects: 1392/1392 [==============================================================]  31830.9/s used: 43.898086ms
     head objects: 116/116 [==============================================================]  1173.5/s  used: 98.932921ms
   delete objects: 116/116 [==============================================================]  433.5/s   used: 267.700097ms
Benchmark finished! block-size: 64 MiB, big-object-size: 1.0 GiB, small-object-size: 128 KiB, small-objects: 100, NumThreads: 12
+--------------------+--------------------+-----------------------+
|        ITEM        |        VALUE       |          COST         |
+--------------------+--------------------+-----------------------+
|     upload objects |       111.46 MiB/s |     6890.55 ms/object |
|   download objects |       232.99 MiB/s |     3296.22 ms/object |
|  put small objects |    35.38 objects/s |      339.20 ms/object |
|  get small objects |    82.23 objects/s |      145.93 ms/object |
|       list objects | 33432.49 objects/s | 41.64 ms/ 116 objects |
|       head objects |  1188.68 objects/s |       10.10 ms/object |
|     delete objects |   437.98 objects/s |       27.40 ms/object |
| change permissions |        not support |           not support |
| change owner/group |        not support |           not support |
|       update mtime |        not support |           not support |
+--------------------+--------------------+-----------------------+