🔥 Blazing fast bulk data transfers between any cloud 🔥
pip install skyplane[aws]
skyplane init
skyplane [sync/cp] [local/s3/gs/azure/scp]://mybucket/big_dataset [local/s3/gs/azure/scp]://mybucket2/
Skyplane is a tool for blazingly fast bulk data transfers between object stores in the cloud. It provisions a fleet of VMs in the cloud to transfer data in parallel while using compression and bandwidth tiering to reduce cost.
Skyplane is:
- 🔥 Blazing fast (110x faster than AWS DataSync)
- 🤑 Cheap (4x cheaper than rsync)
- 🌐 Universal (AWS, Azure, IBM, SCP and GCP)
You can use Skyplane to transfer data:
- between object stores within a cloud provider (e.g. AWS us-east-1 to AWS us-west-2)
- between object stores across multiple cloud providers (e.g. AWS us-east-1 to GCP us-central1)
- between local storage and cloud object stores (experimental)
Skyplane currently supports the following source and destination endpoints (any source and destination can be combined):
| Endpoint | Source | Destination |
|---|---|---|
| AWS S3 | ✅ | ✅ |
| Google Storage | ✅ | ✅ |
| Azure Blob Storage | ✅ | ✅ |
| Cloudflare R2 | ✅ | ✅ |
| Samsung Cloud Platform Object Storage | ✅ | ✅ |
| Local Disk | ✅ | (in progress) |
Skyplane is an actively developed project. It will have 🔪 SHARP EDGES 🔪. Please file an issue or ask the contributors via the #help channel on our Slack if you encounter bugs.