1.0.0

MSC 1.0 is the first stable release of the Python client API. Starting with this release, MSC follows a stricter semantic versioning contract for its stable public API: patch and minor releases should remain compatible, while incompatible API changes are reserved for future major releases.

MSC will continue to evolve after 1.0 with new providers, performance improvements, and higher-level workflows built on top of this stable foundation.

Breaking Changes

• Removed deprecated Python API compatibility shims:
  • StorageClient.list(prefix=...) is no longer accepted. Use StorageClient.list(path=...) or pass the path positionally.
  • follow_symlinks is no longer accepted by StorageClient.list_recursive, StorageClient.sync_from, msc.list, msc.list_recursive, or msc.sync. Use symlink_handling=SymlinkHandling.FOLLOW, SymlinkHandling.SKIP, or SymlinkHandling.PRESERVE.
• Removed support for the deprecated cache config key cache.use_etag. Use cache.check_source_version instead.
• Removed the retired cache experimental feature flags experimental_features.cache_mru_eviction and experimental_features.cache_purge_factor. MRU cache eviction and eviction_policy.purge_factor are now stable cache options and no longer require feature flags.

0.51.0

Multi-Storage Client (MSC)

New Features

• Add a Python API compatibility guardrail.

Bug Fixes

• Make delete_many() idempotent when missing keys are included.
• Retry transient GCS delete_many() failures.
• Reject unsafe cache delete paths.
• Reject unknown top-level config keys correctly.
• Normalize duplicate slashes in MSC URLs.
• Preserve symlink chains.
• Refresh sync progress totals immediately.
• Upgrade vulnerable dependencies from security scans.

Multi-Storage File System (MSFS)

New Features

• Support multi-bucket and multi-backend CSI volumes.
• Add a no-credentials anonymous CSI connection mode.
• Reload AIStore authn_token_file credentials after token rotation.
• Add per-workload IRSA support with CSIDriver tokenRequests and requiresRepublish.
• Add a per-inode disk cache backend.
• Mark MSFS packages with pre-release version metadata.

Bug Fixes

• Use optimistic-lock copy operations to remove warm-read serialization.
• Align manifest directory listings.

0.50.0

Multi-Storage Client (MSC)

Bug Fixes

• Preserve ordering when resolving multi-storage paths.
• Handle buffer-protocol read results when calculating data size metrics.
• Raise an error when SymlinkHandling.FOLLOW encounters a broken symlink.
• Release BytesIO buffers before retrying Rust-backed S3 transfers.
• Document boto3 S3 transfer manager pass-through options for the S3 storage provider.

Multi-Storage File System (MSFS)

New Features

• Add manifest generation and ingest support for MSFS.
• Add CSI driver deployment assets and a Helm chart for Kubernetes deployments.
• Adopt a memory-mapped data cache for file system reads.

0.49.0

Multi-Storage Client (MSC)

New Features

• Add per-item retry handling for MSC batch transfer operations.

Bug Fixes

• Cache path mapping lookups when resolving non-MSC URLs through shortcut APIs.
• Stream dry-run sync output before waiting for the sync producer to finish.
• Prevent metadata attributes from being sent as physical object writes.
• Increase the default storage provider connection pool size.
• Upgrade vulnerable dependencies reported by security scanning.

Multi-Storage File System (MSFS)

New Features

• Adopt fission v4.0.4 for the file system layer.

Bug Fixes

0.48.0

Multi-Storage Client (MSC)

New Features

• Added cache.prefetch_file to let open() use partial file caching by default when configured.
• Use recursive listing in the sync producer to improve sync planning performance.
• Enable parallel recursive listing for the POSIX storage provider.

Bug Fixes

• Allow custom object attributes to use dict[str, Any] values across MSC APIs and providers.

Multi-Storage File System (MSFS)

New Features

• Adopt /dev/fuse multi-reader support for improved FUSE request handling.

Bug Fixes

0.47.1

Multi-Storage Client (MSC)

New Features

• Add a Backstage catalog descriptor to improve service metadata integration.

Bug Fixes

• Parallelize metadata registration during batch uploads to improve upload throughput.

0.47.0

Multi-Storage Client (MSC)

Breaking Changes

• s8k provider no longer pins request_checksum_calculation/response_checksum_validation to when_required; botocore defaults now apply, and botocore >= 1.36.0 adds a CRC32 checksum to every upload.

New Features

• Added checksum_algorithm option to s3 and s8k providers for AWS S3 additional upload checksums (CRC32, CRC32C, SHA1, SHA256, CRC64NVME). When rust_client is enabled, only SHA256 is supported.
• End-to-end symlink preservation for uploads and downloads. MSC now treats symbolic links as a first-class concept across all storage providers, so symlinks in a POSIX source tree can be round-tripped through object storage without losing their link semantics. This adds:
  • A new make_symlink API on every StorageProvider (S3, GCS, Azure, OCI, AIS, POSIX), which writes an empty object whose user metadata carries the link target on cloud backends, and a native symlink on POSIX.
  • A symlink_handling option on list / list_files with three modes: FOLLOW (default, backward-compatible), SKIP (exclude symlinks), and PRESERVE (yield symlinks as leaf entries with ObjectMetadata.symlink_target populated).
  • sync_from / msc sync honor symlink_handling=PRESERVE to recreate symlinks at the destination instead of dereferencing them, so uploading a directory to object storage and downloading it back reconstructs the original link structure.
• Add opt-in client-side MD5 verification to the Azure storage provider.
• Support multipart upload and download in the Azure storage provider.

Bug Fixes

• Sort directory entries in the POSIX storage provider to match S3 lexicographical order.
• Improve msc sync performance by skipping per-object metadata lookups for files up to 16MB, avoiding redundant HEAD requests during synchronization.
• Upgrade fastmcp, cryptography, pygments, and axios to address security advisories.

Multi-Storage File System (MSFS)

New Features

• Add fio benchmark script for MSFS performance testing.

Bug Fixes

• Add globalsStruct lock tracking.
• Additional fixes for the fio benchmark.

0.46.0

Multi-Storage Client (MSC)

New Features

• Add batch download_files and upload_files APIs to StorageClient for efficient multi-file transfers.
• Support presigned URL generation for Azure Blob Storage via SAS tokens.
• Use download_files and upload_files in sync_from for efficient transfers.
• Support non-default AWS profile in S3 storage providers through MSC config.
• Support workload identity in Azure storage provider through rclone config.
• Support origin path in CloudFront signed URLs.
• Support security token file in OCI CLI/SDK configs.

Bug Fixes

• Support credential-less rclone configurations.
• Relax google-cloud-storage dependency to >=2.12,<4.
• Setup non-default HTTP client for AWS SDK for Rust config.
• Add cryptography as vault extra dependency.
• Support custom attributes in upload_files.

Multi-Storage File System (MSFS)

New Features

• Integrate PebbleDB as metadata cache on-disk store.
• Add "pseudo" backend to enable arbitrary directory tree extreme testing.
• Support listObjects with StartAfter.
• Avoid HEAD calls to emulate IfMatch.

0.45.1

Multi-Storage Client (MSC)

Bug Fixes

• Restore cryptography as an optional dependency; it was incorrectly added as a core requirement in 0.45.0.
• Lower the minimum cache refresh interval from 300 seconds to 1 second.
• Fix incorrect handling of short final chunks and improve partial file cache reliability.
• Upgrade grpc-go, pyOpenSSL, and Authlib to address CVE-2026-33186, CVE-2026-27459, and CVE-2026-27962.

Multi-Storage File System (MSFS)

New Features

• Adopt sortedmap for inode management.

0.45.0

Multi-Storage Client (MSC)

New Features

• Add presigned URL generation via msc.generate_presigned_url() shortcut and StorageClient.generate_presigned_url(). Supports S3 native signing and CloudFront signed URLs through a pluggable URLSigner abstraction.
• Add dryrun mode to sync_from and msc.sync() that enumerates and compares objects without performing any copy/delete, streaming results to JSONL files via DryrunResult on SyncResult.

Bug Fixes

• Fix sync worker to protect add_file references from concurrent modification.
• Fix handling of directory paths in ManifestMetadataProvider.get_object_metadata.
• Remove placement group overhead from Ray sync workers.

Multi-Storage File System (MSFS)

New Features

• Integrate sortedmap B+Tree data structure for enhanced scalability.
• Shift to pointer-free inodeStruct for improved memory efficiency.