CASSANDRA-18031: Add configurable snapshot_directory for cross-mount snapshot storage

[sachinAtMS]

JIRA: CASSANDRA-18031 — Make snapshot directory configurable

---

Motivation

When running Cassandra in managed infrastructure environments (e.g., Azure VMSS with Medusa backup tooling), a common deployment pattern uses a single RAID volume (e.g., 4TB) shared between Cassandra data files and backup snapshots. During the backup upload window, both the live SSTable data and the snapshot copies (which are hardlinks occupying logical space on the same filesystem) compete for the same disk. If total data exceeds 50% of the disk, the snapshot hardlinks plus live data can fill the volume entirely, causing Cassandra to halt writes or crash.

The root cause is that Cassandra's current snapshot mechanism always creates snapshots as hardlinks under data_file_directories, making it impossible to redirect snapshot storage to a separate mount/volume without modifying application code.

Real-World Scenario

• Setup: Azure VMSS nodes, 4TB RAID0 disk, Cassandra data + Medusa backup snapshots on the same mount
• Problem: With 2.5TB of data, taking a backup snapshot creates hardlinks that cause the filesystem metadata to report >4TB used, triggering disk-full conditions
• Why hardlinks aren't enough: While hardlinks don't duplicate data blocks, backup tools like Medusa subsequently upload from the snapshot directory. During this window, compaction can remove the original SSTables, causing the hardlinked snapshot files to become the sole owners of those data blocks — effectively doubling disk usage until the upload completes and the snapshot is cleared

Solution

This patch introduces a new optional snapshot_directory configuration parameter in cassandra.yaml. When set:

1. Snapshots are written as file copies to the configured directory instead of hardlinks under data_file_directories
2. Directory structure is mirrored: <snapshot_directory>/<keyspace>/<table-id>/snapshots/<tag>/
3. All snapshot operations work transparently: nodetool snapshot, nodetool listsnapshots, nodetool clearsnapshot
4. Fully backward-compatible: When snapshot_directory is not set, behavior is identical to current Cassandra (hardlinks under data dirs)

Trade-off: Copies vs Hardlinks

Using file copies instead of hardlinks means:

• Pro: Snapshot data lives on a separate mount — data disk is never at risk of filling due to snapshots
• Pro: The separate snapshot mount can be sized independently (e.g., cheaper/slower storage)
• Con: Copies use additional I/O and take longer than hardlink creation
• Con: Copies consume real disk space on the snapshot mount

This trade-off is acceptable for the target use case where disk safety is more critical than snapshot speed.

Configuration

Add to cassandra.yaml:

# Optional: Redirect snapshot storage to a separate mount point.
# When set, snapshots use file copies instead of hardlinks.
# The directory must exist or be creatable by the Cassandra process.
snapshot_directory: /mnt/backup-disk/cassandra-snapshots

Usage Examples

Taking a snapshot (unchanged CLI, new storage location)

# Take a full snapshot of a keyspace
$ nodetool snapshot -t daily_backup_20260303 my_keyspace
Requested creating snapshot(s) for [my_keyspace] with snapshot name [daily_backup_20260303] and target [all tables]

# Files are now at:
# /mnt/backup-disk/cassandra-snapshots/my_keyspace/my_table-<id>/snapshots/daily_backup_20260303/
# NOT at:
# /var/lib/cassandra/data/my_keyspace/my_table-<id>/snapshots/daily_backup_20260303/

Listing snapshots

$ nodetool listsnapshots
Snapshot Details:
Snapshot name          Keyspace name  Column family name  True size  Size on disk  Creation time
daily_backup_20260303  my_keyspace    my_table            6.05 KiB   6.05 KiB      2026-03-03T10:00:00.000Z

Clearing a snapshot

$ nodetool clearsnapshot -t daily_backup_20260303 my_keyspace
Requested clearing snapshot(s) for [my_keyspace] with snapshot name [daily_backup_20260303]

# Files removed from /mnt/backup-disk/cassandra-snapshots/...

Changes

Files Modified (8 files, ~300 lines added)

File	Change
Config.java	New snapshot_directory field with Javadoc
DatabaseDescriptor.java	Startup validation (directory creation/permissions) + getSnapshotDirectory() / hasSnapshotDirectory() accessor methods
Directories.java	Core snapshot path redirection: getSnapshotBaseDirectory(), getSnapshotSearchPaths() helpers; modified getSnapshotDirectory(), listAllSnapshots(), listSnapshotDirsByTag(), snapshotExists(), trueSnapshotsSize(); new 4-arg clearSnapshot() overload
ColumnFamilyStore.java	Modified snapshotWithoutMemtable() to use Files.copy() when configured; new copySSTableToDirectory() helper; clearSnapshot() passes checkSnapshotDirectory=true
SSTableReader.java	New copyToDirectory() method (alternative to createLinks())
SnapshotLoader.java	addSnapshotDirectory() helper to include snapshot_directory in snapshot discovery paths
SnapshotManager.java	Use default SnapshotLoader() constructor (which now includes snapshot_directory)
Keyspace.java	clearSnapshot() passes checkSnapshotDirectory=true to ensure snapshot files on separate mount are cleaned up

Design Principles

• Minimal invasion: Only snapshot-related code paths are modified; no changes to compaction, streaming, repair, or normal read/write paths
• Feature-flagged: All new behavior is gated behind DatabaseDescriptor.hasSnapshotDirectory(); when not configured, every code path falls through to the original logic
• Consistent directory structure: The snapshot directory mirrors the <keyspace>/<table-id>/snapshots/<tag>/ hierarchy from data directories

Test Evidence

Test Environment

• Apache Cassandra 4.1.11-SNAPSHOT (branch: cassandra-4.1)
• OpenJDK 11.0.27 on Ubuntu 20.04 (WSL2)
• Configuration:

  data_file_directories:
      - /tmp/cassandra-data
  snapshot_directory: /tmp/cassandra-snapshots
  commitlog_directory: /tmp/cassandra-commitlog

Test 1: Startup — snapshot_directory recognized

INFO  [main] DatabaseDescriptor.java:694 - Snapshot directory configured: /tmp/cassandra-snapshots.
  Snapshots will use file copies instead of hardlinks.

PASS: Configuration is validated and logged at startup.

Test 2: Snapshot creation — files stored on separate mount

$ nodetool snapshot -t test_snap test_ks

# Verify files are in snapshot_directory, NOT in data_directory:
$ find /tmp/cassandra-snapshots/test_ks -type f | head -5
/tmp/cassandra-snapshots/test_ks/test_table-9b3fce30.../snapshots/test_snap/nb-1-big-Data.db
/tmp/cassandra-snapshots/test_ks/test_table-9b3fce30.../snapshots/test_snap/nb-1-big-Index.db
/tmp/cassandra-snapshots/test_ks/test_table-9b3fce30.../snapshots/test_snap/nb-1-big-Filter.db
/tmp/cassandra-snapshots/test_ks/test_table-9b3fce30.../snapshots/test_snap/schema.cql
/tmp/cassandra-snapshots/test_ks/test_table-9b3fce30.../snapshots/test_snap/manifest.json

$ find /tmp/cassandra-data/test_ks -path '*/snapshots/*' -type f
# (empty — no snapshot files in data directory)

PASS: Snapshot files created in snapshot_directory, data directory is clean.

Test 3: Files are copies, not hardlinks

# Compare inodes between data file and snapshot file:
$ stat /tmp/cassandra-data/test_ks/test_table-.../nb-1-big-Data.db | grep Inode
  Inode: 264531

$ stat /tmp/cassandra-snapshots/test_ks/test_table-.../snapshots/test_snap/nb-1-big-Data.db | grep Inode
  Inode: 264601

PASS: Different inodes confirm these are independent file copies, not hardlinks.

Test 4: nodetool listsnapshots discovers snapshots on separate mount

$ nodetool listsnapshots
Snapshot Details:
Snapshot name  Keyspace name  Column family name  True size  Size on disk  Creation time
final_test     test_ks        test_table          6.05 KiB   6.05 KiB      2026-03-02T17:06:08.221Z

PASS: Snapshots in snapshot_directory are correctly discovered and listed.

Test 5: nodetool clearsnapshot removes files from separate mount

# Before:
$ find /tmp/cassandra-snapshots/test_ks -path '*/clear_test/*' -type f | wc -l
10

# Clear:
$ nodetool clearsnapshot -t clear_test test_ks

# After:
$ find /tmp/cassandra-snapshots/test_ks -path '*/clear_test/*' -type f | wc -l
0

# Other snapshots remain intact:
$ find /tmp/cassandra-snapshots/test_ks -path '*/final_test/*' -type f | wc -l
10

PASS: Snapshot files deleted from separate mount; other snapshots unaffected.

Test 6: nodetool listsnapshots reflects cleared snapshot

$ nodetool listsnapshots
Snapshot Details:
Snapshot name  Keyspace name  Column family name  True size  Size on disk  Creation time
final_test     test_ks        test_table          6.05 KiB   6.05 KiB      2026-03-02T17:06:08.221Z

# Only final_test remains — clear_test is gone.

PASS: Cleared snapshot no longer appears in listing.

Test 7: Backward compatibility (no snapshot_directory set)

When snapshot_directory is not configured in cassandra.yaml:

• DatabaseDescriptor.hasSnapshotDirectory() returns false
• All code paths fall through to original hardlink-based behavior
• No changes to snapshot directory structure under data_file_directories

PASS: Feature is fully opt-in with zero impact on existing deployments.

Build Verification

$ ant jar
BUILD SUCCESSFUL
Total time: 21 seconds

Compatibility

• Backward-compatible: Feature is opt-in; default behavior (no snapshot_directory configured) is identical to upstream
• YAML-compatible: New field is ignored by older Cassandra versions that don't recognize it
• No schema changes: No new system tables or schema modifications

[sachinAtMS]

This PR implements the feature requested in CASSANDRA-18031.

JIRA: https://issues.apache.org/jira/browse/CASSANDRA-18031