Remote Compaction: Can OpenAndCompact work with only input SST files?

[viralmehta-stripe]

We're implementing remote compaction where SST files are transferred from the primary to a compaction worker (no shared storage).

What we want: Transfer only the input SST files (that need to be compacted) specified in CompactionServiceInput, run compaction on the worker, and return the output SST files.

What happens: OpenAndCompact() fails because it opens the DB as secondary and verifies that ALL files in the MANIFEST exist, not just the input files for the compaction job.

Error: Invalid argument: Cannot find matched SST files for the following file numbers: 15 17 19

Is there a way to run OpenAndCompact() with only the input SST files present? Or an alternative API that allows compacting a subset of SST files and producing a valid CompactionServiceResult?

cc: @jaykorean

[viralmehta-stripe]

@pdillinger , @archang19 , @jaykorean , any idea about this?

[jaykorean]

Invalid argument: Cannot find matched SST files for the following file numbers: 15 17 19 --> This might actually be due to a bug that @hx235 recently found in point-in-time recovery for Secondary DB instances. We will provide more details in the PR that will fix this bug, but do you use more than one column family and atomic flush enabled?

[viralmehta-stripe]

Thanks @jaykorean!

Looking closely at our test setup, we're actually using only a single column family (default). The OPTIONS file with multiple column families isn't being used in this test path.

Atomic flush is also not enabled.

[viralmehta-stripe]

Does RocksDB support this usecase of doing remote compaction by running OpenAndCompact on only the input SST files that need to be compacted?

Here is what I want to do: The primary RocksDB instance stores SST files on its local SSD (as our use case is latency sensitive - need p99 latency < 2 ms). For compaction, only the SST files to be compacted are uploaded to AWS S3. The compaction worker downloads only the input SST files from S3 onto its local SSD, compacts the input SST files by calling OpenAndCompact() and then the output SST files are uploaded to S3 which are then downloaded by the primary onto its local SSD.

[jaykorean]

@viralmehta-stripe

If I understand correctly, you are trying to use openAndCompact() as a portable compaction API - essentially to compact selected SST files in an environment where the full LSM tree cannot be reconstructed as in the original setup. Assuming that you are also copying MANIFEST and OPTIONS file, you could technically achieve this by opening the copied partial DB with option best_effort_recovery=true, running manual compaction using CompactFiles() or CompactRange(), and then copying the output back to the source location. You would need to manually construct the CompactionServiceResult for the primary DB to recognize and install the output files, but unfortunately, CompactionServiceResult is not publicly exposed today.

Remote compaction was originally designed to read and compact SST files located in the same place as the primary DB. The API was not intended for input SST files to be copied outside of the original DB path; the DB path is expected to be disaggregated or at least accessible by both the primary and remote hosts. Since the remote compaction worker needs to read the same files used by the primary DB, it opens the DB as a secondary instance.

The only optimization I can foresee is that, currently, the default CF must be opened as part of OpenAndCompact(), even when only a non-default CF is being compacted. Honestly, I don't think we will support OpenAndCompact() in scenarios where other SST files are missing from the same CF. Supporting this would require not relying on the secondary DB for remote compaction and would involve significant changes to several parts of the implementation.