DeleteRange Causes Compatibility Issues Due to kTypeMaxValid Change

[senzzzi]

The DeleteRange feature has a compatibility issue related to kTypeMaxValid in db/dbformat.h. The comment on kTypeMaxValid states that it is only used for validation (line 75), but in reality, it can actually be used when using delete ranges (as another comment indicates).

The issue arises because IsExtendedValueType checks whether a ValueType is greater than or equal to kTypeMaxValid. However, kTypeMaxValid is defined as the last valid ValueType plus one. This means that whenever a new entry is added to the ValueType enum, kTypeMaxValid shifts, potentially making older versions misinterpret previously valid values.

If an older version encounters a manifest entry with a ValueType that is now above its kTypeMaxValid, it will incorrectly classify it as invalid, leading to manifest corruption errors. This can break compatibility when opening databases created with a newer version.

Expected Behavior
Older versions of RocksDB should be able to read manifests created by newer versions without encountering invalid values.

Actual Behavior
When an older version (e.g., 9.0.0) opens a database created by a newer version (e.g., +9.1.0) which made use of the 'delete range' feature, keys present in the manifest may not be parsed, leading to errors opening the database.

I was able to reproduce this issue, and the error message I encountered was:

Corruption: VersionEdit: new-file4 entry  The file MANIFEST-284456 may be corrupted.

Affected File: db/dbformat.h

Issue Introduced in PR: #12419.

[jowlyzhang]

Thanks for the report.

If the files generated by a new RocksDB version is using a new feature (a new db/dbformat.h tag), that the old RocksDB version cannot understand, it's reasonable behavior to not being able to open it with the old RocksDB version because it cannot make sense of the data. RocksDB's forward compatibility is limited to when no new feature is used: https://github.com/facebook/rocksdb/wiki/RocksDB-Compatibility-Between-Different-Releases#format-compatibility

Range deletion related tags are consistently generated and understood by both 9.0.0 and 9.1.0 as this tag
kTypeRangeDeletion = 0xF, // meta block

I don't think making use of "delete range" will cause an issue. Assuming delete range is used in 9.1.0 to generate the manifest, the tag 0xF will be written to the MANIFEST. When you later open the MANIFEST with 9.0.0, it will correctly cast:

and static_cast(0xF) to be kTypeRangeDeletion and these checks hold, there should not be any issues.

However, if 9.1.0 generates this new tag in MANIFEST
kTypeColumnFamilyValuePreferredSeqno = 0x19, // WAL only

when 9.0.0 try to cast it
static_cast(0x19) it will gets to something that is 1 bigger than: kTypeMaxValid and fail this check.

This is however expected, because 9.1.0 is using a new feature, in this case, forward compatibility is not supported.

This means that whenever a new entry is added to the ValueType enum, kTypeMaxValid shifts, potentially making older versions misinterpret previously valid values

Why would older versions misinterpret previously valid values, can you give an example?