fix: Tolerate legacy storage.compression values when opening RocksDB databases - #2037
Merged
Conversation
rocksdb-js >= 2.6 validates the compression option strictly: '' and booleans
throw ("Unsupported compression algorithm"), and unset means the build default
(lz4). Harper's persisted table metadata carries LMDB-era values where defined
falsy (false, '') means explicitly disabled, so map those to 'none' and true to
unset at the single chokepoint every RocksDB open flows through. Also normalize
getDefaultCompression's disabled return to false so raw config garbage ('') is
never persisted, and fix two component test stubs that returned '' for every
unknown config key - which is how CI on the rocksdb-js 2.6.1 bump first hit
this.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces a normalization utility, toRocksCompression, to map legacy LMDB-era compression configuration values to formats compatible with rocksdb-js >= 2.6. Specifically, it maps explicitly disabled values (such as false or empty strings) to 'none', and maps true to undefined to allow the build default to apply. It also updates mock configurations in tests to return undefined instead of empty strings for unknown keys to prevent validation errors, and adds a suite of unit tests to verify these mappings. There are no review comments, and we have no feedback to provide.
Contributor
|
Reviewed; no blockers found. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
#2036 (chore(deps): Update rocksdb-js to 2.6.1) fails unit tests on all Node versions with
Unsupported compression algorithm "". Main is green — the failures are caused by the interaction of the dep bump with two latent Harper issues, fixed here so #2036 can land.Root cause
rocksdb-js 2.5.0 ignored the
compressionoption entirely; 2.6.x validates it strictly and treats unset as "use the build default (lz4)". Harper passes LMDB-era values through to RocksDB opens:getDefaultCompression()returnsLMDB_COMPRESSION && OPTS, i.e. whatever falsy value the config resolves to ('',false,null) when compression is disabled — and that value is persisted into table metadata at table creation, then replayed on every open.storage.compression: falseis a validated config value (configValidator.ts), so on 2.6.1 any instance with compression disabled would fail to open its tables — a real user-facing break, not just a test issue.''for every unknown config key, socompression: ''reached the RocksDB open.Fix
toRocksCompression()maps legacy values at the single chokepoint every RocksDB open flows through (openRocksDatabase): defined-falsy →'none',true→ unset, everything else passes through (2.6.1 already treats an{ threshold, ... }object withoutalgorithmas unset). Internal DBIs (compression never set) keep the build default, matching current behavior.getDefaultCompression()now returnsfalse(not raw config garbage) when disabled, so nothing unnormalizable is persisted going forward.undefinedfor unknown keys.Semantics note (reviewer attention)
On 2.5.0 the option was ignored, so
storage.compression: falseinstances have been writing lz4 anyway. This fix makesfalseactually mean no compression for new writes once 2.6.x lands. Existing lz4 blocks stay readable (RocksDB records compression per block; mixed SSTs are supported). If parity-with-current-behavior is preferred over honoring the config, the'none'mapping can become unset — flagging for review.Verification
componentLoader,startOnMainThreadOnce, newrocksCompressiontests: 16 passing on each).Note: #2036 also shows a failing Windows integration shard (
EPERM: rename ... harper-config.yaml.tmp) — that's an unrelated Windows file-lock flake.Generated by Claude (Fable 5).
🤖 Generated with Claude Code