|
| 1 | +# dbgate-redis-dumper |
| 2 | + |
| 3 | +Standalone, client-agnostic Redis and Valkey dump and restore library for Node.js. |
| 4 | + |
| 5 | +Dumps a server's keys as a script of ordinary Redis commands and restores it back — |
| 6 | +entirely over a Redis connection. **No `redis-cli`, no `redis-server`, no RDB tooling, no |
| 7 | +external process is ever invoked.** Framework-independent: it does not depend on DbGate |
| 8 | +internals and works outside DbGate. |
| 9 | + |
| 10 | +- Node.js >= 20, ESM and CJS builds, full TypeScript types |
| 11 | +- `ioredis` is an **optional** peer dependency, reachable only through the separate |
| 12 | + `dbgate-redis-dumper/ioredis` entry point — the core never imports a driver |
| 13 | +- Binary-safe end to end: keys and values are bytes, never decoded strings |
| 14 | +- Streaming both ways: a dump holds one `SCAN` page at a time and streams large keys in |
| 15 | + chunks, and a restore parses incrementally, so neither ever loads a whole database or |
| 16 | + a whole file (see [memory](docs/known-limitations.md#memory)) |
| 17 | + |
| 18 | +## Two formats, both native |
| 19 | + |
| 20 | +A dump is a list of commands (`SET`, `HSET`, `RPUSH`, `SADD`, `ZADD`, `XADD`, ...). It is |
| 21 | +written in one of two encodings, and **both restore with the stock `redis-cli`**: |
| 22 | + |
| 23 | +| Format | Restore natively with | Why you'd pick it | |
| 24 | +| ---------------- | ------------------------------ | --------------------------------------------------- | |
| 25 | +| `text` (default) | `redis-cli < dump.redis` | One command per line; readable, diffable, greppable | |
| 26 | +| `resp` | `redis-cli --pipe < dump.resp` | Redis's own mass-insertion path; much faster | |
| 27 | + |
| 28 | +There is no custom container, no archive wrapper, and no metadata sidecar. The other way |
| 29 | +round works too: **any `redis-cli` script or RESP mass-insertion file restores with this |
| 30 | +library**, parsed with a byte-exact port of the grammar `redis-cli` itself uses. |
| 31 | + |
| 32 | +Every path is **proven by automated tests against real Redis 6.2, 7.2, 7.4, 8.0 and Valkey |
| 33 | +8.0**, not assumed. Each one ends by reading the restored database back and deep-comparing |
| 34 | +every key, value, expiry, hash-field expiry, stream entry, consumer group and pending entry |
| 35 | +against the source — and a negative control proves the comparison catches a lossy dump. |
| 36 | + |
| 37 | +| Path | Tested | |
| 38 | +| ---------------------------------------------- | -------------------------------------- | |
| 39 | +| this library (`text`) → `redis-cli < dump` | ✅ 6.2, 7.2, 7.4, 8.0, Valkey 8.0 | |
| 40 | +| this library (`resp`) → `redis-cli --pipe` | ✅ 6.2, 7.2, 7.4, 8.0, Valkey 8.0 | |
| 41 | +| this library → this library (both formats) | ✅ 6.2, 7.2, 7.4, 8.0, Valkey 8.0 | |
| 42 | +| hand-written `redis-cli` script → this library | ✅ compared against `redis-cli` itself | |
| 43 | +| Redis 6.2 dump → Redis 8.0 and Valkey 8.0 | ✅ | |
| 44 | + |
| 45 | +See [docs/native-compatibility.md](docs/native-compatibility.md). |
| 46 | + |
| 47 | +## Install |
| 48 | + |
| 49 | +```sh |
| 50 | +npm install dbgate-redis-dumper |
| 51 | +# optional, for the bundled ioredis adapter: |
| 52 | +npm install ioredis |
| 53 | +``` |
| 54 | + |
| 55 | +## Quick start |
| 56 | + |
| 57 | +### Dump |
| 58 | + |
| 59 | +```ts |
| 60 | +import { createWriteStream } from 'node:fs'; |
| 61 | +import { dumpRedis } from 'dbgate-redis-dumper'; |
| 62 | +import { connectIoredis } from 'dbgate-redis-dumper/ioredis'; |
| 63 | + |
| 64 | +const { connection, close } = await connectIoredis({ host: 'localhost', port: 6379 }); |
| 65 | + |
| 66 | +try { |
| 67 | + const result = await dumpRedis( |
| 68 | + connection, |
| 69 | + { databases: 'all' }, |
| 70 | + createWriteStream('backup.redis'), |
| 71 | + event => console.log(event.phase, event.database ?? '', event.totalKeysExported ?? ''), |
| 72 | + ); |
| 73 | + |
| 74 | + console.log(`${result.keysExported} keys in ${result.commandsWritten} commands`); |
| 75 | + for (const warning of result.warnings) { |
| 76 | + console.warn(`[${warning.severity}] ${warning.code}: ${warning.message}`); |
| 77 | + } |
| 78 | +} finally { |
| 79 | + await close(); |
| 80 | +} |
| 81 | +``` |
| 82 | + |
| 83 | +The result is restorable by `redis-cli < backup.redis`. |
| 84 | + |
| 85 | +### Restore |
| 86 | + |
| 87 | +```ts |
| 88 | +import { createReadStream } from 'node:fs'; |
| 89 | +import { restoreRedisDump } from 'dbgate-redis-dumper'; |
| 90 | + |
| 91 | +const result = await restoreRedisDump({ |
| 92 | + connection, |
| 93 | + source: createReadStream('backup.redis'), |
| 94 | + options: { databaseMapping: { 0: 5 } }, // what was database 0 goes into database 5 |
| 95 | + progress: event => console.log(event.phase, event.commandsProcessed), |
| 96 | +}); |
| 97 | + |
| 98 | +console.log(`${result.commandsExecuted} commands`); |
| 99 | +for (const error of result.errors) { |
| 100 | + console.error(`command ${error.commandIndex} (line ${error.location.line}): ${error.message}`); |
| 101 | + console.error(` ${error.commandPreview}`); // truncated, credential-redacted |
| 102 | +} |
| 103 | +``` |
| 104 | + |
| 105 | +`source` accepts a `string`, a `Buffer`, a `Readable`, or any (async) iterable of text or |
| 106 | +`Buffer` chunks. The format is detected from the first byte. |
| 107 | + |
| 108 | +### Using an existing client |
| 109 | + |
| 110 | +```ts |
| 111 | +import Redis from 'ioredis'; |
| 112 | +import { duplicateIoredis, fromIoredis } from 'dbgate-redis-dumper/ioredis'; |
| 113 | + |
| 114 | +const client = new Redis(config); |
| 115 | +const source = duplicateIoredis(client); // a fresh connection per operation (recommended) |
| 116 | +const borrowed = fromIoredis(client); // this very connection, never closed by the library |
| 117 | +``` |
| 118 | + |
| 119 | +A dump `SELECT`s every database it reads, so on a client your application also uses, its |
| 120 | +own commands would run against those databases meanwhile — hence `duplicateIoredis`. |
| 121 | +Either way the connection is put back on the database it had selected. See |
| 122 | +[docs/ioredis-adapter.md](docs/ioredis-adapter.md). |
| 123 | + |
| 124 | +## Public API |
| 125 | + |
| 126 | +| Function | Purpose | |
| 127 | +| ------------------------------------------------------------------------ | ---------------------------------------------------------- | |
| 128 | +| `dumpRedis(connection, options, output, onProgress?, signal?)` | Full pipeline: version → scan → pipelined reads → commands | |
| 129 | +| `restoreRedisDump({ connection, source, options?, progress?, signal? })` | Streaming parser → allow-list → pipelined execution | |
| 130 | +| `analyzeRedisDump(source)` | What a dump contains and needs, without executing anything | |
| 131 | +| `preflightRestore({ connection, requirements })` | What a target server lacks, before anything is written | |
| 132 | +| `isRedisDump(sample)` / `detectDumpFormat(sample)` | Recognize a dump and its encoding from its first bytes | |
| 133 | +| `parseRedisCommands(input)` / `streamRedisCommands(source)` | The parser, usable on its own | |
| 134 | +| `encodeCommand(command, format)` | The encoders, usable on its own | |
| 135 | +| `checkTargetCompatibility(requirements, server)` | Pure version check against a dump's requirements | |
| 136 | +| `fromIoredis` / `duplicateIoredis` / `connectIoredis` | Adapter (from `dbgate-redis-dumper/ioredis`) | |
| 137 | + |
| 138 | +## Fidelity highlights |
| 139 | + |
| 140 | +- **Every core type, and everything around it**: strings, hashes, lists, sets, sorted |
| 141 | + sets and streams — including stream consumer groups, their consumers, each group's |
| 142 | + pending entries (with delivery count and delivery time), `entries-read`, and the |
| 143 | + stream's last ID and counters even when entries were deleted. |
| 144 | +- **Expiries are exact.** Written as absolute `PEXPIREAT` instants (what an RDB file |
| 145 | + stores), read with `PEXPIRETIME` where available and from the _server's_ clock |
| 146 | + otherwise. `relative` and `none` modes are available. |
| 147 | +- **Per-field hash expiration** (Redis 7.4+, Valkey 9.0+) is preserved. |
| 148 | +- **Scores are never parsed.** A sorted-set score is written as the server's own text, |
| 149 | + so `inf`, `-inf`, `5e-324` and 17-digit doubles survive exactly. |
| 150 | +- **Module types** (RedisJSON, RedisBloom, Redis 8 vector sets) are written as `RESTORE` |
| 151 | + payloads, with a warning — never silently dropped. `strategy: 'payload'` writes every |
| 152 | + key that way, for exactness down to the internal encoding. |
| 153 | +- **Large keys are streamed.** A collection above `batchSize` elements is read in chunks |
| 154 | + (`HSCAN`, `LRANGE` windows, ...), a string above `maxCommandBytes` with `GETRANGE` and |
| 155 | + written as `SET` + `APPEND`, so no single key must fit in memory. |
| 156 | +- **Restores are safe by default.** Only data commands run; a dump cannot `FLUSHALL`, |
| 157 | + `CONFIG SET`, `EVAL` or `REPLICAOF` unless you opt in. Error previews redact |
| 158 | + credentials. The connection is handed back on its original database, outside any |
| 159 | + transaction, even when a restore fails or is cancelled. |
| 160 | +- **Preflight before you write.** Every dump records which server features it depends |
| 161 | + on; `preflightRestore` tells you what an older target lacks before a single key lands. |
| 162 | + |
| 163 | +## Documentation |
| 164 | + |
| 165 | +| Document | Contents | |
| 166 | +| ------------------------------------------------------------ | ----------------------------------------------------------------- | |
| 167 | +| [docs/native-compatibility.md](docs/native-compatibility.md) | The two formats, `redis-cli` interoperability, what is reproduced | |
| 168 | +| [docs/dump-api.md](docs/dump-api.md) | `dumpRedis` options, databases, selection, strategies, progress | |
| 169 | +| [docs/restore-api.md](docs/restore-api.md) | `restoreRedisDump`, the parser, allow-list, errors, preflight | |
| 170 | +| [docs/ioredis-adapter.md](docs/ioredis-adapter.md) | Connection ownership, reconnects, writing another adapter | |
| 171 | +| [docs/supported-data-types.md](docs/supported-data-types.md) | Per-type commands, fidelity and chunking | |
| 172 | +| [docs/known-limitations.md](docs/known-limitations.md) | What this package does not do, and why | |
| 173 | +| [docs/round-trip-testing.md](docs/round-trip-testing.md) | Running the Docker-backed matrix; the fixture | |
| 174 | +| [docs/architecture.md](docs/architecture.md) | Layer-by-layer design and the reasoning behind it | |
| 175 | + |
| 176 | +## Development |
| 177 | + |
| 178 | +```sh |
| 179 | +npm install |
| 180 | +npm run typecheck |
| 181 | +npm run lint |
| 182 | +npm test # unit tests, no Docker or network needed |
| 183 | + |
| 184 | +npm run docker:up # Redis 6.2, 7.2, 7.4, 8.0 + Valkey 8.0 |
| 185 | +npm run test:integration # round trips, redis-cli interop, behaviour, cross-version |
| 186 | +npm run docker:down |
| 187 | + |
| 188 | +npm run test:package # builds, then smoke-tests dist/ as ESM and CJS |
| 189 | +``` |
| 190 | + |
| 191 | +Integration tests skip themselves with a clear message when no server is reachable; set |
| 192 | +`REDIS_TEST_REQUIRED=1` (as CI does) to make that a hard error. `REDIS_TEST_TARGETS=redis74` |
| 193 | +runs one version while iterating, and `REDIS_TEST_TARGETS=host REDIS_TEST_HOST_PORT=6379` |
| 194 | +runs against a server you started yourself. |
| 195 | + |
| 196 | +## License |
| 197 | + |
| 198 | +GPL-3.0-only. See [LICENSE](LICENSE). |
0 commit comments