[NEW] Migrate from Redis 7.4.0 to Valkey 7.2.5

[utdrmac]

Describe the bug feature
Attempting to migrate to Valkey from Redis by simply copying the .rdb file from Redis over to Valkey and start Valkey. This was working a few months ago, and now it no longer works.

To reproduce

$ docker run -d --name redis --network valkey -e REDIS_ARGS="--requirepass mypassword" redis/redis-stack-server:latest
$ wget https://github.com/redis-developer/redis-datasets/raw/master/user-database/import_users.redis
-- edit import_users.redis and add `AUTH mypassword` to top of file
$ docker cp import_users.redis redis:import_users.redis
$ docker exec -it redis bash
root@599623d6666c:/# redis-cli < import_users.redis
"5996 Users Created"
root@599623d6666c:/# redis-cli
127.0.0.1:6379> AUTH mypassword
OK
127.0.0.1:6379> LASTSAVE
(integer) 1722353292
127.0.0.1:6379> BGSAVE
Background saving started
127.0.0.1:6379> LASTSAVE
(integer) 1722353319
-- exit redis container
$ docker cp redis:/data/dump.rdb ./dataM1/
Successfully copied 1.44MB to /home/tester/valkey/dataM1/
$ docker run -it --rm --user 1000:1000 --name valkeyM1 --hostname valkeym1 --network valkey -v ${PWD}/dataM1:/data valkey/valkey:7.2.5 /data/valkey.conf
...
1:M 30 Jul 2024 15:33:49.896 * Server initialized
1:M 30 Jul 2024 15:33:49.896 # Can't handle RDB format version 12
1:M 30 Jul 2024 15:33:49.896 # Fatal error loading the DB, check server logs. Exiting.

Expected behavior
To be able to migrate from Redis to Valkey by copying the datafile from Redis. This worked a few months ago.

[madolson]

Redis 7.4 is a proprietary data format, so it's not currently on our roadmap to support migrating to Valkey 7.2 or Valkey 8.0.

To be able to migrate from Redis to Valkey by copying the datafile from Redis. This worked a few months ago.

You were likely using Redis 7.2, which is the open-source data format and the original format we supported.

[utdrmac]

@madolson Thanks for that. What's the best-practice method, then, for migrating from Redis 7.X to Valkey? Replication?

[madolson]

for migrating from Redis 7.X to Valkey?

You can move from 7.0 and 7.2 to Valkey. Moving from 7.4+ to Valkey is not supported currently, my guidance has been for folks using Redis to avoid upgrading to Redis 7.4 because it switches to a proprietary format. We can look into reverse engineering it though. Is there a criticality in the ask to migrate?

[utdrmac]

Hello @madolson I work at Percona, one of Valkey's sponsors in the Linux Foundation. I published a blog on migrating to Valkey earlier this year, along with several others.

Migrations from closed-source/vendor-lockin to open-source is kinda our bread-n-butter at Percona. As companies come to us for Redis/Valkey support, the first goal of most engagements will be to migrate them to Valkey. But if there's no official way to do that, adoption will be difficult.

As it now stands, my blog post is now out of date since you can no longer perform an binary-in-place upgrade to Valkey, if you're using Redis 7.4+.

There needs to be a clear path to Valkey from Redis, regardless of the version. If replication works with any version, great, we can update all our documentation to that effect and that will become 'best practice' for migration.

[hwware]

@utdrmac I am not sure if your clients know the legal issue upgrade from any previous redis version to Redis 7.4, because at the beginning of Redis 7.4, Redis changes from open source software to source available software. Thus if any of your clients wishes to upgrade to Valkey, following is the right way:

1. any version just before Redis 7.0: first upgrade to Redis 7.0 because there are huge difference between Redis 7.0 and its previous version, then after testing everything is fine, upgrade to Redis 7.2, then upgrade to Valkey 7.2.5

2. for the Redis 7.0, upgrade to Redis 7.2 and then upgrade to Valkey 7.2.5

[utdrmac]

@hwware No, most customers do not pay attention to those things. They simply see "next version is out" and they start working on testing the next version and then upgrade soon. This is why being able to migrate from latest Redis to Valkey is functionality that always needs to exist in some way.

Additionally, if we onboard a new customer tomorrow, and they are already running Redis 7.4, there's no way we can tell them to not upgrade to that version. We have to deal with what they have now.

[bof]

This is something that redis itself never supported - downgrade from a newer rdb version to an older one. Consequently also replication from a newer redis version to an older one (different rdb version), never worked.

As a long time redis user I would be very happy to have some "rdb downgrade" rewriter (as far as compatibility can be kept), but I would not expect it to be exist.

[zuiderkwast]

I edited the title since it's not a bug.

We don't support loading/replicating future rdb versions but I understand migrating from proprietary Redis is an important use case.

https://github.com/redis/librdb (MIT license) seems to support this. There is a CLI tool included in the repo that can do RDB to RESP, which maybe can be used to insert the data in a Valkey instance.

https://github.com/tair-opensource/RedisShake from Alibaba is another tool for migrations. It doesn't support redis 7.4 though, but maybe they're interested in adding this feature as well.

[jdork0]

Maybe you can make something work by saving as aof and setting 'aof-use-rdb-preamble no'.

[bigb4ng]

@zuiderkwast You are right about librdb.
You can do this with rdb-cli dump.rdb redis -h <valkey_host> -p <valkey_port>
Tested for Redis 7.4.0 .rdb -> 7.2.6 Valkey

[AndrewWinterman]

Maybe you can make something work by saving as aof and setting 'aof-use-rdb-preamble no'.

I tried this with a sentinel cluster running in kuberentes and it lost the dataset :(

[AndrewWinterman]

Maybe you can make something work by saving as aof and setting 'aof-use-rdb-preamble no'.

I tried this with a sentinel cluster running in kuberentes and it lost the dataset :(

However!

I was able to get redis 7.4 -> valkey 8 by adding the following configs to redis before modifying the image:

aof-use-rdb-preamble no
repl-diskless-load swapdb

I need to repeat this test and try without the aof-use-rdb-preamble parameter, since it looks unnecessary now.

[zuiderkwast]

I need to repeat this test and try without the aof-use-rdb-preamble parameter, since it looks unnecessary now.

• AOF with preamble = RDB dump + AOF file with commands
• AOF without preamble = AOF file with commands

Redis 7.4 bumped the RDB format, so Valkey can't read it. Therefore, I think AOF with preamble doesn't work, but feel free to test it. AOF without preamble is just a sequence of commands.