How to migrate data from old versions to new one?

Problem:

Since v0.20-beta, TickTockDB uses a new binary format. The new binary format improves reads dramatically but unfortunately it is not backward compatible. If you upgrade TickTockDB from an old version to v0.20 or upper, TickTockDB will require you to manually migrate your old data to the new binary format. Otherwise, TickTockDB will exit immediately, as shown in the following.

ylin30@rpi4:~/ticktock $  TickTockDB v0.20.0,  Maintained by
 Yongtao You (yongtao.you@gmail.com) and Yi Lin (ylin30@gmail.com).
 This program comes with ABSOLUTELY NO WARRANTY. It is free software,
 and you are welcome to redistribute it under certain conditions.
 For details, see <https://www.gnu.org/licenses/>.
Initialization failed: ticktockdb.version config missing; TickTockDB version mismatch! Please see
https://github.com/ytyou/ticktock/wiki/How-to-migrate-data-from-old-versions-to-new-one%3F

[1]+  Exit 9                  ./bin/tt -c conf/tt.conf --tsdb.timestamp.resolution millisecond --http.server.port 6182,6183
ylin30@rpi4:~/ticktock $ ls ./data
2022  ticktock.meta
ylin30@rpi4:~/ticktock $

How to migrate data:

We prepare a script <ticktock>/scripts/migrate_data.sh for this purpose. Please read its usage message:

ylin30@rpi4:~/ticktock $ ./scripts/migrate_data.sh -h
Usage  : ./scripts/migrate_data.sh <host> <tcp port for put> <inspect binary> <data file dir>
Note: Inspect binary must be version compatible to data files.
For data files generted by TT from v0.10.0 to v0.12.2, please use v0.12.2 inspect.
Example: ./scripts/migrate_data.sh localhost 6181 /tmp/inspect.0.12.2 /home/usr1/backup/data
ylin30@rpi4:~/ticktock $

There are a few preparation steps before you run migrate_data.sh.

0. Gracefully stop the old version of Ticktock by calling ./admin/stop.sh (highly recommended);

ylin30@rpi4:~/ticktock $ ./admin/stop.sh
Exiting now.
ylin30@rpi4:~/ticktock $ Start shutdown process...
Shutdown process complete

[1]+  Done                    ./bin/tt -c conf/tt.conf --tsdb.data.dir /home/ylin30/ticktock/data --tsdb.timestamp.resolution millisecond --http.server.port 6182,6183
ylin30@rpi4:~/ticktock $

1. Move the old data folder to a secure place (e.g., /tmp/data.0.12.2);

ylin30@rpi4:~/ticktock $ ls ./data
2022  ticktock.meta
ylin30@rpi4:~/ticktock $ mv ./data /tmp/data.0.12.2

NOTE: If you see any .back directories, delete them before proceed.

2. Download or build the v0.12.2 inspect binary and keep it as <inspect binary>, e.g., /home/ylin30/inspect.0.12.2.arm.64bit.

3. Install nc (e.g., sudo apt install netcat in ubuntu) if not installed yet;

4. Make sure the new TT (v0.20.0 or upper) is running and listening at <host> and <tcp port for OpenTSDB put, default 6181>

ylin30@rpi4:~/ticktock $ ./bin/tt -c conf/tt.conf --tsdb.timestamp.resolution millisecond --http.server.port 6182,6183 &
[1] 15606
ylin30@rpi4:~/ticktock $  TickTockDB v0.20.0,  Maintained by
 Yongtao You (yongtao.you@gmail.com) and Yi Lin (ylin30@gmail.com).
 This program comes with ABSOLUTELY NO WARRANTY. It is free software,
 and you are welcome to redistribute it under certain conditions.
 For details, see <https://www.gnu.org/licenses/>.
Writing to log file: /home/ylin30/ticktock/log/ticktock.log
TickTockDB is ready...

ylin30@rpi4:~/ticktock $ ls ./data
config  metrics  ts  WAL
ylin30@rpi4:~/ticktock $

5. Then you can run migrate_data.sh, e.g.:

ylin30@rpi4:~/ticktock $ ./scripts/migrate_data.sh localhost 6181 /home/ylin30/inspect.0.12.2.arm.64bit /tmp/data.0.12.2/
/home/ylin30/inspect.0.12.2.arm.64bit -d /tmp/data.0.12.2/ -r | nc -q 30 localhost 6181
Total number of time series: 99,930
Inspecting tsdb /tmp/data.0.12.2//2022/1/1640995200.1641081600...
Total dps = 206,860
Grand Total = 206,860
ylin30@rpi4:~/ticktock $ ls -hl ./data
total 3.7M
drwxr--r-- 3 ylin30 ylin30 4.0K Apr 17 00:49 2022
-rw-r--r-- 1 ylin30 ylin30   54 Apr 17 00:47 config
-rw-r--r-- 1 ylin30 ylin30  11K Apr 17 00:50 metrics
-rw-r--r-- 1 ylin30 ylin30 3.6M Apr 17 00:50 ts
drwxr--r-- 2 ylin30 ylin30 4.0K Apr 17 00:50 WAL
ylin30@rpi4:~/ticktock $

Please note that migration_data.sh can only migrate data generated by TT v0.10.0 and newer versions. If you are still using older TT than v0.10.0, please don't hesitate to contact us. We will provide you a custom solution. Sorry for the inconvenience.