feat: Optimize data migration

[devchenyan]

• Optimize data migration Magickbase/neuron-public-issues#434

Screen-2025-02-23-204920.mp4

[silySuper]

/package
Packaging for test is done in 13511928473. @silySuper

[silySuper]

Firstly,go through the main flow I understand is that:
1.When resynchronize,choose a new path like XX which will generdate node files.
2.When retain previous data,choose a new path like XX(which will generdate node files also),then next,copy old path data to new path data,click confirm.
3.After copy totally,it will pop up tip migrate successful automatically.

But now in retain previous data，choose a new path like XX will not generdate node files，which may cause next step can not go on.

This is video:

2025-02-25.14.38.49.mov

@devchenyan please check this firstly~.

[devchenyan]

Firstly,go through the main flow I understand is that: 1.When resynchronize,choose a new path like XX which will generdate node files. 2.When retain previous data,choose a new path like XX(which will generdate node files also),then next,copy old path data to new path data,click confirm. 3.After copy totally,it will pop up tip migrate successful automatically.

But now in retain previous data，choose a new path like XX will not generdate node files，which may cause next step can not go on.

This is video:

2025-02-25.14.38.49.mov
@devchenyan please check this firstly~.

1、I will fix it.
2、3、Merging data does not automatically copy data to the new directory; the user needs to manually copy them.

[silySuper]

Firstly,go through the main flow I understand is that: 1.When resynchronize,choose a new path like XX which will generdate node files. 2.When retain previous data,choose a new path like XX(which will generdate node files also),then next,copy old path data to new path data,click confirm. 3.After copy totally,it will pop up tip migrate successful automatically.
But now in retain previous data，choose a new path like XX will not generdate node files，which may cause next step can not go on.
This is video:
2025-02-25.14.38.49.mov
@devchenyan please check this firstly~.

1、I will fix it. 2、3、Merging data does not automatically copy data to the new directory; the user needs to manually copy them.

Yes,I copy by command:rsync -ah --progress /Volumes/My\ Passport/ckb_v0.119.0_aarch64-apple-darwin-portable\ 3/testnet/data /Volumes/My\ Passport/test03-ckb-node/,and after copy finished it will show migrate successful in expect.

[devchenyan]

Firstly,go through the main flow I understand is that: 1.When resynchronize,choose a new path like XX which will generdate node files. 2.When retain previous data,choose a new path like XX(which will generdate node files also),then next,copy old path data to new path data,click confirm. 3.After copy totally,it will pop up tip migrate successful automatically.
But now in retain previous data，choose a new path like XX will not generdate node files，which may cause next step can not go on.
This is video:
2025-02-25.14.38.49.mov
@devchenyan please check this firstly~.

1、I will fix it. 2、3、Merging data does not automatically copy data to the new directory; the user needs to manually copy them.

Yes,I copy by command:rsync -ah --progress /Volumes/My\ Passport/ckb_v0.119.0_aarch64-apple-darwin-portable\ 3/testnet/data /Volumes/My\ Passport/test03-ckb-node/,and after copy finished it will show migrate successful in expect.

I checked, when choose resynchronize, will generdate node files. And when choose Retain previous data, user needs to manually copy files. These are all as expected, so no modifications are needed. Please confirm. @silySuper

[silySuper]

Firstly,go through the main flow I understand is that: 1.When resynchronize,choose a new path like XX which will generdate node files. 2.When retain previous data,choose a new path like XX(which will generdate node files also),then next,copy old path data to new path data,click confirm. 3.After copy totally,it will pop up tip migrate successful automatically.
But now in retain previous data，choose a new path like XX will not generdate node files，which may cause next step can not go on.
This is video:
2025-02-25.14.38.49.mov
@devchenyan please check this firstly~.

1、I will fix it. 2、3、Merging data does not automatically copy data to the new directory; the user needs to manually copy them.

Yes,I copy by command:rsync -ah --progress /Volumes/My\ Passport/ckb_v0.119.0_aarch64-apple-darwin-portable\ 3/testnet/data /Volumes/My\ Passport/test03-ckb-node/,and after copy finished it will show migrate successful in expect.

I checked, when choose resynchronize, will generdate node files. And when choose Retain previous data, user needs to manually copy files. These are all as expected, so no modifications are needed. Please confirm. @silySuper

tip shows only copy data files,not all files.

[devchenyan]

Firstly,go through the main flow I understand is that: 1.When resynchronize,choose a new path like XX which will generdate node files. 2.When retain previous data,choose a new path like XX(which will generdate node files also),then next,copy old path data to new path data,click confirm. 3.After copy totally,it will pop up tip migrate successful automatically.

But now in retain previous data，choose a new path like XX will not generdate node files，which may cause next step can not go on.

This is video:

2025-02-25.14.38.49.mov

@devchenyan please check this firstly~.

1、I will fix it. 2、3、Merging data does not automatically copy data to the new directory; the user needs to manually copy them.

Yes,I copy by command:rsync -ah --progress /Volumes/My\ Passport/ckb_v0.119.0_aarch64-apple-darwin-portable\ 3/testnet/data /Volumes/My\ Passport/test03-ckb-node/,and after copy finished it will show migrate successful in expect.

I checked, when choose resynchronize, will generdate node files. And when choose Retain previous data, user needs to manually copy files. These are all as expected, so no modifications are needed. Please confirm. @silySuper

tip shows only copy data files,not all files.

@Danie0918 Please help to confirm

[Danie0918]

Firstly,go through the main flow I understand is that: 1.When resynchronize,choose a new path like XX which will generdate node files. 2.When retain previous data,choose a new path like XX(which will generdate node files also),then next,copy old path data to new path data,click confirm. 3.After copy totally,it will pop up tip migrate successful automatically.

But now in retain previous data，choose a new path like XX will not generdate node files，which may cause next step can not go on.

This is video:

2025-02-25.14.38.49.mov

@devchenyan please check this firstly~.

1、I will fix it. 2、3、Merging data does not automatically copy data to the new directory; the user needs to manually copy them.

Yes,I copy by command:rsync -ah --progress /Volumes/My\ Passport/ckb_v0.119.0_aarch64-apple-darwin-portable\ 3/testnet/data /Volumes/My\ Passport/test03-ckb-node/,and after copy finished it will show migrate successful in expect.

I checked, when choose resynchronize, will generdate node files. And when choose Retain previous data, user needs to manually copy files. These are all as expected, so no modifications are needed. Please confirm. @silySuper

tip shows only copy data files,not all files.

@Danie0918 Please help to confirm

Text change to: Copy all files to the new path.

[devchenyan]

@silySuper The text revisions have been submitted, please review again.

[silySuper]

/package
Packaging for test is done in 13535774649. @silySuper

[silySuper]

1.In resync,modify path will restore last choosen file which was deleted in local.

2025-02-26.15.08.39.mov

2.In resync,if I choose local files ,then delete the local files .Click browse local files ,no response.

3.mouse tip shows wrong place

4.no fill in tip actually

UI document:

5.In retain previous data,pop up window can not be closed directly,need close window one by one actually.

2025-02-26.15.19.22.mov

6. no migrate successful tip when copy finished.

2025-02-26.15.55.05.mov

[Danie0918]

UI：https://www.figma.com/design/6XNoimRDbFTTNm016rbIdU/Magickbase?node-id=319-4539&p=f&t=WNXTvul4tKjWNuzn-0

Text: The sync file is lost. Please try to recover it and retry, or resync.

Button: Resync/Retry

Requirement: Synchronize or click to open the path to check if the file is lost. A prompt will appear.

• Click 'Resync' to resync.
• Click the 'Retry' button to recheck.

cc @devchenyan @silySuper

[Danie0918]

Also, the button text we used for resync earlier was 'Resynchronize.' This time, please change it to 'Resync'. @devchenyan

[silySuper]

【sync】
7.In resync ,after change path to a new path ，sync stay shows sync is slow ,and log shows error :

part content of main.log:

[2025-02-26T22:19:49.724Z] [info] Monitor: Restarting ckb process success
[2025-02-26T22:19:49.879Z] [error] CKB: run fail: Migration error
[2025-02-26T22:19:49.879Z] [error] CKB: run fail: Internal(
[2025-02-26T22:19:49.879Z] [error] CKB: run fail: DataCorrupted(DB corrupted))

[2025-02-26T22:19:49.885Z] [info] CKB: process closed with code 113
[2025-02-26T22:20:19.467Z] [warn] function call error: TypeError: fetch failed, retry 2 ...
[2025-02-26T22:20:19.571Z] [warn] function call error: TypeError: fetch failed, retry 3 ...
[2025-02-26T22:20:19.675Z] [info] Monitor: is restarting ckb process
[2025-02-26T22:20:19.676Z] [info] CKB: Initializing node...
[2025-02-26T22:20:19.678Z] [info] CKB: init: config file detected, skip ckb init.

main.log

[devchenyan]

【sync】 7.In resync ,after change path to a new path ，sync stay shows sync is slow ,and log shows error :

part content of main.log:

[2025-02-26T22:19:49.724Z] [info] Monitor: Restarting ckb process success [2025-02-26T22:19:49.879Z] [error] CKB: run fail: Migration error [2025-02-26T22:19:49.879Z] [error] CKB: run fail: Internal( [2025-02-26T22:19:49.879Z] [error] CKB: run fail: DataCorrupted(DB corrupted))

[2025-02-26T22:19:49.885Z] [info] CKB: process closed with code 113 [2025-02-26T22:20:19.467Z] [warn] function call error: TypeError: fetch failed, retry 2 ... [2025-02-26T22:20:19.571Z] [warn] function call error: TypeError: fetch failed, retry 3 ... [2025-02-26T22:20:19.675Z] [info] Monitor: is restarting ckb process [2025-02-26T22:20:19.676Z] [info] CKB: Initializing node... [2025-02-26T22:20:19.678Z] [info] CKB: init: config file detected, skip ckb init.

main.log

This is normal, the ckb process has been restarted.

[devchenyan]

All the issues have been resolved. Please test again. @silySuper

[silySuper]

/package
Packaging for test is done in 13560621490. @silySuper

[silySuper]

1.same as before
2.verified
3.verified
4.tip font color is gery in expect,but actually is black,which may mislead user that it can be modified directly.

5.verified
6.migrate successfully tip is better to show after files copy totally,but now is show after click confirm.
Please confirm whether 1 and 6 need to amend in user's situation. @Danie0918

[Danie0918]

1.same as before 2.verified 3.verified 4.tip font color is gery in expect,but actually is black,which may mislead user that it can be modified directly. 5.verified 6.migrate successfully tip is better to show after files copy totally,but now is show after click confirm. Please confirm whether 1 and 6 need to amend in user's situation. @Danie0918

No need for modifications, 1 and 6 are acceptable.

[devchenyan]

1.same as before 2.verified 3.verified 4.tip font color is gery in expect,but actually is black,which may mislead user that it can be modified directly. 5.verified 6.migrate successfully tip is better to show after files copy totally,but now is show after click confirm. Please confirm whether 1 and 6 need to amend in user's situation. @Danie0918

fixed. please check again @silySuper

[silySuper]

/package

Packaging for test is done in 13579867158. @silySuper

[silySuper]

4.verified

[silySuper]

【sync】 7.In resync ,after change path to a new path ，sync stay shows sync is slow ,and log shows error :

part content of main.log:
[2025-02-26T22:19:49.724Z] [info] Monitor: Restarting ckb process success [2025-02-26T22:19:49.879Z] [error] CKB: run fail: Migration error [2025-02-26T22:19:49.879Z] [error] CKB: run fail: Internal( [2025-02-26T22:19:49.879Z] [error] CKB: run fail: DataCorrupted(DB corrupted))
[2025-02-26T22:19:49.885Z] [info] CKB: process closed with code 113 [2025-02-26T22:20:19.467Z] [warn] function call error: TypeError: fetch failed, retry 2 ... [2025-02-26T22:20:19.571Z] [warn] function call error: TypeError: fetch failed, retry 3 ... [2025-02-26T22:20:19.675Z] [info] Monitor: is restarting ckb process [2025-02-26T22:20:19.676Z] [info] CKB: Initializing node... [2025-02-26T22:20:19.678Z] [info] CKB: init: config file detected, skip ckb init.
main.log

This is normal, the ckb process has been restarted.

From yesterday to today,sync is 'sync' or '0.00%',it is hard to confirm whether sync is ok.

[devchenyan]

【sync】 7.In resync ,after change path to a new path ，sync stay shows sync is slow ,and log shows error :

part content of main.log:
[2025-02-26T22:19:49.724Z] [info] Monitor: Restarting ckb process success [2025-02-26T22:19:49.879Z] [error] CKB: run fail: Migration error [2025-02-26T22:19:49.879Z] [error] CKB: run fail: Internal( [2025-02-26T22:19:49.879Z] [error] CKB: run fail: DataCorrupted(DB corrupted))
[2025-02-26T22:19:49.885Z] [info] CKB: process closed with code 113 [2025-02-26T22:20:19.467Z] [warn] function call error: TypeError: fetch failed, retry 2 ... [2025-02-26T22:20:19.571Z] [warn] function call error: TypeError: fetch failed, retry 3 ... [2025-02-26T22:20:19.675Z] [info] Monitor: is restarting ckb process [2025-02-26T22:20:19.676Z] [info] CKB: Initializing node... [2025-02-26T22:20:19.678Z] [info] CKB: init: config file detected, skip ckb init.
main.log

This is normal, the ckb process has been restarted.

From yesterday to today,sync is 'sync' or '0.00%',it is hard to confirm whether sync is ok.

I actually tested it, and it takes about 8 minutes to complete progress reading and actually start syncing. You can check if it’s a network issue.This is the full screen recording of the testing process. @silySuper

https://drive.google.com/file/d/17eOwH6oizouyhLMmjG-qLuUREwdNxK-H/view?usp=drive_link