Skip to content

will not quit if fail to prepare, like disk space is full #13

@july2993

Description

@july2993

run:
./tpccgen all --lightning-ip 172.16.5.70 --importer-ip 172.16.5.69 --data-dir /data2/tpcc_order_line --tidb-ip 172.16.5.70 --tidb-port 4000 --deploy-dir /data1/deploy --db tpcc2 --threads 40 --warehouse 14336 > out

It will not quit if fail to prepare, like disk space is full.
should quit if failed to generate CSV but not continue to restore.

load to warehouse in warehouse 10340

load to stock in warehouse 10340

execute prepare failed, err load stock at warehouse 10241 failed write /data2/tpcc_order_line/tpcc2.stock.0.csv: no space left on device

execute prepare failed, err load stock at warehouse 10354 failed write /data2/tpcc_order_line/tpcc2.stock.33.csv: no space left on device

execute prepare failed, err load stock at warehouse 10294 failed write /data2/tpcc_order_line/tpcc2.stock.13.csv: no space left on device

execute prepare failed, err load stock at warehouse 10287 failed write /data2/tpcc_order_line/tpcc2.stock.6.csv: no space left on device

and start restore:

Skip preparing checking. Please load CSV data into database and check later.

Finished

#============
generate csv files finished
#============
#============
start tikv-importer
#============
/bin/ssh 172.16.5.69 sh /data1/deploy/scripts/stop_importer.sh

lightning log( fail checksum ?):

[2020/03/23 11:35:38.679 +08:00] [INFO] [restore.go:506] [progress] [files="1281/1441 (88.9%)"] [tables="8/9 (88.9%)"] [speed(MiB/s)=84.36864763279368] [state=writing] [remaining=7m30s]
[2020/03/23 11:36:45.720 +08:00] [WARN] [util.go:115] ["compute remote checksum failed but going to try again"] [table=`tpcc2`.`stock`] [query="ADMIN CHECKSUM TABLE `tpcc2`.`stock`"] [retryCnt=0] [error="Error 9005: Region is unavailable"]
[2020/03/23 11:36:45.720 +08:00] [WARN] [util.go:106] ["compute remote checksum retry start"] [table=`tpcc2`.`stock`] [query="ADMIN CHECKSUM TABLE `tpcc2`.`stock`"] [retryCnt=1]
[2020/03/23 11:37:33.689 +08:00] [INFO] [restore.go:1510] ["remote checksum completed"] [table=`tpcc2`.`stock`] [takeTime=4m51.909715449s] []
[2020/03/23 11:37:33.691 +08:00] [ERROR] [restore.go:602] ["restore table failed"] [table=`tpcc2`.`stock`] [takeTime=1h1m48.731959817s] [error="checksum mismatched remote vs local => (checksum: 3775979478809315642 vs 6530941926594385115) (total_kvs: 2062344285 vs 2062344286) (total_bytes:426414213518 vs 426414213563)"]
[2020/03/23 11:37:33.691 +08:00] [ERROR] [restore.go:697] ["restore all tables data failed"] [takeTime=1h1m55.013744875s] [error="checksum mismatched remote vs local => (checksum: 3775979478809315642 vs 6530941926594385115) (total_kvs: 2062344285 vs 2062344286) (total_bytes:426414213518 vs 426414213563)"]
[2020/03/23 11:37:33.691 +08:00] [ERROR] [restore.go:277] ["run failed"] [step=2] [error="checksum mismatched remote vs local => (checksum: 3775979478809315642 vs 6530941926594385115) (total_kvs: 2062344285 vs 2062344286) (total_bytes:426414213518 vs 426414213563)"]
[2020/03/23 11:37:33.691 +08:00] [INFO] [restore.go:475] ["everything imported, stopping periodic actions"]
[2020/03/23 11:37:33.691 +08:00] [ERROR] [restore.go:283] ["the whole procedure failed"] [takeTime=1h1m55.029686236s] [error="checksum mismatched remote vs local => (checksum: 3775979478809315642 vs 6530941926594385115) (total_kvs: 2062344285 vs 2062344286) (total_bytes:426414213518 vs 426414213563)"]
[2020/03/23 11:37:33.691 +08:00] [ERROR] [restore.go:114] ["tables failed to be imported"] [count=1]
[2020/03/23 11:37:33.691 +08:00] [ERROR] [restore.go:116] [-] [table=`tpcc2`.`stock`] [status=checksum] [error="checksum mismatched remote vs local => (checksum: 3775979478809315642 vs 6530941926594385115) (total_kvs: 2062344285 vs 2062344286) (total_bytes:426414213518 vs 426414213563)"]
[2020/03/23 11:37:33.691 +08:00] [ERROR] [main.go:75] ["tidb lightning encountered error"] [error="checksum mismatched remote vs local => (checksum: 3775979478809315642 vs 6530941926594385115) (total_kvs: 2062344285 vs 2062344286) (total_bytes:426414213518 vs 426414213563)"] [errorVerbose="checksum mismatched remote vs local => (checksum: 3775979478809315642 vs 6530941926594385115) (total_kvs: 2062344285 vs 2062344286) (total_bytes:426414213518 vs 426414213563)\ngithub.com/pingcap/tidb-lightning/lightning/restore.(*TableRestore).compareChecksum\n\t/Users/huangjiahao/go/src/github.com/pingcap/tidb-lightning/lightning/restore/restore.go:1452\ngithub.com/pingcap/tidb-lightning/lightning/restore.(*TableRestore).postProcess\n\t/Users/huangjiahao/go/src/github.com/pingcap/tidb-lightning/lightning/restore/restore.go:1044\ngithub.com/pingcap/tidb-lightning/lightning/restore.(*TableRestore).restoreTable\n\t/Users/huangjiahao/go/src/github.com/pingcap/tidb-lightning/lightning/restore/restore.go:747\ngithub.com/pingcap/tidb-lightning/lightning/restore.(*RestoreController).restoreTables.func1\n\t/Users/huangjiahao/go/src/github.com/pingcap/tidb-lightning/lightning/restore/restore.go:601\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"]

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions