Miner start failed now #9801

lbj2004032 · 2022-11-30T17:21:06Z

lbj2004032
Nov 30, 2022

Checklist

This is not a security-related bug/issue. If it is, please follow please follow the security policy.
This is not a question or a support request. If you have any lotus related questions, please ask in the lotus forum.
This is not a new feature request. If it is, please file a feature request instead.
This is not an enhancement request. If it is, please file a improvement suggestion instead.
I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
I did not make any code changes to lotus.

Lotus component

Lotus Version

lotus-miner version 1.18.0+mainnet+git.bd10bdf99

Describe the Bug

I don't have a duplicate path, but I can't start

What should I do?
I have changed V1.18.1 and still cannot start
I could have started it, but there was an error in windowpost starting at about 9:00 on November 30, so I restarted it and found that it could not be started

$ cat storage.json
{
"StoragePaths": [
{
"Path": "/home/lotus/netdisk/mainData/"
}
]
}l

Logging Information

2022-12-01T01:13:07.814+0800    INFO    paramfetch      [email protected]/paramfetch.go:209  Parameter file /mnt/lotus1/data_v28/proofs_new/v28-fil-inner-product-v1.srs is ok
2022-12-01T01:13:17.292+0800    INFO    basichost       basic/natmgr.go:93      DiscoverNAT error:no NAT found
2022-12-01T01:14:02.419+0800    INFO    paramfetch      [email protected]/paramfetch.go:209  Parameter file /mnt/lotus1/data_v28/proofs_new/v28-empty-sector-update-merkletree-poseidon_hasher-8-8-0-3b7f44a9362e3985369454947bc94022e118211e49fd672d52bec1cbfd599d18.params is ok
2022-12-01T01:14:22.166+0800    INFO    paramfetch      [email protected]/paramfetch.go:209  Parameter file /mnt/lotus1/data_v28/proofs_new/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-8-0-sha256_hasher-82a357d2f2ca81dc61bb45f4a762807aedee1b0a53fd6c4e77b46a01bfef7820.params is ok
2022-12-01T01:14:44.685+0800    INFO    paramfetch      [email protected]/paramfetch.go:209  Parameter file /mnt/lotus1/data_v28/proofs_new/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-8-0-0377ded656c6f524f1618760bffe4e0a1c51d5a70c4509eedae8a27555733edc.params is ok
2022-12-01T01:14:44.685+0800    INFO    paramfetch      [email protected]/paramfetch.go:233  parameter and key-fetching complete
2022-12-01T01:14:44.694+0800    INFO    stores  paths/index.go:181      New sector storage: 9ffc8f88-bf07-4723-b619-dd41e89b45b7

Repo Steps

Run '...'
Do '...'
See error '...'
...

TippyFlitsUK · 2022-11-30T17:22:23Z

TippyFlitsUK
Nov 30, 2022
Maintainer

What does journalctl show @lbj2004032?

0 replies

lbj2004032 · 2022-11-30T17:27:21Z

lbj2004032
Nov 30, 2022
Author

journalctl

Previously, it could be started successfully, but now it's not working. I don't know why

0 replies

lbj2004032 · 2022-11-30T17:56:04Z

lbj2004032
Nov 30, 2022
Author

What does journalctl show @lbj2004032?

What to do? I'm in a hurry

0 replies

TippyFlitsUK · 2022-11-30T20:38:48Z

TippyFlitsUK
Nov 30, 2022
Maintainer

Can you please post the journalctl output?

0 replies

lbj2004032 · 2022-12-01T04:35:57Z

lbj2004032
Dec 1, 2022
Author

journalctl

$ journalctl
-- Logs begin at Tue 2022-03-29 16:02:15 CST, end at Thu 2022-12-01 12:35:31 CST. --
Mar 29 16:02:15 miner-172.70.1.12 sshd[3001890]: Received disconnect from 172.70.1.10 port 42632:11: disconnected by user
Mar 29 16:02:15 miner-172.70.1.12 sshd[3001890]: Disconnected from user lotus 172.70.1.10 port 42632
Mar 29 16:15:26 miner-172.70.1.12 sshd[2999847]: Received disconnect from 172.70.1.10 port 42626:11: disconnected by user
Mar 29 16:15:26 miner-172.70.1.12 sshd[2999847]: Disconnected from user lotus 172.70.1.10 port 42626
Mar 29 19:25:30 miner-172.70.1.12 prometheus-node-exporter[2634486]: time="2022-03-29T19:25:30+08:00" level=error msg="error >
Mar 29 19:25:30 miner-172.70.1.12 prometheus-node-exporter[2634486]: time="2022-03-29T19:25:30+08:00" level=error msg="error >
Mar 29 19:25:30 miner-172.70.1.12 prometheus-node-exporter[2634486]: time="2022-03-29T19:25:30+08:00" level=error msg="error >
Mar 29 19:25:30 miner-172.70.1.12 prometheus-node-exporter[2634486]: time="2022-03-29T19:25:30+08:00" level=error msg="error >
Mar 29 19:25:30 miner-172.70.1.12 prometheus-node-exporter[2634486]: time="2022-03-29T19:25:30+08:00" level=error msg="error >
Mar 29 19:25:30 miner-172.70.1.12 prometheus-node-exporter[2634486]: time="2022-03-29T19:25:30+08:00" level=error msg="error >
Mar 29 19:25:30 miner-172.70.1.12 prometheus-node-exporter[2634486]: time="2022-03-29T19:25:30+08:00" level=error msg="error >
Mar 29 19:25:30 miner-172.70.1.12 prometheus-node-exporter[2634486]: time="2022-03-29T19:25:30+08:00" level=error msg="error >
Mar 29 19:25:30 miner-172.70.1.12 prometheus-node-exporter[2634486]: time="2022-03-29T19:25:30+08:00" level=error msg="error >
Mar 29 19:25:30 miner-172.70.1.12 prometheus-node-exporter[2634486]: time="2022-03-29T19:25:30+08:00" level=error msg="error >
Mar 29 19:25:30 miner-172.70.1.12 prometheus-node-exporter[2634486]: time="2022-03-29T19:25:30+08:00" level=error msg="error >
Mar 29 19:25:30 miner-172.70.1.12 prometheus-node-exporter[2634486]: time="2022-03-29T19:25:30+08:00" level=error msg="error >
Mar 29 19:25:30 miner-172.70.1.12 prometheus-node-exporter[2634486]: time="2022-03-29T19:25:30+08:00" level=error msg="error >
Mar 29 19:25:30 miner-172.70.1.12 prometheus-node-exporter[2634486]: time="2022-03-29T19:25:30+08:00" level=error msg="error >
Mar 29 19:25:30 miner-172.70.1.12 prometheus-node-exporter[2634486]: time="2022-03-29T19:25:30+08:00" level=error msg="error >
Mar 29 19:25:30 miner-172.70.1.12 prometheus-node-exporter[2634486]: time="2022-03-29T19:25:30+08:00" level=error msg="error >
Mar 29 19:25:30 miner-172.70.1.12 prometheus-node-exporter[2634486]: time="2022-03-29T19:25:30+08:00" level=error msg="error >
Mar 29 19:25:30 miner-172.70.1.12 prometheus-node-exporter[2634486]: time="2022-03-29T19:25:30+08:00" level=error msg="error >
Mar 29 19:25:30 miner-172.70.1.12 prometheus-node-exporter[2634486]: time="2022-03-29T19:25:30+08:00" level=error msg="error >
Mar 29 19:25:30 miner-172.70.1.12 prometheus-node-exporter[2634486]: time="2022-03-29T19:25:30+08:00" level=error msg="error >
Mar 29 19:25:30 miner-172.70.1.12 prometheus-node-exporter[2634486]: time="2022-03-29T19:25:30+08:00" level=error msg="error >
Mar 29 19:25:30 miner-172.70.1.12 prometheus-node-exporter[2634486]: time="2022-03-29T19:25:30+08:00" level=error msg="error >
Mar 29 19:25:30 miner-172.70.1.12 prometheus-node-exporter[2634486]: time="2022-03-29T19:25:30+08:00" level=error msg="error >
Mar 29 19:25:30 miner-172.70.1.12 prometheus-node-exporter[2634486]: time="2022-03-29T19:25:30+08:00" level=error msg="error >
Mar 29 19:25:30 miner-172.70.1.12 prometheus-node-exporter[2634486]: time="2022-03-29T19:25:30+08:00" level=error msg="error >
Mar 29 19:25:30 miner-172.70.1.12 prometheus-node-exporter[2634486]: time="2022-03-29T19:25:30+08:00" level=error msg="error >
Mar 29 19:25:30 miner-172.70.1.12 prometheus-node-exporter[2634486]: time="2022-03-29T19:25:30+08:00" level=error msg="error >

0 replies

TippyFlitsUK · 2022-12-01T16:17:07Z

TippyFlitsUK
Dec 1, 2022
Maintainer

Are you using a single miner or do you have separate workers?

0 replies

2022-12-05T12:06:02Z

github-actions[bot]
bot Dec 5, 2022

Oops, seems like we needed more information for this issue, please comment with more details or this issue will be closed in 24 hours.

0 replies

lbj2004032 · 2022-12-05T12:46:26Z

lbj2004032
Dec 5, 2022
Author

Oops, seems like we needed more information for this issue, please comment with more details or this issue will be closed in 24 hours.
There are many tasks that cannot be deleted. I don't know if they will affect

Sectors:
Total: 156447
Available: 147857
Packing: 115
PreCommit1: 36
PreCommit2: 2
Committing: 147
CommitFinalize: 300
FinalizeSector: 21
Removed: 7902
SealPreCommit1Failed: 2
CommitFailed: 65

0 replies

TippyFlitsUK · 2022-12-05T14:39:49Z

TippyFlitsUK
Dec 5, 2022
Maintainer

Are you back online @lbj2004032?

0 replies

lbj2004032 · 2022-12-05T14:50:41Z

lbj2004032
Dec 5, 2022
Author

Are you back online @lbj2004032?

yes

0 replies

TippyFlitsUK · 2022-12-05T16:19:55Z

TippyFlitsUK
Dec 5, 2022
Maintainer

Are you using a single miner or do you have separate workers?

0 replies

TippyFlitsUK · 2022-12-05T16:31:54Z

TippyFlitsUK
Dec 5, 2022
Maintainer

Are you starting workers as well or are they completely disabled?

0 replies

TippyFlitsUK · 2022-12-06T04:48:36Z

TippyFlitsUK
Dec 6, 2022
Maintainer

The journalctl logs you posted are from March @lbj2004032!

Can you please post journalctl logs covering the recent period??

0 replies

TippyFlitsUK · 2022-12-06T12:38:26Z

TippyFlitsUK
Dec 6, 2022
Maintainer

Can you also please post the output of lotus-miner sealing workers so we can confirm that all workers have successfully disconnected from the miner?

0 replies

TippyFlitsUK · 2022-12-06T12:52:09Z

TippyFlitsUK
Dec 6, 2022
Maintainer

Post it when you have a chance buddy, I understand you may have a busy sealing queue at the moment 👍

0 replies

TippyFlitsUK · 2022-12-06T14:39:54Z

TippyFlitsUK
Dec 6, 2022
Maintainer

Certainly worth a try @lbj2004032 👍 .

The issue is definitely pointing towards a local config issue though as we have not had any other reports of this precise behaviour. The duplicate paths issue was similar but easily remedied.

Something is having an effect on your declared storage paths but it really isn't clear what seeing as you have already eliminated duplicates.

The main difference between v1.17.0 and v1.18.0 which relates to the error that you are seeing is 9032. Tighter checks on declared storage paths were implemented due to added flexibility in path management at runtime.

This is also reflected in the behaviour that you are seeing when upgrading and downgrading between v1.17.0 and v1.18.0. You are able to start v17 because the additional checks are not present but upgrading to v18 fails the new check when you upgrade and restart.

It's interesting that you can start v18 at all and suggests that the issue doesn't exist at the first run but gets introduced later on.

Trying v1.19.0 certainly won't do any harm but I am doubtful that it will resolve the issue.

I would recommend double-checking all your paths again in every single storage.json file that your system uses. Also make sure that there is no automated script running somewhere that might be calling lotus-miner storage attach. Any non-lotus system commands relating to storage management that you run at regular intervals should also be eliminated.

3 replies

lbj2004032 Dec 6, 2022
Author

I hope the next version can be solved

What I'm worried about is that I won't be able to start again, and then I'll game over

I have 3 nodes with 10P storage capacity, all of which have the same problem

TippyFlitsUK Dec 7, 2022
Maintainer

Did you try the new version @lbj2004032?

TippyFlitsUK Dec 7, 2022
Maintainer

Have you tried version v1.19.0-rc2?

Reiers · 2022-12-07T17:01:16Z

Reiers
Dec 7, 2022

Try to copy / paste the content below in the storage.json @lbj2004032

{
  "StoragePaths": [
    {
      "Path": "/home/lotus/netdisk/mainData/"
    }
  ]
}

From your output above, the syntax is broken due to this symbol -> l

Worth a shot - also please double check the storage.json on all your workers.

If it stops at this stage, I think it something some wrong with storage paths some where.
2022-12-01T01:14:44.694+0800 INFO stores paths/index.go:181 New sector storage: 9ffc8f88-bf07-4723-b619-dd41e89b45b7

4 replies

lbj2004032 Dec 7, 2022
Author

Try to copy / paste the content below in the storage.json @lbj2004032
{
  "StoragePaths": [
    {
      "Path": "/home/lotus/netdisk/mainData/"
    }
  ]
}
From your output above, the syntax is broken due to this symbol -> l

Worth a shot - also please double check the storage.json on all your workers.

If it stops at this stage, I think it something some wrong with storage paths some where. 2022-12-01T01:14:44.694+0800 INFO stores paths/index.go:181 New sector storage: 9ffc8f88-bf07-4723-b619-dd41e89b45b7

The syntax error should be that when I reply, the Enter key hits another key.
In addition, all my wokers have stopped because I have multiple miner nodes with the same problem. I am sure that all the workers have stopped when I started the miner. The worker cluster is indeed different from the miner through the mount to the permanent storage path, but the workers have stopped
I remember that one of the miner nodes could be started successfully during the first upgrade. It was stopped after running for a while and could not be started again

Reiers Dec 7, 2022

The syntax error should be that when I reply, the Enter key hits another key.

Okay, so a typo. Good to know.

In addition, all my wokers have stopped because I have multiple miner nodes with the same problem. I am sure that all the workers have stopped when I started the miner. The worker cluster is indeed different from the miner through the mount to the permanent storage path, but the workers have stopped

Ok, so it's only the "Path": "/home/lotus/netdisk/mainData/" it's trying to connect with.

I remember that one of the miner nodes could be started successfully during the first upgrade. It was stopped after running for a while and could not be started again

Interesting, so it did work after the upgrade at some point.

Since it's only 1 path to troubleshoot.

What kind of storage solution are you using?

Raid, Pools ?
Is it undergoing any maintenance, resilver - scrub etc?
Are all the files intact ?
How is IO R/W?
Is sectorstorage.json been edited ? If, check for mistakes.

Reiers Dec 11, 2022

The patch backported to v1.18.2 does not solve your issue, but I strongly recommend upgrading to the newer version, as v1.18.0/1 have a sealing bug, which can result in the miner not starting.

Reiers Dec 11, 2022

In that case, to avoid potential issues, I would suggest not onboarding deals since that can trigger the AP bug patched in v1.18.2.

Until we can properly identify and address the issue you are describing. There's not much we can do about your situation right now.
We are on the look out for information and other reports.

Miner start failed now #9801

Uh oh!

lbj2004032 Nov 30, 2022

Checklist

Lotus component

Lotus Version

Describe the Bug

Logging Information

Repo Steps

Replies: 17 comments · 7 replies

Uh oh!

TippyFlitsUK Nov 30, 2022 Maintainer

Uh oh!

lbj2004032 Nov 30, 2022 Author

Uh oh!

lbj2004032 Nov 30, 2022 Author

Uh oh!

TippyFlitsUK Nov 30, 2022 Maintainer

Uh oh!

lbj2004032 Dec 1, 2022 Author

Uh oh!

TippyFlitsUK Dec 1, 2022 Maintainer

Uh oh!

github-actions[bot] bot Dec 5, 2022

Uh oh!

lbj2004032 Dec 5, 2022 Author

Uh oh!

TippyFlitsUK Dec 5, 2022 Maintainer

Uh oh!

lbj2004032 Dec 5, 2022 Author

Uh oh!

TippyFlitsUK Dec 5, 2022 Maintainer

Uh oh!

TippyFlitsUK Dec 5, 2022 Maintainer

Uh oh!

TippyFlitsUK Dec 6, 2022 Maintainer

Uh oh!

TippyFlitsUK Dec 6, 2022 Maintainer

Uh oh!

TippyFlitsUK Dec 6, 2022 Maintainer

Uh oh!

TippyFlitsUK Dec 6, 2022 Maintainer

Uh oh!

lbj2004032 Dec 6, 2022 Author

Uh oh!

TippyFlitsUK Dec 7, 2022 Maintainer

Uh oh!

TippyFlitsUK Dec 7, 2022 Maintainer

Uh oh!

Reiers Dec 7, 2022

Uh oh!

Uh oh!

lbj2004032 Dec 7, 2022 Author

Uh oh!

Reiers Dec 7, 2022

Uh oh!

Reiers Dec 11, 2022

Uh oh!

Reiers Dec 11, 2022

lbj2004032
Nov 30, 2022

Replies: 17 comments 7 replies

TippyFlitsUK
Nov 30, 2022
Maintainer

lbj2004032
Nov 30, 2022
Author

lbj2004032
Nov 30, 2022
Author

TippyFlitsUK
Nov 30, 2022
Maintainer

lbj2004032
Dec 1, 2022
Author

TippyFlitsUK
Dec 1, 2022
Maintainer

github-actions[bot]
bot Dec 5, 2022

lbj2004032
Dec 5, 2022
Author

TippyFlitsUK
Dec 5, 2022
Maintainer

lbj2004032
Dec 5, 2022
Author

TippyFlitsUK
Dec 5, 2022
Maintainer

TippyFlitsUK
Dec 5, 2022
Maintainer

TippyFlitsUK
Dec 6, 2022
Maintainer

TippyFlitsUK
Dec 6, 2022
Maintainer

TippyFlitsUK
Dec 6, 2022
Maintainer

TippyFlitsUK
Dec 6, 2022
Maintainer

lbj2004032 Dec 6, 2022
Author

TippyFlitsUK Dec 7, 2022
Maintainer

TippyFlitsUK Dec 7, 2022
Maintainer

Reiers
Dec 7, 2022

lbj2004032 Dec 7, 2022
Author