Miner restored from old backup not aware of newly proven sectors

[shotcollin]

Hi, all. I'm hoping someone can point me in the right direction, as I haven't seen anything here or on slack or elsewhere that addresses this particular situation.

The drive on my miner that stores LOTUS_MINER_PATH data became corrupted and I lost that data. Unfortunately the most recent backup was three or four days old. I restored the miner from that backup, but it's restored to its earlier state and has no record of about 60 sectors that went into Proving state since then. But those "missing" sectors are still on chain and being proven in their daily proving period. And now if I pledge new sectors the miner allocates numbers of the 60 proving, on-chain sectors. Those quickly fail, then the miner removes the sectors, deleting the sealed and cache files and causing those provable sectors to become faults.

I understand from reading some related issues (e.g. #4075) that restoring the miner's database isn't (currently?) possible from on-chain data (although that seems counterintuitive since the miner's only purpose seems to be to put onto the blockchain the very data that's now missing from the miner...).

In any event, does anyone have advice on how to proceed? The ~60 sectors in question are all CC, they don't contain any deals. So should I just pledge them, have them fail, and then restore their data that the miner deletes so they can remain provable? Or is there some better way to handle this?

P.S. I'm also getting errors when the miner starts about two sectors that were in WaitSeed state when the backup was taken: ERROR sectors storage-sealing/states_sealing.go:225 handlePreCommit1: StateSectorPreCommitInfo: api error, not proceeding: sectorNumber is allocated. Those are also now provable sectors, and the miner isn't in danger of deleting them, but I'd really like to see if it's possible to update the miner's database or otherwise do something to get rid of those errors.

[dd45e640b42e6da7da96faee3996ef7c]

first of all: make the permanent sector storage read only.

you might be able to pledge, let the counter increase, abort and delete the newly pledged sectors right away until you reach the count you are looking for.

if this creates more chaos then it helps: import the backup again and you are back at 0 - i think it's at least worth a try

first of all: make the permanent sector storage read only.
first of all: make the permanent sector storage read only.
first of all: make the permanent sector storage read only.
first of all: make the permanent sector storage read only.
first of all: make the permanent sector storage read only.
first of all: make the permanent sector storage read only.

in linux! chmod 500 or some like that.

this! before trying any funny things...

[shotcollin]

Thanks @f8-ptrk, I'll give that a shot.

Stupid question: In regards to making the permanent store read-only, you mean just for this experiment? It shouldn't be read-only in general, right? Because then the miner wouldn't be able to write new sectors to it...? Just want to make sure I didn't somehow miss some important configuration detail or something...

[dd45e640b42e6da7da96faee3996ef7c]

yes. making it read only is just to prevent lotus from deleting them. making a full copy would be even better

[SgtCoin]

@shotcollin Are you still having the same issue?

[shotcollin]

@SgtCoin I worked around the issue by re-pledging the sectors, which appeared at first to work fine... but ended up having to remove them all anyway because after running check --slow they would come back with bad (getting commR: commr is nil). (I was planning to actually write up a slightly more detailed report of the process I went through in case anyone else had this problem, but haven't quite gotten around to that yet.) But definitely curious if there's a better way to fix this?

[dd45e640b42e6da7da96faee3996ef7c]

you are sure to have set this read only before you did that?

[shotcollin]

Yep, I definitely had the sector data set as read-only. The check didn't seem to affect the underlying data in any case; it appears to be complaining about the fact that after re-pledging the sectors, even though the sector data still existed the sectors now had <nil> values in their commit records (or whatever the proper name for the data output by lotus-miner sectors status --log is). I'd guess this error was referencing the CIDcommR value, which was one of those that became <nil> upon re-pledging. (Sample below.)

[SgtCoin]

Hi @shotcollin,

I am not aware of a better way to correct this, however your writeup may help us troubleshoot and recommend a better way. Can you add that write up here?

[shotcollin]

Hi @SgtCoin - happy to provide a write up. My notes get a little sparse toward the end, but here's what I can piece together from notes and memory.

To recap the above: my miner's HD got corrupted and my moist recent backup was about three days old. During that time I had created about 50 new CC sectors, e.g. sectors 1000-1050. Upon restoring from backup, the miner thought we were back at the sector number last sealed before the HD crash, e.g. sector 1000. During the respective proving periods, however, sectors 1000-1050 were still successfully proving, and lotus-miner info reported a total of 1050 sectors. But when I pledged a new sector the miner would reassign sector numbers that existed according to the blockchain, e.g. sector 1001, 1002, etc. The miner would see data sealed/cache data for the "new" sector and delete the data. I made backups of those sectors and also set the sector data to read-only, and then pledged through 1050 and let the miner think they were deleted. That's about where I was after the initial round of troubleshooting based on @f8-ptrk's sage advice.

Here's an example of an actual lotus-miner sectors status --log report for one such sector:

SectorID:	1111
Status:		Removed
CIDcommD:	<nil>
CIDcommR:	<nil>
Ticket:		
TicketH:	0
Seed:		
SeedH:		0
Precommit:	<nil>
Commit:		<nil>
Deals:		[0]
Retries:	0
--------
Event Log:
0.	2021-12-31 13:54:30 +0000 UTC:	[event;sealing.SectorStartCC]	{"User":{"ID":1111,"SectorType":8}}
1.	2021-12-31 18:32:25 +0000 UTC:	[event;sealing.SectorPacked]	{"User":{"FillerPieces":[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq"}}]}}
2.	2021-12-31 18:32:25 +0000 UTC:	[event;sealing.SectorRemove]	{"User":{}}
3.	2021-12-31 18:32:27 +0000 UTC:	[event;sealing.SectorRemoved]	{"User":{}}
4.	2021-12-31 18:43:38 +0000 UTC:	[event;sealing.SectorForceState]	{"User":{"State":"Proving"}}
5.	2022-01-06 10:19:55 +0000 UTC:	[event;sealing.SectorTerminate]	{"User":{}}
6.	2022-01-06 10:29:20 +0000 UTC:	[event;sealing.SectorTerminating]	{"User":{"Message":{"/":"bafy2bzacea6cyy5hfclw6tj37hajg7xmqkd5dh36dxhal54s4k5dccz57zskk"}}}
7.	2022-01-06 10:32:29 +0000 UTC:	[event;sealing.SectorTerminated]	{"User":{"TerminatedAt":1438620}}
8.	2022-01-07 01:13:06 +0000 UTC:	[event;sealing.SectorRemove]	{"User":{}}
9.	2022-01-07 01:13:13 +0000 UTC:	[event;sealing.SectorRemoved]	{"User":{}}

I figured this was fine, the sectors would be removed according to the miner, but the data would still actually be there and be provable according to window post. That was actually the case for several days. What I discovered later was that when running lotus-miner proving check --slow, the re-pledged/removed/still-being-proven sectors returned errors, resulting in this output:

2         0          1104    bad (getting commR: commr is nil)
2         0          1118    bad (getting commR: commr is nil)
2         0          1103    bad (getting commR: commr is nil)

and this in the log:

2022-01-01T23:17:03.316Z	WARN	advmgr	sector-storage/faults.go:109	CheckProvable Sector FAULT: getting commR	{"sector": {"ID":{"Miner":1197354,"Number":1103},"ProofType":8}, "sealed": "/ltstore/sealed/s-t01197354-1103", "cache": "/ltstore/cache/s-t01197354-1103", "err": "commr is nil", "errVerbose": "commr is nil:\n    github.com/filecoin-project/lotus/node/impl.(*StorageMinerAPI).CheckProvable.func1\n        /home/user/lotus/node/impl/storminer.go:963"}
2022-01-01T23:17:03.317Z	WARN	advmgr	sector-storage/faults.go:109	CheckProvable Sector FAULT: getting commR	{"sector": {"ID":{"Miner":1197354,"Number":1104},"ProofType":8}, "sealed": "/ltstore/sealed/s-t01197354-1104", "cache": "/ltstore/cache/s-t01197354-1104", "err": "commr is nil", "errVerbose": "commr is nil:\n    github.com/filecoin-project/lotus/node/impl.(*StorageMinerAPI).CheckProvable.func1\n        /home/user/lotus/node/impl/storminer.go:963"}
2022-01-01T23:17:03.318Z	WARN	advmgr	sector-storage/faults.go:109	CheckProvable Sector FAULT: getting commR	{"sector": {"ID":{"Miner":1197354,"Number":1118},"ProofType":8}, "sealed": "/ltstore/sealed/s-t01197354-1118", "cache": "/ltstore/cache/s-t01197354-1118", "err": "commr is nil", "errVerbose": "commr is nil:\n    github.com/filecoin-project/lotus/node/impl.(*StorageMinerAPI).CheckProvable.func1\n        /home/user/lotus/node/impl/storminer.go:963"}

Note that only a proving check with the --slow flag would cause this; I had run several "non-slow" checks with no errors reported. And only after the proving check identified these errors were the sectors found to be faulty during the proving period. At that point it seemed like the only alternative to never running another proving check --slow again was to remove the remaining sectors that were provable per the blockchain but removed per the miner, since they would eventually become faulty once a --slow check was done.

[ChaiWenyu1983]

I met the same problem. How did you finally solve it?

I performed the deletion:

lotus-miner sectors remove --really-do-it

lotus-miner sectors terminate --really-do-it

But it doesn't work. Windowpost still fails! "Getting commR: commr is nil" is displayed.

[shotcollin]

@ChaiWenyu1983 I was seeing the getting commR: commr is nil errors when doing lotus-miner proving check, and the only way I could get rid of it was to remove the sectors. But it sounds like you've done that already...? Do those sectors show up as removed when you run lotus-miner sectors list with --show-removed flag? I had many times where the remove and terminate seemed like they succeeded but the sector wasn't actually removed. Often just restarting the miner and maybe terminating/removing again would fix that.