Batched publish deals fail other deals if one goes wrong

[RobQuistNL]

Checklist

• This is not a security-related bug/issue. If it is, please follow please follow the security policy.
• This is not a question or a support request. If you have any lotus related questions, please ask in the lotus forum.
• This is not a new feature request. If it is, please file a feature request instead.
• This is not an enhancement request. If it is, please file a improvement suggestion instead.
• I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
• I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
• I did not make any code changes to lotus.

Lotus component

• lotus daemon - chain sync
• lotus miner - mining and block production
• lotus miner/worker - sealing
• lotus miner - proving(WindowPoSt)
• lotus miner/market - storage deal
• lotus miner/market - retrieval deal
• lotus miner/market - data transfer
• lotus client
• lotus JSON-RPC API
• lotus message management (mpool)
• Other

Lotus Version

1.15.2

Describe the Bug

This message:
bafy2bzacedcfrtht7agb7ust456gs7f2yq5voulb6a3af2bebacm4lrdmfmkw
is a batched publish deal.
It contains some deals that were succesful, and another one that was not.

They are unrelated, yet, my storage-deals list shows the unrelated one as FAILED because one of the 6 has failed.

(probably due to no datacap);

Jun  3 21:28:01  true   bafyreidjaq5lcamj6ap7mjquh6qd4rs5yrixxg3p5ifxruoryh26adev7q  6766456  StorageDealError              f1pkrmygbvweykpjcut36lf7ewgqdfhjklbhvepda                                               4GiB    0 FIL             1533678                                                                                                                                 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1828500): failed to look up deal on chain: comparing publish deal message bafy2bzacedcfrtht7agb7ust456gs7f2yq5voulb6a3af2bebacm4lrdmfmkw proposal to deal proposal: resolve address f13ctc37hbnjhpz5vnksphpjfptfwrnqud2f445oq: actor not found

Logging Information

-

Repo Steps

[jennijuju]

(won't fix in lotus)

[RobQuistNL]

@LexLuthr

[LexLuthr]

@RobQuistNL We discussed this and it seems to be coming from specs-actor code base. Dirk will write up his thoughts and send it to correct repo. We can keep following up there till we have a fix.

[dirkmc]

Looking more closely at the error message I'm not sure that this is to do with batched publish deals.
The error message indicates that it was not possible to look up the address of the deal proposal client on chain.

It looks like the address is on chain: https://filfox.info/en/address/f13ctc37hbnjhpz5vnksphpjfptfwrnqud2f445oq

So maybe the miner got out of sync with the chain?

[RobQuistNL]

Eh no, the error is completely wrong, which is why I opened the ticket. Please read my description and check the message on chain.

[LexLuthr]

@RobQuistNL Do you have those older logs? They might be able to point to the code we should be looking at.

[dirkmc]

The steps that lotus takes when it receives a storage deal are:

1. Accept deal proposal
2. Data transfer
3. Verify data
4. Reserve funds
5. Publish deal
6. Hand-off deal to sealing subsystem
7. Wait for pre-commit message on chain

You can see these stages in the code here:
https://github.com/filecoin-project/go-fil-markets/blob/7eb34e4b68d6e3f1859b3eb041fd8b0d652d3ca6/storagemarket/impl/providerstates/provider_fsm.go#L251-L267

The error message error awaiting deal pre-commit occurs at stage 7. So that implies that this deal was successfully published (in step 5) or we would not have got to step 7.

The reason you see the words "comparing publish deal message" in the error message

comparing publish deal message bafy2bzacedcfrtht7agb7ust456gs7f2yq5voulb6a3af2bebacm4lrdmfmkw proposal to deal proposal

is because in step 7 lotus looks up the deal proposal by examining the publish deal message on chain.

Looking at the publish message on chain it looks like all 6 deals were published successfully:
https://filfox.info/en/message/bafy2bzacedcfrtht7agb7ust456gs7f2yq5voulb6a3af2bebacm4lrdmfmkw

{
  "IDs": [
    6766453,
    6766454,
    6766455,
    6766456,
    6766457,
    6766458
  ],
  "ValidDeals": "0-5"
}

The error message resolve address f13ctc37hbnjhpz5vnksphpjfptfwrnqud2f445oq: actor not found indicates that it was not possible to look up the address of client actor in the the deal proposal on chain.
However it looks like the address is on chain: https://filfox.info/en/address/f13ctc37hbnjhpz5vnksphpjfptfwrnqud2f445oq
That's why I suspect that somehow the lotus instance got out of sync with the chain.