renewed deals in old format not able to index

[honghaoq]

PiKNiK reported that indexing for renewed deals (the old format deals using graphsync) seem to be failing, while the new boost format deals seem to be working fine with indexing.
For old deals, those ones without an unsealed sector, they try to recover it and the miner tries to unseal the copy, but the formatting is incompatible with that indexing function with the following error:
failed to read/generate CAR Index: invalid header: malformed stream: invalid appearance of int token; expected start of map
The impact is currently they have more than 600 old deals unable to index today because of this. This seems to be happening for other SPs too and impacting indexing for thousands of renewed deals.

[willscott]

This isn't an issue in this repo. i think the only question is where to move it: is the underlying issue in index provider, dag store, boost, or go-fil-markets

[dirkmc]

they try to recover it and the miner tries to unseal the copy, but the formatting is incompatible with that indexing function with the following error

It would help us to diagnose the problem if we could have repro steps. What are the commands that they run?

[dirkmc]

@stuberman reports

I am using lotus v1.15.3 + Boost

Problem #1 - Evergreen deals use the legacy format (GraphSync) when I retrieve those CAR files they do not announce after publishing and the dagstore shard stays in the ‘new’ state until I manually initialize them.

Problem #2 - After the dagstore migration (months ago) I have more than 600 dagstore shards in ‘new’ or ‘errored’ state. Most of the related sectors no longer have unsealed sectors since we (many SPs) would delete them to save space expecting that if a retrieval was ever required we could simply unseal those sectors. Now when I try to initialize those related shards, I get an error and then I try to use a dagstore recover command. That triggers an unseal of the sealed sector. When the sector is unsealed the shard is still unable to be initialized due to the error you see above.

Notice that in both cases many SPs will not be indexing a portion of their deals due to these two problems. Trying to initialize-all is not a solution since this also triggers unsealing which is problematic.

[dirkmc]

@stuberman provided us with access to one of the original files on disk (in unsealed fr32-encoded format).

I wrote a script to remove fr32 padding and parse the file. After removing the fr32 padding, when the script attempts to index the CAR file it returns the same error as @stuberman was seeing:

invalid header: malformed stream: invalid appearance of uint token; expected map key

When running the script over an unsealed file known to be well-formed, it was possible to index the file correctly. The CBOR-encoded header of the well-formed file has a "roots" and a "version" field. In JSON format it looks like:

{
  "roots": [42(h'0012204962CEDEBB4BC0B4F67BC331564AECBA6BEAF7431DCF83A5BC4A249414F0AD3F')],
  "version": 1
}

The CBOR-encoded header of the file that cannot be indexed has a "roots" field but not a "version" field. In JSON format it looks like:

{
  "roots": [42(h'000170A0E402203EF79696452DDD725C13669D133B6E6ECB9CA7BAC198A805315528D9EA01C5AC')],
  18: h'B38FF776E4F3BD937CB4EC23AE35FD81C60FCD9D456000'
}

[phantom-rabbit]

I have the same problem.

I reset our miner-repo for some reasons.
Then I wrote the pieceinfo information back into/market/pieces/1/baga6ea4seaqkohlc7s7dbgxx5fraonzn2jfdcoizxdrbvfvdppbuaquu4rtkcbq , I execute lotus-miner dagstore recover-shard baga6ea4seaqkohlc7s7dbgxx5fraonzn2jfdcoizxdrbvfvdppbuaquu4rtkcbq
ERROR: failed to recover shard: failed to read/generate CAR Index: invalid header: malformed stream: invalid appearance of int token; expected start of map。
I think it has something to do with dagstore's database, but I don't know how to fix it