SPX v1.15.2-rc3 - Testing & Call Outs

[Reiers]

Hello SPX Fellows

Same old, you know the drill - upgrade to: v1.15.2-rc3 and fill out the Reply card - report any bugs, if any.

❗You need to upgrade Go to v1.17.9 or higher.

🚀 This post if you are participating in this testing round.
If you aren't able to join for reasons, please reach out to @jennijuju or me on Slack!!!

New features and Improvements

• --max-piece-size in set-ask are no longer required when changing the price

For more improvements & bug fixes see here.

Reply card:

• If you have successfully sealed some new sectors on this tag.
• If you have successfully made some Storage Deals on this tag.
• If you have successfully made some Snap Deals on this tag.
• If you passed your first WindowPoSt on this tag.
• If the log is back to normal.
• The resource manager has not caused any connection problems.

Reminders and Risk Mitigations

Questions are welcomed in this discussion, the team will be monitoring this discussion and check-in regularly.

Issues can be reported in https://github.com/filecoin-project/lotus/issues. Issues should be submitted with:

• Daemon logs / set RUST_LOG=debug if possible.
• go routine dumps lotus pprof goroutines and lotus-miner pprof goroutines
• the operation/step at which the problem occurred
• The stack trace or logs as relevant. As always, for large log files, you may upload it in this Google Disk folder and link it in the issue!

❗ If you think the v1.15.2-rc3 branch is not working well, especially the resource manager, keep all the logs for describing the issue and disable the resource manager with LOTUS_RCMGR=0, if the problem still persists revert back to the stable v1.15.1 release.

[tmyuu]

go version

$ go version
go version go1.18.1 linux/amd64

lotus version

$ lotus-miner version
Daemon:  1.15.2-rc2+mainnet+git.94d9dee0d+api1.5.0
Local: lotus-miner version 1.15.2-rc2+mainnet+git.94d9dee0d
$ lotus version
Daemon:  1.15.2-rc2+mainnet+git.94d9dee0d+api1.5.0
Local: lotus version 1.15.2-rc2+mainnet+git.94d9dee0d

boost version

$ boostd -v
boostd version 0.0.1+git.067a458.dirty

check box

• If you have successfully sealed some new sectors on this tag.
• If you have successfully made some Storage Deals on this tag.
• If you have successfully made some Snap Deals on this tag.
• If you passed your first WindowPoSt on this tag.
• If the log is back to normal.
• The resource manager has not caused any connection problems.

[jamerduhgamer]

• If you have successfully sealed some new sectors on this tag.
• If you have successfully made some Storage Deals on this tag.
• If you have successfully made some Snap Deals on this tag.
• If you passed your first WindowPoSt on this tag.
• If the log is back to normal.
• The resource manager has not caused any connection problems.

[SBudo]

go version

$ go version
go version go1.18.1 linux/amd64

lotus version

$ lotus-miner version
Daemon:  1.15.2-rc3+mainnet+git.cfe201450.dirty+api1.5.0
Local: lotus-miner version 1.15.2-rc3+mainnet+git.cfe201450.dirty


$ lotus version
Daemon:  1.15.2-rc3+mainnet+git.cfe201450.dirty+api1.5.0
Local: lotus version 1.15.2-rc3+mainnet+git.cfe201450.dirty

boost version

$ boost -v
boost version 0.0.1+git.10c5ec8

Checks

• If you have successfully sealed some new sectors on this tag.
• If you have successfully made some Storage Deals on this tag.
• If you have successfully made some Snap Deals on this tag.
• If you passed your first WindowPoSt on this tag.
• If the log is back to normal.
• The resource manager has not caused any connection problems.

[cryptowhizzard]

go version

$ go version
go version go1.18.1 linux/amd64 

lotus version

$ lotus-miner version
Local: lotus-miner version 1.15.2-rc3+mainnet+git.cfe201450

$ lotus version
Daemon:  1.15.2-rc2+mainnet+git.94d9dee0d+api1.5.0

boost version

$ boostd -v
boostd version 0.0.1+git.b11e3ea

check box

• If you have successfully sealed some new sectors on this tag.
• If you have successfully made some Storage Deals on this tag.
• If you have successfully made some Snap Deals on this tag.
• If you passed your first WindowPoSt on this tag.
• If the log is back to normal.
• The resource manager has not caused any connection problems.

Upgraded and running. Will populate the results whenever they are ready.

[benjaminh83]

go version

$ go version
go version go1.17.9 linux/amd64 

lotus version

$ lotus-miner version
Daemon:  1.15.2-rc3+mainnet+git.cfe201450+api1.5.0
Local: lotus-miner version 1.15.2-rc3+mainnet+git.cfe201450

boost version

$ boostd -v
boostd version 0.0.1+git.e9db5d6

check box

• If you have successfully sealed some new sectors on this tag.
• If you have successfully made some Storage Deals on this tag.
• If you have successfully made some Snap Deals on this tag.
• If you passed your first WindowPoSt on this tag.
• If the log is back to normal.
• The resource manager has not caused any connection problems.

Sealed sectors and WindowPoSt done on 1.15.2-rc2. Now upgraded and running again on rc3. Will populate the results whenever they are ready.

[benjaminh83]

@Reiers Still seeing the error here in rc3:

[flexi_logger][ERRCODE::Time] flexi_logger has to work with UTC rather than with local time, caused by IndeterminateOffset
    See https://docs.rs/flexi_logger/latest/flexi_logger/error_info/index.html#time

But its not showing %Y-%m-%dT%H:%M:%S%.3f anymore, but does 2022-05-04T08:49:21.097, so I'm happy, as it is usable again.

[Meest-NAM]

lotus version

Daemon:  1.15.2-rc3+mainnet+git.cfe201450+api1.5.0
Local: lotus-miner version 1.15.2-rc3+mainnet+git.cfe201450

boost version

boostd version 0.0.1+git.067a458

go version

go version go1.17.9 linux/amd64

• If you have successfully sealed some new sectors on this tag.
• If you have successfully made some Storage Deals on this tag.
• If you have successfully made some Snap Deals on this tag.
• If you passed your first WindowPoSt on this tag.
• If the log is back to normal.
• The resource manager has not caused any connection problems.

[Meest-NAM]

2022-05-06T16:58:02.602+0900	WARN	storageminer	storage/wdpost_run.go:235	Checked sectors	{"checked": 2165, "good": 2165}
2022-05-06T16:58:02.649+0900	INFO	storageminer	storage/wdpost_run.go:647	running window post	{"chain-random": "7fToR5EOlu4sueQmV8Z7H/vpmjeIrQoF6rW32PX7RKo=", "deadline": {"CurrentEpoch":1783914,"PeriodStart":1783920,"Index":0,"Open":1783920,"Close":1783980,"Challenge":1783900,"FaultCutoff":1783850,"WPoStPeriodDeadlines":48,"WPoStProvingPeriod":2880,"WPoStChallengeWindow":60,"WPoStChallengeLookback":20,"FaultDeclarationCutoff":70}, "height": "1783914", "skipped": 0}
2022-05-06T16:58:02.649+0900	INFO	advmgr	sector-storage/manager_post.go:81	GenerateWindowPoSt run at lotus-miner
2022-05-06T07:58:02.680 INFO filcrypto::proofs::api > generate_window_post: start
2022-05-06T07:58:02.686 INFO filcrypto::proofs::api > generate_window_post: finish
2022-05-06T16:58:02.689+0900	INFO	storageminer	storage/wdpost_run.go:662	computing window post	{"batch": 0, "elapsed": 0.039357451, "skip": 0, "err": "could not read from path=\"p_aux\"\n\nCaused by:\n    No such file or directory (os error 2)\n\nStack backtrace:\n   0: anyhow::context::<impl anyhow::Context<T,E> for core::result::Result<T,E>>::with_context\n   1: filecoin_proofs::types::private_replica_info::PrivateReplicaInfo<Tree>::new\n   2: filecoin_proofs_api::post::generate_window_post_inner\n   3: filecoin_proofs_api::post::generate_window_post\n   4: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once\n   5: ffi_toolkit::catch_panic_response\n   6: fil_generate_window_post\n   7: _cgo_7059698500c4_Cfunc_fil_generate_window_post\n   8: runtime.asmcgocall\n             at /usr/local/go/src/runtime/asm_amd64.s:765", "errVerbose": "could not read from path=\"p_aux\"\n\nCaused by:\n    No such file or directory (os error 2)\n\nStack backtrace:\n   0: anyhow::context::<impl anyhow::Context<T,E> for core::result::Result<T,E>>::with_context\n   1: filecoin_proofs::types::private_replica_info::PrivateReplicaInfo<Tree>::new\n   2: filecoin_proofs_api::post::generate_window_post_inner\n   3: filecoin_proofs_api::post::generate_window_post\n   4: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once\n   5: ffi_toolkit::catch_panic_response\n   6: fil_generate_window_post\n   7: _cgo_7059698500c4_Cfunc_fil_generate_window_post\n   8: runtime.asmcgocall\n             at /usr/local/go/src/runtime/asm_amd64.s:765\ngithub.com/filecoin-project/filecoin-ffi.GenerateWindowPoSt\n\t/root/lotus/extern/filecoin-ffi/proofs.go:687\ngithub.com/filecoin-project/lotus/extern/sector-storage/ffiwrapper.(*Sealer).GenerateWindowPoSt\n\t/root/lotus/extern/sector-storage/ffiwrapper/verifier_cgo.go:48\ngithub.com/filecoin-project/lotus/extern/sector-storage.(*Manager).GenerateWindowPoSt\n\t/root/lotus/extern/sector-storage/manager_post.go:82\ngithub.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).runPoStCycle\n\t/root/lotus/storage/wdpost_run.go:660\ngithub.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).runGeneratePoST\n\t/root/lotus/storage/wdpost_run.go:96\ngithub.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).startGeneratePoST.func1\n\t/root/lotus/storage/wdpost_run.go:80\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581"}
2022-05-06T16:58:02.689+0900	ERROR	storageminer	storage/wdpost_run.go:664	error generating window post: could not read from path="p_aux"

Caused by:
    No such file or directory (os error 2)

Stack backtrace:
   0: anyhow::context::<impl anyhow::Context<T,E> for core::result::Result<T,E>>::with_context
   1: filecoin_proofs::types::private_replica_info::PrivateReplicaInfo<Tree>::new
   2: filecoin_proofs_api::post::generate_window_post_inner
   3: filecoin_proofs_api::post::generate_window_post
   4: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
   5: ffi_toolkit::catch_panic_response
   6: fil_generate_window_post
   7: _cgo_7059698500c4_Cfunc_fil_generate_window_post
   8: runtime.asmcgocall
             at /usr/local/go/src/runtime/asm_amd64.s:765
2022-05-06T16:58:02.689+0900	ERROR	storageminer	storage/wdpost_run.go:98	runPoStCycle failed: running window post failed:
    github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).runPoStCycle
        /root/lotus/storage/wdpost_run.go:725
  - could not read from path="p_aux"
    
    Caused by:
        No such file or directory (os error 2)
    
    Stack backtrace:
       0: anyhow::context::<impl anyhow::Context<T,E> for core::result::Result<T,E>>::with_context
       1: filecoin_proofs::types::private_replica_info::PrivateReplicaInfo<Tree>::new
       2: filecoin_proofs_api::post::generate_window_post_inner
       3: filecoin_proofs_api::post::generate_window_post
       4: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
       5: ffi_toolkit::catch_panic_response
       6: fil_generate_window_post
       7: _cgo_7059698500c4_Cfunc_fil_generate_window_post
       8: runtime.asmcgocall
                 at /usr/local/go/src/runtime/asm_amd64.s:765
    github.com/filecoin-project/filecoin-ffi.GenerateWindowPoSt
    	/root/lotus/extern/filecoin-ffi/proofs.go:687
    github.com/filecoin-project/lotus/extern/sector-storage/ffiwrapper.(*Sealer).GenerateWindowPoSt
    	/root/lotus/extern/sector-storage/ffiwrapper/verifier_cgo.go:48
    github.com/filecoin-project/lotus/extern/sector-storage.(*Manager).GenerateWindowPoSt
    	/root/lotus/extern/sector-storage/manager_post.go:82
    github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).runPoStCycle
    	/root/lotus/storage/wdpost_run.go:660
    github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).runGeneratePoST
    	/root/lotus/storage/wdpost_run.go:96
    github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).startGeneratePoST.func1
    	/root/lotus/storage/wdpost_run.go:80
    runtime.goexit
    	/usr/local/go/src/runtime/asm_amd64.s:1581

I don't know why
I'm doing this all of a sudden

[Meest-NAM]

I checked all sectors of "Storage"

/storage/sealed/cache# ls | wc -l
2956

lotus-miner proving deadlines

deadline  partitions  sectors (faults)  proven partitions
0         1           2349 (2164)       0

[jennijuju]

Note - rc3 patch the bug where we got the wrong actor bundle that broke the FVM running.

[Angelo-gh3990]

Anybody getting or seen this error when compiling rc3 ? :

go build  -ldflags="-X=github.com/filecoin-project/lotus/build.CurrentCommit=+git.cfe201450" -o lotus ./cmd/lotus
go: downloading github.com/marten-seemann/qtls-go1-17 v0.1.1
# runtime/internal/sys
/usr/local/go/src/runtime/internal/sys/stubs.go:9:7: PtrSize redeclared in this block
        /usr/local/go/src/runtime/internal/sys/arch.go:24:38: previous declaration
/usr/local/go/src/runtime/internal/sys/stubs.go:10:24: undefined: Uintreg
/usr/local/go/src/runtime/internal/sys/stubs.go:16:7: StackGuardMultiplier redeclared in this block
        /usr/local/go/src/runtime/internal/sys/arch.go:27:74: previous declaration
make: *** [Makefile:87: lotus] Error 2

[Angelo-gh3990]

/lotus# go version
go version go1.17.9 linux/amd64

[Angelo-gh3990]

Found it :

rm -rf /usr/local/go
wget -c https://golang.org/dl/go1.17.9.linux-amd64.tar.gz -O - | sudo tar -xz -C /usr/local

First I had to remove the previous install of Go (Is a known go issue apparently)

[benjaminh83]

Yes... you need to remove first. Same thing when we moved to v1.16 a while ago

[TippyFlitsUK]

• If you have successfully sealed some new sectors on this tag.
• If you have successfully made some Storage Deals on this tag.
• If you have successfully made some Snap Deals on this tag.
• If you passed your first WindowPoSt on this tag.
• If the log is back to normal.
• The resource manager has not caused any connection problems.