MinerX FF Edition: Curio Storage - Testing Assignment Week 1

[Reiers]

Hello MinerX Fellows !

Week 1 is about getting familiar with Curio Storage. Everyone should read our docs and setup a local-dev-net.
Follow the guide below. Looking forward to the feedback and answers. Thank you!

Requirement

• OS: Linux or macOS
• Docker
• Read our Curio Documentation

🚀 This post if you are participating in this testing round.
If you aren't able to join for some reason, please reach out to me @Reiers here or on Slack!

Task assignment

1. Install Docker if you haven't already - the desktop version should work as well.

git clone https://github.com/filecoin-project/lotus.git curio-docker
cd curio-docker
git checkout master

❗ unset FFI_BUILD_FROM_SOURCE (if you have it in your environments, check with env)

make docker/curio-devnet
make curio-devnet/up

This can take a couple of minutes, so be patient.

Curio UI: http://localhost:4701/ - (This will also be displayed in the logs)

To stop and remove all the containers to start fresh:

make curio-devnet/down

Make sure to still be in curio-docker and start it up again, you don't have to build it again:

make curio-devnet/up

Test Operations

• curio seal start --now --cc --actor <Miner actor ID>
• Test thesptool cmd - we want you to have the option to use familiar commands with Curio, so test all the commands, let's us know if anything is missing or if something is broken:

NAME:
   sptool - Manage Filecoin Miner Actor

USAGE:
   sptool [global options] command [command options] [arguments...]

VERSION:
   1.27.1-dev+debug+git.64686d9bc

COMMANDS:
   actor    Manage Filecoin Miner Actor Metadata
   info     Print miner actor info
   sectors  interact with sector store
   proving  View proving information
   help, h  Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --log-level value  (default: "info")
   --actor value      miner actor to manage [$SP_ADDRESS]
   --help, -h         show help
   --version, -v      print the version

Reply with when completed:

• - 🦭If you have successfully sealed some new sectors.
• - 💻 If you passed your first WindowPoSt.

UI Feedback and Alerting

• What would like on the first overview page?
• What features or changes would you suggest to improve the UI?
• Is there anything you felt was missing that should be included?
• What would you like to have alerts on, and where?

❗ Create an issue if you think there is a bug, don't forget to attach the logs! 🐛

If you have any questions or need help, join:

• For dev questions : # fil-curio-dev
• For general help : # fil-curio-help

[stuberman]

• How soon can we start?
• When is the deadline to finish this test?

[stuberman]

• - 🦭If you have successfully sealed some new sectors.
• - 💻 If you passed your first WindowPoSt.

[stuberman]

Posted errors in curio logs to #fil-curio-help Slack channel

[stuberman]

Overview Page: What would you like to see on the first overview page?

• I would like to see disk space indicators (always concerned about available capacity)
• I like to monitor the status of the five most recent sectors being sealed or completed
• I like to see the amount of available ram, CPU and GPU utilization
• I like to see all of the wallet balances
• I like to see messages waiting in mpool

Improvements: What features or changes would you suggest to improve the UI?

• Still getting used to the new concept, too early to say

Missing Elements: Is there anything you felt was missing that should be included?

• Blocks won

Alerts: What would you like to have alerts on, and where should they appear?

• Process node (chain, PoST worker, sealer, etc) down displayed on main page
• Low on space/capacity, mpool messages stuck for more than 10 minutes displayed on main page

[stuberman]

stuart@minerx:~$ docker -v
Docker version 26.1.3, build b72abbb

stuart@minerx:~$ docker exec -it curio /bin/bash
root@35b1887e18cd:/app# curio -v
curio version 1.27.1-dev+debug+git.af31126ea

[stuberman]

docker info

Client: Docker Engine - Community
Version: 26.1.3
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.14.0
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.27.0
Path: /usr/libexec/docker/cli-plugins/docker-compose

Server:
Containers: 4
Running: 4
Paused: 0
Stopped: 0
Images: 10
Server Version: 26.1.3
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: e377cd56a71523140ca6ae87e30244719194a521
runc version: v1.1.12-0-g51d5e94
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.5.0-35-generic
Operating System: Ubuntu 22.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 24
Total Memory: 125.7GiB
Name: minerx
ID: 22b44b91-2fe1-4263-83e5-0087068bbdbd
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

[TippyFlitsUK]

• - 🦭If you have successfully sealed some new sectors.
• - 💻 If you passed your first WindowPoSt.

[amughal]

Confirmed both.

[SBudo]

Feedback/note to add to the installation process:
I'm not using Docker Desktop, but instead already had Docker Engine installed. Docker Engine doesn't have Buildx installed by default like Docker Desktop, so the build failed at the very end complaining that Buildx wasn't installed.
Would be good to note this in the future installation documentation.

[stuberman]

I did not want to install Docker Desktop since I prefer running Ubuntu Server (CLI only). So I installed Docker CE (Community Edition) and that worked well.

https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-22-04

[SBudo]

I had an additional "problem" with Docker Engine:
Docker Compose doesn't exist, so instead had to change the Makefile to use docker-compose

[SBudo]

• 🦭If you have successfully sealed some new sectors.
• 💻 If you passed your first WindowPoSt.

[herrehesse]

Soon..

[herrehesse]

I encountered small issues while installing:

• The download of parameters got stuck at 150kbit. I had to import the parameters from a local machine, which is not optimal for new users.
• The lotus host was not added to the /etc/hosts file. As a result, I was unable to connect. Although Curio was present, the binary treats lotus as localhost.

• 🦭If you have successfully sealed some new sectors.
• 💻 If you passed your first WindowPoSt.

[LexLuthr]

Param download is probably a network thing.
Could you clarify the host thing? I am not sure I understand what you mean by access here? Just to clarify, devnet is meant to be self contained. All commands should be run from inside the container to avoid any unnecessary variable exports.

[herrehesse]

@LexLuthr if the assignment was to create a new sector, how to do this from within the docker then?

[LexLuthr]

If you have docker UI then just access from UI otherwise get access to docker shell with docker exec -it <name> /bin/bash then run whatever command you wish.

[herrehesse]

Should we include instructions on how to work from within the Docker container itself (using the command docker exec -it lotus /bin/bash) compared to our usual methods?

@LexLuthr you assisted me this time, but perhaps useful for new miners.

[LexLuthr]

@Reiers Can we add this to the top?^ How to access a container from your machine instructions.

[nathandragun]

• - 🦭If you have successfully sealed some new sectors.
• - 💻 If you passed your first WindowPoSt.

[SBudo]

UI Feedback and Alerting

This is the overview I've built (below) that gives me all the information about the environment.
Would be great to get some or all the information available...

From an alerting perspective:

• node out of sync
• messages stuck in the mpool
• WdPoSt faults
• WinningPoSt faults
• Sealing errors
• Wallets threshold (especially for control and publish wallet)

[LexLuthr]

Thank you @SBudo
I have already planned for most of the alerts you pointed out. This will help prioritise.

[johansealstorage]

• 🦭If you have successfully sealed some new sectors.
• 💻 If you passed your first WindowPoSt.

[johansealstorage]

@LexLuthr , I'll re-produce that wdpost issue and let you know about the database records on it. I restarted the docker compose and it thus cleared the data from the previous run.

[johansealstorage]

@LexLuthr , at least it was easy enought to re-produce.

This is what is in the Database on wdpost_partition_tasks:

task_id              sp_id                proving_period_start deadline_index       partition_index      
-------------------- -------------------- -------------------- -------------------- -------------------- 
1371                 1002                 -171                 13                   0                    
592                  1002                 -171                 12                   0                    
2010                 1002                 -171                 14                   0                    
              

And just for completeness, this is the corresponding log entry:

2024-05-17T13:33:30.401Z        ERROR   harmonytask     harmonytask/task_type_handler.go:197    Do() returned error     {"type": "WdPost", "id": "592", "error": "can't scan into dest[1]: -171 is less than minimum value for uint64"}

[LexLuthr]

Does this this happen only at start up or always(after 4 hours)?

[johansealstorage]

I have left curio running after I've re-produced it, and it has now gone through all the deadlines and started at deadline 0 again. The interesting thing is that it is always "-171", no matter how long I wait before re-producing it. The -171 seems to be the "constant" that it fails with.

Now after it wrapped back to deadline 0, WDPost is going trough for those deadlines. There are a view more to go before it hits the ones with -171 again and I'll keep an eye on those.
For now, wdpost_partition_tasks contains this:

task_id              sp_id                proving_period_start deadline_index       partition_index      
-------------------- -------------------- -------------------- -------------------- -------------------- 
1371                 1002                 -171                 13                   0                    
3820                 1002                 -171                 28                   0                    
592                  1002                 -171                 12                   0                    
3224                 1002                 -171                 24                   0                    
6425                 1002                 2709                 3                    0                    
1884                 1002                 -171                 18                   0                    
4270                 1002                 -171                 34                   0                    
2327                 1002                 -171                 16                   0                    
6215                 1002                 2709                 0                    0                    
2010                 1002                 -171                 14                   0                    
5805                 1002                 -171                 43                   0                    
2682                 1002                 -171                 21                   0                    
3292                 1002                 -171                 26                   0                    
1975                 1002                 -171                 17                   0                    
2593                 1002                 -171                 20                   0                    
2617                 1002                 -171                 19                   0                    
3789                 1002                 -171                 29                   0                    
5506                 1002                 -171                 41                   0                    
4419                 1002                 -171                 33                   0                    
2943                 1002                 -171                 22                   0                    
4461                 1002                 -171                 35                   0                    
3875                 1002                 -171                 30                   0                    
4213                 1002                 -171                 32                   0                    
3631                 1002                 -171                 27                   0                    
5324                 1002                 -171                 40                   0                    
6178                 1002                 2709                 1                    0                    
4844                 1002                 -171                 38                   0                    
4645                 1002                 -171                 36                   0                    
4056                 1002                 -171                 31                   0                    
2974                 1002                 -171                 23                   0                    
3251                 1002                 -171                 25                   0                    
5209                 1002                 -171                 39                   0                    
6348                 1002                 2709                 2                    0                    
1595                 1002                 -171                 15                   0                    
5393                 1002                 -171                 42                   0                    
4922                 1002                 -171                 37                   0                    

[johansealstorage]

@LexLuthr , just an update now that it's been running again for more than 4 hours. The WDPost managed to clear some of the deadlines, but not all of them.

But looking at the database, there are duplicates deadline_index. One value with the proving_period_start set to -171 and one value with proving_period_start set to 2709

This is the output of select * from curio.wdpost_partition_tasks order by deadline_index;

task_id              sp_id                proving_period_start deadline_index       partition_index      
-------------------- -------------------- -------------------- -------------------- -------------------- 
6215                 1002                 2709                 0                    0                    
6178                 1002                 2709                 1                    0                    
6348                 1002                 2709                 2                    0                    
6425                 1002                 2709                 3                    0                    
6449                 1002                 2709                 4                    0                    
6808                 1002                 2709                 5                    0                    
7101                 1002                 2709                 6                    0                    
6498                 1002                 2709                 7                    0                    
7224                 1002                 2709                 8                    0                    
7249                 1002                 2709                 9                    0                    
7425                 1002                 2709                 10                   0                    
7455                 1002                 2709                 11                   0                    
592                  1002                 -171                 12                   0                    
7385                 1002                 2709                 12                   0                    
1371                 1002                 -171                 13                   0                    
7584                 1002                 2709                 13                   0                    
2010                 1002                 -171                 14                   0                    
8118                 1002                 2709                 14                   0                    
7878                 1002                 2709                 15                   0                    
1595                 1002                 -171                 15                   0                    
8063                 1002                 2709                 16                   0                    
2327                 1002                 -171                 16                   0                    
1975                 1002                 -171                 17                   0                    
8528                 1002                 2709                 17                   0                    
1884                 1002                 -171                 18                   0                    
8349                 1002                 2709                 18                   0                    
8276                 1002                 2709                 19                   0                    
2617                 1002                 -171                 19                   0                    
2593                 1002                 -171                 20                   0                    
8817                 1002                 2709                 20                   0                    
2682                 1002                 -171                 21                   0                    
8861                 1002                 2709                 21                   0                    
9234                 1002                 2709                 22                   0                    
2943                 1002                 -171                 22                   0                    
2974                 1002                 -171                 23                   0                    
9503                 1002                 2709                 23                   0                    
3224                 1002                 -171                 24                   0                    
3251                 1002                 -171                 25                   0                    
3292                 1002                 -171                 26                   0                    
3631                 1002                 -171                 27                   0                    
3820                 1002                 -171                 28                   0                    
3789                 1002                 -171                 29                   0                    
3875                 1002                 -171                 30                   0                    
4056                 1002                 -171                 31                   0                    
4213                 1002                 -171                 32                   0                    
4419                 1002                 -171                 33                   0                    
4270                 1002                 -171                 34                   0                    
4461                 1002                 -171                 35                   0                    
4645                 1002                 -171                 36                   0                    
4922                 1002                 -171                 37                   0                    
4844                 1002                 -171                 38                   0                    
5209                 1002                 -171                 39                   0                    
5324                 1002                 -171                 40                   0                    
5506                 1002                 -171                 41                   0                    
5393                 1002                 -171                 42                   0                    
5805                 1002                 -171                 43                   0                    

And this is the output of sptool --actor t01002 proving deadlines. It seems to also pick up the first record with -171 and reporting it as a faulty sector?

root@e5634e11197e:/app# sptool --actor t01002 proving deadlines
Miner: t01002
deadline  open      partitions  sectors (faults)  proven partitions
0         15:54:39  1           2 (0)             0
1         15:58:39  1           2 (0)             0
2         16:02:39  1           2 (0)             0
3         16:06:39  1           2 (0)             0
4         16:10:39  1           2 (0)             0
5         16:14:39  1           2 (0)             0
6         16:18:39  1           2 (0)             0
7         16:22:39  1           2 (0)             0
8         16:26:39  1           2 (0)             0
9         16:30:39  1           2 (0)             0
10        16:34:39  1           2 (0)             0
11        16:38:39  1           2 (0)             0
12        16:42:39  1           2 (0)             0
13        16:46:39  1           2 (0)             0
14        16:50:39  1           2 (0)             0
15        16:54:39  1           2 (0)             0
16        16:58:39  1           2 (0)             0
17        17:02:39  1           2 (2)             0
18        17:06:39  1           2 (2)             0
19        17:10:39  1           2 (2)             0
20        17:14:39  1           2 (2)             0
21        17:18:39  1           2 (2)             0
22        17:22:39  1           2 (2)             0
23        17:26:39  1           2 (2)             0  (current)
24        17:30:39  1           2 (2)             0
25        17:34:39  1           2 (2)             0
26        17:38:39  1           2 (2)             0
27        17:42:39  1           2 (2)             0
28        17:46:39  1           2 (2)             0
29        17:50:39  1           2 (2)             0
30        17:54:39  1           2 (2)             0
31        17:58:39  1           2 (2)             0
32        18:02:39  1           2 (2)             0
33        18:06:39  1           2 (2)             0
34        18:10:39  1           2 (2)             0
35        18:14:39  1           2 (2)             0
36        18:18:39  1           2 (2)             0
37        18:22:39  1           2 (2)             0
38        18:26:39  1           2 (2)             0
39        18:30:39  1           2 (2)             0
40        18:34:39  1           2 (2)             0
41        18:38:39  1           2 (2)             0
42        18:42:39  1           2 (2)             0
43        18:46:39  1           2 (2)             0
44        18:50:39  0           0 (0)             0
45        18:54:39  0           0 (0)             0
46        18:58:39  0           0 (0)             0
47        19:02:39  0           0 (0)             0

The UI also shows these deadlines as failing:

The logs seems to be clear and not logging any more -171's for WDPost though.

[marshyonline]

• 🦭If you have successfully sealed some new sectors.
• 💻 If you passed your first WindowPoSt.

Slow going with some issues with the proof's downloading fairly slowly - but left overnight and it came good.

[marshyonline]

One thing i did notice is that the event log for all 100 sectors I created was empty

root@10b7a2e5ab6c:/app# sptool --actor t01002 sectors status -l 2

Sector On Chain Info
SealProof:              6
Activation:             19311 (9 minutes 48 seconds ago)
Expiration:             3699017 (in 24 weeks 2 days)
DealWeight:             0
VerifiedDealWeight:             0
InitialPledge:          0.00024414062116864 FIL
SectorID:               {Miner: 1002, Number: 2}
--------
Event Log:
root@10b7a2e5ab6c:/app# sptool --actor t01002 sectors status -l 3

Sector On Chain Info
SealProof:              6
Activation:             19350 (7 minutes 12 seconds ago)
Expiration:             3699017 (in 24 weeks 2 days)
DealWeight:             0
VerifiedDealWeight:             0
InitialPledge:          0.00024414062116864 FIL
SectorID:               {Miner: 1002, Number: 3}
--------
Event Log:

[marshyonline]

In terms of the UI

IMO there needs to filtering options added to almost every page. Filter my Miner, Status etc etc
On big miners, these pages are going to get long quickly.

The config edit page is really nice, I like the UI method of editing layers - but I think they should all default to collapsed(or at least have a collapse/expand all button so make getting around quick) and mabey even a search.

[LexLuthr]

sptool will not have events as this tool does not talk to Curio. It fetches all the data from chain. We will remove the "Event Log" words from output.

[LexLuthr]

#12036 has the fix for event logs in sptool.

[johansealstorage]

Starting a new discussion for some new errors on Winning Post. @LexLuthr , should I log this as an issue or just discuss in the channel?

After running for 4+ hours, WinPost starts to fail:

The Fail message is:

failed to submit block: <!!> SLASH FILTER DETECTED FAULT due to witness bafy2bzacecqfuqj7h3gtrf2di3vp4jugyd7fsdvnwk6ybzy6h4jeorarkczh4

The first log entry when WinPost starts to fail is:

2024-05-17T16:54:58.226Z        ERROR   harmonytask     harmonytask/task_type_handler.go:197    Do() returned error     {"type": "WinPost", "id": "7886", "error": "failed to submit block: sync to submitted block failed: collectChain failed: collectChain syncMessages: message processing failed: validating block bafy2bzacedtuawltu7qpl3gmrma562po6sajbdelxcj76lvktyqwgmbcy3ez4: 1 error occurred:\n\t* invalid election post:\n    github.com/filecoin-project/lotus/chain/consensus/filcns.(*FilecoinEC).ValidateBlock.func6\n        /opt/curio/chain/consensus/filcns/filecoin.go:282\n  - [INSECURE-POST-VALIDATION] winning post was invalid:\n    github.com/filecoin-project/lotus/chain/consensus/filcns.(*FilecoinEC).VerifyWinningPoStProof\n        /opt/curio/chain/consensus/filcns/filecoin.go:333\n\n", "errorVerbose": "failed to submit block:\n    github.com/filecoin-project/lotus/curiosrc/winning.(*WinPostTask).Do\n        /opt/curio/curiosrc/winning/winning_task.go:417\n  - sync to submitted block failed: collectChain failed: collectChain syncMessages: message processing failed: validating block bafy2bzacedtuawltu7qpl3gmrma562po6sajbdelxcj76lvktyqwgmbcy3ez4: 1 error occurred:\n\t* invalid election post:\n    github.com/filecoin-project/lotus/chain/consensus/filcns.(*FilecoinEC).ValidateBlock.func6\n        /opt/curio/chain/consensus/filcns/filecoin.go:282\n  - [INSECURE-POST-VALIDATION] winning post was invalid:\n    github.com/filecoin-project/lotus/chain/consensus/filcns.(*FilecoinEC).VerifyWinningPoStProof\n        /opt/curio/chain/consensus/filcns/filecoin.go:333\n\n"}                                                                                              

And then after that, all the WinPost log entries with errors looks like this:

2024-05-17T16:55:01.642Z        ERROR   harmonytask     harmonytask/task_type_handler.go:197    Do() returned error     {"type": "WinPost", "id": "7886", "error": "failed to submit block: <!!> SLASH FILTER DETECTED FAULT due to witness bafy2bzacedtuawltu7qpl3gmrma562po6sajbdelxcj76lvktyqwgmbcy3ez4", "errorVerbose": "failed to submit block:\n    github.com/filecoin-project/lotus/curiosrc/winning.(*WinPostTask).Do\n        /opt/curio/curiosrc/winning/winning_task.go:417\n  - <!!> SLASH FILTER DETECTED FAULT due to witness bafy2bzacedtuawltu7qpl3gmrma562po6sajbdelxcj76lvktyqwgmbcy3ez4"}                                                                                                                                                                                                                                                          

I managed to re-produce this same error with just 1 CC sector sealed and having curio up for longer than 4 hours and I've also seen this happen again after I re-produces the WDPost error and leaving Curio running for longer than 4 hours on Devnet.

[tmyuu]

• 🦭If you have successfully sealed some new sectors.
• 💻 If you passed your first WindowPoSt.
  

[amughal]

The first sector on the main page shows OnChain No, however clicking Details it shows Green.

[LexLuthr]

Fixed in master

[amughal]

Just thinking on the order of widgets on the main page based on their usefulness. My thought is that Actor Summary should come after the Chain Connectivity.
Widgets could be in a group, which can be hidden or open up.
I have larger 27" Screen. Details from Stephane would be nice to see in one widget group.

[johansealstorage]

On curio devnet, why does the one address on PoRep Pipeline start with an "f" and the same curio address on Actor Summary start with a "t"

[LexLuthr]

Fixed in master

[johansealstorage]

Yip, somehow my lotus binary was mainnet build. I did a rebuild and it seems OK now.

[johansealstorage]

The online documentation shows some curio and cli commands that are not available in the actual latest curio. Is this expected? https://lotus.filecoin.io/storage-providers/curio/cli/
vs.
curio --help

[LexLuthr]

@johansealstorage We are merging changes on daily basis so docs might be different than what is being shipped at the moment. We will realign the docs before GA.

[NDLABS-Leo]

Reply with when completed:

• 🦭If you have successfully sealed some new sectors.
• 💻 If you passed your first WindowPoSt.

Confirmed

What would like on the first overview page?
Actor Summary:
1.Add Market Balance: Locked: Available:
2.Add Raw Byte Power
3.Add Control List

Recently Finished Tasks:
Added page-turning functionality and message can support replication so that operations can be maintained to track log problems

finally:
Add a work machine list sealing to see work machine tasks

What features or changes would you suggest to improve the UI?
none

Is there anything you felt was missing that should be included?
Block rewards and orphan blocks

What would you like to have alerts on, and where?
1.An orphan block occurs or the parent block is lost
2.Daemon node synchronization abnormality
3.windpost processing reminder timeout
4.mpool messages remain for more than 10 minutes and disk alarms occur