[Support Ticket]: Boost stops accepting deals, performing retrievals, deals are stuck in 'Announcing' state, 'Piece Doctor' page does not load.

[Shekelme]

Boost component

• boost daemon - storage providers
• boost client
• boost UI
• boost data-transfer
• boost index-provider
• booster-http
• booster-bitswap
• LID Database - Yugabyte/LevelDB
• boostd-data
• Other

Boost Version

boost version 2.1.1+mainnet+git.e9d18ac
boostd version 2.1.1+mainnet+git.e9d18ac
boostd-data version 1.5.1+mainnet+git.e9d18ac
booster-bitswap version 2.1.1+mainnet+git.e9d18ac

Describe the problem

After some time of more or less successful work (it can be several hours or a day or two), something goes wrong - boostd stops accepting deals, performing retrievals, deals are stuck in 'Announcing' state, 'Piece Doctor' page does not load.

Yugabyte setup details

YugabyteDB v2.18.2.1 (replication factor 3, 3 nodes)
YugabyteDB nodes are running on Lenovo P620 workstations. HW specs of 3 hosts:

Host	OS	YB NVME	FS	CPU	RAM	Role
len4	Ubuntu 22.04.3 LTS	WD Black SN750 2TB WDS200T3X0C-00SJG0	XFS	AMD Ryzen Threadripper PRO 3955WX 16-Cores	512GB DDR4-3200 ECC	PR2/C2 worker
len91	Ubuntu 22.04.3 LTS	WD Black SN750 2TB WDS200T3X0C-00SJG0	XFS	AMD Ryzen Threadripper PRO 3945WX 12-Cores	256GB DDR4-3200 ECC	lotus-miner
len101	Ubuntu 22.04.3 LTS	WD Black SN750 2TB WDS200T3X0C-00SJG0	XFS	AMD Ryzen Threadripper PRO 3995WX 64-Cores	1024GB DDR4-3200 ECC	RU/PC2 worker

In addition to `lotus` software (on all servers) and `boost` (lun91), `subspace` nodes are also installed on two workers, but separate drives are allocated for Yugabyte, and processor resources are not even half occupied. Sealing is carried out intermittently, from time to time.

How does your boostd-data talks to yugabyte

I use boostd-data service:

Description=boostd-data
After=network-online.target
After=yb-tserver.service
Requires=network-online.target
Wants=yb-tserver.service

[Service]
ExecStart=/usr/local/bin/boostd-data run yugabyte --hosts 192.168.11.91,192.168.11.14,192.168.11.101 --connect-string="postgresql://yugabyte:yugabyte@192.168.11.91:5433?sslmode=disable" --addr 0.0.0.0:8044
Restart=always
RestartSec=5
TimeoutStartSec=20
TimeoutStopSec=20
User=adm
Group=adm
Type=simple


[Install]
WantedBy=multi-user.target

I've tried using another option as well, ?sslmode=disable&load_balance=true, wuth the same result.

Boostd logs and boostd-data logs

boostd-data.log
boostd-data's log attached. The boost upgrade to v2.1.1 was carried out on 11.12.2023
Boostd log - please download from here: https://disk.yandex.ru/d/ugDMZaikacDfVQ

Recent error, warning and INFO logs from Yugabyte

Attached.
yb-master-tserver-logs.zip

Logging Information

Logs has been attached as files

Repo Steps

1. Install lotus-miner v1.25.1+boost v2.2.1+YugabyteDB v2.18.2.1 (replication factor 3, 3 nodes)
2. Start all processes
3. Wait for the described symptoms to appear.
   ...

[Shekelme]

boost goroutines atttached
boost-15-12-2023.zip

[Shekelme]

There are at least three miners affected by this problem: 2 big ones (YugabyteDB, though on second miner there are no errors (timeouts) in cutrrent boostd-data log) - I can open the "Local Index Directory" page, but not "Piece Doctor". On third small miner (LevelDB) I can't even open the "Local Index Directory", no errors or timeouts in cutrrent boostd-data log.

[ischasny]

Do deals get "unstuck" from Announcing after boostd restart?

[Shekelme]

I don't remember trying to do this, however, when restarting boost-data service, the Local Index Directory page does not start displaying until you restart the entire set of boost software.
I checked that right now.

[ischasny]

I don't remember trying to do this, however, when restarting boost-data service, the Local Index Directory page does not start displaying until you restart the entire set of boost software.

I checked that right now.

Do deals get unstuck from the announcing state then?

[Shekelme]

There is no way to verify this yet. Let's wait for some deals...

[ischasny]

What I mean is, as per your description -

deals are stuck in 'Announcing' state, 'Piece Doctor' page does not load

I wonder if those get unstuck after restart?

[Shekelme]

Yes, after stop and start Boost processess (we do not touch the YugabyteDB services) the deals get unstuck and become 'Proving' after a while.

[Shekelme]

After yesterday's restart, the problem was repeated. No new deals have been received yet. I will be glad for advice on which logs to look at.
For which log subsystems should I set 'set-level' to some more detailed level?

[ischasny]

CC @LexLuthr

[ischasny]

If no new deals came in - what did get stuck in announcing then?

[ischasny]

@Shekelme I suggest to look into debug logs for piecedirectory. The log lines that start with add index: .... and podsi: .... Both will have breakdown of time / concurrency settings that should help to understand the problem further.