[bootnode] - leveldb corruption after restart from systemd service

[mur-me]

Describe the bug
Bootnode coudn't start after restart from the systemd:
Default systemd service behavior is:

graph LR
    A[SIGTERM] --> B{Did the process exit?}
    B -- Yes --> C[Service stops cleanly]
    B -- No (after 90s) --> D[SIGKILL]
    D --> E[Service forcibly stopped]

Loading

To Reproduce
Steps to reproduce the behavior:

1. Let bootnode to work at least 6 hours to collect the dht routes, this isn't a problem for a fresh levelDB
2. Restart it via sudo service bootnode restart
3. Check the binary output via sudo journalctl --unit=bootnode.service -n 50 --no-pager
4. You will see either:

harmony[25794]: failed to create data store: failed to open leveldb db for peerstore: file missing
harmony[25794]: cannot initialize network

Or

harmony[1291]: file does not exist for table 286
harmony[1291]: cannot open Badger data store at .dht-0.0.0.0-9867
harmony[1291]: github.com/harmony-one/harmony/p2p/discovery.getDataStoreOption
harmony[1291]:         /home/uladzislau/.gvm/pkgsets/go1.22.5/global/src/github.com/harmony-one/harmony/p2p/discovery/option.go:63
harmony[1291]: github.com/harmony-one/harmony/p2p/discovery.DHTConfig.GetLibp2pRawOptions
harmony[1291]:         /home/uladzislau/.gvm/pkgsets/go1.22.5/global/src/github.com/harmony-one/harmony/p2p/discovery/option.go:32
harmony[1291]: github.com/harmony-one/harmony/p2p.NewHost
harmony[1291]:         /home/uladzislau/.gvm/pkgsets/go1.22.5/global/src/github.com/harmony-one/harmony/p2p/host.go:322
harmony[1291]: main.main
harmony[1291]:         /home/uladzislau/.gvm/pkgsets/go1.22.5/global/src/github.com/harmony-one/harmony/cmd/bootnode/main.go:161
harmony[1291]: runtime.main
harmony[1291]:         /home/uladzislau/.gvm/pkgsets/go1.22.5/global/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/runtime/proc.go:283
harmony[1291]: runtime.goexit
harmony[1291]:         /home/uladzislau/.gvm/pkgsets/go1.22.5/global/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/runtime/asm_amd64.s:1700
harmony[1291]: failed to get data store option
harmony[1291]: initialize libp2p raw options failed
harmony[1291]: github.com/harmony-one/harmony/p2p.NewHost
harmony[1291]:         /home/uladzislau/.gvm/pkgsets/go1.22.5/global/src/github.com/harmony-one/harmony/p2p/host.go:325
harmony[1291]: main.main
harmony[1291]:         /home/uladzislau/.gvm/pkgsets/go1.22.5/global/src/github.com/harmony-one/harmony/cmd/bootnode/main.go:161
harmony[1291]: runtime.main
harmony[1291]:         /home/uladzislau/.gvm/pkgsets/go1.22.5/global/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/runtime/proc.go:283
harmony[1291]: runtime.goexit
harmony[1291]:         /home/uladzislau/.gvm/pkgsets/go1.22.5/global/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/runtime/asm_amd64.s:1700
harmony[1291]: cannot initialize network

5. Setting systemd timeout to bigger value via TimeoutStopSec=300, e.g. 5 minutes isn't helping, result is the same

Expected behavior
I'm expecting that bootnode can stop via SIGTERM in 90 seconds limit

1. Close all the connections
2. Safely stop all the actions with leveldb

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• OS: linux, ubuntu
• Go environment: 1.24.2

Additional context
We still have a workaround with a fresh levelDB, but this isn't a permanent solution, this is a workaround.

[GheisMohammadi]

PR #4885

[mur-me]

Fixed via #4885 - it was a deadlock in the dht

[mur-me]

Issue is still reproducible - mitigation, start bootnode with clear DB

[mur-me]

#4925 split dht and peer store in different locations