Page load time scales with peer/channel count on production nodes

[ronballesteros]

Hi everyone...

We've been running peerswap-web on a production LND node with a decent number of peers and noticed the index page takes 30+ seconds to load after login, sometimes triggering a 504 from our load balancer. On a fresh node with no peers it loads near instantly.

Looking at the code, indexHandler makes several sequential gRPC calls with context.Background() (no timeout)...including ln.FeeReport which returns fee data for every channel on the node. Our hypothesis is the cumulative load time scales with peer/channel count, but we haven't added timing instrumentation to confirm which call is the actual bottleneck.

Would love to know if this is a known issue or if there's a recommended approach before we dig deeper.

Environment:

• LND, mainnet, Kubernetes deployment (separate containers deployed on diff pods)
• Elementsd
• Peerswap
• Peerswap UI

Observed behavior:
On a production node with a significant number of peerswap peers and Lightning channels, the index page (and post-login redirect) takes 30+ seconds to load. On a fresh node with no peers the page loads near-instantly. Based on this, it took about 46 seconds...

However, Bbypassing any ingress and loadbalancer and directly connecting to the peerswap-ui pod still takes about 36 seconds to login.

On my dev/sandbox node with no peers, login happens instantly.

Each call takes longer as peer/channel count grows. With no peers these return instantly. With many peers the cumulative time easily exceeds 30 seconds, which also causes load balancer timeouts (504s) in hosted environments.

Thanks!

Ron

[Impa10r]

Hi @ronballesteros please test the branch or docker image v5.0.5 and let me know if it solved the problem

[Impa10r]

How long do lncli listpeers and listchannels take? If they are fast, the bottleneck is in peerswapd. Could you open the issue at https://github.com/ElementsProject/peerswap please?

There is no pagination on either call. Not sure what I can do from the ui.

For a long swaps history the solution could be to copy the entries and then delete the swaps database file.

For listpeers, we do want to see the latest channel balances, so it must be fresh. I could make it refresh in the background, say every 10 minutes, to speed up index page, but it may cost some cpu...

[ronballesteros]

Listing peers and and channels on LND is quite fast...

Opened and issue with the ElementsProject: ElementsProject/peerswap#439.

Thanks for the assist with this...

[ronballesteros]

@Impa10r The PR here ElementsProject/peerswap#440 should fix the listpeers issue:

time grpcurl -plaintext \
    -import-path /Users/ronellb/Documents/strike/repos/peerswap/peerswaprpc \
    -proto peerswaprpc.proto \
    -max-msg-sz 209715200 \
    localhost:8080 peerswap.PeerSwap/ListPeers | wc -c
   33457
grpcurl -plaintext -import-path  -proto peerswaprpc.proto -max-msg-sz     0.01s user 0.02s system 1% cpu 2.322 total
wc -c  0.00s user 0.00s system 0% cpu 2.322 total

For listswaps, peerswap does support pagination:

No page_size:

time grpcurl -plaintext \
    -import-path /tmp/peerswaprpc \
    -proto peerswaprpc.proto \
    -max-msg-sz 209715200 \
    localhost:8080 peerswap.PeerSwap/ListSwaps | wc -c

 34362815
grpcurl -plaintext -import-path  -proto peerswaprpc.proto -max-msg-sz     0.88s user 0.14s system 7% cpu 14.336 total
wc -c  0.00s user 0.00s system 0% cpu 14.335 total

With page_size:

time grpcurl -plaintext \
    -import-path /tmp/peerswaprpc \
    -proto peerswaprpc.proto \
    -max-msg-sz 209715200 \
    -d '{"page_size": 50}' \
    localhost:8080 peerswap.PeerSwap/ListSwaps | wc -c

   33017
grpcurl -plaintext -import-path  -proto peerswaprpc.proto -max-msg-sz  -d     0.02s user 0.02s system 2% cpu 1.930 total
wc -c  0.00s user 0.00s system 0% cpu 1.930 total

[ronballesteros]

Been getting sub `~5-6 seconds on the list swaps which isn't too bad. Way better than it was prior to....

time grpcurl -plaintext \
    -import-path /tmp/peerswaprpc \
    -proto peerswaprpc.proto \
    -max-msg-sz 209715200 \
    localhost:8080 peerswap.PeerSwap/ListSwaps | wc -c
    
 34363439
grpcurl -plaintext -import-path  -proto peerswaprpc.proto -max-msg-sz     0.84s user 0.11s system 14% cpu 6.410 total
wc -c  0.00s user 0.00s system 0% cpu 6.409 total

But logging via psweb is such as improvement:

[Impa10r]

Hi Ron, please test the new v5.0.5 image and l let me know if you spot any bugs. It includes the latest peerswap commit.