Skip to content

Bug: electrs won't close stale sockets #1066

Open
@luisschwab

Description

@luisschwab

version: v0.10.5

electrs wouldn't respond to new connections, but the process was running just fine. When checking for open connections, I came across a lot of open sockets in the CLOSE_WAIT state. After restarting the process (thus killing all stale sockets), it responded again.

~$ sudo lsof -i tcp:50001
COMMAND  PID USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
electrs  737 zeus    4u  IPv4    22111      0t0  TCP *:50001 (LISTEN)
electrs  737 zeus    5u  IPv4 83943302      0t0  TCP bitcoin:50001->10.10.20.1:42906 (ESTABLISHED)
electrs  737 zeus   19u  IPv4    22972      0t0  TCP localhost:50001->localhost:36888 (CLOSE_WAIT)
electrs  737 zeus   22u  IPv4  5239520      0t0  TCP localhost:50001->localhost:55850 (ESTABLISHED)
electrs  737 zeus   33u  IPv4  5239520      0t0  TCP localhost:50001->localhost:55850 (ESTABLISHED)
electrs  737 zeus   69u  IPv4 83943305      0t0  TCP bitcoin:50001->10.10.20.1:59618 (ESTABLISHED)
electrs  737 zeus   70u  IPv4 11814999      0t0  TCP bitcoin:50001->10.10.20.1:54110 (CLOSE_WAIT)
electrs  737 zeus   71u  IPv4 15344781      0t0  TCP bitcoin:50001->10.10.20.1:40570 (CLOSE_WAIT)
electrs  737 zeus   72u  IPv4 17749816      0t0  TCP bitcoin:50001->10.10.20.1:43464 (CLOSE_WAIT)
electrs  737 zeus   73u  IPv4 23868900      0t0  TCP bitcoin:50001->10.10.20.1:43468 (CLOSE_WAIT)
electrs  737 zeus   74u  IPv4 23868905      0t0  TCP bitcoin:50001->10.10.20.1:48882 (CLOSE_WAIT)
electrs  737 zeus   76u  IPv4 23868906      0t0  TCP bitcoin:50001->10.10.20.1:48886 (CLOSE_WAIT)
electrs  737 zeus   80u  IPv4 23868908      0t0  TCP bitcoin:50001->10.10.20.1:53436 (CLOSE_WAIT)
electrs  737 zeus   81u  IPv4 23868909      0t0  TCP bitcoin:50001->10.10.20.1:46194 (CLOSE_WAIT)
electrs  737 zeus   82u  IPv4 23868910      0t0  TCP bitcoin:50001->10.10.20.1:58866 (CLOSE_WAIT)
electrs  737 zeus   83u  IPv4 23868923      0t0  TCP bitcoin:50001->10.10.20.1:45890 (CLOSE_WAIT)
electrs  737 zeus   84u  IPv4 23868929      0t0  TCP bitcoin:50001->10.10.20.1:34478 (CLOSE_WAIT)
electrs  737 zeus   85u  IPv4 23868931      0t0  TCP bitcoin:50001->10.10.20.1:38604 (CLOSE_WAIT)
electrs  737 zeus   86u  IPv4 23868934      0t0  TCP bitcoin:50001->10.10.20.1:60560 (CLOSE_WAIT)
electrs  737 zeus   87u  IPv4 23868936      0t0  TCP bitcoin:50001->10.10.20.1:54056 (CLOSE_WAIT)
electrs  737 zeus   88u  IPv4 23868938      0t0  TCP bitcoin:50001->10.10.20.1:45968 (CLOSE_WAIT)
electrs  737 zeus   89u  IPv4 23868941      0t0  TCP bitcoin:50001->10.10.20.1:51538 (CLOSE_WAIT)
electrs  737 zeus   90u  IPv4 23868942      0t0  TCP bitcoin:50001->10.10.20.1:42442 (CLOSE_WAIT)
electrs  737 zeus   91u  IPv4 23868945      0t0  TCP bitcoin:50001->10.10.20.1:50278 (CLOSE_WAIT)
electrs  737 zeus   92u  IPv4 23868948      0t0  TCP bitcoin:50001->10.10.20.1:54234 (CLOSE_WAIT)
electrs  737 zeus   93u  IPv4 27566904      0t0  TCP bitcoin:50001->10.10.20.1:55326 (CLOSE_WAIT)
electrs  737 zeus   94u  IPv4 27788913      0t0  TCP bitcoin:50001->10.10.20.1:42202 (CLOSE_WAIT)
electrs  737 zeus   95u  IPv4 27880904      0t0  TCP bitcoin:50001->10.10.20.1:42760 (CLOSE_WAIT)
electrs  737 zeus   96u  IPv4 28044791      0t0  TCP bitcoin:50001->10.10.20.1:38232 (CLOSE_WAIT)
electrs  737 zeus   97u  IPv4 28230711      0t0  TCP bitcoin:50001->10.10.20.1:38112 (CLOSE_WAIT)
electrs  737 zeus   98u  IPv4 28414562      0t0  TCP bitcoin:50001->10.10.20.1:45354 (CLOSE_WAIT)
electrs  737 zeus   99u  IPv4 28622991      0t0  TCP bitcoin:50001->10.10.20.1:35196 (CLOSE_WAIT)
electrs  737 zeus  100u  IPv4 28633725      0t0  TCP bitcoin:50001->10.10.20.1:60678 (CLOSE_WAIT)
electrs  737 zeus  101u  IPv4 28815135      0t0  TCP bitcoin:50001->10.10.20.1:34374 (CLOSE_WAIT)
electrs  737 zeus  102u  IPv4 28995418      0t0  TCP bitcoin:50001->10.10.20.1:39930 (CLOSE_WAIT)
electrs  737 zeus  103u  IPv4 29109424      0t0  TCP bitcoin:50001->10.10.20.1:48748 (CLOSE_WAIT)
electrs  737 zeus  104u  IPv4 29369266      0t0  TCP bitcoin:50001->10.10.20.1:59396 (CLOSE_WAIT)
electrs  737 zeus  105u  IPv4 29491772      0t0  TCP bitcoin:50001->10.10.20.1:43352 (CLOSE_WAIT)
electrs  737 zeus  106u  IPv4 29606323      0t0  TCP bitcoin:50001->10.10.20.1:58346 (CLOSE_WAIT)
electrs  737 zeus  107u  IPv4 29756051      0t0  TCP bitcoin:50001->10.10.20.1:44606 (CLOSE_WAIT)
electrs  737 zeus  108u  IPv4 29776109      0t0  TCP bitcoin:50001->10.10.20.1:52578 (CLOSE_WAIT)
electrs  737 zeus  109u  IPv4 29783029      0t0  TCP bitcoin:50001->10.10.20.1:59434 (CLOSE_WAIT)
electrs  737 zeus  110u  IPv4 29939285      0t0  TCP bitcoin:50001->10.10.20.1:45402 (CLOSE_WAIT)
electrs  737 zeus  111u  IPv4 30164354      0t0  TCP bitcoin:50001->10.10.20.1:50072 (CLOSE_WAIT)
electrs  737 zeus  112u  IPv4 30316322      0t0  TCP bitcoin:50001->10.10.20.1:58690 (CLOSE_WAIT)
electrs  737 zeus  113u  IPv4 30353369      0t0  TCP bitcoin:50001->10.10.20.1:58138 (CLOSE_WAIT)
electrs  737 zeus  114u  IPv4 30376331      0t0  TCP bitcoin:50001->10.10.20.1:35728 (CLOSE_WAIT)
electrs  737 zeus  115u  IPv4 30587933      0t0  TCP bitcoin:50001->10.10.20.1:52234 (CLOSE_WAIT)
electrs  737 zeus  116u  IPv4 30794296      0t0  TCP bitcoin:50001->10.10.20.1:41866 (CLOSE_WAIT)
electrs  737 zeus  117u  IPv4 30809384      0t0  TCP bitcoin:50001->10.10.20.1:60152 (CLOSE_WAIT)
electrs  737 zeus  118u  IPv4 30970005      0t0  TCP bitcoin:50001->10.10.20.1:52972 (CLOSE_WAIT)
electrs  737 zeus  119u  IPv4 30983317      0t0  TCP bitcoin:50001->10.10.20.1:52610 (CLOSE_WAIT)
electrs  737 zeus  120u  IPv4 31158246      0t0  TCP bitcoin:50001->10.10.20.1:48234 (CLOSE_WAIT)
electrs  737 zeus  121u  IPv4 31288124      0t0  TCP bitcoin:50001->10.10.20.1:56040 (CLOSE_WAIT)
electrs  737 zeus  122u  IPv4 31346158      0t0  TCP bitcoin:50001->10.10.20.1:56772 (CLOSE_WAIT)
electrs  737 zeus  123u  IPv4 42355261      0t0  TCP bitcoin:50001->10.10.20.1:36056 (CLOSE_WAIT)
electrs  737 zeus  124u  IPv4 60903529      0t0  TCP bitcoin:50001->10.10.20.1:47746 (CLOSE_WAIT)
electrs  737 zeus  125u  IPv4 60903530      0t0  TCP bitcoin:50001->10.10.20.1:55020 (CLOSE_WAIT)
electrs  737 zeus  126u  IPv4 60903559      0t0  TCP bitcoin:50001->10.10.20.1:55028 (CLOSE_WAIT)
electrs  737 zeus  127u  IPv4 60903746      0t0  TCP bitcoin:50001->10.10.20.1:51036 (CLOSE_WAIT)
electrs  737 zeus  128u  IPv4 60903839      0t0  TCP bitcoin:50001->10.10.20.1:33432 (CLOSE_WAIT)
electrs  737 zeus  129u  IPv4 60904002      0t0  TCP bitcoin:50001->10.10.20.1:37154 (CLOSE_WAIT)
electrs  737 zeus  130u  IPv4 60904003      0t0  TCP bitcoin:50001->10.10.20.1:45286 (CLOSE_WAIT)
electrs  737 zeus  131u  IPv4 60904098      0t0  TCP bitcoin:50001->10.10.20.1:41214 (CLOSE_WAIT)
electrs  737 zeus  132u  IPv4 60904149      0t0  TCP bitcoin:50001->10.10.20.1:56424 (CLOSE_WAIT)
electrs  737 zeus  133u  IPv4 60904158      0t0  TCP bitcoin:50001->10.10.20.1:44260 (CLOSE_WAIT)
electrs  737 zeus  134u  IPv4 60904159      0t0  TCP bitcoin:50001->10.10.20.1:46186 (CLOSE_WAIT)
electrs  737 zeus  135u  IPv4 60904160      0t0  TCP bitcoin:50001->10.10.20.1:53340 (CLOSE_WAIT)
electrs  737 zeus  136u  IPv4 60904161      0t0  TCP bitcoin:50001->10.10.20.1:55378 (CLOSE_WAIT)
electrs  737 zeus  137u  IPv4 60904162      0t0  TCP bitcoin:50001->10.10.20.1:48826 (CLOSE_WAIT)
electrs  737 zeus  138u  IPv4 60904163      0t0  TCP bitcoin:50001->10.10.20.1:48836 (CLOSE_WAIT)
electrs  737 zeus  139u  IPv4 60904164      0t0  TCP bitcoin:50001->10.10.20.1:60528 (CLOSE_WAIT)
electrs  737 zeus  140u  IPv4 60904165      0t0  TCP bitcoin:50001->10.10.20.1:49546 (CLOSE_WAIT)
electrs  737 zeus  141u  IPv4 60904168      0t0  TCP bitcoin:50001->10.10.20.1:42526 (CLOSE_WAIT)
electrs  737 zeus  142u  IPv4 60904169      0t0  TCP bitcoin:50001->10.10.20.1:46386 (CLOSE_WAIT)
electrs  737 zeus  143u  IPv4 60908409      0t0  TCP bitcoin:50001->10.10.20.1:49938 (CLOSE_WAIT)
electrs  737 zeus  144u  IPv4 60904364      0t0  TCP bitcoin:50001->10.10.20.1:49942 (CLOSE_WAIT)
electrs  737 zeus  145u  IPv4 60904366      0t0  TCP bitcoin:50001->10.10.20.1:41512 (CLOSE_WAIT)
electrs  737 zeus  146u  IPv4 60904367      0t0  TCP bitcoin:50001->10.10.20.1:47094 (CLOSE_WAIT)
electrs  737 zeus  147u  IPv4 60908446      0t0  TCP bitcoin:50001->10.10.20.1:38480 (CLOSE_WAIT)
electrs  737 zeus  148u  IPv4 60908447      0t0  TCP bitcoin:50001->10.10.20.1:49742 (CLOSE_WAIT)
electrs  737 zeus  149u  IPv4 60908448      0t0  TCP bitcoin:50001->10.10.20.1:60894 (CLOSE_WAIT)
electrs  737 zeus  150u  IPv4 60914765      0t0  TCP bitcoin:50001->10.10.20.1:58272 (CLOSE_WAIT)
electrs  737 zeus  151u  IPv4 60914861      0t0  TCP bitcoin:50001->10.10.20.1:60374 (CLOSE_WAIT)
electrs  737 zeus  152u  IPv4 60914862      0t0  TCP bitcoin:50001->10.10.20.1:41598 (CLOSE_WAIT)
electrs  737 zeus  153u  IPv4 60914866      0t0  TCP bitcoin:50001->10.10.20.1:47820 (CLOSE_WAIT)
electrs  737 zeus  154u  IPv4 60914867      0t0  TCP bitcoin:50001->10.10.20.1:39514 (CLOSE_WAIT)
electrs  737 zeus  155u  IPv4 60914868      0t0  TCP bitcoin:50001->10.10.20.1:54478 (CLOSE_WAIT)
electrs  737 zeus  156u  IPv4 60914869      0t0  TCP bitcoin:50001->10.10.20.1:51464 (CLOSE_WAIT)
electrs  737 zeus  157u  IPv4 60914870      0t0  TCP bitcoin:50001->10.10.20.1:39052 (CLOSE_WAIT)
electrs  737 zeus  158u  IPv4 60914871      0t0  TCP bitcoin:50001->10.10.20.1:37304 (CLOSE_WAIT)
electrs  737 zeus  159u  IPv4 60914872      0t0  TCP bitcoin:50001->10.10.20.1:38114 (CLOSE_WAIT)
electrs  737 zeus  160u  IPv4 60914873      0t0  TCP bitcoin:50001->10.10.20.1:33250 (CLOSE_WAIT)
electrs  737 zeus  161u  IPv4 60914874      0t0  TCP bitcoin:50001->10.10.20.1:33256 (CLOSE_WAIT)
electrs  737 zeus  162u  IPv4 60914876      0t0  TCP bitcoin:50001->10.10.20.1:34970 (CLOSE_WAIT)
electrs  737 zeus  163u  IPv4 60914904      0t0  TCP bitcoin:50001->10.10.20.1:43028 (CLOSE_WAIT)
electrs  737 zeus  164u  IPv4 60914907      0t0  TCP bitcoin:50001->10.10.20.1:35102 (CLOSE_WAIT)
electrs  737 zeus  165u  IPv4 60914912      0t0  TCP bitcoin:50001->10.10.20.1:35440 (CLOSE_WAIT)
electrs  737 zeus  166u  IPv4 60914913      0t0  TCP bitcoin:50001->10.10.20.1:57826 (CLOSE_WAIT)
electrs  737 zeus  167u  IPv4 60914927      0t0  TCP bitcoin:50001->10.10.20.1:51424 (CLOSE_WAIT)
electrs  737 zeus  168u  IPv4 60914975      0t0  TCP bitcoin:50001->10.10.20.1:55226 (CLOSE_WAIT)
electrs  737 zeus  169u  IPv4 60915000      0t0  TCP bitcoin:50001->10.10.20.1:51052 (CLOSE_WAIT)
electrs  737 zeus  170u  IPv4 60915030      0t0  TCP bitcoin:50001->10.10.20.1:32796 (CLOSE_WAIT)
electrs  737 zeus  171u  IPv4 60915175      0t0  TCP bitcoin:50001->10.10.20.1:35048 (CLOSE_WAIT)
electrs  737 zeus  172u  IPv4 60915176      0t0  TCP bitcoin:50001->10.10.20.1:55420 (CLOSE_WAIT)
electrs  737 zeus  173u  IPv4 60915177      0t0  TCP bitcoin:50001->10.10.20.1:55428 (CLOSE_WAIT)
electrs  737 zeus  174u  IPv4 60915187      0t0  TCP bitcoin:50001->10.10.20.1:47890 (CLOSE_WAIT)
electrs  737 zeus  175u  IPv4 60915188      0t0  TCP bitcoin:50001->10.10.20.1:37906 (CLOSE_WAIT)
electrs  737 zeus  176u  IPv4 60915209      0t0  TCP bitcoin:50001->10.10.20.1:37762 (CLOSE_WAIT)
electrs  737 zeus  177u  IPv4 60915380      0t0  TCP bitcoin:50001->10.10.20.1:42320 (CLOSE_WAIT)
electrs  737 zeus  178u  IPv4 60915407      0t0  TCP bitcoin:50001->10.10.20.1:57256 (CLOSE_WAIT)
electrs  737 zeus  179u  IPv4 60915410      0t0  TCP bitcoin:50001->10.10.20.1:40734 (CLOSE_WAIT)
electrs  737 zeus  180u  IPv4 60915433      0t0  TCP bitcoin:50001->10.10.20.1:57960 (CLOSE_WAIT)
electrs  737 zeus  181u  IPv4 60915461      0t0  TCP bitcoin:50001->10.10.20.1:55458 (CLOSE_WAIT)
electrs  737 zeus  182u  IPv4 60915480      0t0  TCP bitcoin:50001->10.10.20.1:38250 (CLOSE_WAIT)
electrs  737 zeus  183u  IPv4 60924993      0t0  TCP bitcoin:50001->10.10.20.1:37768 (CLOSE_WAIT)
electrs  737 zeus  184u  IPv4 60925081      0t0  TCP bitcoin:50001->10.10.20.1:53362 (CLOSE_WAIT)
electrs  737 zeus  185u  IPv4 60925318      0t0  TCP bitcoin:50001->10.10.20.1:53364 (CLOSE_WAIT)
electrs  737 zeus  186u  IPv4 60925431      0t0  TCP bitcoin:50001->10.10.20.1:36976 (CLOSE_WAIT)
electrs  737 zeus  187u  IPv4 60925432      0t0  TCP bitcoin:50001->10.10.20.1:47562 (CLOSE_WAIT)
electrs  737 zeus  188u  IPv4 60925434      0t0  TCP bitcoin:50001->10.10.20.1:40048 (CLOSE_WAIT)
electrs  737 zeus  189u  IPv4 60925436      0t0  TCP bitcoin:50001->10.10.20.1:47676 (CLOSE_WAIT)
electrs  737 zeus  190u  IPv4 62562853      0t0  TCP bitcoin:50001->10.10.20.1:55854 (CLOSE_WAIT)
electrs  737 zeus  191u  IPv4 63084925      0t0  TCP bitcoin:50001->10.10.20.1:55448 (CLOSE_WAIT)
electrs  737 zeus  192u  IPv4 69786180      0t0  TCP bitcoin:50001->10.10.20.1:38508 (CLOSE_WAIT)
electrs  737 zeus  193u  IPv4 75729332      0t0  TCP bitcoin:50001->10.10.20.1:37852 (CLOSE_WAIT)
electrs  737 zeus  194u  IPv4 80210966      0t0  TCP bitcoin:50001->10.10.20.1:43732 (CLOSE_WAIT)
electrs  737 zeus  195u  IPv4 83922343      0t0  TCP bitcoin:50001->10.10.20.1:43748 (CLOSE_WAIT)
electrs  737 zeus  196u  IPv4 83922344      0t0  TCP bitcoin:50001->10.10.20.1:43082 (CLOSE_WAIT)
electrs  737 zeus  197u  IPv4 83922345      0t0  TCP bitcoin:50001->10.10.20.1:43096 (CLOSE_WAIT)
electrs  737 zeus  198u  IPv4 83922346      0t0  TCP bitcoin:50001->10.10.20.1:59954 (CLOSE_WAIT)
electrs  737 zeus  199u  IPv4 83922347      0t0  TCP bitcoin:50001->10.10.20.1:36542 (CLOSE_WAIT)
electrs  737 zeus  200u  IPv4 83922348      0t0  TCP bitcoin:50001->10.10.20.1:36344 (CLOSE_WAIT)
electrs  737 zeus  201u  IPv4 83922350      0t0  TCP bitcoin:50001->10.10.20.1:46922 (CLOSE_WAIT)
electrs  737 zeus  202u  IPv4 83922351      0t0  TCP bitcoin:50001->10.10.20.1:46284 (CLOSE_WAIT)
electrs  737 zeus  203u  IPv4 83922352      0t0  TCP bitcoin:50001->10.10.20.1:41902 (CLOSE_WAIT)
electrs  737 zeus  204u  IPv4 83922354      0t0  TCP bitcoin:50001->10.10.20.1:41054 (CLOSE_WAIT)
electrs  737 zeus  205u  IPv4 83922355      0t0  TCP bitcoin:50001->10.10.20.1:47348 (CLOSE_WAIT)
electrs  737 zeus  206u  IPv4 83922870      0t0  TCP bitcoin:50001->10.10.20.1:35698 (CLOSE_WAIT)
electrs  737 zeus  207u  IPv4 83941263      0t0  TCP bitcoin:50001->10.10.20.1:35714 (CLOSE_WAIT)
electrs  737 zeus  209u  IPv4 83942620      0t0  TCP bitcoin:50001->10.10.20.1:37314 (CLOSE_WAIT)
electrs  737 zeus  210u  IPv4 83943113      0t0  TCP bitcoin:50001->10.10.20.1:38736 (CLOSE_WAIT)
electrs  737 zeus  211u  IPv4 83943301      0t0  TCP bitcoin:50001->10.10.20.1:44056 (CLOSE_WAIT)

To Reproduce
Run it for some time with a lot of connections and disconnections?

Expected behavior
electrs should clean up stale sockets.

Configuration

# electrs.toml
auth="<redacted>:<redacted>"
electrum_rpc_addr = "<redacted>:50001"

System running electrs

  • Deplyment: followed guide on romanz/electrs
  • OS: Ubuntu Server 22.04.4 LTS x86_64

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions