Skip to content

Bug: electrs won't close stale sockets #1066

Open
@luisschwab

Description

version: v0.10.5

electrs wouldn't respond to new connections, but the process was running just fine. When checking for open connections, I came across a lot of open sockets in the CLOSE_WAIT state. After restarting the process (thus killing all stale sockets), it responded again.

~$ sudo lsof -i tcp:50001
COMMAND  PID USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
electrs  737 zeus    4u  IPv4    22111      0t0  TCP *:50001 (LISTEN)
electrs  737 zeus    5u  IPv4 83943302      0t0  TCP bitcoin:50001->10.10.20.1:42906 (ESTABLISHED)
electrs  737 zeus   19u  IPv4    22972      0t0  TCP localhost:50001->localhost:36888 (CLOSE_WAIT)
electrs  737 zeus   22u  IPv4  5239520      0t0  TCP localhost:50001->localhost:55850 (ESTABLISHED)
electrs  737 zeus   33u  IPv4  5239520      0t0  TCP localhost:50001->localhost:55850 (ESTABLISHED)
electrs  737 zeus   69u  IPv4 83943305      0t0  TCP bitcoin:50001->10.10.20.1:59618 (ESTABLISHED)
electrs  737 zeus   70u  IPv4 11814999      0t0  TCP bitcoin:50001->10.10.20.1:54110 (CLOSE_WAIT)
electrs  737 zeus   71u  IPv4 15344781      0t0  TCP bitcoin:50001->10.10.20.1:40570 (CLOSE_WAIT)
electrs  737 zeus   72u  IPv4 17749816      0t0  TCP bitcoin:50001->10.10.20.1:43464 (CLOSE_WAIT)
electrs  737 zeus   73u  IPv4 23868900      0t0  TCP bitcoin:50001->10.10.20.1:43468 (CLOSE_WAIT)
electrs  737 zeus   74u  IPv4 23868905      0t0  TCP bitcoin:50001->10.10.20.1:48882 (CLOSE_WAIT)
electrs  737 zeus   76u  IPv4 23868906      0t0  TCP bitcoin:50001->10.10.20.1:48886 (CLOSE_WAIT)
electrs  737 zeus   80u  IPv4 23868908      0t0  TCP bitcoin:50001->10.10.20.1:53436 (CLOSE_WAIT)
electrs  737 zeus   81u  IPv4 23868909      0t0  TCP bitcoin:50001->10.10.20.1:46194 (CLOSE_WAIT)
electrs  737 zeus   82u  IPv4 23868910      0t0  TCP bitcoin:50001->10.10.20.1:58866 (CLOSE_WAIT)
electrs  737 zeus   83u  IPv4 23868923      0t0  TCP bitcoin:50001->10.10.20.1:45890 (CLOSE_WAIT)
electrs  737 zeus   84u  IPv4 23868929      0t0  TCP bitcoin:50001->10.10.20.1:34478 (CLOSE_WAIT)
electrs  737 zeus   85u  IPv4 23868931      0t0  TCP bitcoin:50001->10.10.20.1:38604 (CLOSE_WAIT)
electrs  737 zeus   86u  IPv4 23868934      0t0  TCP bitcoin:50001->10.10.20.1:60560 (CLOSE_WAIT)
electrs  737 zeus   87u  IPv4 23868936      0t0  TCP bitcoin:50001->10.10.20.1:54056 (CLOSE_WAIT)
electrs  737 zeus   88u  IPv4 23868938      0t0  TCP bitcoin:50001->10.10.20.1:45968 (CLOSE_WAIT)
electrs  737 zeus   89u  IPv4 23868941      0t0  TCP bitcoin:50001->10.10.20.1:51538 (CLOSE_WAIT)
electrs  737 zeus   90u  IPv4 23868942      0t0  TCP bitcoin:50001->10.10.20.1:42442 (CLOSE_WAIT)
electrs  737 zeus   91u  IPv4 23868945      0t0  TCP bitcoin:50001->10.10.20.1:50278 (CLOSE_WAIT)
electrs  737 zeus   92u  IPv4 23868948      0t0  TCP bitcoin:50001->10.10.20.1:54234 (CLOSE_WAIT)
electrs  737 zeus   93u  IPv4 27566904      0t0  TCP bitcoin:50001->10.10.20.1:55326 (CLOSE_WAIT)
electrs  737 zeus   94u  IPv4 27788913      0t0  TCP bitcoin:50001->10.10.20.1:42202 (CLOSE_WAIT)
electrs  737 zeus   95u  IPv4 27880904      0t0  TCP bitcoin:50001->10.10.20.1:42760 (CLOSE_WAIT)
electrs  737 zeus   96u  IPv4 28044791      0t0  TCP bitcoin:50001->10.10.20.1:38232 (CLOSE_WAIT)
electrs  737 zeus   97u  IPv4 28230711      0t0  TCP bitcoin:50001->10.10.20.1:38112 (CLOSE_WAIT)
electrs  737 zeus   98u  IPv4 28414562      0t0  TCP bitcoin:50001->10.10.20.1:45354 (CLOSE_WAIT)
electrs  737 zeus   99u  IPv4 28622991      0t0  TCP bitcoin:50001->10.10.20.1:35196 (CLOSE_WAIT)
electrs  737 zeus  100u  IPv4 28633725      0t0  TCP bitcoin:50001->10.10.20.1:60678 (CLOSE_WAIT)
electrs  737 zeus  101u  IPv4 28815135      0t0  TCP bitcoin:50001->10.10.20.1:34374 (CLOSE_WAIT)
electrs  737 zeus  102u  IPv4 28995418      0t0  TCP bitcoin:50001->10.10.20.1:39930 (CLOSE_WAIT)
electrs  737 zeus  103u  IPv4 29109424      0t0  TCP bitcoin:50001->10.10.20.1:48748 (CLOSE_WAIT)
electrs  737 zeus  104u  IPv4 29369266      0t0  TCP bitcoin:50001->10.10.20.1:59396 (CLOSE_WAIT)
electrs  737 zeus  105u  IPv4 29491772      0t0  TCP bitcoin:50001->10.10.20.1:43352 (CLOSE_WAIT)
electrs  737 zeus  106u  IPv4 29606323      0t0  TCP bitcoin:50001->10.10.20.1:58346 (CLOSE_WAIT)
electrs  737 zeus  107u  IPv4 29756051      0t0  TCP bitcoin:50001->10.10.20.1:44606 (CLOSE_WAIT)
electrs  737 zeus  108u  IPv4 29776109      0t0  TCP bitcoin:50001->10.10.20.1:52578 (CLOSE_WAIT)
electrs  737 zeus  109u  IPv4 29783029      0t0  TCP bitcoin:50001->10.10.20.1:59434 (CLOSE_WAIT)
electrs  737 zeus  110u  IPv4 29939285      0t0  TCP bitcoin:50001->10.10.20.1:45402 (CLOSE_WAIT)
electrs  737 zeus  111u  IPv4 30164354      0t0  TCP bitcoin:50001->10.10.20.1:50072 (CLOSE_WAIT)
electrs  737 zeus  112u  IPv4 30316322      0t0  TCP bitcoin:50001->10.10.20.1:58690 (CLOSE_WAIT)
electrs  737 zeus  113u  IPv4 30353369      0t0  TCP bitcoin:50001->10.10.20.1:58138 (CLOSE_WAIT)
electrs  737 zeus  114u  IPv4 30376331      0t0  TCP bitcoin:50001->10.10.20.1:35728 (CLOSE_WAIT)
electrs  737 zeus  115u  IPv4 30587933      0t0  TCP bitcoin:50001->10.10.20.1:52234 (CLOSE_WAIT)
electrs  737 zeus  116u  IPv4 30794296      0t0  TCP bitcoin:50001->10.10.20.1:41866 (CLOSE_WAIT)
electrs  737 zeus  117u  IPv4 30809384      0t0  TCP bitcoin:50001->10.10.20.1:60152 (CLOSE_WAIT)
electrs  737 zeus  118u  IPv4 30970005      0t0  TCP bitcoin:50001->10.10.20.1:52972 (CLOSE_WAIT)
electrs  737 zeus  119u  IPv4 30983317      0t0  TCP bitcoin:50001->10.10.20.1:52610 (CLOSE_WAIT)
electrs  737 zeus  120u  IPv4 31158246      0t0  TCP bitcoin:50001->10.10.20.1:48234 (CLOSE_WAIT)
electrs  737 zeus  121u  IPv4 31288124      0t0  TCP bitcoin:50001->10.10.20.1:56040 (CLOSE_WAIT)
electrs  737 zeus  122u  IPv4 31346158      0t0  TCP bitcoin:50001->10.10.20.1:56772 (CLOSE_WAIT)
electrs  737 zeus  123u  IPv4 42355261      0t0  TCP bitcoin:50001->10.10.20.1:36056 (CLOSE_WAIT)
electrs  737 zeus  124u  IPv4 60903529      0t0  TCP bitcoin:50001->10.10.20.1:47746 (CLOSE_WAIT)
electrs  737 zeus  125u  IPv4 60903530      0t0  TCP bitcoin:50001->10.10.20.1:55020 (CLOSE_WAIT)
electrs  737 zeus  126u  IPv4 60903559      0t0  TCP bitcoin:50001->10.10.20.1:55028 (CLOSE_WAIT)
electrs  737 zeus  127u  IPv4 60903746      0t0  TCP bitcoin:50001->10.10.20.1:51036 (CLOSE_WAIT)
electrs  737 zeus  128u  IPv4 60903839      0t0  TCP bitcoin:50001->10.10.20.1:33432 (CLOSE_WAIT)
electrs  737 zeus  129u  IPv4 60904002      0t0  TCP bitcoin:50001->10.10.20.1:37154 (CLOSE_WAIT)
electrs  737 zeus  130u  IPv4 60904003      0t0  TCP bitcoin:50001->10.10.20.1:45286 (CLOSE_WAIT)
electrs  737 zeus  131u  IPv4 60904098      0t0  TCP bitcoin:50001->10.10.20.1:41214 (CLOSE_WAIT)
electrs  737 zeus  132u  IPv4 60904149      0t0  TCP bitcoin:50001->10.10.20.1:56424 (CLOSE_WAIT)
electrs  737 zeus  133u  IPv4 60904158      0t0  TCP bitcoin:50001->10.10.20.1:44260 (CLOSE_WAIT)
electrs  737 zeus  134u  IPv4 60904159      0t0  TCP bitcoin:50001->10.10.20.1:46186 (CLOSE_WAIT)
electrs  737 zeus  135u  IPv4 60904160      0t0  TCP bitcoin:50001->10.10.20.1:53340 (CLOSE_WAIT)
electrs  737 zeus  136u  IPv4 60904161      0t0  TCP bitcoin:50001->10.10.20.1:55378 (CLOSE_WAIT)
electrs  737 zeus  137u  IPv4 60904162      0t0  TCP bitcoin:50001->10.10.20.1:48826 (CLOSE_WAIT)
electrs  737 zeus  138u  IPv4 60904163      0t0  TCP bitcoin:50001->10.10.20.1:48836 (CLOSE_WAIT)
electrs  737 zeus  139u  IPv4 60904164      0t0  TCP bitcoin:50001->10.10.20.1:60528 (CLOSE_WAIT)
electrs  737 zeus  140u  IPv4 60904165      0t0  TCP bitcoin:50001->10.10.20.1:49546 (CLOSE_WAIT)
electrs  737 zeus  141u  IPv4 60904168      0t0  TCP bitcoin:50001->10.10.20.1:42526 (CLOSE_WAIT)
electrs  737 zeus  142u  IPv4 60904169      0t0  TCP bitcoin:50001->10.10.20.1:46386 (CLOSE_WAIT)
electrs  737 zeus  143u  IPv4 60908409      0t0  TCP bitcoin:50001->10.10.20.1:49938 (CLOSE_WAIT)
electrs  737 zeus  144u  IPv4 60904364      0t0  TCP bitcoin:50001->10.10.20.1:49942 (CLOSE_WAIT)
electrs  737 zeus  145u  IPv4 60904366      0t0  TCP bitcoin:50001->10.10.20.1:41512 (CLOSE_WAIT)
electrs  737 zeus  146u  IPv4 60904367      0t0  TCP bitcoin:50001->10.10.20.1:47094 (CLOSE_WAIT)
electrs  737 zeus  147u  IPv4 60908446      0t0  TCP bitcoin:50001->10.10.20.1:38480 (CLOSE_WAIT)
electrs  737 zeus  148u  IPv4 60908447      0t0  TCP bitcoin:50001->10.10.20.1:49742 (CLOSE_WAIT)
electrs  737 zeus  149u  IPv4 60908448      0t0  TCP bitcoin:50001->10.10.20.1:60894 (CLOSE_WAIT)
electrs  737 zeus  150u  IPv4 60914765      0t0  TCP bitcoin:50001->10.10.20.1:58272 (CLOSE_WAIT)
electrs  737 zeus  151u  IPv4 60914861      0t0  TCP bitcoin:50001->10.10.20.1:60374 (CLOSE_WAIT)
electrs  737 zeus  152u  IPv4 60914862      0t0  TCP bitcoin:50001->10.10.20.1:41598 (CLOSE_WAIT)
electrs  737 zeus  153u  IPv4 60914866      0t0  TCP bitcoin:50001->10.10.20.1:47820 (CLOSE_WAIT)
electrs  737 zeus  154u  IPv4 60914867      0t0  TCP bitcoin:50001->10.10.20.1:39514 (CLOSE_WAIT)
electrs  737 zeus  155u  IPv4 60914868      0t0  TCP bitcoin:50001->10.10.20.1:54478 (CLOSE_WAIT)
electrs  737 zeus  156u  IPv4 60914869      0t0  TCP bitcoin:50001->10.10.20.1:51464 (CLOSE_WAIT)
electrs  737 zeus  157u  IPv4 60914870      0t0  TCP bitcoin:50001->10.10.20.1:39052 (CLOSE_WAIT)
electrs  737 zeus  158u  IPv4 60914871      0t0  TCP bitcoin:50001->10.10.20.1:37304 (CLOSE_WAIT)
electrs  737 zeus  159u  IPv4 60914872      0t0  TCP bitcoin:50001->10.10.20.1:38114 (CLOSE_WAIT)
electrs  737 zeus  160u  IPv4 60914873      0t0  TCP bitcoin:50001->10.10.20.1:33250 (CLOSE_WAIT)
electrs  737 zeus  161u  IPv4 60914874      0t0  TCP bitcoin:50001->10.10.20.1:33256 (CLOSE_WAIT)
electrs  737 zeus  162u  IPv4 60914876      0t0  TCP bitcoin:50001->10.10.20.1:34970 (CLOSE_WAIT)
electrs  737 zeus  163u  IPv4 60914904      0t0  TCP bitcoin:50001->10.10.20.1:43028 (CLOSE_WAIT)
electrs  737 zeus  164u  IPv4 60914907      0t0  TCP bitcoin:50001->10.10.20.1:35102 (CLOSE_WAIT)
electrs  737 zeus  165u  IPv4 60914912      0t0  TCP bitcoin:50001->10.10.20.1:35440 (CLOSE_WAIT)
electrs  737 zeus  166u  IPv4 60914913      0t0  TCP bitcoin:50001->10.10.20.1:57826 (CLOSE_WAIT)
electrs  737 zeus  167u  IPv4 60914927      0t0  TCP bitcoin:50001->10.10.20.1:51424 (CLOSE_WAIT)
electrs  737 zeus  168u  IPv4 60914975      0t0  TCP bitcoin:50001->10.10.20.1:55226 (CLOSE_WAIT)
electrs  737 zeus  169u  IPv4 60915000      0t0  TCP bitcoin:50001->10.10.20.1:51052 (CLOSE_WAIT)
electrs  737 zeus  170u  IPv4 60915030      0t0  TCP bitcoin:50001->10.10.20.1:32796 (CLOSE_WAIT)
electrs  737 zeus  171u  IPv4 60915175      0t0  TCP bitcoin:50001->10.10.20.1:35048 (CLOSE_WAIT)
electrs  737 zeus  172u  IPv4 60915176      0t0  TCP bitcoin:50001->10.10.20.1:55420 (CLOSE_WAIT)
electrs  737 zeus  173u  IPv4 60915177      0t0  TCP bitcoin:50001->10.10.20.1:55428 (CLOSE_WAIT)
electrs  737 zeus  174u  IPv4 60915187      0t0  TCP bitcoin:50001->10.10.20.1:47890 (CLOSE_WAIT)
electrs  737 zeus  175u  IPv4 60915188      0t0  TCP bitcoin:50001->10.10.20.1:37906 (CLOSE_WAIT)
electrs  737 zeus  176u  IPv4 60915209      0t0  TCP bitcoin:50001->10.10.20.1:37762 (CLOSE_WAIT)
electrs  737 zeus  177u  IPv4 60915380      0t0  TCP bitcoin:50001->10.10.20.1:42320 (CLOSE_WAIT)
electrs  737 zeus  178u  IPv4 60915407      0t0  TCP bitcoin:50001->10.10.20.1:57256 (CLOSE_WAIT)
electrs  737 zeus  179u  IPv4 60915410      0t0  TCP bitcoin:50001->10.10.20.1:40734 (CLOSE_WAIT)
electrs  737 zeus  180u  IPv4 60915433      0t0  TCP bitcoin:50001->10.10.20.1:57960 (CLOSE_WAIT)
electrs  737 zeus  181u  IPv4 60915461      0t0  TCP bitcoin:50001->10.10.20.1:55458 (CLOSE_WAIT)
electrs  737 zeus  182u  IPv4 60915480      0t0  TCP bitcoin:50001->10.10.20.1:38250 (CLOSE_WAIT)
electrs  737 zeus  183u  IPv4 60924993      0t0  TCP bitcoin:50001->10.10.20.1:37768 (CLOSE_WAIT)
electrs  737 zeus  184u  IPv4 60925081      0t0  TCP bitcoin:50001->10.10.20.1:53362 (CLOSE_WAIT)
electrs  737 zeus  185u  IPv4 60925318      0t0  TCP bitcoin:50001->10.10.20.1:53364 (CLOSE_WAIT)
electrs  737 zeus  186u  IPv4 60925431      0t0  TCP bitcoin:50001->10.10.20.1:36976 (CLOSE_WAIT)
electrs  737 zeus  187u  IPv4 60925432      0t0  TCP bitcoin:50001->10.10.20.1:47562 (CLOSE_WAIT)
electrs  737 zeus  188u  IPv4 60925434      0t0  TCP bitcoin:50001->10.10.20.1:40048 (CLOSE_WAIT)
electrs  737 zeus  189u  IPv4 60925436      0t0  TCP bitcoin:50001->10.10.20.1:47676 (CLOSE_WAIT)
electrs  737 zeus  190u  IPv4 62562853      0t0  TCP bitcoin:50001->10.10.20.1:55854 (CLOSE_WAIT)
electrs  737 zeus  191u  IPv4 63084925      0t0  TCP bitcoin:50001->10.10.20.1:55448 (CLOSE_WAIT)
electrs  737 zeus  192u  IPv4 69786180      0t0  TCP bitcoin:50001->10.10.20.1:38508 (CLOSE_WAIT)
electrs  737 zeus  193u  IPv4 75729332      0t0  TCP bitcoin:50001->10.10.20.1:37852 (CLOSE_WAIT)
electrs  737 zeus  194u  IPv4 80210966      0t0  TCP bitcoin:50001->10.10.20.1:43732 (CLOSE_WAIT)
electrs  737 zeus  195u  IPv4 83922343      0t0  TCP bitcoin:50001->10.10.20.1:43748 (CLOSE_WAIT)
electrs  737 zeus  196u  IPv4 83922344      0t0  TCP bitcoin:50001->10.10.20.1:43082 (CLOSE_WAIT)
electrs  737 zeus  197u  IPv4 83922345      0t0  TCP bitcoin:50001->10.10.20.1:43096 (CLOSE_WAIT)
electrs  737 zeus  198u  IPv4 83922346      0t0  TCP bitcoin:50001->10.10.20.1:59954 (CLOSE_WAIT)
electrs  737 zeus  199u  IPv4 83922347      0t0  TCP bitcoin:50001->10.10.20.1:36542 (CLOSE_WAIT)
electrs  737 zeus  200u  IPv4 83922348      0t0  TCP bitcoin:50001->10.10.20.1:36344 (CLOSE_WAIT)
electrs  737 zeus  201u  IPv4 83922350      0t0  TCP bitcoin:50001->10.10.20.1:46922 (CLOSE_WAIT)
electrs  737 zeus  202u  IPv4 83922351      0t0  TCP bitcoin:50001->10.10.20.1:46284 (CLOSE_WAIT)
electrs  737 zeus  203u  IPv4 83922352      0t0  TCP bitcoin:50001->10.10.20.1:41902 (CLOSE_WAIT)
electrs  737 zeus  204u  IPv4 83922354      0t0  TCP bitcoin:50001->10.10.20.1:41054 (CLOSE_WAIT)
electrs  737 zeus  205u  IPv4 83922355      0t0  TCP bitcoin:50001->10.10.20.1:47348 (CLOSE_WAIT)
electrs  737 zeus  206u  IPv4 83922870      0t0  TCP bitcoin:50001->10.10.20.1:35698 (CLOSE_WAIT)
electrs  737 zeus  207u  IPv4 83941263      0t0  TCP bitcoin:50001->10.10.20.1:35714 (CLOSE_WAIT)
electrs  737 zeus  209u  IPv4 83942620      0t0  TCP bitcoin:50001->10.10.20.1:37314 (CLOSE_WAIT)
electrs  737 zeus  210u  IPv4 83943113      0t0  TCP bitcoin:50001->10.10.20.1:38736 (CLOSE_WAIT)
electrs  737 zeus  211u  IPv4 83943301      0t0  TCP bitcoin:50001->10.10.20.1:44056 (CLOSE_WAIT)

To Reproduce
Run it for some time with a lot of connections and disconnections?

Expected behavior
electrs should clean up stale sockets.

Configuration

# electrs.toml
auth="<redacted>:<redacted>"
electrum_rpc_addr = "<redacted>:50001"

System running electrs

  • Deplyment: followed guide on romanz/electrs
  • OS: Ubuntu Server 22.04.4 LTS x86_64

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions