Double-Free of UserData when gracefully closing a Websocket from Server-Side

[DerAlbiG]

Hi,
i am trying to implement a graceful server Shutdown. Either i am doing something wrong, or there is a fundamental bug in the implementation. ASAN complains (rightfully) about a double-free attempt.

My Shutdown-Code involves:

loop->defer([this]
{
    for (auto* con: websocketConnections)
        con->end(1000, "Server shutting down");
    if (appPtr)
        appPtr->close();
});

The code of con->end() currently finishes with the following lines:

        /* Emit close event */
        if (webSocketContextData->closeHandler) {
            webSocketContextData->closeHandler(this, code, message);
        }
        ((USERDATA *) this->getUserData())->~USERDATA();

As we can see, the destructor of UserData is called at the very end.

However webSocketContextData->closeHandler() already brings me into uWS::TemplatedApp::ws in App.hpp, line 263 - 439.
Within this function there is this snippet:

        /* Copy all handlers */
        webSocketContext->getExt()->openHandler = std::move(behavior.open);
        webSocketContext->getExt()->messageHandler = std::move(behavior.message);
        webSocketContext->getExt()->drainHandler = std::move(behavior.drain);
        webSocketContext->getExt()->subscriptionHandler = std::move(behavior.subscription);
        webSocketContext->getExt()->closeHandler = std::move([closeHandler = std::move(behavior.close)](WebSocket<SSL, true, UserData> *ws, int code, std::string_view message) mutable {
            if (closeHandler) {
                closeHandler(ws, code, message);
            }

            /* Destruct user data after returning from close handler */
            ((UserData *) ws->getUserData())->~UserData();   // <<<<<<<<<<<<<<<<<<<<<<<<<-----
        });
        webSocketContext->getExt()->pingHandler = std::move(behavior.ping);
        webSocketContext->getExt()->pongHandler = std::move(behavior.pong);

Where i have marked the destructor-call.

So, i am running into a code-path where the destructor is definitely called twice.

This is the full call-stack

[nelson] <lambda#1>::operator()(uWS::WebSocket<true, true, WebsocketData> *, int, std::string_view) App.h:383 <----------------- is calling the destructor
[nelson] std::__invoke_impl<void, <lambda#1> &, uWS::WebSocket<true, true, WebsocketData> *, int, std::string_view>(std::__invoke_other, <lambda#1> &, uWS::WebSocket<true, true, WebsocketData> *&&, int &&, std::string_view &&) invoke.h:63
[nelson] std::__invoke_r<void, <lambda#1> &, uWS::WebSocket<true, true, WebsocketData> *, int, std::string_view>(<lambda#1> &, uWS::WebSocket<true, true, WebsocketData> *&&, int &&, std::string_view &&) invoke.h:113
[nelson] std::move_only_function<void (uWS::WebSocket<true, true, WebsocketData> *, int, std::string_view)>::_S_invoke<<lambda#1> >(std::_Mofunc_base *, uWS::WebSocket<true, true, WebsocketData> *, int, std::string_view &&) mofunc_impl.h:219
[nelson] std::move_only_function<void (uWS::WebSocket<true, true, WebsocketData> *, int, std::string_view)>::operator() mofunc_impl.h:184
[nelson] uWS::WebSocket<true, true, WebsocketData>::end WebSocket.h:266  <----------------- will call destructor again once this callstack finishes.
[nelson] <lambda#1>::operator()() const server.cpp:234
[nelson] std::__invoke_impl<void, <lambda#1> &>(std::__invoke_other, <lambda#1> &) invoke.h:63
[nelson] std::__invoke_r<void, <lambda#1> &>(<lambda#1> &) invoke.h:113
[nelson] std::move_only_function<void ()>::_S_invoke<<lambda#1> >(std::_Mofunc_base *) mofunc_impl.h:219
[nelson] std::move_only_function<void ()>::operator() mofunc_impl.h:184
[nelson] uWS::Loop::wakeupCb Loop.h:50
[nelson] us_loop_run epoll_kqueue.c:147
[nelson] uWS::Loop::run Loop.h:216
[nelson] uWS::run Loop.h:233
[nelson] uWS::TemplatedApp<true>::run App.h:571
[nelson] ServerImpl::runApp server.cpp:215
[nelson] std::__invoke_impl<void, void (ServerImpl::*)(std::string), ServerImpl *, std::string> invoke.h:76
[nelson] std::__invoke<void (ServerImpl::*)(std::string), ServerImpl *, std::string> invoke.h:98
[nelson] std::thread::_Invoker<std::tuple<void (ServerImpl::*)(std::string), ServerImpl *, std::string> >::_M_invoke<0ul, 1ul, 2ul> std_thread.h:303
[nelson] std::thread::_Invoker<std::tuple<void (ServerImpl::*)(std::string), ServerImpl *, std::string> >::operator() std_thread.h:310
[nelson] std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (ServerImpl::*)(std::string), ServerImpl *, std::string> > >::_M_run std_thread.h:255
[libstdc++.so.6] std::execute_native_thread_routine 0x00007ffff64e51a4
[libasan.so.8] asan_thread_start 0x00007ffff785e29b
[libc.so.6] <unknown> 0x00007ffff62a57eb
[libc.so.6] <unknown> 0x00007ffff632918c

I setup the Websocket-functionality with

        // WebSocket handlers
        app.ws<WebsocketData>("/:0", {
            .upgrade = std::bind_front(&ServerImpl::upgradeWebsocket, this),
            .open = std::bind_front(&ServerImpl::onWebsocketOpen, this),
            .message = [](auto *ws, std::string_view msg, uWS::OpCode op)
            {
                ws->send(msg, op); // Echo
            },
            .close = std::bind_front(&ServerImpl::onWebsocketClose, this)
        });

With WebsocketData containing a std::string which is attempted to be destructed twice.

[uNetworkingAB]

Note to self:

Could be WebSocket->end followed by WebSocket->close that doesn't work.

[uNetworkingAB]

I don't remember the details but I think App.close will call WebSocket::close on all websockets. So the bug can probably be reproduced with any call to WebSocket::end followed immediately by WebSocket::close

[uNetworkingAB]

You really don't mix end and close. Either you end, or you close. Not both. So it's probably missing a check in .close.

[DerAlbiG]

Just doing appPtr->close(); crashes ASAN too.
Just doing con->end(...) is also enough to trigger the bug.

In fact, my application never reaches appPtr->close() with ASAN enabled because con->end() kills it via ASAN.
So, its not about the mixing.

I just wanted to provide a nice close-code and a reason. That is why I wanted to close the connections manually.
I feel like con->end(code, reason) is fundamentally broken as soon as WebsocketData has non-trivial members.
So far I commented out the destructor-call at the end of end(). I rather deal with a leak than a double free.

[uNetworkingAB]

can you post a minimal reproducer in full

[DerAlbiG]

Give me a few minutes please. Thats... uff.

[DerAlbiG]

Connect to it via wscat -c ws://localhost:1234 or a similar tool. As soon as you connect it will defer a close operation with code and reason. While doing so, it will destruct WebsocketData twice. Either you compile with ASAN and crash directly, or the destructor of WebsocketData triggers the __builint_trap()
The crash occurs with #if 1 or #if 0. It does not matter.

//
// Created by albi on 13.06.25.
//

#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wdeprecated-declarations"
#pragma GCC diagnostic ignored "-Wshadow"
#pragma GCC diagnostic ignored "-Wnrvo"
#include "App.h"
#pragma GCC diagnostic pop

#include <sanitizer/asan_interface.h>
#include <string>
#include <iostream>


int instanceCount = 0;

struct WebsocketData
{
    std::string string;
    volatile bool destructed = false;
    WebsocketData():
        string("default-constructed-"+std::to_string(instanceCount++))
    {
        std::cout << "WebsocketData " << string << std::endl;
    }

    WebsocketData(const WebsocketData& other):
        string("Copy of \"" + other.string+"\"")
    {
        std::cout << "WebsocketData " << string << std::endl;
    }
    WebsocketData(WebsocketData&& other) noexcept:
        string("Moved from \"" + other.string + "\"")
    {
        std::cout << "WebsocketData " << string << std::endl;
        other.string = "empty/moved from";
    }
    ~WebsocketData()
    {
        std::cout << "WebsocketData destructed: " << string << std::endl;
        if (destructed)
            __builtin_trap();
        destructed = true;
    }
    WebsocketData& operator=(const WebsocketData& other) = delete;
    WebsocketData& operator=(WebsocketData&& other) = delete;
};

uWS::WebSocket<false, true, WebsocketData>* connection = nullptr;
uWS::App* appPtr = nullptr;

int main(const int argc, const char *argv[])
{
#ifdef __SANITIZE_ADDRESS__
    __asan_set_error_report_callback([](const char*) { __builtin_trap(); });
#endif
    uWS::App app{};
    appPtr = &app;
    app.ws<WebsocketData>("/*",
    {
        .open = [](uWS::WebSocket<false, true, WebsocketData> *ws)
        {
            std::cout << "Client connected" << std::endl;
            connection = ws;
            uWS::Loop::get()->defer([]
            {
#if 1 
                if (connection)
                    connection->end(1000, "Bye Bye");
#else
                if (appPtr)
                    appPtr->close();
#endif
            });
        },
        .close = [](uWS::WebSocket<false, true, WebsocketData> *ws, int, std::string_view)
        {
            std::cout << "Client disconnected" << std::endl;
            connection = nullptr;
        }
    }).listen(1234, [](auto *listenSocket)
        {
            if (listenSocket)
                std::cout << "Server listening on port 1234" << std::endl;
            else
                std::cerr << "Failed to listen on port 1234" << std::endl;
        }).run();

    appPtr = nullptr;
    return 0;
}

[DerAlbiG]

Pls let me know if this replicates for you.