Description
Info
I am trying to perform some tests but tperf seems to crash as soon as the Client connects to the server.
This was not a problem a couple months ago (specifically on Commit 4b03b63 (March 19th))
When I try to perform local measurements using tperf (e.g. via localhost) it works fine on both ends (Server&Client).
When trying to use another OS for the Client (On Ubuntu 20) the Server-Side still crashes which leads me to believe the problem lies somewhere with Debian 12.
The Current Setup is as follows:
On Commit: df1b77c (On Client and Server)
OS-Info: (Server)
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
OS-Info (Client):
NAME="Fedora Linux"
VERSION="39 (Workstation Edition)"
ID=fedora
VERSION_ID=39
VERSION_CODENAME=""
PLATFORM_ID="platform:f39"
PRETTY_NAME="Fedora Linux 39 (Workstation Edition)"
The Server & Client Outputs are as follows:
Server output:
$ ./tperf -mode=server -host=0.0.0.0 -v=4
I20240505 13:34:54.775065 3747126 QuicServer.cpp:311] No valid takenover fd found for address=0.0.0.0:6666. binding on worker=0x55b328683d70 workerId=0 processId=0
I20240505 13:34:54.775225 3747126 QuicServer.cpp:322] Initialized all workers in the eventbase
I20240505 13:34:54.775493 3747126 tperf.cpp:167] TPerfAcceptObserver attached
I20240505 13:34:54.775604 3747125 tperf.cpp:497] tperf server started at: 0.0.0.0:6666
I20240505 13:34:57.816308 3747126 QuicServerWorker.cpp:882] Creating new connection for client=2.210.213.55:4543, routingInfo=CID=746e50ebcc63a8e2, cidVersion=1, workerId=0, processId=0, hostId=0, threadId=140222459406016, cidVersion in packet=1, workerId in packet=67, processId in packet=1, hostId in packet=53689,
I20240505 13:34:57.819614 3747126 QuicServerWorker.cpp:1290] Adding into connectionIdMap_ for CID=4000001da19ad191 client CID= server CID=4000001da19ad191 peer address=2.210.213.55:4543
I20240505 13:34:57.819643 3747126 tperf.cpp:239] Starting sends to client.
I20240505 13:34:57.819713 3747126 QuicPacketScheduler.cpp:815] Wrote crypto frame offset=0 bytesWritten=90 client CID= server CID=4000001da19ad191 peer address=2.210.213.55:4543
I20240505 13:34:57.819876 3747126 QuicPacketScheduler.cpp:815] Wrote crypto frame offset=0 bytesWritten=713 client CID= server CID=4000001da19ad191 peer address=2.210.213.55:4543
I20240505 13:34:57.819984 3747126 QuicPacketScheduler.cpp:558] Wrote stream frame stream=3 offset=0 bytesWritten=1430 finWritten=0 client CID= server CID=4000001da19ad191 peer address=2.210.213.55:4543
I20240505 13:34:57.820039 3747126 QuicPacketScheduler.cpp:558] Wrote stream frame stream=3 offset=1430 bytesWritten=1428 finWritten=0 client CID= server CID=4000001da19ad191 peer address=2.210.213.55:4543
I20240505 13:34:57.820072 3747126 QuicPacketScheduler.cpp:558] Wrote stream frame stream=3 offset=2858 bytesWritten=1428 finWritten=0 client CID= server CID=4000001da19ad191 peer address=2.210.213.55:4543
I20240505 13:34:57.820120 3747126 QuicPacketScheduler.cpp:558] Wrote stream frame stream=3 offset=4286 bytesWritten=1428 finWritten=0 client CID= server CID=4000001da19ad191 peer address=2.210.213.55:4543
I20240505 13:34:57.820158 3747126 QuicPacketScheduler.cpp:558] Wrote stream frame stream=3 offset=5714 bytesWritten=1428 finWritten=0 client CID= server CID=4000001da19ad191 peer address=2.210.213.55:4543
I20240505 13:34:57.820201 3747126 QuicPacketScheduler.cpp:558] Wrote stream frame stream=3 offset=7142 bytesWritten=1428 finWritten=0 client CID= server CID=4000001da19ad191 peer address=2.210.213.55:4543
I20240505 13:34:57.820233 3747126 QuicPacketScheduler.cpp:558] Wrote stream frame stream=3 offset=8570 bytesWritten=1428 finWritten=0 client CID= server CID=4000001da19ad191 peer address=2.210.213.55:4543
I20240505 13:34:57.820304 3747126 IoBufQuicBatch.cpp:146] Error writing to the socket Input/output error, 2.210.213.55:4543
I20240505 13:34:57.821170 3747126 QuicTransportBase.cpp:3201] writeSocketDataAndCatchError on socket write Input/output error, client CID= server CID=4000001da19ad191 peer address=2.210.213.55:4543
I20240505 13:34:57.821202 3747126 QuicTransportBase.cpp:2886] Clearing datagram callback
I20240505 13:34:57.821210 3747126 QuicTransportBase.cpp:2889] Clearing ping callback
I20240505 13:34:57.821218 3747126 QuicTransportBase.cpp:2892] Clearing 0 peek callbacks
E20240505 13:34:57.821234 3747126 tperf.cpp:231] Conn errorCoded=Internal Error, errorMsg=Error on socket write Input/output error,
F20240505 13:34:57.821369 3747126 tperf.cpp:265] Connection closed
*** Check failure stack trace: ***
*** Aborted at 1714908897 (Unix time, try 'date -d @1714908897') ***
*** Signal 6 (SIGABRT) (0x392d35) received by PID 3747125 (pthread TID 0x7f8815e1e6c0) (linux TID 3747126) (maybe from PID 3747125, UID 0) (code: -6), stack trace: ***
@ 000000000017cdd7 folly::symbolizer::(anonymous namespace)::signalHandler(int, siginfo_t*, void*)
/tmp/fbcode_builder_getdeps-ZhomeZusernameZmvfstZbuildZfbcode_builder-root/repos/github.com-facebook-folly.git/folly/experimental/symbolizer/SignalHandler.cpp:453
@ 000000000003c04f (unknown)
@ 000000000008ae2c (unknown)
@ 000000000003bfb1 gsignal
@ 0000000000026471 abort
@ 0000000000070c9d folly::(anonymous namespace)::wrapped_abort()
/tmp/fbcode_builder_getdeps-ZhomeZusernameZmvfstZbuildZfbcode_builder-root/repos/github.com-facebook-folly.git/folly/init/Init.cpp:62
@ 000000000000efcc google::LogMessage::Fail()
/tmp/fbcode_builder_getdeps-ZhomeZusernameZmvfstZbuildZfbcode_builder-root/extracted/glog-v0.5.0.tar.gz/glog-0.5.0/src/logging.cc:1920
@ 00000000000114ed google::LogMessage::SendToLog()
/tmp/fbcode_builder_getdeps-ZhomeZusernameZmvfstZbuildZfbcode_builder-root/extracted/glog-v0.5.0.tar.gz/glog-0.5.0/src/logging.cc:1877
@ 000000000000eb17 google::LogMessage::Flush()
/tmp/fbcode_builder_getdeps-ZhomeZusernameZmvfstZbuildZfbcode_builder-root/extracted/glog-v0.5.0.tar.gz/glog-0.5.0/src/logging.cc:1737
@ 0000000000011a68 google::LogMessageFatal::~LogMessageFatal()
/tmp/fbcode_builder_getdeps-ZhomeZusernameZmvfstZbuildZfbcode_builder-root/extracted/glog-v0.5.0.tar.gz/glog-0.5.0/src/logging.cc:2496
@ 00000000000bbd61 quic::tperf::ServerStreamHandler::notifyDataForStream(unsigned long)::{lambda()#1}::operator()() const
/home/username/mvfst/quic/tools/tperf/tperf.cpp:265
@ 00000000000fdb2a folly::EventBase::FunctionLoopCallback::runLoopCallback()
/tmp/fbcode_builder_getdeps-ZhomeZusernameZmvfstZbuildZfbcode_builder-root/repos/github.com-facebook-folly.git/folly/Function.h:370
-> /tmp/fbcode_builder_getdeps-ZhomeZusernameZmvfstZbuildZfbcode_builder-root/repos/github.com-facebook-folly.git/folly/io/async/EventBase.cpp
@ 00000000000f7598 folly::EventBase::runLoopCallbacks(boost::intrusive::list<folly::EventBase::LoopCallback, boost::intrusive::constant_time_size >&)
/tmp/fbcode_builder_getdeps-ZhomeZusernameZmvfstZbuildZfbcode_builder-root/repos/github.com-facebook-folly.git/folly/io/async/EventBase.cpp:913
@ 00000000000f7718 folly::EventBase::runLoopCallbacks()
/tmp/fbcode_builder_getdeps-ZhomeZusernameZmvfstZbuildZfbcode_builder-root/repos/github.com-facebook-folly.git/folly/io/async/EventBase.cpp:932
@ 00000000000f9407 folly::EventBase::loopMain(int, folly::EventBase::LoopOptions)
/tmp/fbcode_builder_getdeps-ZhomeZusernameZmvfstZbuildZfbcode_builder-root/repos/github.com-facebook-folly.git/folly/io/async/EventBase.cpp:586
@ 00000000000f9d6d folly::EventBase::loopBody(int, folly::EventBase::LoopOptions)
/tmp/fbcode_builder_getdeps-ZhomeZusernameZmvfstZbuildZfbcode_builder-root/repos/github.com-facebook-folly.git/folly/io/async/EventBase.cpp:474
@ 00000000000f9e07 folly::EventBase::loop()
/tmp/fbcode_builder_getdeps-ZhomeZusernameZmvfstZbuildZfbcode_builder-root/repos/github.com-facebook-folly.git/folly/io/async/EventBase.cpp:435
@ 00000000000fbed6 folly::EventBase::loopForever()
/tmp/fbcode_builder_getdeps-ZhomeZusernameZmvfstZbuildZfbcode_builder-root/repos/github.com-facebook-folly.git/folly/io/async/EventBase.cpp:732
@ 0000000000404ba7 folly::run(folly::EventBaseManager*, folly::EventBase*, folly::Baton<true, std::atomic>, folly::Range<char const> const&)
/tmp/fbcode_builder_getdeps-ZhomeZusernameZmvfstZbuildZfbcode_builder-root/repos/github.com-facebook-folly.git/folly/io/async/ScopedEventBaseThread.cpp:40
@ 00000000000d44a2 (unknown)
@ 0000000000089133 (unknown)
@ 00000000001097db (unknown)
Client output:
$ ./tperf -mode=client -host=xxx.xxx.xxx.xxx -v=4
I0505 13:34:57.794664 156127 QuicClientTransport.cpp:86] initial dcid: 746e50ebcc63a8e2
I0505 13:34:57.794744 156127 tperf.cpp:710] TPerfClient connecting to xxx.xxx.xxx.xxx:6666
I0505 13:34:57.797061 156127 QuicPacketScheduler.cpp:815] Wrote crypto frame offset=0 bytesWritten=209 client CID= server CID=None peer address=xxx.xxx.xxx.xxx:6666
I0505 13:34:57.836753 156127 QuicReadCodec.cpp:404] Client cannot read key phase zero packet
I0505 13:34:57.836798 156127 QuicClientTransport.cpp:590] Client closed by peer reason=writeSocketDataAndCatch() error client CID= server CID=4000001da19ad191 peer address=xxx.xxx.xxx.xxx:6666
I0505 13:34:57.836817 156127 QuicTransportBase.cpp:2886] Clearing datagram callback
I0505 13:34:57.836822 156127 QuicTransportBase.cpp:2889] Clearing ping callback
I0505 13:34:57.836823 156127 QuicTransportBase.cpp:2892] Clearing 0 peek callbacks
E0505 13:34:57.836830 156127 tperf.cpp:636] TPerfClient error: Internal Error
E0505 13:34:57.836993 156127 QuicTransportBase.cpp:434] close threw exception Cannot encrypt (insufficient space for tag) client CID= server CID=4000001da19ad191 peer address=xxx.xxx.xxx.xxx:6666
E0505 13:34:57.838029 156127 EventHandler.cpp:90] EventBase: failed to register event handler for fd 9: Resource temporarily unavailable
Any help would be greatly appreciated.
Thanks in advance!
Activity