exit-node: fix RST/EPERM spam and yandex transport handshake race - #54
Open
D13tre33 wants to merge 2 commits into
Open
exit-node: fix RST/EPERM spam and yandex transport handshake race#54D13tre33 wants to merge 2 commits into
D13tre33 wants to merge 2 commits into
Conversation
The exit node's TCP stack lives entirely in userspace (gVisor), so the kernel has no socket for these connections and fires its own spurious RSTs at them - entrypoint.sh's OUTPUT DROP rule exists specifically to swallow those. But that rule matched *all* outbound RSTs, including the ones WritePackets legitimately sends via the raw socket to close real connections on the application's behalf. A local netfilter DROP on a raw-socket send surfaces back to the caller as EPERM rather than silently dropping, which is what produced the recurring "[RAW-NIC2] Sendto failed: operation not permitted" errors - observed firing over a thousand times in 5 minutes under real traffic. Fix: tag every packet sent through the raw socket with SO_MARK=100 (rawSocketMark), and exempt that mark from the DROP rule in entrypoint.sh via a RETURN rule that runs first. Now only truly kernel-originated RSTs (unmarked, since they never go through our socket) get dropped; our own legitimate RSTs go out normally. Verified on a live exit node: EPERM count went from ~1166/5min to 0/5min under equivalent traffic, with iptables counters confirming the split (RETURN rule matching our marked RSTs, DROP rule still catching the kernel's unmarked ones).
connectToDoc() was writing the socket.io "40{token}" connect packet
and the "42[...]" auth event immediately after the WebSocket upgrade
completed, without ever reading anything from the server first. Per
the engine.io/socket.io protocol the server sends its own OPEN packet
("0{"sid":...,"pingInterval":...}") first, and this raced it: writing
our packets before reading that OPEN packet got the connection closed
by the server with "websocket: close 1005 (no status)" within roughly
50-100ms of connecting, almost every time.
Confirmed by logging otherwise-unhandled incoming messages: the
server's "0{...}" packet was consistently the very next thing to
arrive, immediately followed by the close - i.e. the server appears to
treat receiving our packets before its own handshake packet as a
protocol violation and drops the connection.
Fix: block on a single ReadMessage() for that OPEN packet before
sending "40"/"42". With this, handshakes now reliably complete (auth
ack, license, waitAuth, documentOpen all arrive in order) and real
tunnel traffic flows - previously this only happened by chance, when
the write and the server's OPEN packet happened to land in the right
order on their own.
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two correctness bugs in the exit-node path, found while running a
local exit node for a while and watching the logs closely.
1.
[RAW-NIC2] Sendto failed: operation not permittedspam (tunnel/rawsocket_linux.go,docker/entrypoint.sh)The exit node's TCP stack runs entirely in userspace (gVisor), so the
kernel has no socket for these connections and answers with its own
spurious RSTs -
entrypoint.sh'sOUTPUTDROP rule exists to swallowexactly those. But it matched all outbound RSTs, including the ones
WritePacketslegitimately sends via the raw socket to close realconnections. A local netfilter DROP on a raw-socket send surfaces back
to the caller as
EPERMinstead of silently dropping - that's thesource of the repeated log line. Under real traffic this fired well
over a thousand times in 5 minutes.
Fix: mark every packet the raw socket sends with
SO_MARK=100, andadd a
RETURNrule ahead of the DROP rule that exempts that mark.Only truly kernel-originated RSTs (never marked, since they don't go
through our socket) still get dropped.
Verified live: EPERM count went from ~1166/5min to 0/5min at
equivalent traffic volume; iptables counters confirm the split
(RETURN rule catching our marked RSTs, DROP rule still catching the
kernel's unmarked ones).
2. yandex transport: handshake race causes near-instant
close 1005(transport/yandex/yandex.go)connectToDoc()sent the socket.io40{token}connect packet and the42[...]auth event immediately after the WebSocket upgrade, withoutever reading anything from the server first. The server sends its own
engine.io OPEN packet (
0{"sid":...}) first per protocol - writingbefore reading that raced it, and the server closed the connection
with
websocket: close 1005 (no status)within ~50-100ms almost everytime. This meant a working session only ever happened by luck, when
the timing happened to land in the right order on its own.
Confirmed by temporarily logging otherwise-unhandled incoming
messages: the OPEN packet consistently arrived right before the close.
Fix: block on one
ReadMessage()for the OPEN packet before sending40/42. Handshakes now reliably complete (auth ack, license,waitAuth, documentOpen all arrive in order) and real tunnel traffic
flows afterward.
Test plan
CGO_ENABLED=0 go buildsucceeds (built via the repo's ownDockerfile)
exit-nodecompose profile against a realYandex Disk share link; confirmed via
docker logs+iptables -L OUTPUT -n -vthat RSTs split correctly between theRETURN and DROP rules under real traffic
documentOpenandcarries real tunnel traffic, where before it essentially never
did deterministically
Not touched: a separate, deeper issue where even a successful yandex
session eventually gets an explicit server-side
disconnectReason(code 4007, "drop") after roughly a minute - that looks like Yandex's
own collab-session participant limits, not a client bug, and is out of
scope here.