[Bug] Node deletion does not close the active long-poll session; the client gets no notification and cannot recover without manual re-auth

[nekohasekai]

Is this a support request?

• This is not a support request

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Headscale can delete a node while the node's client is connected. Examples:
headscale nodes delete, ephemeral garbage collection. When this occurs,
headscale does not close the node's active /machine/map long-poll session.
This causes three failures. We reproduced all three failures with official
tailscaled 1.98.10 and headscale v0.29.3.

1. The client of the deleted node becomes a zombie. The mapSession
   continues to run. It continues to send keep-alives. (The keep-alive ticker
   in hscontrol/poll.go is independent of the batcher.) Thus the client
   thinks that it is connected and healthy. tailscale status shows the
   tailnet as up. tailscaled logs nothing. The poll TCP session stays
   ESTABLISHED on the two sides. We observed this state for minutes while
   headscale nodes list was already empty. Headscale never tells the client
   to authenticate again. The client never registers again.

2. After the long-poll breaks, the client cannot recover. We restarted
   headscale. The client sent a new poll to /machine/map with its old node
   key. Headscale replied with a bare HTTP 404: node not found. Upstream
   tailscaled thinks that each non-200 map response is a temporary error. It
   retries forever with backoff. It stays loggedIn=true. It never goes to
   the NeedsLogin state:

   2026/08/03 17:17:23 control: PollNetMap: initial fetch failed 404: node not found
   2026/08/03 17:17:23 control: sendStatus: mapRoutine1: loggedIn=true inMapPoll=false
   2026/08/03 17:17:23 control: mapRoutine: loggedIn=true
   2026/08/03 17:17:23 Received error: PollNetMap: initial fetch failed 404: node not found
   2026/08/03 17:17:23 control: mapRoutine: backoff: 8503 msec
   

   We counted 56 iterations in 4 minutes. The loop does not stop. The node
   stays dead until a person runs tailscale up --force-reauth. That command
   recovers the node in seconds.

3. The orphan session blocks graceful shutdown. The session of the
   deleted node stayed open. We sent SIGTERM to headscale. Headscale did not
   exit in 30 seconds. A normal shutdown takes approximately 1 second. We had
   to send SIGKILL.

The direct cause of failure (1) is in hscontrol/mapper/batcher.go, function
addToBatch. On change.NodeRemoved, the batcher only does
b.nodes.LoadAndDelete(removedID). It removes its bookkeeping entry. It does
not close the node's mapSession. It does not notify the client.

Impact: a routine restart or upgrade of headscale becomes a permanent outage
for each node that headscale deletes during the downtime. The most common
example is an ephemeral node that the garbage collector removes. These nodes
reconnect into the 404 loop. They do not come back without manual repair.
(A tsnet-based client reported this downstream as SagerNet/sing-box#4371.
This report shows that official tailscaled has the same behavior.)

Expected Behavior

• When headscale deletes a node, headscale immediately closes the node's
  active long-poll session.
• When a client sends a node key that headscale does not know, headscale
  sends a protocol-level rejection. The rejection makes the client go to the
  NeedsLogin state. The official control plane does this when an
  administrator deletes a device. A bare 404 is not sufficient, because
  upstream clients retry it forever.
• Sessions of deleted nodes do not block graceful shutdown.

Steps To Reproduce

All components run on localhost of one Linux machine (an OrbStack Linux
machine on macOS). There is no reverse proxy. This is the full headscale
configuration:

server_url: http://127.0.0.1:8080
listen_addr: 127.0.0.1:8080
metrics_listen_addr: 127.0.0.1:9090
grpc_listen_addr: 127.0.0.1:50443
private_key_path: ./private.key
noise:
  private_key_path: ./noise_private.key
prefixes:
  v4: 100.64.0.0/10
  v6: fd7a:115c:a1e0::/48
database:
  type: sqlite
  sqlite:
    path: ./db.sqlite
derp:
  server:
    enabled: false
  urls:
    - https://controlplane.tailscale.com/derpmap/default
  auto_update_enabled: false
  update_frequency: 24h
disable_check_updates: true
dns:
  magic_dns: false
  base_domain: example.com
  nameservers:
    global:
      - 1.1.1.1
unix_socket: ./headscale.sock
logtail:
  enabled: false

1. Start headscale serve with the configuration above. Create a user:
   headscale users create test. Create a reusable pre-auth key:
   headscale preauthkeys create --user 1 --reusable --expiration 24h.

2. Start tailscaled --tun=userspace-networking. Run
   tailscale up --login-server=http://127.0.0.1:8080 --auth-key=<key>.
   The node registers and is online.

3. Run headscale nodes delete -i 1 --force while the client is connected
   (17:09:27 in the logs below).

4. Wait 150 seconds. Then examine the system. tailscale status shows a
   healthy node. The poll session stays ESTABLISHED. headscale nodes list
   is empty. The tailscaled log shows no disconnect and no error:

   17:12:06 --- tailscale status:
   100.64.0.1  arch  test  linux  -
   17:12:06 --- headscale nodes list: (empty)
   17:12:06 --- ss -tnp | grep 8080:
   ESTAB 0 0 127.0.0.1:8080  127.0.0.1:40388 users:(("headscale",pid=14314,fd=16))
   ESTAB 0 0 127.0.0.1:40388 127.0.0.1:8080  users:(("tailscaled",pid=14367,fd=16))
   

5. Send SIGTERM to headscale. Headscale continues to run for more than 30
   seconds. Send SIGKILL. Start headscale again.

6. tailscaled goes into the initial fetch failed 404: node not found
   backoff loop shown above. The loop continues without limit. We observed
   it for more than 4 minutes. The maximum backoff is approximately 30
   seconds. tailscale status shows the node as offline.

7. Run tailscale up --auth-key=<key> --force-reauth. The node registers
   again and is online in seconds.

Environment

• OS: Arch Linux (aarch64), kernel 7.0.11 (OrbStack VM)
• Headscale version: 0.29.3 (official release binary)
• Tailscale version: 1.98.10
• Number of nodes: 1 (minimal reproduction)

Runtime environment

• Headscale is behind a (reverse) proxy
• Headscale runs in a container

Debug information

Full timeline from the reproduction logs:

17:09:26 node online, connected
17:09:27 headscale nodes delete -i 1 --force  → "Node deleted"
17:12:06 client still healthy, session ESTABLISHED, node list empty,
         tailscaled log silent since 17:09:16
17:12:48 SIGTERM headscale
17:13:18 headscale still running (30 s) → SIGKILL
17:13:33 headscale restarted
17:13:3x → 17:17:42 tailscaled loops "initial fetch failed 404: node not
         found" (56 times), loggedIn=true throughout, never re-registers
17:18:03 tailscale up --force-reauth → online within seconds

tailscaled ran with --verbose=1. headscale ran with the default log level.
We can supply the full logs on request.