feat(swarm): implement autoscaling (scale to zero/one) for Docker Swarm services by SCBionicle · Pull Request #564 · itzg/mc-router

SCBionicle · 2026-07-06T23:39:06Z

This pull request implements full auto-scaling support (scale-to-zero and scale-to-one) for Docker Swarm services, supporting both VIP (Virtual IP) and DNSRR (DNS Round-Robin) modes.

The Problem: Premature Connection Severing in VIP Mode

In Swarm's standard VIP mode, Swarm manages traffic through kernel-level load balancing (IPVS). When a service replica scales down to 0, Swarm immediately cuts all active TCP streams routed through the VIP. This ignores the Minecraft server's stop timeout/grace period, preventing it from performing a clean shutdown (e.g. broadcasting warnings, saving player states).

The Solution: Direct-to-Task Routing

To solve this, the Swarm autoscaler bypasses the VIP when scaling is active by discovering and routing directly to the task container's overlay IP:

When a service is scaled down to 0, its route is registered in the routing table with an empty backend (""), which instantly serves the "Asleep" MOTD on status checks without waiting for network timeouts.
When a player logs in, the Waker scales the service up to 1, polls the Swarm Task API (TaskList) to wait for a task to transition to running, extracts the task's concrete overlay IP, dials it to ensure it is healthy, and routes the player connection directly to it.
Because the connection is direct, Swarm does not sever the TCP socket when a scale-down command is issued. The Minecraft server can utilize its full graceful shutdown grace period before being stopped.

Detailed Changes

1. Configuration & Downscaler Wiring

server/server.go: Enables the connection DownScaler when InDockerSwarm is active.

2. Swarm Watcher Updates (`server/docker_swarm.go`)

Struct Decoupling: Introduced a Swarm-specific routableSwarmService struct to cleanly encapsulate Swarm metadata without coupling to the Kubernetes data models.
Autoscale Label Parsing: Updated parseServiceData and listServices to accept ResolutionModeDNSRR as a valid endpoint resolution mode, read autoscale labels (mc-router.auto-scale-up, mc-router.auto-scale-down, and custom asleep/loading MOTDs), and check if current replicas are 0 (setting backend IP to "").
Waker (makeWakerFunc):
- Automatically scales the Swarm service replicas from 0 to 1 when accessed.
- Polls the Swarm TaskList API to wait for the task state to enter running.
- Extracts the container overlay IP from the task's NetworksAttachments.
- Verifies health via TCP dial before returning the direct task address to the connector.
Sleeper (makeSleeperFunc): Scales down the replicated service replicas from 1 to 0 once all active connections have closed and the idle period (DownAfter) has passed.
Reconciliation (reconcileServices): Registered Swarm wakers, sleepers, custom MOTDs, and used the Swarm serviceID as the scalingTarget to ensure correct connection counting.

Update (July 8, 2026)

After testing the initial implementation, we refined the state machine evaluation to handle Docker Swarm's asynchronous scheduling behavior and resolved two transient race conditions:

1. State Machine Refactoring (Actual vs. Desired States)

Desired State Synchronization: We updated the waker and watcher to inspect both the task's actual state (task.Status.State) and desired state (task.DesiredState).
- A task is now only considered Running if both actual and desired states are running (preventing routing to stopping tasks).
- A task is identified as in Restart Delay if both actual and desired states are ready.
- We detect if Swarm has Given Up on scheduling by checking if the latest task's desired state is shutdown (with no other active tasks present).

2. Resolving Scale-Up Race Conditions (Stale History)

The Problem: Right after scaling up (replicas 0 -> 1), Swarm takes 1–2 seconds to create the new task record. During this window, querying TaskList only returned stale tasks from the previous run. Because the latest old task was shutdown, the waker instantly failed.
The Solution: We introduced timestamp-based filtering. The waker records filterTime = time.Now() right before updating the service (and the watcher uses service.Meta.UpdatedAt). We ignore any stale terminal tasks created before filterTime. The waker now cleanly waits for the new task to be scheduled.

3. VIP Propagation Bypass (Direct Task Routing)

Since Minecraft services run as a single instance (replicas == 1), we bypass the 5-10s IPVS load balancer VIP propagation delay by resolving and routing directly to the running container's direct overlay IP. Players and server status pings connect instantly without transient connection refused warnings during the start-up transition.

4. Documentation & MOTD Polishing

Swarm States Documentation: Added docs/docker-swarm-states.md mapping Docker Swarm's task states to mc-router routing actions for developer reference.
Default Failed MOTD: Defaults to "Server failed to start." if mc-router.auto-scale-failed-motd is not set.

The above is vibe coded and is being tested on my own server set up before being unmarked as a draft.

Fixes #561

…esolution

itzg · 2026-07-07T01:02:46Z

Took a really quick look and am liking that the changes are contained almost entirely within docker swarm file. Keep me posted as you verify further.

SCBionicle · 2026-07-07T01:06:22Z

Thank you. Fixing a routing bug right now where the script routed to an IP of the first network it saw.

I did notice a possible bug while working on this that docker socket path does not have a default and seems to need to be explicitly defined, at least in swarm mode. I'm not going to fix that in this PR because I'm trying to keep the scope of the PR narrow, but it's something worth looking out for and/or posting in the documentation.

itzg · 2026-07-07T01:16:48Z

I did notice a possible bug while working on this that docker socket path does not have a default and seems to need to be explicitly defined, at least in swarm mode.

Thanks for keeping the PR scope narrow. If you get a chance, can you open an issue track this?

SCBionicle · 2026-07-07T15:55:37Z

If you get a chance, can you open an issue track this?

I have opened an issue at #567.

Regarding progress so far. VIP mode for auto spool down and auto spool up is working perfectly in my live environment. However, I have yet to test DNSRR endpoint mode and I'm running into the issue where the sleeping and waking custom MOTD are not displaying in the client. So, that's what I'm working on now.

…low status interception

SCBionicle · 2026-07-07T16:27:42Z

Okay, so custom MOTD fixed and tested on my system. So, the last thing I need to test is DNSRR deployments and then this PR would be ready for review.

SCBionicle · 2026-07-07T17:00:06Z

@itzg While testing the auto-scaling implementation on a live cluster with a heavy modpack (NeoForge), two edge cases were identified that would make the Swarm autoscaler much more production-ready:

1. Configurable Startup Timeout (via Service Label)

The issue: Heavy modpacks can take 2–3 minutes to boot up and bind to the TCP port. The current hardcoded 60s timeout is too short for these servers, causing the waker to give up prematurely.
The solution: Introduce a mc-router.auto-scale-wait-timeout service label (e.g., "5m"). This allows admins to define how long the router waits for slow-starting servers, matching the existing Kubernetes annotation mc-router.itzg.me/autoScaleWaitTimeout.

2. Failed-Task / Crash-Loop Detection (Fail-Fast)

The issue: If the container is permanently misconfigured and crashlooping, we want the router to fail fast if Swarm gives up.
The solution: In the polling loop, query all tasks for the service. If a finite restart policy is configured (e.g., max_attempts: 3) and has been exhausted, Swarm will stop spawning new tasks. The router can detect that there are no active tasks left in the scheduler (new, pending, preparing, running, etc.) and fail immediately rather than making the player wait for the full timeout duration.

To further improve player UX:

3. Custom MOTD for Exhausted / Failed State

The issue: If a server fails to start, players currently see the generic "Sleeping" or "Loading" MOTD on their server list, leading them to repeatedly try joining a broken server.
The solution: Introduce a mc-router.auto-scale-failed-motd label (e.g., "Server failed to start. Contact admin!"). If the watcher detects that the service is scaled to 1 but all tasks have terminated (exhausted restart policy), it maps the backend to "" and displays this custom failed MOTD.

Are you okay for me to tack this into this PR?

itzg · 2026-07-07T18:13:09Z

Are you okay for me to tack this into this PR?

Yes, that all sounds quite reasonable. As you can guess, purposely avoid implementing the same for regular docker and kubernetes modes. That can be addressed separately/later.

…sing

…rash loop

…untdowns

…rser

… TaskEventType events

…late for dynamic formatting

…letions

SCBionicle · 2026-07-08T00:13:45Z

I got most of the core logic down, but still working through edge cases.

- Refactor Swarm task parsing to inspect both task.Status.State and task.DesiredState. - Treat a task as fully running only if both actual and desired states are "running". - Trust the "ready" state (actual or desired) directly to identify tasks in a restart delay. - Track the most recently created task to detect when Swarm has given up (latestTask.DesiredState == "shutdown"). - Add a default fallback permanently failed MOTD ("Server failed to start.") when no custom MOTD is configured.

SCBionicle · 2026-07-11T02:19:02Z

@itzg I got a question about your vision regarding the join process when the server is sleeping or waking. Were you preferring the UX where the player waits in the connecting screen while the server wakes up or were you envisioning players being immediately disconnected if the server isn't ready as part of a failfast approach?

Reason I ask is that I cannot fully eliminate race conditions in the Swarm autoscale up to make it smooth and my testing environment guarentees that the server takes more than 30 seconds to start (from scale up to game server ready).

itzg · 2026-07-11T02:37:08Z

Personally I don't feel strongly either way. It looks like the Minecraft client will only wait 30 seconds for the server to respond, so it's inherently a risky assumption that a waking server would startup within 30 seconds anyway. So failing fast on the connection is actually a more deterministic UX anyway.

If the behavior needs to different (or aimpler) for Docker Swarm mode then that's ok.

…r stateless

SCBionicle · 2026-07-12T16:31:54Z

Progress

Core Routing

Routes to server while server is healthy: Pass (automatically bypasses VIP loadbalancer)
Holds connections while server is stopping: Pass
Holds connections while server is starting: Pass
Wakes server while server is sleeping: Pass
Autoscales Down Server when Connections are idle at timeout: Pass
Allows currently connected players to remain connected to server while server is in graceful shutdown state while preventing new connections: Pass

MOTD Messages

Shows server MotD while server is healthy: Pass (will always show the server MotD if it is accepting routes to the server)
Shows waking MotD while server is starting or pending: Pass
Shows restart delay MotD while restart delay in progress: Pass
Shows permanently failed MotD if Swarm is not scheduling or planning new service start: Pass
Shows sleeping MotD while while server is sleeping (replicas=0): Pass

Fast Fail is not implemented. The feature is now done. I just need to update the docs and clean up, and this is ready.

itzg · 2026-07-12T17:07:11Z

Thanks for the update @SCBionicle . Move out of draft when you're ready.

SCBionicle · 2026-07-13T16:46:10Z

@itzg, just following up. I have marked this PR as ready for review.

itzg · 2026-07-13T16:48:56Z

Thanks for letting me know @SCBionicle . Apparently my assumption was wrong that github would notify me when a PR goes to ready.

itzg · 2026-07-15T23:49:23Z

As you've probably seen from the govulncheck, the docker client module seems to have a CVE they're never going to fix. As such I am pondering switch to the moby/client module in #568 . The non-Swarm code seems fairly simple to switch over with mainly argument changes for options, etc. Any thoughts on the impact for the Swarm operations?

itzg

Looks great! Just the refactoring of that lengthy parseServiceData and I'm good to merge.

itzg · 2026-07-16T00:04:19Z

-	for _, network := range service.Spec.TaskTemplate.Networks {
-		networkAliases[network.Target] = network.Aliases
-	}
+func (w *dockerSwarmWatcherImpl) parseServiceData(ctx context.Context, service *swarm.Service, networkMap map[string]*network.Inspect) (data parsedDockerServiceData, ok bool) {


This function has now gotten pretty long. Can you refactor it out just slightly into a couple or so functions? I realized after a while that I was still reading the content of the same function 😄

I can do that once I get the chance.

I was wondering what was going on, didn't get an email, and decided to check up on this PR 🫤. Looks like Github isn't notifying either of us. 😅

I also don't mind refactoring it in a follow up. The Swarm logic seems get intense pretty quick, so it's not like the length of the function was unexpected.

I was wondering what was going on, didn't get an email, and decided to check up on this PR 🫤. Looks like Github isn't notifying either of us. 😅

Very strange.

I probably need to give it a stricter look over because there's really only few things happening, caching the state of the Swarm service to detect changes, polling the Swarm service state based on the diagram I put in the docs dirs, and updating routing rules and modifying the MotD that is returned from the server ping. It got really messy because there's a lot of things I learned about Swarm that should've been covered in their official documentation.

Yeah, that would be good if you don't mind. Take you time.

Okay, I did a refactor. I renamed parseServiceData to evaluateSwarmService and broke a lot of the code into smaller pieces. I applied an optimization in the first step to use a switch statement instead of a long chain of if statements. And then elsewhere, I applied a bugfix so it only updates the service definition set replicas to 1 only if replicas are set to 0.

I've yet to test this, it's been a long week, and people are playing on my MC server now, so it'll be trickier to do a standardize test than to just deploy this patch to the wild on my cluster and hope it doesn't crash.

Awesome. As always, no hurry on the last bit of testing.

Discovered a bug with the motd restart delay and permafailure and that's now fixed and tested.

If you want to rereview, it should be ready again.

SCBionicle · 2026-07-17T04:13:20Z

As you've probably seen from the govulncheck, the docker client module seems to have a CVE they're never going to fix. As such I am pondering switch to the moby/client module in #568 . The non-Swarm code seems fairly simple to switch over with mainly argument changes for options, etc. Any thoughts on the impact for the Swarm operations?

Something I noticed with the godocker package is that they had things in the API that the Swarm cluster doesn't actually support, like filer for events that were never implemented into Swarm. I'm not familiar with coding Go, so I found this really difficult to build. The best thing we could try is to test a migration and see it breaks anything.

Endpoint IP is based off of the service definition and task state polling should be off the Task API. In theory, you shouldn't need a full blown cluster to test this, I only had enough spare compute to test it on my homelab cluster. In theory, you should be able to do the initial testing in docker desktop and we can probably test the edge cases on my homelab which is a multi node environment.

Those are my thoughts, unfortunately, no straight answer.

itzg · 2026-07-17T12:52:17Z

I'm not familiar with coding Go, so I found this really difficult to build.

It's not you, there's something weird or perhaps overly "generic" that they've done with their SDK. Even non-Swarm I'm finding the moby client still needs constants from the docker module. So it's like both directions are not fully portable and not fully platform-aware.

So, yep, probably just need to try it out when I'm ready to do that SDK switch.

…t function

…from being stopped while server was in play (active players)

SCBionicle · 2026-07-21T02:37:34Z

Had an issue where the router would scale down the server while people are playing on it due to a logic oversight in the waker function. I fixed it and is now ready again for review.

itzg

The code is certainly complex, but that's the nature of the beast. As such, it looks great. I just have a curiosity question. Otherwise, ready for me to merge?

itzg · 2026-07-21T23:47:07Z

+			(service.Spec.EndpointSpec.Mode != swarmtypes.ResolutionModeVIP &&
+				service.Spec.EndpointSpec.Mode != swarmtypes.ResolutionModeDNSRR) {


There seem to be only two possible resolution modes

https://pkg.go.dev/github.com/docker/docker/api/types/swarm@v28.5.2+incompatible#ResolutionMode

Were there cases where a junk value would show up?

SCBionicle added 6 commits July 6, 2026 18:22

feat(server): enable downscaler for docker swarm watcher

cf0eae7

refactor(swarm): introduce routableSwarmService struct

d00f6d1

feat(swarm): parse autoscale and MOTD labels for services

2913289

feat(swarm): implement service scale-down sleeper function

8551a0a

feat(swarm): implement service scale-up waker function with task IP r…

72929ac

…esolution

feat(swarm): map wakers, sleepers, and custom MOTDs to route table

859a768

fix(swarm): resolve correct task IP by matching container network ID

cf90b66

itzg linked an issue Jul 7, 2026 that may be closed by this pull request

Docker Swarm Auto Scale #561

Open

fix(swarm): register empty backend endpoint when replicas are 0 to al…

8bd5d5c

…low status interception

SCBionicle added 10 commits July 7, 2026 16:34

feat(swarm): parse timeout and failed MOTD labels

cc2d803

feat(swarm): detect restart delays and crash loops during service par…

6f3ab89

…sing

feat(swarm): extend waker timeout on restart delay and fail fast on c…

db8cfcd

…rash loop

docs(readme): document Swarm wait-timeout and failed-motd labels

00d6855

fix(swarm): resolve loading MOTD bypassing and stuck restart delay co…

3eb7e38

…untdowns

refactor(swarm): remove unused maxAttempts variable from waker and pa…

3821c56

…rser

feat(swarm): resolve dynamic MOTD countdowns on-the-fly and listen to…

c184593

… TaskEventType events

fix(swarm): remove event action filters and preserve failed MOTD temp…

7534567

…late for dynamic formatting

feat(swarm): separate restart delay motd from permanent failed motd

a64b933

fix(routes): enforce case insensitivity in all mapping lookups and de…

9a8a1e9

…letions

SCBionicle added 3 commits July 8, 2026 12:32

fix(swarm): simplify state machine using undocumented task StateReady

1f29679

docs: document Docker Swarm task states mapping logic for autoscaling

7607543

SCBionicle added 2 commits July 10, 2026 21:45

fix(swarm): remove fragile timestamp filters to make waker and watche…

38e5d68

…r stateless

fix(swarm): add state transition logging to reconcileServices

c9dc320

docs(swarm): document wait timeout client limit and api polling details

7a2f9ec

SCBionicle marked this pull request as ready for review July 12, 2026 20:05

itzg reviewed Jul 15, 2026

View reviewed changes

Comment thread docs/docker-swarm-states.md

itzg requested changes Jul 16, 2026

View reviewed changes

SCBionicle added 9 commits July 18, 2026 15:10

refractor: convert to switch statement for label parsing

b0ef162

fix: update service update to only scale up when replicas is set to 0.

2e8dd41

refractor: update parseServiceData symbol name to accurately represen…

f7f0a04

…t function

chore: clean whitespace

b07ccc6

refactor: extract resolveTargetNetwork helper from evaluateSwarmService

89e327a

fix: build networkAliases once via getNetworkAliases helper

1d3a37e

refactor: extract queryServiceTasks helper from evaluateSwarmService

1e37154

refactor: extract classifyServiceState helper from evaluateSwarmService

496b60e

fix: restart delay and permafailture motd not showing

ba992c4

SCBionicle marked this pull request as draft July 20, 2026 23:39

fix: changed statement that might have prevented the scaledown timer …

3d9e107

…from being stopped while server was in play (active players)

SCBionicle marked this pull request as ready for review July 21, 2026 02:15

SCBionicle requested a review from itzg July 21, 2026 05:35

itzg approved these changes Jul 21, 2026

View reviewed changes

		(service.Spec.EndpointSpec.Mode != swarmtypes.ResolutionModeVIP &&
		service.Spec.EndpointSpec.Mode != swarmtypes.ResolutionModeDNSRR) {

Uh oh!

Conversation

SCBionicle commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The Problem: Premature Connection Severing in VIP Mode

The Solution: Direct-to-Task Routing

Detailed Changes

1. Configuration & Downscaler Wiring

2. Swarm Watcher Updates (server/docker_swarm.go)

Update (July 8, 2026)

1. State Machine Refactoring (Actual vs. Desired States)

2. Resolving Scale-Up Race Conditions (Stale History)

3. VIP Propagation Bypass (Direct Task Routing)

4. Documentation & MOTD Polishing

Uh oh!

itzg commented Jul 7, 2026

Uh oh!

SCBionicle commented Jul 7, 2026

Uh oh!

itzg commented Jul 7, 2026

Uh oh!

SCBionicle commented Jul 7, 2026

Uh oh!

SCBionicle commented Jul 7, 2026

Uh oh!

SCBionicle commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

1. Configurable Startup Timeout (via Service Label)

2. Failed-Task / Crash-Loop Detection (Fail-Fast)

3. Custom MOTD for Exhausted / Failed State

Uh oh!

itzg commented Jul 7, 2026

Uh oh!

SCBionicle commented Jul 8, 2026

Uh oh!

SCBionicle commented Jul 11, 2026

Uh oh!

itzg commented Jul 11, 2026

Uh oh!

SCBionicle commented Jul 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Progress

Core Routing

MOTD Messages

Uh oh!

itzg commented Jul 12, 2026

Uh oh!

SCBionicle commented Jul 13, 2026

Uh oh!

itzg commented Jul 13, 2026

Uh oh!

itzg commented Jul 15, 2026

Uh oh!

Uh oh!

itzg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SCBionicle commented Jul 17, 2026

Uh oh!

itzg commented Jul 17, 2026

Uh oh!

SCBionicle commented Jul 21, 2026

SCBionicle commented Jul 6, 2026 •

edited

Loading

2. Swarm Watcher Updates (`server/docker_swarm.go`)

SCBionicle commented Jul 7, 2026 •

edited

Loading

SCBionicle commented Jul 12, 2026 •

edited

Loading