-
-
Notifications
You must be signed in to change notification settings - Fork 63
feat(swarm): implement autoscaling (scale to zero/one) for Docker Swarm services #564
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
SCBionicle
wants to merge
38
commits into
itzg:main
Choose a base branch
from
SCBionicle:swarm-autoscale
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 28 commits
Commits
Show all changes
38 commits
Select commit
Hold shift + click to select a range
cf0eae7
feat(server): enable downscaler for docker swarm watcher
SCBionicle d00f6d1
refactor(swarm): introduce routableSwarmService struct
SCBionicle 2913289
feat(swarm): parse autoscale and MOTD labels for services
SCBionicle 8551a0a
feat(swarm): implement service scale-down sleeper function
SCBionicle 72929ac
feat(swarm): implement service scale-up waker function with task IP r…
SCBionicle 859a768
feat(swarm): map wakers, sleepers, and custom MOTDs to route table
SCBionicle cf90b66
fix(swarm): resolve correct task IP by matching container network ID
SCBionicle 8bd5d5c
fix(swarm): register empty backend endpoint when replicas are 0 to al…
SCBionicle cc2d803
feat(swarm): parse timeout and failed MOTD labels
SCBionicle 6f3ab89
feat(swarm): detect restart delays and crash loops during service par…
SCBionicle db8cfcd
feat(swarm): extend waker timeout on restart delay and fail fast on c…
SCBionicle 00d6855
docs(readme): document Swarm wait-timeout and failed-motd labels
SCBionicle 3eb7e38
fix(swarm): resolve loading MOTD bypassing and stuck restart delay co…
SCBionicle 3821c56
refactor(swarm): remove unused maxAttempts variable from waker and pa…
SCBionicle c184593
feat(swarm): resolve dynamic MOTD countdowns on-the-fly and listen to…
SCBionicle 7534567
fix(swarm): remove event action filters and preserve failed MOTD temp…
SCBionicle a64b933
feat(swarm): separate restart delay motd from permanent failed motd
SCBionicle 9a8a1e9
fix(routes): enforce case insensitivity in all mapping lookups and de…
SCBionicle 1f29679
fix(swarm): simplify state machine using undocumented task StateReady
SCBionicle 7607543
docs: document Docker Swarm task states mapping logic for autoscaling
SCBionicle ac8671a
fix(swarm): simplify state machine using actual and desired task states
SCBionicle 1f57e33
fix(swarm): resolve scale-up race conditions and VIP routing propagat…
SCBionicle c631f9d
fix(swarm): document reconciliation block and add settling delay to t…
SCBionicle d805558
fix(swarm): replace task events with 5-second background polling ticker
SCBionicle 690c462
docs: update README to clarify support for native Swarm scaling and V…
SCBionicle 38e5d68
fix(swarm): remove fragile timestamp filters to make waker and watche…
SCBionicle c9dc320
fix(swarm): add state transition logging to reconcileServices
SCBionicle 7a2f9ec
docs(swarm): document wait timeout client limit and api polling details
SCBionicle b0ef162
refractor: convert to switch statement for label parsing
SCBionicle 2e8dd41
fix: update service update to only scale up when replicas is set to 0.
SCBionicle f7f0a04
refractor: update parseServiceData symbol name to accurately represen…
SCBionicle b07ccc6
chore: clean whitespace
SCBionicle 89e327a
refactor: extract resolveTargetNetwork helper from evaluateSwarmService
SCBionicle 1d3a37e
fix: build networkAliases once via getNetworkAliases helper
SCBionicle 1e37154
refactor: extract queryServiceTasks helper from evaluateSwarmService
SCBionicle 496b60e
refactor: extract classifyServiceState helper from evaluateSwarmService
SCBionicle ba992c4
fix: restart delay and permafailture motd not showing
SCBionicle 3d9e107
fix: changed statement that might have prevented the scaledown timer …
SCBionicle File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,55 @@ | ||
| # Docker Swarm Task States for Autoscaling | ||
|
|
||
| When building plugins or routers (like `mc-router`) that autoscale and monitor Docker Swarm services, interpreting task scheduling states is historically challenging due to sparse Docker documentation. | ||
|
|
||
| This document defines how to map Swarm's **Actual Task Status State** (`task.Status.State`) and **Desired Task State** (`task.DesiredState`) to logical route status classifications. | ||
|
|
||
| --- | ||
|
|
||
| ## Task State Matrix | ||
|
|
||
| | State | Status State (`task.Status.State`) | Desired State (`task.DesiredState`) | Description | | ||
| | :--- | :--- | :--- | :--- | | ||
| | **Running** | `running` | `running` | The container is alive, healthy, and fully running. Connections can be routed directly. | | ||
| | **Restart Delay** | `ready` | `ready` | The service task crashed and Swarm is holding the next task retry until the restart delay timer expires. | | ||
| | **Starting / Waking** | `new`, `pending`, `assigned`, `preparing`, `ready`, `starting` | `running` | Swarm has scaled the replicas to `> 0` and is actively preparing or starting the container. | | ||
| | **Stopping** | `running` | `shutdown` or `remove` | Swarm has commanded the task to shut down or be removed, but the container is still cleaning up. | | ||
| | **Permanently Failed** | `failed`, `shutdown`, `rejected` | `shutdown` | Swarm has commanded a shutdown on the last failed container and is no longer attempting to schedule a new task. | | ||
|
|
||
| --- | ||
|
|
||
| ## State Classifications & Logic | ||
|
|
||
| ### 1. Sleeping State | ||
| * **Condition**: `replicas == 0` | ||
| * **Interpretation**: The service is explicitly scaled down to zero replicas. | ||
| * **Router Action**: Keep the backend route empty (`""`) and display the configured asleep MOTD to prompt a wake-up on join. | ||
|
|
||
| ### 2. Running (Healthy) State | ||
| * **Condition**: At least one task has `Status.State == running` and `DesiredState == running`. | ||
| * **Interpretation**: The server is fully online and ready for traffic. | ||
| * **Router Action**: Resolve the route to the service's Virtual IP (VIP) or DNS Round Robin (DNSRR) name. | ||
|
|
||
| ### 3. Restart Delay State | ||
| * **Condition**: `replicas > 0` AND at least one task has `Status.State == ready` and `DesiredState == ready`. | ||
| * **Interpretation**: Swarm has scheduled a retry but is waiting for the configured restart policy delay to expire before spawning the container. | ||
| * **Router Action**: Keep the backend route empty (`""`) to route connection attempts to the waker, and display the restart delay countdown MOTD. | ||
|
|
||
| ### 4. Permanently Failed State | ||
| * **Condition**: `replicas > 0` AND the most recently updated task in history has `DesiredState == shutdown` AND no active tasks exist. | ||
| * **Interpretation**: Swarm has exhausted its restart attempts and stopped trying. | ||
| * **Router Action**: Keep the backend route empty (`""`) and display the custom permanently failed MOTD. | ||
|
|
||
| ### 5. Starting / Waking State | ||
| * **Condition**: `replicas > 0` AND the desired state of the task is `running` but the actual status is not yet `running` (and not delayed or failed). | ||
| * **Interpretation**: The container is actively being assigned, prepared, or started by Swarm. | ||
| * **Router Action**: Keep the backend route empty (`""`) to preserve the waker, and display the loading/waking MOTD. | ||
|
|
||
| --- | ||
|
|
||
| ## Polling Implementation Details | ||
|
|
||
| > [!NOTE] | ||
| > **Why Polling is Used Instead of API Events:** | ||
| > Although `mc-router` subscribes to the Docker Event Stream, the Docker Engine API does **not** broadcast task-level scheduling transitions (such as `task.Status.State` moving from `preparing` to `ready` or `running`) over the event system. | ||
| > To reliably capture these low-level task state changes and resolve the direct task IP without transient routing failures, `mc-router` utilizes a background polling ticker that queries the Swarm Service and Task API every 5 seconds. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.