Skip to content

Document limitations of AKS stop/start with KAITO#451

Open
ramanjk wants to merge 1 commit into
MicrosoftDocs:mainfrom
ramanjk:patch-1
Open

Document limitations of AKS stop/start with KAITO#451
ramanjk wants to merge 1 commit into
MicrosoftDocs:mainfrom
ramanjk:patch-1

Conversation

@ramanjk

@ramanjk ramanjk commented Jul 16, 2026

Copy link
Copy Markdown

Added note about AKS stop/start operations and potential issues with KAITO Workspaces.

Added note about AKS stop/start operations and potential issues with KAITO Workspaces.
@prmerger-automator

Copy link
Copy Markdown
Contributor

@ramanjk : Thanks for your contribution! The author(s) and reviewer(s) have been notified to review your proposed change.

@prmerger-automator

Copy link
Copy Markdown
Contributor

@ramanjk : Thanks for your contribution! The author(s) and reviewer(s) have been notified to review your proposed change.

@learn-build-service-prod

Copy link
Copy Markdown
Contributor

Learn Build status updates of commit 8fae71c:

✅ Validation status: passed

File Status Preview URL Details
articles/aks/ai-toolchain-operator.md ✅Succeeded

For more details, please refer to the build report.

@ramanjk

ramanjk commented Jul 16, 2026

Copy link
Copy Markdown
Author

Background
A product behavior has been identified and confirmed through GitHub issue (Issue #2075) where KAITO workspace controllers may begin reconciliation immediately after an AKS cluster start operation, before AKS has fully completed its node pool recovery workflow.

This can result in node pool update conflicts and cluster provisioning failures during AKS stop/start scenarios. The behavior is currently considered a known KAITO limitation and should be documented to improve customer awareness and avoid unexpected operational issues.

Affected Documentation
Deploy an AI model on Azure Kubernetes Service (AKS) with the AI toolchain operator add-on [Deploy an...soft Learn | Learn.Microsoft.com]
GitHub Reference
GitHub Issue #2075: "Workspace controller races with AKS start operation causing OriginalEtagNotExpect failure" [github.com]
Customer Impact
When customers:

Deploy KAITO Workspaces on AKS.
Stop the AKS cluster using az aks stop.
Start the AKS cluster using az aks start.
The KAITO workspace controller may attempt to reconcile and manage GPU node pools before the AKS start operation has fully completed.

Potential outcomes include:

AKS cluster enters a Failed provisioning state.
OriginalEtagNotExpect or similar node pool update conflicts.
KAITO-created GPU node pools becoming inconsistent with cluster state.
Orphaned VMSS resources continuing to run and incur unexpected costs.
Manual reconciliation may be required to restore the cluster to a healthy state. [github.com]

Proposed Documentation Update
Add a new bullet under Limitations:

AKS stop and start operations (az aks stop / az aks start) aren't fully supported for clusters with active KAITO Workspaces. In some scenarios, startup operations can result in node pool reconciliation conflicts, provisioning failures, or orphaned compute resources.

@v-dirichards
v-dirichards requested a review from Copilot July 16, 2026 15:30
@v-dirichards

Copy link
Copy Markdown
Contributor

Hi @ramanjk - This pull request was opened in the public repo. Microsoft contributors should work in the private repo, per the Microsoft Docs contributor guide. We can keep this PR open for review and merge, but would you make future content updates in the private repo? Thank you!


@schaffererin

Can you review the proposed changes?

Important: When the changes are ready for publication, adding a #sign-off comment is the best way to signal that the PR is ready for the review team to merge.

#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a documented limitation for using AKS stop/start with KAITO, warning about potential reconciliation/provisioning issues when workspaces are active.

Changes:

  • Documented that AKS stop/start isn’t fully supported with active KAITO workspaces and can lead to reconciliation conflicts and orphaned resources.

* `Windows` OS SKU is not currently supported.
* AMD GPU VM sizes are not supported `instanceType` in a KAITO workspace.
* AI toolchain operator add-on is supported in **public** Azure regions.
* AKS stop and start operations (az aks stop / az aks start) aren't fully supported for clusters with active KAITO Workspaces. In some scenarios, startup operations can result in node pool reconciliation conflicts, provisioning failures, or orphaned compute resources.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants