Skip to content

Commit b2ee986

Browse files
Merge pull request #21248 from MicrosoftDocs/main
Auto Publish – main to live - 2026-07-14 17:01 UTC
2 parents fa30be6 + 031ea7a commit b2ee986

4 files changed

Lines changed: 138 additions & 4 deletions

File tree

AKS-Arc/TOC.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -201,6 +201,8 @@
201201
items:
202202
- name: Issues after deleting storage volumes
203203
href: delete-storage-volume.md
204+
- name: Kubernetes version image not ready
205+
href: aks-arc-image-not-ready.md
204206
- name: Security
205207
items:
206208
- name: Entra authentication prompts when running kubectl

AKS-Arc/aks-arc-image-not-ready.md

Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,132 @@
1+
---
2+
title: Troubleshoot AKS on Azure Local issues when Kubernetes version image isn't ready
3+
description: Learn how to mitigate cluster and node pool issues when a Kubernetes version reports image not ready on AKS enabled by Azure Arc on Azure Local.
4+
ms.topic: troubleshooting
5+
author: davidsmatlak
6+
ms.author: davidsmatlak
7+
ms.date: 07/01/2026
8+
ms.reviewer: srikantsarwa
9+
10+
# Intent: As an IT admin, I want to troubleshoot and resolve Kubernetes version image readiness issues in AKS on Azure Local.
11+
# Keyword: Kubernetes version image not ready AKS Azure Local
12+
---
13+
14+
# Troubleshoot AKS on Azure Local issues when Kubernetes version image isn't ready
15+
16+
This article describes how to mitigate cluster and node pool issues when a Kubernetes version reports image not ready on AKS enabled by Azure Arc on Azure Local.
17+
18+
## Symptoms
19+
20+
You might observe one or more of the following error scenarios when creating or updating an AKS Arc cluster:
21+
22+
### Error 1: Gallery image not found during cluster creation
23+
24+
```output
25+
Provisioning the AKSArc cluster. This operation might take a while...
26+
27+
(ResourceDeploymentFailure) The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.
28+
29+
(ClusterPrecheckFailed) Error: Aks Arc cluster creation precheck failed. Detailed message: 1 error occurred:
30+
* rpc error: code = NotFound desc = Type[GalleryImage], Location[MocLocation], Name[linux-cblmariner-0.11.26.10605]: Not Found
31+
```
32+
33+
### Error 2: Kubernetes version validation failure
34+
35+
```output
36+
Provisioning the AKSArc cluster. This operation might take a while...
37+
38+
(TemplateDeploymentValidationFailed) Validation failed for 'Microsoft.HybridContainerService/provisionedClusterInstances'.
39+
40+
(PreflightValidationFailed) Preflight validation failed for one or more resources
41+
42+
(PreflightValidationFailed) admission webhook "vhybridakscluster.kb.io" denied the request: {
43+
"result": "Failed",
44+
"validationChecks": [
45+
{
46+
"name": "K8sVersionValidation",
47+
"message": "Kubernetes version 1.33.5 isn't ready for use on Linux. Please go to https://aka.ms/aksarccheckk8sversions for details of how to check the readiness of Kubernetes versions.",
48+
"recommendation": "Please check https://aka.ms/AKSArcValidationErrors/K8sVersionValidation for recommendations"
49+
}
50+
]
51+
}
52+
```
53+
54+
For more information on troubleshooting the K8sVersionValidation error code, see [Troubleshoot K8sVersionValidation error code](cluster-k8s-version.md).
55+
56+
## Cause
57+
58+
It's possible that the image provisioning is still in progress. AKS on Azure Local depends on image readiness for the selected Kubernetes version and OS SKU. If the required image variant isn't fully provisioned or available, the Kubernetes version remains not ready and dependent operations can fail.
59+
60+
## Workaround
61+
62+
Use the following steps to recover image readiness.
63+
64+
### 1. Verify available Kubernetes versions
65+
66+
Run the following command to check available Kubernetes versions:
67+
68+
```azurecli
69+
az aksarc get-versions --resource-group <resource-group-name> --custom-location <custom-location-name>
70+
```
71+
72+
Confirm that your target version is listed and supported for your intended OS SKU.
73+
74+
### 2. Check storage path health
75+
76+
Run the following command to verify storage path status:
77+
78+
```azurecli
79+
az stack-hci-vm storagepath list --resource-group <resource-group-name> --custom-location <custom-location-resource-id>
80+
```
81+
82+
If the expected storage path is missing or unhealthy, recreate or update it by following the guidance in [Create storage path for Azure Local](/azure/azure-local/manage/create-storage-path).
83+
84+
### 3. Refresh storage path metadata
85+
86+
Run the following command to refresh the storage path:
87+
88+
```azurecli
89+
az stack-hci-vm storagepath update --resource-group <resource-group-name> --custom-location <custom-location-resource-id> --name <storage-path-name> --location <azure-region>
90+
```
91+
92+
This action can trigger revalidation and image resynchronization.
93+
94+
### 4. Run support module validation
95+
96+
Use the Support.AksArc diagnostic tool to validate that all tests are passing. For installation and usage instructions, see [Support.AksArc diagnostic and remediation tool](support-module.md).
97+
98+
Ensure all validation checks pass before proceeding with cluster operations.
99+
100+
### 5. Wait and retry the AKS operation
101+
102+
After recovering the storage path and image sync, retry the failed cluster or node pool operation.
103+
104+
### 6. If the issue persists
105+
106+
If the problem continues after following the previous steps, collect diagnostic logs and open a support request.
107+
108+
Include the following information:
109+
110+
- Resource group
111+
- Custom location
112+
- Azure Local resource ID
113+
- Full error message
114+
- Correlation IDs from the error output
115+
116+
## Prevention
117+
118+
To prevent this issue:
119+
120+
- Keep at least one storage path for AKS Arc cluster to work.
121+
- Validate storage path status is healthy (green status).
122+
- Never rename or change the folder path name locally on the Azure Local machine.
123+
- Avoid deleting storage volumes or storage paths used by AKS on Azure Local.
124+
- Validate storage path availability before cluster upgrades.
125+
- Use supported Kubernetes versions and OS SKUs for your target environment.
126+
- Apply platform updates and retry transient image provisioning states after a short interval.
127+
128+
## Related content
129+
130+
- [Troubleshoot AKS on Azure Local issues after deleting storage volumes](delete-storage-volume.md)
131+
- [Create storage path for Azure Local](/azure/azure-local/manage/create-storage-path)
132+
- [AKS Arc troubleshooting](aks-troubleshoot.md)

azure-local/small-form-factor/small-form-factor-faq.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,4 +39,4 @@ Run:
3939
az provisionedmachine reset-os --name <NAME_OF_MACHINE> --resource-group <RESOURCE_GROUP_NAME>
4040
```
4141

42-
After the reset finishes, you can provision the machine again by using either the Azure portal or the Azure CLI.
42+
After the reset finishes, you can provision the machine again by using either the Azure portal or the Azure CLI.

azure-local/small-form-factor/small-form-factor-known-issues.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -19,13 +19,13 @@ The following known issues apply to Azure Local small form factor deployments ru
1919

2020
| Feature area | Issue | Impact | Workaround |
2121
|---|---|---|---|
22-
| Subscription setup | New required resource providers were added since the last release. | Deployment can fail if the required providers aren't registered. | Validate your subscription against the current required provider list before deployment. |
22+
| Subscription setup | New required resource providers were added since the last release. | Deployment can fail if the required providers aren't registered. | [Validate your subscription against the current required provider list before deployment](small-form-factor-subscription-setup.md#register-the-required-resource-providers). |
2323
| Subscription policy | Azure policies that restrict publicly accessible storage accounts can block deployment. | Provisioned machine creation can fail during storage account-related steps. | If you hit storage account errors, review subscription policies that restrict publicly accessible storage accounts. |
2424
| ROE install | NUC 15 Pro doesn't show status messages on the CLI during ROE install. | This behavior can be confusing during testing because the installation might appear stalled. | When you're testing on a NUC 15 Pro, treat a blank screen with a blinking cursor as a successful in-progress state. |
2525
| OS provisioning | During OS provisioning, the Azure portal may display the status **Action Required!** even though provisioning is progressing successfully. | Users may incorrectly assume that provisioning has failed and could file unnecessary support requests or ICMs. | No action is required. OS provisioning continues to run successfully despite the incorrect status message. Monitor the provisioning workflow until completion. |
26-
| OS provisioning | OS provisioning can intermittently fail because the `LinuxEdgeObservability` extension times out during deployment. The extension attempts to retrieve the GCS configuration file after observability tenant registration, which can take up to eight minutes. However, the Azure Arc extension manager imposes a five-minute timeout on the enable operation. | OS provisioning may intermittently fail during deployment. | Retry the OS provisioning operation. This issue is intermittent and occurs when the observability tenant registration exceeds the extension manager timeout window. |
26+
| OS provisioning | OS provisioning can intermittently fail because the `LinuxEdgeObservability` extension times out during deployment. The extension attempts to retrieve the GCS configuration file after observability tenant registration, which can take up to eight minutes. However, the Azure Arc extension manager imposes a five-minute timeout on the enable operation. | OS provisioning may intermittently fail during deployment. | [Retry the OS provisioning operation](../deploy/troubleshoot-simplified-machine-provisioning.md#reattempt-a-failed-os-provisioning). This issue is intermittent and occurs when the observability tenant registration exceeds the extension manager timeout window. |
2727
| Reset OS | After running `reset-os`, the provisioned machine state in the Azure portal may display as **Unknown** instead of a resetting state. | Users may be unsure whether the reset operation is progressing successfully. | This behavior is expected in version 2604. Reset status flows and reset operations are not yet fully supported in the Azure portal for small form factor deployments. The portal currently defaults unsupported states to **Unknown**. |
28-
| Retesting | TPM slot exhaustion can happen after repeated deployments, usually after about 15 runs. | Retesting can fail when no TPM slots remain. | Manually clear the TPM in BIOS before you retest. |
28+
| Retesting | TPM slot exhaustion can happen after repeated deployments, usually after about 15 runs. | Retesting can fail when no TPM slots remain. | [Manually clear the TPM in BIOS before you retest](small-form-factor-troubleshoot.md#tpm-isnt-writable-during-provisioning). |
2929

3030
## Related content
3131

0 commit comments

Comments
 (0)