XCP-NG VM/Kubernetes node dynamic scale failed.

[furyflash777]

ISSUE TYPE

• Bug Report

COMPONENT NAME

UI

CLOUDSTACK VERSION

4.18.0 and 4.18.1

CONFIGURATION

OS / ENVIRONMENT

XCP-NG 8.2.1

Current setup:
cpu.overprovisioning.factor = 2
mem.overprovisioning.factor = 2

same result with:
cpu.overprovisioning.factor = 1
mem.overprovisioning.factor = 1

test VM 1 CPU x 0.50 Ghz and 512 MB memory

SUMMARY

Unable to dynamic scale VM from different templates (Debian, Ubuntu, Centos)

GUI error:
Unhandled exception

Log error:
2023-12-08 04:13:57,203 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-27:ctx-ff0ece9e job-1340/job-13
42 ctx-23ed9399) (logid:15e9aadd) Invocation exception, caused by: com.cloud.utils.exception.CloudRuntimeExcep
tion: Unable to scale vm due to Catch exception com.cloud.utils.exception.CloudRuntimeException when scaling V
M:i-8-254-VM due to com.cloud.utils.exception.CloudRuntimeException: Cannot scale up the vm because of memory
constraint violation: 0 <= memory-static-min(268435456) <= memory-dynamic-min(2147483648) <= memory-dynamic-ma
x(4294967296) <= memory-static-max(536870912)

Cannot scale up the vm because of memory constraint violation: 0 <= memory-static-min(268435456) <= memory-d
ynamic-min(2147483648) <= memory-dynamic-max(4294967296) <= memory-static-max(536870912)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateReConfigureVm(VirtualMachineManagerImpl.java:4648
)
at com.cloud.vm.VirtualMachineManagerImpl.reConfigureVm(VirtualMachineManagerImpl.java:4579)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateReconfigure(VirtualMachineManagerImpl.java:5522)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.jav
a:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5536)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.ja
va:620)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:
55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedCont
ext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedConte
xt.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)

STEPS TO REPRODUCE

Create VM with 1 or more CPUs and select dynamic scalable in VM, Template, Global properties. 
Try to scale running VM.

EXPECTED RESULTS

VM scaled successfully

ACTUAL RESULTS

VM scale failed result

[DaanHoogland]

@furyflash777 , can you have a look at your report, please? something is off; the message Cannot scale up the vm because of memory constraint violation: is created in a class called CitrixResourceBase, which does not occur in your stack-trace. the message I would expect is "Unable to scale vm due to and I think it would be a valid response from the hypervisor.

Can you also explain what your intentions are? In kubernetes the number of nodes would scale, not the size of the VM. Is this what your try?

[furyflash777]

I am trying to scale VM size (change compute offering). Dynamic scaling also is not working for instances.

044: Sending { Cmd , MgmtId: 20156575640318, via: 6, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.ScaleVmCo
mmand":{"vm":{"id":"1656","name":"i-6-1656-VM","type":"User","cpus":"4","minSpeed":"725","maxSpeed":"2900","mi
nRam":"(4.00 GB) 4294967296","maxRam":"(8.00 GB) 8589934592","enableHA":"false","limitCpuUse":"false","enableD
ynamicallyScaleVm":"false","uuid":"9ad5863b-d170-4d7b-8ea1-04f7efcc8fa0","enterHardwareSetup":"false","configD
riveLocation":"SECONDARY","guestOsDetails":{},"extraConfig":{}},"vmName":"i-6-1656-VM","cpus":"4","minSpeed":"
725","maxSpeed":"2900","minRam":"(4.00 GB) 4294967296","maxRam":"(8.00 GB) 8589934592","wait":"0","bypassHostM
aintenance":"false"}}] }
2024-01-27 03:11:00,466 DEBUG [c.c.a.t.Request] (AgentManager-Handler-14:null) (logid:) Seq 6-5214042468588252
044: Executing: { Cmd , MgmtId: 20156575640318, via: 6, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.ScaleV
mCommand":{"vm":{"id":"1656","name":"i-6-1656-VM","type":"User","cpus":"4","minSpeed":"725","maxSpeed":"2900",
"minRam":"(4.00 GB) 4294967296","maxRam":"(8.00 GB) 8589934592","enableHA":"false","limitCpuUse":"false","enab
leDynamicallyScaleVm":"false","uuid":"9ad5863b-d170-4d7b-8ea1-04f7efcc8fa0","enterHardwareSetup":"false","conf
igDriveLocation":"SECONDARY","guestOsDetails":{},"extraConfig":{}},"vmName":"i-6-1656-VM","cpus":"4","minSpeed
":"725","maxSpeed":"2900","minRam":"(4.00 GB) 4294967296","maxRam":"(8.00 GB) 8589934592","wait":"0","bypassHo
stMaintenance":"false"}}] }

2024-01-27 03:11:00,535 DEBUG [c.c.h.x.r.w.x.CitrixScaleVmCommandWrapper] (DirectAgent-27:ctx-310bdcf2) (logid
:d2eeb120) Catch exception com.cloud.utils.exception.CloudRuntimeException when scaling VM:i-6-1656-VM due to
com.cloud.utils.exception.CloudRuntimeException: Cannot scale up the vm because of memory constraint violation
: 0 <= memory-static-min(2147483648) <= memory-dynamic-min(4294967296) <= memory-dynamic-max(8589934592) <= me
mory-static-max(4294967296)

It's very strange. In log I see "enableDynamicallyScaleVm":"false"

but it is enabled in VM, Template and Compute offering

Other logs:

2024-01-27 03:11:00,538 DEBUG [c.c.a.t.Request] (AgentManager-Handler-11:null) (logid:) Seq 6-5214042468588252
044: Processing: { Ans: , MgmtId: 20156575640318, via: 6, Ver: v1, Flags: 110, [{"com.cloud.agent.api.ScaleVm
Answer":{"result":"false","details":"Catch exception com.cloud.utils.exception.CloudRuntimeException when scal
ing VM:i-6-1656-VM due to com.cloud.utils.exception.CloudRuntimeException: Cannot scale up the vm because of m
emory constraint violation: 0 <= memory-static-min(2147483648) <= memory-dynamic-min(4294967296) <= memory-dyn
amic-max(8589934592) <= memory-static-max(4294967296)","wait":"0","bypassHostMaintenance":"false"}}] }

2024-01-27 03:11:00,539 ERROR [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-103:ctx-636242c2 job-29223/
job-29224 ctx-7eebd461) (logid:d2eeb120) Unable to scale vm due to Catch exception com.cloud.utils.exception.C
loudRuntimeException when scaling VM:i-6-1656-VM due to com.cloud.utils.exception.CloudRuntimeException: Canno
t scale up the vm because of memory constraint violation: 0 <= memory-static-min(2147483648) <= memory-dynamic
-min(4294967296) <= memory-dynamic-max(8589934592) <= memory-static-max(4294967296)

2024-01-27 03:11:00,552 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-103:ctx-636242c2 job-29223/job-
29224 ctx-7eebd461) (logid:d2eeb120) Invocation exception, caused by: com.cloud.utils.exception.CloudRuntimeEx
ception: Unable to scale vm due to Catch exception com.cloud.utils.exception.CloudRuntimeException when scalin
g VM:i-6-1656-VM due to com.cloud.utils.exception.CloudRuntimeException: Cannot scale up the vm because of mem
ory constraint violation: 0 <= memory-static-min(2147483648) <= memory-dynamic-min(4294967296) <= memory-dynam
ic-max(8589934592) <= memory-static-max(4294967296)

2024-01-27 03:11:00,552 INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-103:ctx-636242c2 job-29223/job-
29224 ctx-7eebd461) (logid:d2eeb120) Rethrow exception com.cloud.utils.exception.CloudRuntimeException: Unable
to scale vm due to Catch exception com.cloud.utils.exception.CloudRuntimeException when scaling VM:i-6-1656-V
M due to com.cloud.utils.exception.CloudRuntimeException: Cannot scale up the vm because of memory constraint
violation: 0 <= memory-static-min(2147483648) <= memory-dynamic-min(4294967296) <= memory-dynamic-max(85899345
92) <= memory-static-max(4294967296)

2024-01-27 03:11:00,553 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-103:ctx-636242c2 job-29223/job-29
224) (logid:d2eeb120) Unable to complete AsyncJobVO: {id:29224, userId: 2, accountId: 2, instanceType: null, i
nstanceId: null, cmd: com.cloud.vm.VmWorkReconfigure, cmdInfo: xxxxxxxxxxx, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0,
resultCode: 0, result: null, initMsid: 20156575640318, completeMsid: null, lastUpdated: null, lastPolled: nul
l, created: Sat Jan 27 03:10:59 MSK 2024, removed: null}, job origin:29223
com.cloud.utils.exception.CloudRuntimeException: Unable to scale vm due to Catch exception com.cloud.utils.exc
eption.CloudRuntimeException when scaling VM:i-6-1656-VM due to com.cloud.utils.exception.CloudRuntimeExceptio
n: Cannot scale up the vm because of memory constraint violation: 0 <= memory-static-min(2147483648) <= memory
-dynamic-min(4294967296) <= memory-dynamic-max(8589934592) <= memory-static-max(4294967296)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateReConfigureVm(VirtualMachineManagerImpl.java:4648
)
at com.cloud.vm.VirtualMachineManagerImpl.reConfigureVm(VirtualMachineManagerImpl.java:4579)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateReconfigure(VirtualMachineManagerImpl.java:5522)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.jav
a:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5536)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.ja
va:620)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:
55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedCont
ext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedConte
xt.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)

2024-01-27 03:11:00,668 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-72:ctx-3d19bf6d job-29223) (logi
d:d2eeb120) Unexpected exception while executing org.apache.cloudstack.api.command.user.kubernetes.cluster.Sca
leKubernetesClusterCmd
java.lang.RuntimeException: Unhandled exception
at com.cloud.vm.VirtualMachineManagerImpl.reConfigureVm(VirtualMachineManagerImpl.java:4594)
at com.cloud.vm.VirtualMachineManagerImpl.reConfigureVm(VirtualMachineManagerImpl.java:271)
at com.cloud.vm.UserVmManagerImpl.upgradeRunningVirtualMachine(UserVmManagerImpl.java:2044)
at com.cloud.vm.UserVmManagerImpl.upgradeVirtualMachine(UserVmManagerImpl.java:1900)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.jav
a:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvoca
tion.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.jav
a:163)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.
java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.jav
a:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy185.upgradeVirtualMachine(Unknown Source)
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleKubernetesClusterOffer
ing(KubernetesClusterScaleWorker.java:301)
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleCluster(KubernetesClus
terScaleWorker.java:464)
at com.cloud.kubernetes.cluster.KubernetesClusterManagerImpl.scaleKubernetesCluster(KubernetesClusterM
anagerImpl.java:1328)
at org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd.execute(ScaleKu
bernetesClusterCmd.java:156)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:163)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.ja
va:620)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:
55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedCont
ext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedConte
xt.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to scale vm due to Catch exception com.clou
d.utils.exception.CloudRuntimeException when scaling VM:i-6-1656-VM due to com.cloud.utils.exception.CloudRunt
imeException: Cannot scale up the vm because of memory constraint violation: 0 <= memory-static-min(2147483648
) <= memory-dynamic-min(4294967296) <= memory-dynamic-max(8589934592) <= memory-static-max(4294967296)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateReConfigureVm(VirtualMachineManagerImpl.java:4648
)
at com.cloud.vm.VirtualMachineManagerImpl.reConfigureVm(VirtualMachineManagerImpl.java:4579)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateReconfigure(VirtualMachineManagerImpl.java:5522)
... 19 more

2024-01-27 03:11:00,674 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-72:ctx-3d19bf6d job-29223) (
logid:d2eeb120) Complete async job-29223, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.ap
i.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Unhandled exception"}

[furyflash777]

When k8 cluster has stopped state. I can change compute offering for cluster successfully. Seems problem with Dynamic scaling for XCP-NG VMs.

[DaanHoogland]

ok @furyflash777, get it.
I wonder if you really want scaling up of machines as you can easily add and remove worker nodes.
That said you may have hit a XCPNG limitation.
So can you try if you can autoscale a separate VM on your setup?

[furyflash777]

Dynamic scaling isn't working for any XCP-NG VMs started with Cloudstack.

[furyflash777]

@DaanHoogland maybe I should set
memory-dynamic-max,
memory-dynamic-min
in to VM settings or template.

I found https://cwiki.apache.org/confluence/display/CLOUDSTACK/Enable+additional+configuration+metadata+to+virtual+machines

but I don't understand what should I put in to VM settings.

i tried put in to platform and with separate metadata, but didn't changed.

[DaanHoogland]

maybe an obvious one, but do you have xentools installed in the vm?

[furyflash777]

Yes I have