BUSY_ENTITY -> Java Exception

[MarcelMertens]

Describe the bug

Hello,
we are a german MSP and offers Cloud Director Services to our customer.
As an MSP "extra service" we add some data (OrgVDC Name etc) to each VMs guestinfo.* variables via an API call.
This is working fine and the customers loves these information about the cloud infrastructure in there VM.
The issue is, that update of the guestinfo.* variables locks the VM for a few seconds.
And here is the issue:
Some customers deploy their VMs with terraform-provider of vCD and they randomly report errors:
If the VM is locked by our "guestinfo update process" and terroform tries to do something with the VM they receive an

The entity Ref: com.vmware.vcloud.entity.vm:740ea4a2-6904-4bdd-b97d-9a0ecc8e79d9 is busy completing an operation VAPP_UPDATE_VM. VAPP_UPDATE_VM(com.vmware.vcloud.entity.task:4eefa495-c6ed-4f0f-8ff0-fd98d8ac05a1)" majorErrorCode="400" minorErrorCode="BUSY_ENTITY"/>

Besides the main issue that an error code 400 is wrong (because it's not an client error) (503 or 429 is more appropiate)
is it possible to add a retries in the case of "BUSY ENTITY" to the vCD sdk?

Kind Regards,

Marcel

Reproduction steps

1. Update / Change a VM from Client A
2. In the same moment update/ Change a VM from Client B
3. Client B receives a JavaException because of "BUSY ENTITY"

Expected behavior

If "BUSY ENTITY" is received, wait a few seconds and try again instead throw an exception

Additional context

Complete Log

2023/09/06 09:22:58 --------------------------------------------------------------------------------
2023/09/06 09:22:58 Request caller: govcd.(*VM).PowerOn-->govcd.(*VM).PowerOn-->govcd.executeRequestWithApiVersion-->govcd.(*Client).executeTaskRequest-->govcd.executeRequestCustomErr-->govcd.executeRequestCustomErr-->govcd.(*Client).newRequest
2023/09/06 09:22:58 POST https://vcdurl/api/vApp/vm-740ea4a2-6904-4bdd-b97d-9a0ecc8e79d9/power/action/powerOn
2023/09/06 09:22:58 --------------------------------------------------------------------------------
2023/09/06 09:22:58 Req header:
2023/09/06 09:22:58     Accept: [application/*+xml;version=36.0]
2023/09/06 09:22:58     X-Vmware-Vcloud-Token-Type: [Bearer]
2023/09/06 09:22:58     Authorization: [********]
2023/09/06 09:22:58     User-Agent: [terraform-provider-vcd/v3.9.0 (linux/amd64; isProvider:true)]
2023/09/06 09:22:58     X-Vmware-Vcloud-Access-Token: [********]
2023/09/06 09:22:58 ################################################################################
2023/09/06 09:22:58 Response caller govcd.(*VM).PowerOn-->govcd.(*VM).PowerOn-->govcd.executeRequestWithApiVersion-->govcd.(*Client).executeTaskRequest-->govcd.executeRequestCustomErr-->govcd.checkRespWithErrType-->govcd.ParseErr-->govcd.decodeBody
2023/09/06 09:22:58 Response status 400 Bad Request
2023/09/06 09:22:58 ################################################################################
2023/09/06 09:22:58 Response header:
2023/09/06 09:22:58     Set-Cookie: [xxxxprodhttps=srv-TsxeLcdz0qbtju9N5qhG2A|ZPgoz; path=/]
2023/09/06 09:22:58     Date: [Wed, 06 Sep 2023 07:22:58 GMT]
2023/09/06 09:22:58     Cache-Control: [no-store, must-revalidate]
2023/09/06 09:22:58     X-Vmware-Vcloud-Request-Execution-Time: [166]
2023/09/06 09:22:58     X-Vmware-Vcloud-Request-Id: [87118440-6114-492b-8da2-9b7c7a698ceb]
2023/09/06 09:22:58     Content-Type: [application/vnd.vmware.vcloud.error+xml;version=36.0]
2023/09/06 09:22:58 Response text: [17394]
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Error xmlns="http://www.vmware.com/vcloud/v1.5" xmlns:vmext="http://www.vmware.com/vcloud/extension/v1.5" xmlns:ovf="http://schemas.dmtf.org/ovf/envelope/1" xmlns:vssd="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_VirtualSystemSettingData" xmlns:common="http://schemas.dmtf.org/wbem/wscim/1/common" xmlns:rasd="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData" xmlns:vmw="http://www.vmware.com/schema/ovf" xmlns:ovfenv="http://schemas.dmtf.org/ovf/environment/1" xmlns:ns9="http://www.vmware.com/vcloud/versions" stackTrace="com.vmware.vcloud.api.presentation.service.BusyEntityException: The entity Ref: com.vmware.vcloud.entity.vm:740ea4a2-6904-4bdd-b97d-9a0ecc8e79d9 is busy completing an operation VAPP_UPDATE_VM.&#10; at com.vmware.ssdc.backendbase.dao.impl.BusyObjectDaoImpl.createBusyEntityException(BusyObjectDaoImpl.java:73)&#10;     at jdk.internal.reflect.GeneratedMethodAccessor22432.invoke(Unknown Source)&#10;        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)&#10;        at java.base/java.lang.reflect.Method.invoke(Method.java:566)&#10;      at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)&#10;      at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)&#10;       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)&#10;       at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123)&#10;     at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388)&#10; at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)&#10;      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)&#10;       at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)&#10;        at com.sun.proxy.$Proxy286.createBusyEntityException(Unknown Source)&#10;       at com.vmware.ssdc.backendbase.services.impl.BusyObjectServiceImpl.testAndSetBusy(BusyObjectServiceImpl.java:142)&#10;  at com.vmware.vcloud.backendbase.management.system.TaskServiceImpl.createBusyTaskWithQuotaContext(TaskServiceImpl.java:1215)&#10;       at jdk.internal.reflect.GeneratedMethodAccessor9813.invoke(Unknown Source)&#10; at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)&#10;        at java.base/java.lang.reflect.Method.invoke(Method.java:566)&#10;      at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)&#10;      at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)&#10;        at com.sun.proxy.$Proxy300.createBusyTaskWithQuotaContext(Unknown Source)&#10;  at com.vmware.vcloud.vapp.impl.VAppServiceImpl.createDeployTask(VAppServiceImpl.java:1555)&#10; at com.vmware.vcloud.vapp.impl.VAppServiceImpl.deployVm(VAppServiceImpl.java:1532)&#10; at com.vmware.vcloud.vapp.impl.VAppServiceImpl.deploy(VAppServiceImpl.java:1420)&#10;   at jdk.internal.reflect.GeneratedMethodAccessor9791.invoke(Unknown Source)&#10; at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)&#10;        at java.base/java.lang.reflect.Method.invoke(Method.java:566)&#10;      at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)&#10;      at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)&#10;       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)&#10;       at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:58)&#10;   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)&#10;       at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:58)&#10;   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)&#10;       at com.vmware.vcloud.common.validation.AbstractMethodInterceptor.invoke(AbstractMethodInterceptor.java:41)&#10; at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)&#10;       at com.vmware.ssdc.backendbase.annotation.PresentationApiDiagnosticsInterceptor.invoke(PresentationApiDiagnosticsInterceptor.java:52)&#10;      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)&#10;       at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)&#10;        at com.sun.proxy.$Proxy377.deploy(Unknown Source)&#10;  at jdk.internal.reflect.GeneratedMethodAccessor9791.invoke(Unknown Source)&#10; at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)&#10;        at java.base/java.lang.reflect.Method.invoke(Method.java:566)&#10;      at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)&#10;      at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)&#10;       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)&#10;       at com.vmware.vcloud.common.validation.AbstractMethodInterceptor.invoke(AbstractMethodInterceptor.java:41)&#10; at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)&#10;       at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)&#10;        at com.sun.proxy.$Proxy618.deploy(Unknown Source)&#10;  at com.vmware.vcloud.api.presentation.service.impl.VAppServiceAdapterImpl.deploy(VAppServiceAdapterImpl.java:530)&#10;  at jdk.internal.reflect.GeneratedMethodAccessor9791.invoke(Unknown Source)&#10; at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)&#10;        at java.base/java.lang.reflect.Method.invoke(Method.java:566)&#10;      at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)&#10;      at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)&#10;       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)&#10;       at com.vmware.vcloud.common.validation.AbstractMethodInterceptor.invoke(AbstractMethodInterceptor.java:41)&#10; at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)&#10;       at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)&#10;        at com.sun.proxy.$Proxy619.deploy(Unknown Source)&#10;  at com.vmware.vcloud.api.rest.handlers.VAppHandler.deployWithPowerOnParams(VAppHandler.java:359)&#10;   at com.vmware.vcloud.api.rest.handlers.VAppHandler.powerOnVApp(VAppHandler.java:348)&#10;       at jdk.internal.reflect.GeneratedMethodAccessor12938.invoke(Unknown Source)&#10;        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)&#10;        at java.base/java.lang.reflect.Method.invoke(Method.java:566)&#10;      at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:179)&#10;      at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)&#10;  at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:201)&#10; at com.vmware.vcloud.api.rest.jaxrs.ContextClassLoaderInvoker.invoke(ContextClassLoaderInvoker.java:27)&#10;    at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:104)&#10; at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)&#10;   at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96)&#10;   at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307)&#10;  at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)&#10;    at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:265)&#10;  at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234)&#10;        at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208)&#10;   at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160)&#10;   at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:225)&#10;       at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:304)&#10;        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:217)&#10;       at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)&#10;    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:279)&#10;      at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1419)&#10;       at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)&#10;  at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665)&#10;    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:352)&#10;        at com.vmware.vcloud.api.framework.web.ExtensibilityFilter.doFilter(ExtensibilityFilter.java:159)&#10;  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)&#10;        at com.vmware.vcloud.security.filters.ValidityExceptionFilter.doFilterHttp(ValidityExceptionFilter.java:47)&#10;        at com.vmware.vcloud.security.filters.HttpFilterBean.doFilter(HttpFilterBean.java:35)&#10;      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)&#10;        at com.vmware.vcloud.security.web.RestSessionFilter.doFilter(RestSessionFilter.java:235)&#10;   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)&#10;        at com.vmware.vcloud.security.web.CellLivenessCheckFilter.doFilter(CellLivenessCheckFilter.java:54)&#10;        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)&#10;        at com.vmware.vcloud.security.web.SecurityContextCleanerFilter.doFilter(SecurityContextCleanerFilter.java:32)&#10;      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)&#10;        at com.vmware.vcloud.api.rest.versioning.AcceptHeaderFilter.doFilter(AcceptHeaderFilter.java:78)&#10;   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)&#10;        at com.vmware.vcloud.security.filters.ConversationFilter$1.run(ConversationFilter.java:39)&#10; at com.vmware.vcloud.security.filters.ConversationFilter$1.run(ConversationFilter.java:37)&#10; at com.vmware.vcloud.common.persist.ConversationContextExecutor.execute(ConversationContextExecutor.java:67)&#10;       at com.vmware.vcloud.security.filters.ConversationFilter.doFilter(ConversationFilter.java:45)&#10;      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)&#10;        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:225)&#10;   at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:190)&#10;   at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354)&#10;     at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267)&#10;   at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210)&#10;  at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)&#10;       at com.vmware.vcloud.common.web.deployment.impl.HostCheckerFilter.doFilter(HostCheckerFilter.java:107)&#10;     at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)&#10;  at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)&#10;       at com.vmware.vcloud.common.web.deployment.impl.RequestDiagnosticFilter.doFilter(RequestDiagnosticFilter.java:143)&#10; at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)&#10;  at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)&#10;       at com.vmware.vcloud.common.web.deployment.impl.ConfigurableSniffyFilter.doFilter(ConfigurableSniffyFilter.java:48)&#10;        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)&#10;  at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)&#10;       at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)&#10;      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)&#10;   at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578)&#10;     at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)&#10; at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)&#10;       at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1571)&#10;      at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)&#10;       at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1383)&#10;      at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)&#10;        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)&#10;       at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1544)&#10;       at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)&#10;        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1305)&#10;       at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)&#10;   at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:192)&#10;     at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:51)&#10;        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:141)&#10;   at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)&#10; at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822)&#10;  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)&#10; at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301)&#10;        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)&#10; at org.eclipse.jetty.server.Server.handle(Server.java:563)&#10; at org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505)&#10;      at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762)&#10;     at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497)&#10;       at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282)&#10;     at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)&#10;     at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)&#10;       at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:558)&#10;     at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:379)&#10;       at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:146)&#10;      at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)&#10;       at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)&#10; at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)&#10;    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)&#10;        at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)&#10; at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.lambda$new$0(AdaptiveExecutionStrategy.java:139)&#10;       at com.vmware.vcloud.common.threading.ExtendedJettyQueuedThreadPool$CleaningRunnable.run(ExtendedJettyQueuedThreadPool.java:58)&#10;    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)&#10;        at com.vmware.vcloud.common.threading.ExtendedJettyQueuedThreadPool$CleaningRunnable.run(ExtendedJettyQueuedThreadPool.java:58)&#10;    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:933)&#10;        at com.vmware.vcloud.common.threading.ExtendedJettyQueuedThreadPool.runJob(ExtendedJettyQueuedThreadPool.java:43)&#10;  at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1077)&#10;   at java.base/java.lang.Thread.run(Thread.java:829)&#10;" message="[ 87118440-6114-492b-8da2-9b7c7a698ceb ] The entity Ref: com.vmware.vcloud.entity.vm:740ea4a2-6904-4bdd-b97d-9a0ecc8e79d9 is busy completing an operation VAPP_UPDATE_VM. VAPP_UPDATE_VM(com.vmware.vcloud.entity.task:4eefa495-c6ed-4f0f-8ff0-fd98d8ac05a1)" majorErrorCode="400" minorErrorCode="BUSY_ENTITY"/>

[adezxc]

Hi,

Thanks for submitting this issue.

The described behaviour is not really a bug. The API entities/resources like the vm are not intended to be managed by different applications at the same time. (e.g Terraform and UI) One would need to implement such checks himself as otherwise it would convolute the SDK and it would become much harder to keep track of the state of an entity.

As a reference, here is the way we created a retry function for testing, maybe you could utilise it in your process: https://github.com/vmware/go-vcloud-director/blob/main/govcd/api_vcd_test.go#L2037-L2054

[MarcelMertens]

Hi, i completely agree with you that it isn't a "bug" and that resources should NOT be changed at the same time by different applications. But it can happen and i think every application should handle it. So maybe "enhancement" is a better label ;-)

As i already wrote: From my point of view the error message "400" is totaly wrong. Because its not a client error (Except that it sent the request to the wrong time). 503 would fit much better here.

[adezxc]

SDK only provides an interface to the VCD API and is not in control of its responses, it is our intent to make it as accurate as possible and we wouldn't want to change that.

While with Terraform, even though it could be possible to have these checks, again, we would need to re-think all of our resources if there are any concurrent operations happening outside of Terraform, which undermines the (probably) biggest benefit of using the tool of having somewhat stable state that one can expect to be true.

The bottom line is: we shouldn't try to convert into concurrent an operation that was designed to be serialized in VCD.
While a retry mechanism is not suitable for the SDK, your client applications can attempt dealing with that to handle concurrent usage from different applications.