VR Mising

[hstan77]

I have my VR that deleted during restart network, however it not automatically created back.

Now all the VM that under this GuestNetwork are unable to start, it shows Insufficient Capacity due to no VR . Even though I want to delete this network from the VM and change another one also cannot delete it.

The only thing I can do is destroy the whole vm .

I am using Cloudstack 4.15 and Ubuntu 18.

[weizhouapache]

please share the logs, thanks

[hstan77]

Starting VM that missing VR
025-03-14 18:57:40,347 INFO [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-81:ctx-e1df6ca7 job-487558/job-487559 ctx-ad6c35ae) (logid:ed251ac2) Unable to contact resource.
com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is unreachable: Can't find at least one running router!
at com.cloud.network.element.VirtualRouterElement.prepare(VirtualRouterElement.java:288)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareElement(NetworkOrchestrator.java:1485)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareNic(NetworkOrchestrator.java:1840)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(NetworkOrchestrator.java:1774)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1149)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5464)
at jdk.internal.reflect.GeneratedMethodAccessor424.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5631)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)

=======+Restarting network that missing VR ===
com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is unreachable: Can't find all necessary running routers!
at com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:251)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1392)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElementsAndResources(NetworkOrchestrator.java:1327)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:3166)
at com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1994)
at com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:2012)
at jdk.internal.reflect.GeneratedMethodAccessor716.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy256.restartNetwork(Unknown Source)
at org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2025-03-14 18:59:52,200 WARN [c.c.n.NetworkServiceImpl] (API-Job-Executor-100:ctx-310b6a64 job-487561 ctx-99dba8c4) (logid:5b299583) Network id=217 failed to restart.
2025-03-14 18:59:52,223 DEBUG [c.c.a.ApiServlet] (qtp1886491834-1071:ctx-467ab645) (logid:7657

[hstan77]

Any body can let me know to manual get the VR up . or Some Db record need to update it .

[DaanHoogland]

@hstan77 , is there enough capacity to start the VR ?

[weizhouapache]

@hstan77

I think there are some exceptions before the log you shared. can you have a look ?

also, can you restart the network with cleanup ?

[hstan77]

Hi

It is enaugh to start VR, a lot more

Cleaning up , it will say no VR to clean up

[hstan77]

i grep WARN and ERR

**restart Network

2025-03-14 19:55:30,234 WARN [c.c.a.ApiServer] (qtp1886491834-1084:ctx-584ca2a8 ctx-a2113a51) (logid:6033a140) Unknown API command: listAndSwitchSamlAccount
2025-03-14 19:55:46,110 WARN [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-24:ctx-52d1ff6b job-488646 ctx-faec8381) (logid:2394dcae) Failed to implement network Ntwk[217|Guest|6] elements and resources as a part of network restart due to
2025-03-14 19:55:46,112 WARN [c.c.n.NetworkServiceImpl] (API-Job-Executor-24:ctx-52d1ff6b job-488646 ctx-faec8381) (logid:2394dcae) Network id=217 failed to restart.

2025-03-14 19:57:30,843 ERROR [o.c.n.r.d.RouterDeploymentDefinition] (API-Job-Executor-100:ctx-8c1177ba job-488649 ctx-3c1be356) (logid:7d78035b) Didn't support redundant virtual router without public network!

** restart network and clean up
2025-03-14 19:56:58,533 ERROR [o.c.n.r.d.RouterDeploymentDefinition] (API-Job-Executor-87:ctx-8d18e4fc job-488648 ctx-d509981e) (logid:518baef4) Didn't support redundant virtual router without public network!
2025-03-14 19:56:58,560 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-87:ctx-8d18e4fc job-488648) (logid:518baef4) Unexpected exception while executing org.apache.cloudstack.api.command.user.network.RestartNetworkCmd

[hstan77]

This is detail with debug log

2025-03-14 19:58:24,165 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-50:ctx-8c72f876 job-488650) (logid:345a719c) Add job-488650 into job monitoring
2025-03-14 19:58:24,169 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 8844, power state: PowerOn
2025-03-14 19:58:24,173 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (qtp1886491834-1113:ctx-cdaaea5e ctx-3c16e021) (logid:a4093f5d) submit async job-488650, details: AsyncJobVO {id:488650, userId: 2, accountId: 2, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.user.network.RestartNetworkCmd, cmdInfo: {"makeredundant":"false","cleanup":"false","response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"3079878","id":"a3cca856-973e-4a6a-aa9d-d4c908938cc7","ctxDetails":"{"interface com.cloud.network.Network":"a3cca856-973e-4a6a-aa9d-d4c908938cc7"}","ctxAccountId":"2","uuid":"a3cca856-973e-4a6a-aa9d-d4c908938cc7","cmdEventType":"NETWORK.RESTART"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 2130706689, completeMsid: null, lastUpdated: null, lastPolled: null, created: null, removed: null}
2025-03-14 19:58:24,175 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 8848, power state: PowerOn
2025-03-14 19:58:24,176 DEBUG [c.c.a.ApiServlet] (qtp1886491834-1113:ctx-cdaaea5e ctx-3c16e021) (logid:a4093f5d) ===END=== 180.75.249.24 -- GET id=a3cca856-973e-4a6a-aa9d-d4c908938cc7&cleanup=false&makeredundant=false&command=restartNetwork&response=json
2025-03-14 19:58:24,180 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 1940, power state: PowerOn
2025-03-14 19:58:24,181 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-50:ctx-8c72f876 job-488650) (logid:3fef167a) Executing AsyncJobVO {id:488650, userId: 2, accountId: 2, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.user.network.RestartNetworkCmd, cmdInfo: {"makeredundant":"false","cleanup":"false","response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"3079878","id":"a3cca856-973e-4a6a-aa9d-d4c908938cc7","ctxDetails":"{"interface com.cloud.network.Network":"a3cca856-973e-4a6a-aa9d-d4c908938cc7"}","ctxAccountId":"2","uuid":"a3cca856-973e-4a6a-aa9d-d4c908938cc7","cmdEventType":"NETWORK.RESTART"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 2130706689, completeMsid: null, lastUpdated: null, lastPolled: null, created: null, removed: null}
2025-03-14 19:58:24,185 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 9621, power state: PowerOn
2025-03-14 19:58:24,191 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 8857, power state: PowerOn
2025-03-14 19:58:24,197 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 9628, power state: PowerOn
2025-03-14 19:58:24,204 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 1181, power state: PowerOn
2025-03-14 19:58:24,210 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 1951, power state: PowerOn
2025-03-14 19:58:24,215 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 1445, power state: PowerOn
2025-03-14 19:58:24,221 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 9642, power state: PowerOn
2025-03-14 19:58:24,228 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 1196, power state: PowerOn
2025-03-14 19:58:24,235 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 1199, power state: PowerOn
2025-03-14 19:58:24,242 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 433, power state: PowerOn
2025-03-14 19:58:24,249 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 8627, power state: PowerOn
2025-03-14 19:58:24,253 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-50:ctx-8c72f876 job-488650 ctx-e17cb53d) (logid:3fef167a) Restarting network 217...
2025-03-14 19:58:24,255 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 7352, power state: PowerOn
2025-03-14 19:58:24,258 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-50:ctx-8c72f876 job-488650 ctx-e17cb53d) (logid:3fef167a) Implementing the network Ntwk[217|Guest|6] elements and resources as a part of network restart without cleanup
2025-03-14 19:58:24,260 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 1209, power state: PowerOn
2025-03-14 19:58:24,266 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 1212, power state: PowerOn
2025-03-14 19:58:24,271 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 7358, power state: PowerOn
2025-03-14 19:58:24,277 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 8640, power state: PowerOn
2025-03-14 19:58:24,283 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 960, power state: PowerOn
2025-03-14 19:58:24,289 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 7620, power state: PowerOn
2025-03-14 19:58:24,295 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 965, power state: PowerOn
2025-03-14 19:58:24,300 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 967, power state: PowerOn
2025-03-14 19:58:24,306 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 970, power state: PowerOn
2025-03-14 19:58:24,311 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 5834, power state: PowerOn
2025-03-14 19:58:24,317 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 8139, power state: PowerOn
2025-03-14 19:58:24,323 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 2000, power state: PowerOn
2025-03-14 19:58:24,325 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-50:ctx-8c72f876 job-488650 ctx-e17cb53d) (logid:3fef167a) Asking SecurityGroupProvider to implemenet Ntwk[217|Guest|6]
2025-03-14 19:58:24,328 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 980, power state: PowerOn
2025-03-14 19:58:24,333 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 981, power state: PowerOn
2025-03-14 19:58:24,339 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 983, power state: PowerOn
2025-03-14 19:58:24,345 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 2007, power state: PowerOn
2025-03-14 19:58:24,351 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 984, power state: PowerOn
2025-03-14 19:58:24,356 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 6106, power state: PowerOn
2025-03-14 19:58:24,361 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 2012, power state: PowerOn
2025-03-14 19:58:24,367 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 991, power state: PowerOn
2025-03-14 19:58:24,372 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 2017, power state: PowerOn
2025-03-14 19:58:24,375 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-50:ctx-8c72f876 job-488650 ctx-e17cb53d) (logid:3fef167a) Asking VirtualRouter to implemenet Ntwk[217|Guest|6]
2025-03-14 19:58:24,378 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 2019, power state: PowerOn
2025-03-14 19:58:24,384 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 996, power state: PowerOn
2025-03-14 19:58:24,388 ERROR [o.c.n.r.d.RouterDeploymentDefinition] (API-Job-Executor-50:ctx-8c72f876 job-488650 ctx-e17cb53d) (logid:3fef167a) Didn't support redundant virtual router without public network!
2025-03-14 19:58:24,389 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 7142, power state: PowerOn
2025-03-14 19:58:24,395 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 7147, power state: PowerOn
2025-03-14 19:58:24,395 WARN [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-50:ctx-8c72f876 job-488650 ctx-e17cb53d) (logid:3fef167a) Failed to implement network Ntwk[217|Guest|6] elements and resources as a part of network restart due to
com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is unreachable: Can't find all necessary running routers!
at com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:251)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1392)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElementsAndResources(NetworkOrchestrator.java:1327)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:3166)
at com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1994)
at com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:2012)
at jdk.internal.reflect.GeneratedMethodAccessor716.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy256.restartNetwork(Unknown Source)
at org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2025-03-14 19:58:24,397 WARN [c.c.n.NetworkServiceImpl] (API-Job-Executor-50:ctx-8c72f876 job-488650 ctx-e17cb53d) (logid:3fef167a) Network id=217 failed to restart.

[hstan77]

When restart network, it show error cannot find router, of cause it cant find, cause the VR missing.

When restart with Clenaup, It show cannot find VR to clean also. and that is right cause VR is missing.

However my question is the VR not auto created, and how can we create it back ?

[weizhouapache]

When restart network, it show error cannot find router, of cause it cant find, cause the VR missing.

When restart with Clenaup, It show cannot find VR to clean also. and that is right cause VR is missing.

However my question is the VR not auto created, and how can we create it back ?

Can you upload the full log when start /creatr the VR ?

The logs you uploaded do not give enough information.

[hstan77]

Hi

Issue now is the VR not created automatically , i can only do is retart network as above.

The whole Cloudstack , there is no function to create VR to the GuestNetwork . Or Manual activate Create VR .

The case now is when VR not automatically create, it will be trouble. How to create back VR?

[hstan77]

I do change IP those function:
2025-03-15 03:31:34,288 ERROR [o.c.n.r.d.RouterDeploymentDefinition] (Work-Job-Executor-151:ctx-e7cd657e job-489150/job-489152 ctx-ec97e82e) (logid:8b532c8b) Didn't support redundant virtual router without public network!
com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is unreachable: Can't find at least one running router!

it just not auto create back the router.

[hstan77]

Finally i found the issue . This is Bug of Cloudstack.

The GuestNetwork , redundant=1 in Mysql, and when this happen , the VR will not auto create back . It keep say cannot create redundant . And the webinterface is no way to the status of the GuestNetwork is in redundant or not , and there is no way to turn off the redundant.

I edit mysql and set redundant=0, and restart network , and VR created back.

[weizhouapache]

Finally i found the issue . This is Bug of Cloudstack.

The GuestNetwork , redundant=1 in Mysql, and when this happen , the VR will not auto create back . It keep say cannot create redundant . And the webinterface is no way to the status of the GuestNetwork is in redundant or not , and there is no way to turn off the redundant.

I edit mysql and set redundant=0, and restart network , and VR created back.

a bit strange, why is redundant 1 ?

is it a shared network or isolated network ? what's the network offering ?

[hstan77]

I am in shared network .

Network offering : [DefaultSharedNetworkOfferingWithSGService]

i think it may cause by when restart network time, i clicked "redundant" and "clean up", after VR destroyed, it cannot up back. I guess is this, need to test how to simulate this back.