New feature: Reconcile commands (CopyCommand, MigrateCommand, MigrateVolumeCommand) #10514

weizhouapache · 2025-03-06T08:57:46Z

Description

This PR aims to improve the process on some agent commands and answers.

Current process

Many cloudstack operations require the communication between management server and cloudstack agent.
The normal process is

management server --> send commands to agents --> agents process the commands ->
agents send the answers to management server --> management server process the answers

Each operation might have one or more processes above.

Issues in some scenarios

Normally the process works fine. However, there are some issues in some scenarios

agent lost connectivity to management server
agent crashes
agent is stopped or restarted
management server is restarted (not in scope of FR doc)

Consider the following examples

agent has processed the command completely, and sent the answer to management server but the answer is not received on management server (connectivity issue or worker threads are all in use), therefore timed out on management server
agent has asked 3rd party (for example libvirt or scaleio) to process the command, but agent crashes, which leads to some inconsistent state of resources. For example vm has been migrated, volume has been copied to destination pool, but management server does not get the results.
management is restarted while vm/volume is being migrated, which leads vm/volume to be stuck at Migrating state.

Operations to address

This FR focuses on the following operations

migrate vm
migrate vm with volumes
migrate volume to another pool

The backend processes can be found at
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=337678693#AsyncAgentCommandReconciliation-4.1BackendcommandsofVMandvolumemigrations

Main changes

Design doc: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Async+Agent+Command+Reconciliation

Global settings

reconcile.commands.enabled (default: false)
reconcile.commands.interval (default: 60)
reconcile.commands.max.attempts (default: 30)
reconcile.commands.workers (default: 100)

New terminology: Reconcile commands

Add command state: CommandInfo.State
- Management server: CREATED -> COMPLETED/FAILED or INTERRUPTED/TIMED_OUT -> RECONCILING -> RECONCILED/RECONCILE_RETRY/RECONCILE_FAILED
- Agent: STARTED -> PROCESSING -> COMPLETED/FAILED/INTERRUPTED
Add property to Command: isReconcile (false by default). true for 3 commands
- CopyCommand
- MigrateCommand
- MigrateVolumeCommand

How it works

management server creates a record in reconcile_commands table when create the command
agent updates the command/answer in a JSON file while process the command
agent syncs with management server every 60 seconds (ping.interval)
when management server receives the update from agent, it updates the database
when agent receives the PingAnswer form management server, it removes the JSON file with state=COMPLETED/FAILED

For reconcile commands, during stop/start of mgmt server and agent

when agent is stopped/started, it updates the state (by agent) to INTERUPTED in JSON files and sync with mgmt server
when mgmt server is stopped/started, it updates the state (by management) to INTERUPTED in database
Every minute, management server loads the reconcile commands and reconcile the command in INTERUPTED or TIMED_OUT state

Improvement on management server when wait for the answer of reconcile commands

meanwhile, every 60 seconds, it reads the state and answer from database
if state is INTERUPTED or FAILED, it terminates the process
if state is COMPLETED and answer is null, it processes the answer as normal
This fixes the intermittent connection failure between agent and management server
when times out, it updates the state (by mgmt) to TIMED_OUT in database for further reconciliation

Improvement on VM migration w/wo volumes

If the operation fails with operation timeout exception, it might because of the connection failure between soource host and management server
check if the vm is Running on destination host, if yes, consider the migration succceed
if succeed, consider migration as success, and commit the network and volume changes.
if not, destroy the vm on destination host and rollback the network/volume changes

Fixes after Volume migration

on NFS, fix the new volumes at Creating state after failed migration
on Powerflex, fix the read-only VM issue by checking if volume has been migrated by checking via ScaleIO gateway

Improvement on Agent

Add MigrationCancelHook to terminate vm migration jobs when agent is disconnected or restarted
Add VolumeMigrationCancelHook to terminate block copy jobs when agent is disconnected or restarted

Test results

It has been tested by dev on NFS and Powerflex

Refer to https://cwiki.apache.org/confluence/display/CLOUDSTACK/Async+Agent+Command+Reconciliation#AsyncAgentCommandReconciliation-4.3Summaryoftestresults

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

codecov · 2025-03-06T09:05:31Z

Codecov Report

Attention: Patch coverage is 11.11686% with 1719 lines in your changes missing coverage. Please review.

Project coverage is 16.28%. Comparing base (4c31f9d) to head (b4bc020).

Files with missing lines	Patch %	Lines
...loudstack/command/ReconcileCommandServiceImpl.java	0.00%	751 Missing ⚠️
...source/wrapper/LibvirtReconcileCommandWrapper.java	0.62%	160 Missing ⚠️
.../apache/cloudstack/command/ReconcileCommandVO.java	0.00%	96 Missing ⚠️
...ache/cloudstack/command/ReconcileCommandUtils.java	37.30%	73 Missing and 6 partials ⚠️
...ervisor/kvm/resource/LibvirtComputingResource.java	26.16%	76 Missing and 3 partials ⚠️
...loudstack/command/dao/ReconcileCommandDaoImpl.java	0.00%	70 Missing ⚠️
...ain/java/com/cloud/agent/manager/AgentAttache.java	3.70%	52 Missing ⚠️
...java/com/cloud/agent/manager/AgentManagerImpl.java	3.03%	32 Missing ⚠️
...torage/motion/StorageSystemDataMotionStrategy.java	0.00%	32 Missing ⚠️
...atastore/driver/ScaleIOPrimaryDataStoreDriver.java	21.62%	28 Missing and 1 partial ⚠️
... and 42 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #10514      +/-   ##
============================================
- Coverage     16.30%   16.28%   -0.03%     
- Complexity    13449    13483      +34     
============================================
  Files          5676     5695      +19     
  Lines        499208   501081    +1873     
  Branches      60374    60657     +283     
============================================
+ Hits          81414    81612     +198     
- Misses       408722   410376    +1654     
- Partials       9072     9093      +21

Flag	Coverage Δ
uitests	`3.99% <ø> (ø)`
unittests	`17.14% <11.11%> (-0.03%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…VolumeCommand)

weizhouapache · 2025-03-06T09:40:24Z

@blueorangutan package

blueorangutan · 2025-03-06T09:42:06Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-03-06T11:11:15Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12678

weizhouapache · 2025-03-06T11:13:04Z

@blueorangutan test matrix

blueorangutan · 2025-03-06T11:14:04Z

@weizhouapache a [SL] Trillian-Jenkins matrix job (EL8 mgmt + EL8 KVM, Ubuntu22 mgmt + Ubuntu22 KVM, EL8 mgmt + VMware 7.0u3, EL9 mgmt + XCP-ng 8.2 ) has been kicked to run smoke tests

blueorangutan · 2025-03-06T11:49:19Z

[SF] Trillian Build Failed (tid-12593)

blueorangutan · 2025-03-06T13:27:24Z

[SF] Trillian Build Failed (tid-12594)

blueorangutan · 2025-03-07T04:11:57Z

[SF] Trillian test result (tid-12592)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 58377 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10514-t12592-kvm-ol8.zip
Smoke tests completed. 140 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestSharedNetworkWithConfigDrive>:setup	`Error`	1521.32	test_network.py

blueorangutan · 2025-03-07T09:17:29Z

[SF] Trillian test result (tid-12595)
Environment: xcpng82 (x2), Advanced Networking with Mgmt server ol9
Total time taken: 77369 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10514-t12595-xcpng82.zip
Smoke tests completed. 135 look OK, 6 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestClusterDRS>:setup	`Error`	0.00	test_cluster_drs.py
test_01_prepare_and_cancel_maintenance	`Error`	0.11	test_ms_maintenance_and_safe_shutdown.py
ContextSuite context=TestSharedNetworkWithConfigDrive>:setup	`Error`	10.65	test_network.py
test_01_non_strict_host_anti_affinity	`Error`	241.59	test_nonstrict_affinity_group.py
test_02_non_strict_host_affinity	`Error`	121.60	test_nonstrict_affinity_group.py
test_02_create_volume	`Error`	5.30	test_resource_names.py
test_05_scale_vm_dont_allow_disk_offering_change	`Failure`	69.74	test_scale_vm.py

weizhouapache · 2025-03-13T09:16:51Z

@blueorangutan package

blueorangutan · 2025-03-13T09:18:03Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-03-13T10:29:42Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12766

rohityadavcloud · 2025-03-16T07:00:43Z

@blueorangutan test matrix

- add column resource_id and resource_type - set resource_id and resource_type based on the srcData - skip Migrating VMs with reconcile commands - clean up volumes attached to Migrating vms without reconcile commands - skip Migrating volumes with reconcile commands - skip Migrating volumes attached to vms with reconcile commands - cleanup volumes in Migrating/Creating with last_id - add vmName to ReconcileMigrateAnswer

weizhouapache · 2025-03-26T10:07:07Z

@blueorangutan package

blueorangutan · 2025-03-26T10:08:04Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-03-26T11:20:06Z

Packaging result [SF]: ✔️ el8 ✖️ el9 ✔️ debian ✖️ suse15. SL-JID 12892

weizhouapache · 2025-03-26T11:35:01Z

@blueorangutan package

blueorangutan · 2025-03-26T11:36:03Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-03-26T12:41:21Z

Packaging result [SF]: ✔️ el8 ✖️ el9 ✔️ debian ✖️ suse15. SL-JID 12894

weizhouapache · 2025-03-26T13:05:28Z

@blueorangutan package

blueorangutan · 2025-03-26T13:06:03Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-03-26T14:23:03Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12895

weizhouapache · 2025-03-28T08:03:34Z

@blueorangutan package

blueorangutan · 2025-03-28T08:04:05Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-03-28T09:08:16Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12914

blueorangutan · 2025-03-29T05:02:54Z

[SF] Trillian test result (tid-12844)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 55913 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10514-t12844-kvm-ol8.zip
Smoke tests completed. 141 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

vladimirpetrov

LGTM based on manual testing. I tested the following scenarios:

Migrate volume from one storage to another during connection failures
Migrate volume from one storage to another while agent crashes
Migrate volume from one storage to another while agent restarts
Migrate volume from one storage to another while agent times out
Migrate volume from one storage to another while management server restarts
Migrate VM to another host during connection failures
Migrate VM to another host while agent crashes
Migrate VM to another host while agent restarts
Migrate VM to another host while agent times out
Migrate VM to another host while management server restarts
Migrate VM with volumes during connection failures
Migrate VM with volumes while agent crashes
Migrate VM with volumes while agent restarts
Migrate VM with volumes while agent times out
Migrate VM with volumes while management server restarts

with NFS, Ceph and local storages.

harikrishna-patnala

code LGTM, very good detailing in all the related docs @weizhouapache thanks.

weizhouapache · 2025-04-09T11:49:41Z

@blueorangutan package

blueorangutan · 2025-04-09T11:52:03Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-04-09T13:14:42Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12993

weizhouapache · 2025-04-09T13:49:54Z

@blueorangutan test

blueorangutan · 2025-04-09T13:52:04Z

@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-04-10T22:44:22Z

[SF] Trillian test result (tid-12928)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 115341 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10514-t12928-kvm-ol8.zip
Smoke tests completed. 104 look OK, 37 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_events_resource	`Error`	7.85	test_events_resource.py
test_01_events_resource	`Error`	7.85	test_events_resource.py
ContextSuite context=TestAccounts>:setup	`Error`	0.00	test_accounts.py
ContextSuite context=TestAddVmToSubDomain>:setup	`Error`	0.00	test_accounts.py
test_DeleteDomain	`Error`	14.51	test_accounts.py
test_forceDeleteDomain	`Failure`	14.22	test_accounts.py
ContextSuite context=TestRemoveUserFromAccount>:setup	`Error`	16.10	test_accounts.py
ContextSuite context=TestTemplateHierarchy>:setup	`Error`	1536.22	test_accounts.py
ContextSuite context=TestDeployVmWithAffinityGroup>:setup	`Error`	0.00	test_affinity_groups_projects.py
test_replace_acl_of_network	`Error`	4.63	test_global_acls.py
ContextSuite context=TestAnnotations>:setup	`Error`	0.00	test_annotations.py
ContextSuite context=TestAsyncJob>:setup	`Error`	0.00	test_async_job.py
test_00_deploy_vm_root_resize	`Error`	1.52	test_deploy_vm_root_resize.py
ContextSuite context=TestMultipleVolumeAttach>:setup	`Error`	0.00	test_attach_multiple_volumes.py
test_01_condensed_drs_algorithm	`Error`	1.27	test_cluster_drs.py
test_02_balanced_drs_algorithm	`Error`	0.11	test_cluster_drs.py
ContextSuite context=TestClusterDRS>:teardown	`Error`	1.20	test_cluster_drs.py
ContextSuite context=TestDeployVmWithUserData>:setup	`Error`	0.00	test_deploy_vm_with_userdata.py
ContextSuite context=TestConsoleEndpoint>:setup	`Error`	0.00	test_console_endpoint.py
ContextSuite context=TestDeployVirtioSCSIVM>:setup	`Error`	0.00	test_deploy_virtio_scsi_vm.py
ContextSuite context=TestInternalLb>:setup	`Error`	0.00	test_internal_lb.py
test_02_deploy_vm_with_extraconfig_kvm	`Error`	1.39	test_deploy_vm_extra_config_data.py
test_03_update_vm_with_extraconfig_kvm	`Error`	1.37	test_deploy_vm_extra_config_data.py
ContextSuite context=TestDeployVmWithVariedPlanners>:setup	`Error`	0.00	test_deploy_vms_with_varied_deploymentplanners.py
ContextSuite context=TestDeployVMFromISO>:setup	`Error`	0.00	test_deploy_vm_iso.py
ContextSuite context=TestDeployVMsInParallel>:setup	`Error`	0.00	test_deploy_vms_in_parallel.py
ContextSuite context=TestIpv4Routing>:setup	`Error`	0.00	test_ipv4_routing.py
ContextSuite context=TestDeployVMFromISOWithUefi>:setup	`Error`	0.00	test_deploy_vm_iso_uefi.py
ContextSuite context=TestRemoteDiagnostics>:setup	`Error`	0.00	test_diagnostics.py
test_01_deploy_vm_from_direct_download_template_nfs_storage	`Error`	1.38	test_direct_download.py
ContextSuite context=TestDirectDownloadTemplates>:teardown	`Error`	1.10	test_direct_download.py
ContextSuite context=TestDomainsServiceOfferings>:setup	`Error`	1519.80	test_domain_service_offerings.py
test_03_create_vpc_domain_vpc_offering	`Error`	17.55	test_domain_vpc_offerings.py
test_updating_nics_on_two_shared_networks	`Error`	1.57	test_gateway_on_shared_networks.py
ContextSuite context=TestGatewayOnSharedNetwork>:teardown	`Error`	3.73	test_gateway_on_shared_networks.py
ContextSuite context=TestHostControlState>:setup	`Error`	53.35	test_host_control_state.py
test_01_browser_migrate_template	`Error`	65.76	test_image_store_object_migration.py
ContextSuite context=TestImportAndUnmanageVolumes>:setup	`Error`	0.00	test_import_unmanage_volumes.py
test_01_1_create_iso_with_checksum_sha1_negative	`Error`	66.57	test_iso.py
test_01_create_iso_with_checksum_sha1	`Error`	66.51	test_iso.py
test_01_create_iso_with_checksum_sha1	`Error`	66.52	test_iso.py
test_02_1_create_iso_with_checksum_sha256_negative	`Error`	66.57	test_iso.py
test_02_create_iso_with_checksum_sha256	`Error`	66.54	test_iso.py
test_02_create_iso_with_checksum_sha256	`Error`	66.54	test_iso.py
test_03_1_create_iso_with_checksum_md5_negative	`Error`	66.55	test_iso.py
test_03_create_iso_with_checksum_md5	`Error`	66.55	test_iso.py
test_03_create_iso_with_checksum_md5	`Error`	66.55	test_iso.py
test_04_create_iso_with_no_checksum	`Error`	66.53	test_iso.py
test_04_create_iso_with_no_checksum	`Error`	66.54	test_iso.py
test_01_create_iso	`Failure`	1519.01	test_iso.py
ContextSuite context=TestISO>:setup	`Error`	3036.83	test_iso.py
test_01_invalid_upgrade_kubernetes_cluster	`Failure`	0.01	test_kubernetes_clusters.py
test_02_upgrade_kubernetes_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_03_deploy_and_scale_kubernetes_cluster	`Failure`	0.01	test_kubernetes_clusters.py
test_04_autoscale_kubernetes_cluster	`Failure`	0.01	test_kubernetes_clusters.py
test_05_basic_lifecycle_kubernetes_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_06_delete_kubernetes_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_08_upgrade_kubernetes_ha_cluster	`Failure`	0.01	test_kubernetes_clusters.py
test_10_vpc_tier_kubernetes_cluster	`Failure`	0.01	test_kubernetes_clusters.py
test_11_test_unmanaged_cluster_lifecycle	`Error`	0.01	test_kubernetes_clusters.py
ContextSuite context=TestRouterServices>:setup	`Error`	0.00	test_routers.py
ContextSuite context=TestSnapshotRootDisk>:setup	`Error`	0.00	test_snapshots.py
test_01_create_template	`Error`	1.16	test_templates.py
test_CreateTemplateWithDuplicateName	`Error`	1.15	test_templates.py
test_02_create_template_with_checksum_sha1	`Error`	65.64	test_templates.py
ContextSuite context=TestTemplates>:setup	`Error`	295.78	test_templates.py
test_01_snapshot_usage	`Error`	31.01	test_usage.py
test_01_template_usage	`Error`	1.22	test_usage.py
test_01_volume_usage	`Error`	129.77	test_usage.py
test_01_scale_up_verify	`Error`	45.12	test_vm_autoscaling.py
test_02_update_vmprofile_and_vmgroup	`Failure`	245.72	test_vm_autoscaling.py
test_03_scale_down_verify	`Failure`	304.60	test_vm_autoscaling.py
test_04_stop_remove_vm_in_vmgroup	`Failure`	0.03	test_vm_autoscaling.py
test_06_autoscaling_vmgroup_on_project_network	`Error`	52.71	test_vm_autoscaling.py
test_06_autoscaling_vmgroup_on_project_network	`Error`	52.72	test_vm_autoscaling.py
test_07_autoscaling_vmgroup_on_vpc_network	`Error`	7.41	test_vm_autoscaling.py
ContextSuite context=TestVmAutoScaling>:teardown	`Error`	19.54	test_vm_autoscaling.py
test_01_deploy_vm_on_specific_host	`Error`	1.45	test_vm_deployment_planner.py
test_02_deploy_vm_on_specific_cluster	`Error`	1.34	test_vm_deployment_planner.py
test_03_deploy_vm_on_specific_pod	`Error`	1.37	test_vm_deployment_planner.py
test_04_deploy_vm_on_host_override_pod_and_cluster	`Error`	1.42	test_vm_deployment_planner.py
test_05_deploy_vm_on_cluster_override_pod	`Error`	1.41	test_vm_deployment_planner.py
ContextSuite context=TestDeployVM>:setup	`Error`	0.00	test_vm_life_cycle.py
test_01_migrate_VM_and_root_volume	`Error`	1.45	test_vm_life_cycle.py
test_02_migrate_VM_with_two_data_disks	`Error`	1.42	test_vm_life_cycle.py
test_01_secure_vm_migration	`Error`	78.62	test_vm_life_cycle.py
test_02_unsecure_vm_migration	`Error`	223.99	test_vm_life_cycle.py
test_03_secured_to_nonsecured_vm_migration	`Error`	149.31	test_vm_life_cycle.py
test_04_nonsecured_to_secured_vm_migration	`Error`	150.44	test_vm_life_cycle.py
ContextSuite context=TestVMLifeCycle>:setup	`Error`	6.70	test_vm_life_cycle.py
ContextSuite context=TestVMSchedule>:setup	`Error`	0.00	test_vm_schedule.py

boring-cyborg bot added component:agent component:api component:orchestration component:storage labels Mar 6, 2025

New feature: Reconcile commands (CopyCommand, MigrateCommand, Migrate…

f9bba6e

…VolumeCommand)

weizhouapache force-pushed the 4.21-reconcile-commands branch from 5dcdcb8 to f9bba6e Compare March 6, 2025 09:21

weizhouapache added this to the 4.21.0 milestone Mar 6, 2025

JoaoJandre self-requested a review March 6, 2025 12:01

Reconcile: fix java.lang.ArrayIndexOutOfBoundsException

3359ec8

weizhouapache added 2 commits March 13, 2025 10:15

Merge remote-tracking branch 'apache/main' into 4.21-reconcile-commands

0f6aedf

Reconcile: default reconcile.commands.enabled to false

7952497

rohityadavcloud assigned vladimirpetrov Mar 16, 2025

rohityadavcloud added the status:needs-testing label Mar 16, 2025

rohityadavcloud requested review from shwstppr, harikrishna-patnala and sureshanaparti March 16, 2025 07:00

weizhouapache added 3 commits March 26, 2025 11:00

Merge remote-tracking branch 'apache/main' into 4.21-reconcile-commands

6d8d3b0

Reconcile: process volume states if reconcile copy command is skipped

307e4fb

Reconcile: fix resource_type cannot be null

39979a6

vladimirpetrov approved these changes Apr 9, 2025

View reviewed changes

rohityadavcloud removed the status:needs-testing label Apr 9, 2025

rohityadavcloud approved these changes Apr 9, 2025

View reviewed changes

rohityadavcloud marked this pull request as ready for review April 9, 2025 10:01

rohityadavcloud assigned harikrishna-patnala Apr 9, 2025

harikrishna-patnala approved these changes Apr 9, 2025

View reviewed changes

Merge branch 'main' into 4.21-reconcile-commands

b4bc020

New feature: Reconcile commands (CopyCommand, MigrateCommand, MigrateVolumeCommand) #10514

Are you sure you want to change the base?

New feature: Reconcile commands (CopyCommand, MigrateCommand, MigrateVolumeCommand) #10514

Conversation

weizhouapache commented Mar 6, 2025 • edited Loading

Description

Current process

Issues in some scenarios

Operations to address

Main changes

Global settings

New terminology: Reconcile commands

Improvement on management server when wait for the answer of reconcile commands

Improvement on VM migration w/wo volumes

Fixes after Volume migration

Improvement on Agent

Test results

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

codecov bot commented Mar 6, 2025 • edited Loading

Codecov Report

weizhouapache commented Mar 6, 2025

blueorangutan commented Mar 6, 2025

blueorangutan commented Mar 6, 2025

weizhouapache commented Mar 6, 2025

blueorangutan commented Mar 6, 2025

blueorangutan commented Mar 6, 2025

blueorangutan commented Mar 6, 2025

blueorangutan commented Mar 7, 2025

blueorangutan commented Mar 7, 2025

weizhouapache commented Mar 13, 2025

blueorangutan commented Mar 13, 2025

blueorangutan commented Mar 13, 2025

rohityadavcloud commented Mar 16, 2025

weizhouapache commented Mar 26, 2025

blueorangutan commented Mar 26, 2025

blueorangutan commented Mar 26, 2025

weizhouapache commented Mar 26, 2025

blueorangutan commented Mar 26, 2025

blueorangutan commented Mar 26, 2025

weizhouapache commented Mar 26, 2025

blueorangutan commented Mar 26, 2025

blueorangutan commented Mar 26, 2025

weizhouapache commented Mar 28, 2025

blueorangutan commented Mar 28, 2025

blueorangutan commented Mar 28, 2025

blueorangutan commented Mar 29, 2025

vladimirpetrov left a comment

Choose a reason for hiding this comment

harikrishna-patnala left a comment

Choose a reason for hiding this comment

weizhouapache commented Apr 9, 2025

blueorangutan commented Apr 9, 2025

blueorangutan commented Apr 9, 2025

weizhouapache commented Apr 9, 2025

blueorangutan commented Apr 9, 2025

blueorangutan commented Apr 10, 2025

weizhouapache commented Mar 6, 2025 •

edited

Loading

codecov bot commented Mar 6, 2025 •

edited

Loading