Skip to content

ui,api,server: template categorization based on os #10773

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 28 commits into
base: 4.20
Choose a base branch
from

Conversation

shwstppr
Copy link
Contributor

@shwstppr shwstppr commented Apr 24, 2025

Description

Adds new interface for image selection (template/iso) for an instance in UI.
Old interface can still be used and it can be configured using UI configuration (config.json)

OS categories/Guest OS categories have been improved with ability to create new categories, delete an existing category, and marking a category as featured to allow it to show up in the UI in the image selection interface.

New APIs added:

  • addOsCategory
  • deleteOsCategory
  • updateOsCategory

APIs updated:

  • updateOsType
  • listTemplates
  • listOsCategories

Several improvements in UI especially related to forms - DeloyVM, ReinstallVM, CreateVnfAppliance, AddAutoscaleGroup.

DeployVM form can now be opened from template/ISO details view with query params.

Reorganized (removed and added some) OS categories to the following (in the same order):

1. Ubuntu
2. Debian
3. Fedora
4. CentOS
5. Rocky Linux
6. Alma Linux
7. Oracle
8. RedHat
9. SUSE
10. Windows
11. Other

Documentation PR: apache/cloudstack-documentation#500

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • build/CI
  • test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

image

How Has This Been Tested?

How did you try to break this feature and the system with this change?

@shwstppr shwstppr changed the title Deployvm improvements ui,api,server: template categorization based on os Apr 24, 2025
Copy link

codecov bot commented Apr 24, 2025

Codecov Report

Attention: Patch coverage is 57.03971% with 119 lines in your changes missing coverage. Please review.

Project coverage is 16.14%. Comparing base (7e2aa0e) to head (9434391).
Report is 2 commits behind head on 4.20.

Files with missing lines Patch % Lines
...in/java/com/cloud/server/ManagementServerImpl.java 67.36% 27 Missing and 4 partials ⚠️
.../command/admin/guest/DeleteGuestOsCategoryCmd.java 0.00% 15 Missing ⚠️
.../command/admin/guest/UpdateGuestOsCategoryCmd.java 40.00% 15 Missing ⚠️
...api/command/admin/guest/AddGuestOsCategoryCmd.java 36.84% 12 Missing ⚠️
...ain/java/com/cloud/api/query/QueryManagerImpl.java 0.00% 11 Missing ⚠️
...ain/java/com/cloud/storage/dao/GuestOSDaoImpl.java 0.00% 9 Missing ⚠️
...main/java/com/cloud/storage/GuestOSCategoryVO.java 41.66% 7 Missing ⚠️
...i/command/user/guest/ListGuestOsCategoriesCmd.java 81.81% 4 Missing ⚠️
.../java/com/cloud/storage/dao/VMTemplateDaoImpl.java 87.50% 2 Missing and 2 partials ⚠️
...tack/api/command/admin/guest/UpdateGuestOsCmd.java 50.00% 3 Missing ⚠️
... and 3 more
Additional details and impacted files
@@             Coverage Diff              @@
##               4.20   #10773      +/-   ##
============================================
+ Coverage     16.13%   16.14%   +0.01%     
- Complexity    13217    13255      +38     
============================================
  Files          5651     5660       +9     
  Lines        496748   497474     +726     
  Branches      60184    60370     +186     
============================================
+ Hits          80137    80323     +186     
- Misses       407688   408221     +533     
- Partials       8923     8930       +7     
Flag Coverage Δ
uitests 3.96% <ø> (-0.05%) ⬇️
unittests 17.01% <57.03%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@DaanHoogland
Copy link
Contributor

looks nice. would be great for end-users.

Adds new interface for image selection (template/iso) for an instance in
UI.
Old interface can still be used and it can be configured using UI
configuration (config.json)

OS categories/Guest OS categories have been improved with ability to
create new categories, delete an existing category, and marking a
category as featured to allow it to show up in the UI in the image
selection interface.

New APIs added:
- addOsCategory
- deleteOsCategory
- updateOsCategory

APIs updated:
- updateOsType
- listTemplates
- listOsCategories

Several improvements in UI especially related to forms - DeloyVM,
ReinstallVM, CreateVnfAppliance, AddAutoscaleGroup.

DeployVM form can now be opened from template details view with query
params.

Signed-off-by: Abhishek Kumar <[email protected]>
@shwstppr shwstppr force-pushed the deployvm-improvements branch from f9852f0 to 90294a6 Compare April 28, 2025 12:35
Signed-off-by: Abhishek Kumar <[email protected]>
@shwstppr
Copy link
Contributor Author

@blueorangutan package

@blueorangutan
Copy link

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13202

show only required elements when query params are passed

Signed-off-by: Abhishek Kumar <[email protected]>
@shwstppr
Copy link
Contributor Author

@blueorangutan package

@blueorangutan
Copy link

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13221

Signed-off-by: Abhishek Kumar <[email protected]>
@shwstppr shwstppr requested a review from vishesh92 April 30, 2025 08:54
@vishesh92 vishesh92 requested a review from Copilot April 30, 2025 11:34
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces an enhanced OS categorization interface for both the UI and API by adding new endpoints and updating existing ones to support operations such as adding, updating, and deleting OS categories. Key changes include:

  • New API commands for managing OS categories (add, update, delete).
  • Enhancements in the DAO and VO layers to support the new category properties such as featured and sortKey.
  • Updates to tests and response objects to incorporate the new fields.

Reviewed Changes

Copilot reviewed 55 out of 55 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
engine/schema/src/main/java/com/cloud/storage/dao/GuestOSDaoImpl.java Added a new method to list IDs by category.
engine/schema/src/main/java/com/cloud/storage/GuestOSCategoryDaoImpl.java Adjusted SQL count query for consistency.
engine/schema/src/main/java/com/cloud/storage/GuestOSCategoryVO.java Included new columns (featured, sortKey) with their getters/setters.
api/.../*.java Introduced and updated API commands and responses for managing OS categories.
api/src/main/java/org/apache/cloudstack/api/ApiConstants.java Added new constant for ISO support.
api/src/main/java/com/cloud/server/ResourceTag.java Added a new resource tag for Guest OS category.
api/src/main/java/org/apache/cloudstack/api/ResponseGenerator.java Added response generation method for Guest OS categories.
api/src/main/java/org/apache/cloudstack/api/ManagementService.java Declared new methods for OS category operations.

@@ -71,7 +77,7 @@ public String getOsDisplayName() {
return osDisplayName;
}

public Map getDetails() {
public Map<String, String> getDetails() {
Copy link
Preview

Copilot AI Apr 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The getDetails() method initializes a new empty map and then checks if it is not empty, which will always return false, effectively ignoring the 'details' parameter intended to be processed. Also, remove the redundant semicolon in the map initialization ("new HashMap<>();;").

Copilot uses AI. Check for mistakes.

Signed-off-by: Abhishek Kumar <[email protected]>
Signed-off-by: Abhishek Kumar <[email protected]>
@shwstppr
Copy link
Contributor Author

@blueorangutan package

@blueorangutan
Copy link

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13300

@shwstppr
Copy link
Contributor Author

shwstppr commented May 8, 2025

@blueorangutan test

@blueorangutan
Copy link

@shwstppr a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

@apache apache deleted a comment from blueorangutan May 8, 2025
@apache apache deleted a comment from blueorangutan May 8, 2025
@shwstppr
Copy link
Contributor Author

shwstppr commented May 8, 2025

@blueorangutan package

@blueorangutan
Copy link

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13305

@blueorangutan
Copy link

[SF] Trillian test result (tid-13239)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 58969 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10773-t13239-kvm-ol8.zip
Smoke tests completed. 141 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File

@github-project-automation github-project-automation bot moved this from In Progress to Done in ACS 4.20.1 May 9, 2025
@rohityadavcloud
Copy link
Member

@shwstppr is this ready for review/testing now, or is this still dev in progress?

@shwstppr
Copy link
Contributor Author

shwstppr commented May 9, 2025

@rohityadavcloud this is largely ready for review/testing but awaiting internal testing. Also, I may add some fixes if I find any issues

Copy link

github-actions bot commented May 9, 2025

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

@shwstppr
Copy link
Contributor Author

shwstppr commented May 9, 2025

@blueorangutan package

@blueorangutan
Copy link

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13316

@shwstppr
Copy link
Contributor Author

@blueorangutan test

@blueorangutan
Copy link

@shwstppr a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

@blueorangutan
Copy link

[SF] Trillian test result (tid-13253)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 64178 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10773-t13253-kvm-ol8.zip
Smoke tests completed. 140 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
test_02_deploy_vm_with_extraconfig_kvm Error 104.90 test_deploy_vm_extra_config_data.py

@shwstppr
Copy link
Contributor Author

An unrelated, weird exception failed the test,

2025-05-10 11:42:27,682 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-146:[ctx-34767c04, job-458/job-461, ctx-84b3b693]) (logid:bf45ed95) Host does not have enough reserved CPU available, cannot allocate to this host.
2025-05-10 11:42:27,682 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-146:[ctx-34767c04, job-458/job-461, ctx-84b3b693]) (logid:bf45ed95) Checking if host: Host {"id":1,"name":"pr10773-t13253-kvm-ol8-kvm1","type":"Routing","uuid":"8a91c98f-44c0-4de8-9ecd-0b939865d323"} has enough capacity for requested CPU: 100 and requested RAM: (256.00 MB) 268435456 , cpuOverprovisioningFactor: 2.0
2025-05-10 11:42:27,683 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-146:[ctx-34767c04, job-458/job-461, ctx-84b3b693]) (logid:bf45ed95) Hosts's actual total CPU: 6300 and CPU after applying overprovisioning: 12600
2025-05-10 11:42:27,683 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-146:[ctx-34767c04, job-458/job-461, ctx-84b3b693]) (logid:bf45ed95) Free CPU: 11600 , Requested CPU: 100
2025-05-10 11:42:27,683 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-146:[ctx-34767c04, job-458/job-461, ctx-84b3b693]) (logid:bf45ed95) Free RAM: (5.26 GB) 5648531456 , Requested RAM: (256.00 MB) 268435456
2025-05-10 11:42:27,683 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-146:[ctx-34767c04, job-458/job-461, ctx-84b3b693]) (logid:bf45ed95) Host has enough CPU and RAM available
2025-05-10 11:42:27,683 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-146:[ctx-34767c04, job-458/job-461, ctx-84b3b693]) (logid:bf45ed95) STATS: Can alloc CPU from host: Host {"id":1,"name":"pr10773-t13253-kvm-ol8-kvm1","type":"Routing","uuid":"8a91c98f-44c0-4de8-9ecd-0b939865d323"}, used: 1000, reserved: 0, actual total: 6300, total with overprovisioning: 12600; requested cpu: 100, alloc_from_last_host?: false, considerReservedCapacity?: true
2025-05-10 11:42:27,683 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-146:[ctx-34767c04, job-458/job-461, ctx-84b3b693]) (logid:bf45ed95) STATS: Can alloc MEM from host: Host {"id":1,"name":"pr10773-t13253-kvm-ol8-kvm1","type":"Routing","uuid":"8a91c98f-44c0-4de8-9ecd-0b939865d323"}, used: (1.50 GB) 1610612736, reserved: (0 bytes) 0, total: (6.76 GB) 7259144192; requested mem: (256.00 MB) 268435456, alloc_from_last_host?: false, considerReservedCapacity?: true
2025-05-10 11:42:27,693 DEBUG [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-146:[ctx-34767c04, job-458/job-461, ctx-84b3b693]) (logid:bf45ed95) Cleaning up resources for the vm VM instance {"id":45,"instanceName":"i-2-45-VM","state":"Starting","type":"User","uuid":"166483d6-2e6e-4ce2-9b82-2636d3592147"} in Starting state
2025-05-10 11:42:27,702 DEBUG [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-146:[ctx-34767c04, job-458/job-461, ctx-84b3b693]) (logid:bf45ed95) VM instance {"id":45,"instanceName":"i-2-45-VM","state":"Starting","type":"User","uuid":"166483d6-2e6e-4ce2-9b82-2636d3592147"} is a [User], returning null for control Nic IP.
2025-05-10 11:42:27,703 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (Work-Job-Executor-146:[ctx-34767c04, job-458/job-461, ctx-84b3b693]) (logid:bf45ed95) Wait time setting on com.cloud.agent.api.StopCommand is 1800 seconds
2025-05-10 11:42:27,704 DEBUG [c.c.a.m.ClusteredAgentAttache] (Work-Job-Executor-146:[ctx-34767c04, job-458/job-461, ctx-84b3b693]) (logid:bf45ed95) Seq 4-1773855303230554692: Routed from 32987664089784
2025-05-10 11:42:27,704 DEBUG [c.c.a.t.Request] (Work-Job-Executor-146:[ctx-34767c04, job-458/job-461, ctx-84b3b693]) (logid:bf45ed95) Seq 1-1773855303230554692: Sending  { Cmd , MgmtId: 32987664089784, via: 1(pr10773-t13253-kvm-ol8-kvm1), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":"false","checkBeforeCleanup":"false","forceStop":"false","volumesToDisconnect":[],"vmName":"i-2-45-VM","executeInSequence":"false","wait":"0","bypassHostMaintenance":"false"}}] }
2025-05-10 11:42:27,750 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-0ef90b21]) (logid:30023ce3) HA health check task is running...
2025-05-10 11:42:27,977 DEBUG [c.c.a.t.Request] (AgentManager-Handler-4:[]) (logid:) Seq 1-1773855303230554692: Processing:  { Ans: , MgmtId: 32987664089784, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}] }
2025-05-10 11:42:27,977 DEBUG [c.c.a.t.Request] (Work-Job-Executor-146:[ctx-34767c04, job-458/job-461, ctx-84b3b693]) (logid:bf45ed95) Seq 1-1773855303230554692: Received:  { Ans: , MgmtId: 32987664089784, via: 1(pr10773-t13253-kvm-ol8-kvm1), Ver: v1, Flags: 10, { StopAnswer } }
2025-05-10 11:42:27,985 WARN  [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-146:[ctx-34767c04, job-458/job-461, ctx-84b3b693]) (logid:bf45ed95) Unable to release some network resources for the VM VM instance {"id":45,"instanceName":"i-2-45-VM","state":"Starting","type":"User","uuid":"166483d6-2e6e-4ce2-9b82-2636d3592147"} in Starting state java.lang.NullPointerException: Cannot invoke "com.cloud.network.dao.NetworkVO.getId()" because "vmNetwork" is null
        at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.releaseNic(NetworkOrchestrator.java:2490)
        at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.release(NetworkOrchestrator.java:2433)
        at com.cloud.vm.VirtualMachineManagerImpl.releaseVmResources(VirtualMachineManagerImpl.java:1983)
        at com.cloud.vm.VirtualMachineManagerImpl.cleanup(VirtualMachineManagerImpl.java:1973)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1442)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5473)
        at jdk.internal.reflect.GeneratedMethodAccessor710.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)

Trying again
@blueorangutan test

@blueorangutan
Copy link

@shwstppr a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

@blueorangutan
Copy link

[SF] Trillian test result (tid-13256)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 55682 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10773-t13256-kvm-ol8.zip
Smoke tests completed. 141 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

6 participants