Skip to content

Netris Network Plugin Integration with CloudStack #10458

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 64 commits into
base: main
Choose a base branch
from

Conversation

Pearl1594
Copy link
Contributor

Description

This PR adds a new network plugin - Netris

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • build/CI
  • test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Copy link

codecov bot commented Feb 24, 2025

Codecov Report

Attention: Patch coverage is 22.85992% with 3965 lines in your changes missing coverage. Please review.

Project coverage is 16.38%. Comparing base (69cf299) to head (e02262f).
Report is 71 commits behind head on main.

Files with missing lines Patch % Lines
...apache/cloudstack/service/NetrisApiClientImpl.java 3.12% 1355 Missing and 8 partials ⚠️
...a/org/apache/cloudstack/service/NetrisElement.java 12.06% 426 Missing and 4 partials ⚠️
...g/apache/cloudstack/service/NetrisServiceImpl.java 3.06% 315 Missing and 1 partial ⚠️
...ain/java/com/cloud/network/vpc/VpcManagerImpl.java 1.56% 311 Missing and 3 partials ⚠️
...va/com/cloud/network/element/NetrisProviderVO.java 0.00% 146 Missing ⚠️
...org/apache/cloudstack/resource/NetrisResource.java 44.27% 139 Missing and 7 partials ⚠️
...che/cloudstack/service/NetrisGuestNetworkGuru.java 32.25% 100 Missing and 26 partials ⚠️
...java/com/cloud/network/SDNProviderNetworkRule.java 46.25% 122 Missing ⚠️
.../cloudstack/service/NetrisProviderServiceImpl.java 43.47% 61 Missing and 4 partials ⚠️
...va/com/cloud/network/netris/NetrisNetworkRule.java 3.77% 51 Missing ⚠️
... and 113 more
Additional details and impacted files
@@             Coverage Diff              @@
##               main   #10458      +/-   ##
============================================
+ Coverage     16.15%   16.38%   +0.22%     
- Complexity    13273    13708     +435     
============================================
  Files          5666     5713      +47     
  Lines        498081   508508   +10427     
  Branches      60267    62749    +2482     
============================================
+ Hits          80475    83306    +2831     
- Misses       408593   415984    +7391     
- Partials       9013     9218     +205     
Flag Coverage Δ
uitests 3.93% <ø> (-0.08%) ⬇️
unittests 17.24% <22.85%> (+0.23%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

@borisstoyanov
Copy link
Contributor

@blueorangutan package

@blueorangutan
Copy link

@borisstoyanov a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 12617

Pearl1594 and others added 11 commits February 28, 2025 09:30
…d API class through CloudStack (#7)

* Delegate API classes creation to the SDK and simply invoke the desired API class through CloudStack

* Pass default auth scheme for now
)

* Support adding netris provider to CloudStack

* revert marvin change

* add license and perform session check when provider is added

* add license and remove unused import

* fix build failure - uunused imports

* address comments

* fix provider name

* add Netris network element

* add license

* Add netris management APIs and netris service provider

* add license

* revert change

* remove other network elements from Netris element

* fix api name in doc generator

* remove logs

* move session alive check to CheckHealthCommand exec

* Fix zone creation wizard to configure netris provider

* Upgrade GSON version - from PR 8756

* Add additional parametes to the add Netris provider API

* add netris as a host

* add additional params to the resoponse and update UI

* Rename site to site_name

* Create Netris VPC (#8)

* Delegate API classes creation to the SDK and simply invoke the desired API class through CloudStack (#7)

* Delegate API classes creation to the SDK and simply invoke the desired API class through CloudStack

* Pass default auth scheme for now

* Drop for_nsx and for_tungten columns in favour of checking the provider on the ntwserviceprovider map table

* Remove missing setForTungsten occurrence

* Remove forNsx from VPC offerings

* Create Netris VPC

* Fix VPC offerings listing and remove unused dao

* Create VPC fixes

* Upgrade GSON version - from PR 8756

* Fix VPC creation response by using the latest SDK code

* Fix unit test

* Remove unused import

* Fix NSX unit tests after refactoring

* Add Netris key to the VLAN Details table (#10)

* Add Netris key to the VLAN Details table

* update for_<provider> column to be generic

* Fix VPC and add IPAM allocation for the VPC CIDR (#9)

* Fix VPC and add IPAM allocation for the VPC CIDR

* Remove VPC logic

* Use zoneId accountId and domainId on resources creation

* Fix naming

* Fix VR public nic issue

* Fix Netris Public IP for VPC source NAT allocation

* Add Netris VPC Subnets and vNets (#11)

* Add Netris VPC Subnets and vNets

* fix compilation errors

* Add netris subnet

* refactor naming convention to differentiate between VPC tiers and Isolated networks

* revert marvin change

* fix constructor - build failure

* Add support to filter netris offerings, delete netris provider when zone is being deleted

* Fix build

* Fix VPC creation

* Fix vnet creation

* unnecesary log

Authored-by: Pearl Dsilva <[email protected]>
* Add support to delete VNets and Subnets

* Add support to delete vnet resources

* Add support to delete vnet resources

* extract code to method

---------

Co-authored-by: nvazquez <[email protected]>
* Set up Netris Public range on new zone addition

* Add dependency to calculate subnet containing a start and end IP

* Remove unused import

* Move dependency to the netris module

* Rename Netris IP range

* Refactor logic

* Revert "Refactor logic"

This reverts commit 7ec36a81320444c37e7bb914dd895060b663411b.

* Fix setup range after adding Netris Provider

* Fix VXLAN range adding on zone creation
* add zone params to accepts management vnet

* Release vxlan associated to the netris broadcast domain type

* handle update network broadcast uri
* Update Subnet purpose for Netris Public Traffic

* search for existing subnet of common purpose type
nvazquez and others added 8 commits March 24, 2025 09:45
* Update netris VPC and tier name

* add support to update vpc tier name

* add license

* support editing names of dual stack VPCs
…ce with minimum network_id (#43)

* server: fix NPE when deploy vm on isolated network

* vpn: fix s2s vpn status is not updated

Prior to this fix
```
java.lang.IllegalArgumentException: Class com.cloud.agent.api.CheckS2SVpnConnectionsAnswer declares multiple JSON fields named 'details'; conflict is caused by fields com.cloud.agent.api.CheckS2SVpnConnectionsAnswer#details and com.cloud.agent.api.Answer#details
	at com.cloud.agent.transport.ResponseTest.testCheckS2SVpnConnectionsAnswer(ResponseTest.java:42)
```

* test: fix test_01_vpn_usage as now it is only possible to create VPN on Source NAT if it uses VR

* VR: fix unable to create remote access VPN on regular isolated network

the error is
```
  File "/opt/cloud/bin/configure.py", line 1242, in process
    self.remoteaccessvpn_iptables(self.dbag['public_interface'], public_ip, self.dbag[public_ip])
                                  ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
KeyError: 'public_interface'
```

* VR/server: configure default gateway and RA/S2S VPN on the IP/interface with minimum network_id
…th NATTED mode (#75)

fixes
```
2025-03-21T10:42:55,039 ERROR [c.c.a.ApiServer] (qtp1513608173-21:[ctx-f9c7f002, ctx-bcfe846d]) (logid:e12e798f) unhandled exception executing api command: [Ljava.lang.String;@3a1416cd java.lang.UnsupportedOperationException
        at java.base/java.util.AbstractList.add(AbstractList.java:153)
        at java.base/java.util.AbstractList.add(AbstractList.java:111)
        at org.apache.cloudstack.api.command.admin.vpc.CreateVPCOfferingCmd.getServiceProviderMapForExternalProvider(CreateVPCOfferingCmd.java:248)
```
@nvazquez
Copy link
Contributor

@blueorangutan package

@blueorangutan
Copy link

@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 12874

@nvazquez nvazquez closed this Mar 24, 2025
@nvazquez nvazquez reopened this Mar 24, 2025
@nvazquez
Copy link
Contributor

@blueorangutan package

@blueorangutan
Copy link

@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12875

@nvazquez
Copy link
Contributor

@blueorangutan package

@blueorangutan
Copy link

@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@nvazquez nvazquez closed this Mar 24, 2025
@nvazquez nvazquez reopened this Mar 24, 2025
@blueorangutan
Copy link

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 12877

@nvazquez
Copy link
Contributor

@blueorangutan package

@blueorangutan
Copy link

@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12879

Copy link

Quality Gate Failed Quality Gate failed

Failed conditions
15.4% Coverage on New Code (required ≥ 40%)
C Reliability Rating on New Code (required ≥ B)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

@nvazquez
Copy link
Contributor

@blueorangutan test

@blueorangutan
Copy link

@nvazquez a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

@blueorangutan
Copy link

[SF] Trillian test result (tid-12821)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 125411 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10458-t12821-kvm-ol8.zip
Smoke tests completed. 107 look OK, 34 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
test_DeleteDomain Error 216.47 test_accounts.py
test_DeleteDomain Error 216.48 test_accounts.py
test_forceDeleteDomain Failure 221.32 test_accounts.py
test_forceDeleteDomain Error 221.33 test_accounts.py
ContextSuite context=TestMultipleVolumeAttach>:teardown Error 34.98 test_attach_multiple_volumes.py
test_04_verify_guest_lspci Error 810.62 test_deploy_virtio_scsi_vm.py
test_06_verify_guest_lspci_again Error 810.58 test_deploy_virtio_scsi_vm.py
ContextSuite context=TestDeployVirtioSCSIVM>:teardown Error 818.89 test_deploy_virtio_scsi_vm.py
test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80 Error 227.09 test_internal_lb.py
test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80 Error 227.10 test_internal_lb.py
test_02_internallb_roundrobin_1RVPC_3VM_HTTP_port80 Error 318.10 test_internal_lb.py
test_02_internallb_roundrobin_1RVPC_3VM_HTTP_port80 Error 318.11 test_internal_lb.py
test_03_vpc_internallb_haproxy_stats_on_all_interfaces Error 165.50 test_internal_lb.py
test_03_vpc_internallb_haproxy_stats_on_all_interfaces Error 165.51 test_internal_lb.py
test_04_rvpc_internallb_haproxy_stats_on_all_interfaces Error 233.24 test_internal_lb.py
test_04_rvpc_internallb_haproxy_stats_on_all_interfaces Error 233.25 test_internal_lb.py
ContextSuite context=TestDomainsServiceOfferings>:teardown Error 807.10 test_domain_service_offerings.py
ContextSuite context=TestImportAndUnmanageVolumes>:teardown Error 20.30 test_import_unmanage_volumes.py
test_01_invalid_upgrade_kubernetes_cluster Failure 164.35 test_kubernetes_clusters.py
test_02_upgrade_kubernetes_cluster Failure 166.53 test_kubernetes_clusters.py
test_03_deploy_and_scale_kubernetes_cluster Failure 167.39 test_kubernetes_clusters.py
test_04_autoscale_kubernetes_cluster Failure 163.38 test_kubernetes_clusters.py
test_05_basic_lifecycle_kubernetes_cluster Failure 168.36 test_kubernetes_clusters.py
test_06_delete_kubernetes_cluster Failure 165.34 test_kubernetes_clusters.py
test_08_upgrade_kubernetes_ha_cluster Failure 237.51 test_kubernetes_clusters.py
test_10_vpc_tier_kubernetes_cluster Error 156.74 test_kubernetes_clusters.py
ContextSuite context=TestListVolumes>:teardown Error 15.61 test_list_volumes.py
test_list_management_server_metrics Error 0.23 test_metrics_api.py
test_01_verify_ipv6_network Error 589.75 test_network_ipv6.py
test_01_verify_ipv6_network Error 589.78 test_network_ipv6.py
test_04_deploy_vm_for_other_user_and_test_vm_operations Failure 62.48 test_network_permissions.py
ContextSuite context=TestNetworkPermissions>:teardown Error 1.46 test_network_permissions.py
test_delete_account Error 124.27 test_network.py
test_01_port_fwd_on_src_nat Error 3.30 test_network.py
test_02_port_fwd_on_non_src_nat Error 11.69 test_network.py
test_02_port_fwd_on_non_src_nat Error 11.69 test_network.py
ContextSuite context=TestPortForwarding>:teardown Error 25.10 test_network.py
test_reboot_router Error 227.31 test_network.py
test_releaseIP Error 105.67 test_network.py
test_releaseIP_using_IP Error 107.83 test_network.py
test_network_rules_acquired_public_ip_2_nat_rule Error 6.48 test_network.py
ContextSuite context=TestRouterRules>:teardown Error 50.71 test_network.py
test_01_deployVMInSharedNetwork Error 131.17 test_network.py
test_01_deployVMInSharedNetwork Error 131.19 test_network.py
test_isolate_network_password_server Error 3.33 test_password_server.py
ContextSuite context=TestIsolatedNetworksPasswdServer>:teardown Error 12.65 test_password_server.py
test_01_create_delete_portforwarding_fornonvpc Error 158.33 test_portforwardingrules.py
test_01_create_delete_portforwarding_fornonvpc Error 158.33 test_portforwardingrules.py
test_01_primary_storage_nfs Error 0.21 test_primary_storage.py
ContextSuite context=TestStorageTags>:setup Error 0.34 test_primary_storage.py
test_01_primary_storage_scope_change Error 0.13 test_primary_storage_scope.py
test_02_vpc_privategw_static_routes Error 268.59 test_privategw_acl.py
test_02_vpc_privategw_static_routes Error 268.61 test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanup Error 267.90 test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanup Error 267.92 test_privategw_acl.py
test_04_rvpc_privategw_static_routes Error 423.75 test_privategw_acl.py
test_04_rvpc_privategw_static_routes Error 423.77 test_privategw_acl.py
test_deploy_vm_with_registered_userdata Error 113.19 test_register_userdata.py
test_deploy_vm_with_registered_userdata Error 113.19 test_register_userdata.py
test_deploy_vm_with_registered_userdata_with_override_policy_allow Error 117.41 test_register_userdata.py
test_deploy_vm_with_registered_userdata_with_override_policy_allow Error 117.42 test_register_userdata.py
test_deploy_vm_with_registered_userdata_with_override_policy_append Error 114.24 test_register_userdata.py
test_deploy_vm_with_registered_userdata_with_override_policy_append Error 114.24 test_register_userdata.py
test_deploy_vm_with_registered_userdata_with_params Error 119.20 test_register_userdata.py
test_deploy_vm_with_registered_userdata_with_params Error 119.21 test_register_userdata.py
test_router_dhcphosts Error 4.40 test_router_dhcphosts.py
ContextSuite context=TestRouterDHCPHosts>:teardown Error 27.03 test_router_dhcphosts.py
test_router_dhcp_opts Error 4.49 test_router_dhcphosts.py
ContextSuite context=TestRouterDHCPOpts>:teardown Error 30.21 test_router_dhcphosts.py
test_router_dns_guestipquery Error 3.28 test_router_dns.py
ContextSuite context=TestRouterDns>:teardown Error 16.64 test_router_dns.py
test_router_dns_guestipquery Error 3.33 test_router_dnsservice.py
ContextSuite context=TestRouterDnsService>:teardown Error 13.64 test_router_dnsservice.py
test_01_single_VPC_iptables_policies Error 186.41 test_routers_iptables_default_policy.py
test_01_isolate_network_FW_PF_default_routes_egress_true Error 109.57 test_routers_network_ops.py
test_01_isolate_network_FW_PF_default_routes_egress_true Error 109.58 test_routers_network_ops.py
test_02_isolate_network_FW_PF_default_routes_egress_false Error 116.72 test_routers_network_ops.py
test_01_RVR_Network_FW_PF_SSH_default_routes_egress_true Error 181.26 test_routers_network_ops.py
test_01_RVR_Network_FW_PF_SSH_default_routes_egress_true Error 181.27 test_routers_network_ops.py
test_02_RVR_Network_FW_PF_SSH_default_routes_egress_false Error 197.59 test_routers_network_ops.py
test_01_service_offering_cpu_limit_use Error 811.22 test_service_offerings.py
test_04_change_offering_small Failure 913.25 test_service_offerings.py
ContextSuite context=TestSharedFSLifecycle>:setup Error 0.00 test_sharedfs_lifecycle.py
ContextSuite context=TestSnapshotRootDisk>:teardown Error 60.38 test_snapshots.py
ContextSuite context=TestSnapshotStandaloneBackup>:setup Error 345.87 test_snapshots.py
test_01_nat_usage Error 5.23 test_usage.py
test_01_secure_vm_migration Error 343.96 test_vm_life_cycle.py
test_02_unsecure_vm_migration Error 446.11 test_vm_life_cycle.py
test_03_secured_to_nonsecured_vm_migration Error 384.68 test_vm_life_cycle.py
test_04_nonsecured_to_secured_vm_migration Error 352.02 test_vm_life_cycle.py
test_09_expunge_vm Failure 427.71 test_vm_life_cycle.py
test_10_attachAndDetach_iso Failure 816.96 test_vm_life_cycle.py
test_13_destroy_and_expunge_vm Error 58.76 test_vm_life_cycle.py
test_01_create_vm_snapshots Failure 802.42 test_vm_snapshots.py
test_02_revert_vm_snapshots Failure 810.67 test_vm_snapshots.py
test_03_delete_vm_snapshots Failure 0.02 test_vm_snapshots.py
test_01_create_volume Failure 815.09 test_volumes.py
ContextSuite context=TestCreateVolume>:teardown Error 828.51 test_volumes.py
test_01_root_volume_encryption Failure 1102.71 test_volumes.py
test_01_root_volume_encryption Error 1102.72 test_volumes.py
test_02_data_volume_encryption Failure 1035.71 test_volumes.py
test_02_data_volume_encryption Error 1035.72 test_volumes.py
test_03_root_and_data_volume_encryption Failure 1027.58 test_volumes.py
test_03_root_and_data_volume_encryption Error 1027.58 test_volumes.py
test_02_attach_volume Failure 1667.67 test_volumes.py
test_02_attach_volume Failure 1667.68 test_volumes.py
test_03_download_attached_volume Failure 861.17 test_volumes.py
test_04_delete_attached_volume Failure 860.12 test_volumes.py
test_05_detach_volume Failure 957.38 test_volumes.py
test_06_download_detached_volume Failure 1041.48 test_volumes.py
test_07_resize_fail Failure 857.89 test_volumes.py
test_08_resize_volume Failure 860.36 test_volumes.py
test_09_delete_detached_volume Failure 860.76 test_volumes.py
test_10_list_volumes Failure 855.18 test_volumes.py
test_11_attach_volume_with_unstarted_vm Failure 973.48 test_volumes.py
test_12_resize_volume_with_only_size_parameter Failure 862.35 test_volumes.py
test_13_migrate_volume_and_change_offering Failure 993.89 test_volumes.py
test_14_delete_volume_delete_protection Failure 858.55 test_volumes.py
ContextSuite context=TestVolumes>:teardown Error 874.16 test_volumes.py
test_01_verify_ipv6_vpc Error 644.45 test_vpc_ipv6.py
test_01_verify_ipv6_vpc Error 644.49 test_vpc_ipv6.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL Error 387.65 test_vpc_redundant.py
test_02_redundant_VPC_default_routes Error 387.70 test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers Error 259.94 test_vpc_redundant.py
test_04_rvpc_network_garbage_collector_nics Error 220.02 test_vpc_redundant.py
test_05_rvpc_multi_tiers Error 325.00 test_vpc_redundant.py
test_05_rvpc_multi_tiers Error 325.02 test_vpc_redundant.py
test_01_VPC_nics_after_destroy Error 188.62 test_vpc_router_nics.py
test_02_VPC_default_routes Error 183.87 test_vpc_router_nics.py
test_01_redundant_vpc_site2site_vpn Failure 387.56 test_vpc_vpn.py
test_01_redundant_vpc_site2site_vpn Error 387.58 test_vpc_vpn.py
test_01_vpc_site2site_vpn_multiple_options Failure 214.24 test_vpc_vpn.py
test_01_vpc_site2site_vpn_multiple_options Error 214.26 test_vpc_vpn.py
test_01_vpc_site2site_vpn Failure 244.31 test_vpc_vpn.py
test_01_vpc_site2site_vpn Error 244.33 test_vpc_vpn.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants