0.8.0rc1 (#804)

* start of refactor

* vpc working

* split vrfs and networks

* add deploy param to create vrfs and networks workflow

* comment out vrfs and networks deploy param in create workflow

* updates for vrfs and networks functionality

* refactor deploy

* enable create pipeline for policy

* updates for tags and remove

* refactor to abstract objs

* Add update hostname pipeline step

* initial updates for tor

* modify action plugin executor

* update create resources

* update config save for mcfg

* many updates - renames, tor, switch deploy, etc

* close gaps on missing functionality for manual allocation and unmanaging policy

* fix tag bug with tors and update data model check flag

* cleanup and refactor imports

* rename objects to resources

* add banner for pipeline task to know where execution is in create and remove

* update remove for ints, vrf, networks when diff run disabled

* optimize config save runs

* update resources

* update display printing

* updates for msd resources

* updates for msd

* update role tag issue

* enable ebgp

* update color

* remove meta end play

* Fix remove when config list empty with state overridden

* Diff Run Fixes

* Update ebgp fabric

* update deferred msite overlay data render

* update create and remove resources yaml

* Update create_resources.yml

* updates for fabric links

* update data model check for ibgp vxlan

* update for policy and fabric links

* clean up lint

* update vrf and network removal

* fixes to vrfs and networks

* fix for child fabric removal

* update config save for fabric

* adjust runtime status msgs

* update vrf loopback attachment to execute only if valid loopback or freeform

* update changed status check for vrf loopback attach

* update msd and mcfg vrf loopback templates

* update ebgp and bring in mcfg

* update change flag for config save tasks

* resolve remove for links and update data model guard for inventory config save

* updates for deferred msd and mcfg overlay processing

* enable edge connections

* enable edge connections

* Update existing_links_check.py (#779)

* updates for msite diff run

* updates for mcfg

* comment out config save request debugs

* Refactor skip validation fabric (#780)

* Update pipeline_base.py

Add Skip_validation for ISN fabric

* Update ndfc_executor.py

add skip_validation flag for dcnm_fabric

* Update pipeline_base.py

add use case for external fabric

* Update pipeline_base.py

add eBGP_VXLAN use case

* Update create_resources.yml

* Update pipeline_base.py

* Update pipeline_base.py

* Update ndfc_executor.py

* updates to fabric deployer

* add display in deployer

* Improve role validate performance (#789)

* Improve role validate performance

* Improve mark stage complete tasks

* Build resources performance improvements

* resolve lint & sanity errors

* Update ndfc_vpc_domain_id_resource.j2 (#792)

* comment out unused discovery tasks

* Refactor var name for underlay ip / bgw anycast (#793)

* Update ndfc_underlay_ip_address.j2

* Update ndfc_bgw_anycast_vip.j2

* Update create_resources.yml

* Bulk Update Switch Hostname Policy (#795)

* Bulk API for Hostname Fabric Policy Get

* Fix Sanity

* Fix Pep8

* Diff underlay ip address (#773)

* Update create_resources.yml

* Update manage_resources.py

* Create underlay_ip_manual_allocation_filter.py

* Update create_resources.yml

* Update manage_resources.py

* Update manage_resources.py

* Update underlay_ip_manual_allocation_filter.py

* Address PR #773 review comments: single-pass optimization, remove Tier 3 fallback, fix verbosity, remove dead code

- Merge extract/filter/index into single-pass loop (review comment #1)
- Move pools list logging from -vv to -vvv (review comment #2)
- Remove Tier 3 pool+resource fallback, keep two-tier only (review comment #3)
- Remove unused underlay_ip_audit_pools and underlay_ip_audit_scope_filter (review comment #4)

Note: After dcnm_resource_manager updates pool allocations, a Recalculate
and Deploy is required for changes to be applied to device configuration.

* fix lint errors

* Update main.yml

Enable CI tests on refactor_collection.

Remove before merge to develop

* Refactor - enhance diff controller links (#794)

* Update manage_resources.py

* Move fabric links profile comparison into existing_links_check

- Move value-level profile comparison from manage_resources.py prefilter
  into existing_links_check.py during endpoint matching (single pass)
- Add _link_profile_changed() method with forward check (desired vs controller)
  and reverse check (field removal detection for peer1_cmds/peer2_cmds)
- Remove redundant second-pass prefilter from manage_resources.py
- Both YAML and Python pipeline callers now benefit from profile filtering

Addresses review comments on PR #794.

* Update manage_resources.py

* Refactor - Diff policies (#784)

* Update diff_compare.py for policy

* Update manage_resources.py add remote policies diff

* Update create_resources.yml

* Update resource_types.yml

* Update manage_resources.py

* Update diff_compare.py

* Update manage_resources.py

* Address PR #784 review: clean policy diff, resolve merge conflict, add use_desc_as_key

- Remove extra },  in diff_compare.py (review comment #1)
- Rewrite manage_resources.py policy diff to use per-switch API (review comment #2)
- Remove dead code: _fetch_all_policy_pages, urlencode import, ndfc_internal_keys
- Remove all DEBUG blocks from manage_resources.py and pipeline_base.py
- Resolve merge conflict: integrate _underlay_ip_remote_diff from upstream
- Add use_desc_as_key=True for dcnm_policy in ndfc_executor.py (fixes changed=False bug)

* Update diff_compare.py

* Update manage_resources.py

* Fix E131 pep8: use parenthesized ternaries for hanging indent

* Fix MCFG Fabric Handling (#798)

* Fix MCFG Fabric Handling

* Fix Lint

* remove docstring line

---------

Co-authored-by: mwiebe <mwiebe@cisco.com>
Co-authored-by: Charly Coueffe <75327499+ccoueffe@users.noreply.github.com>
Co-authored-by: Akshayanat C S <83760554+AKDRG@users.noreply.github.com>

Author: 4 people

.github/workflows/main.yml

@@ -9,10 +9,12 @@ on:
     branches:
       - develop
       - main
+      - refactor_collection
   pull_request:
     branches:
       - develop
       - main
+      - refactor_collection
 
   # Allows you to run this workflow manually from the Actions tab
   workflow_dispatch:

plugins/action/common/get_credentials.py

@@ -207,4 +207,7 @@ def run(self, tmp=None, task_vars=None):
                     display.vvv(f"Using group_vars discovery credentials from model data for device {device_ip}")
 
         results['updated_inv_list'] = updated_inv_list
+        # Convention: post-hooks that produce module-ready data set 'module_data'
+        # so build_resource_data can pass it directly to downstream NDFC modules.
+        results['module_data'] = updated_inv_list
         return results

plugins/action/common/nac_dc_validate.py

@@ -89,10 +89,12 @@ def run(self, tmp=None, task_vars=None):
             schema = DEFAULT_SCHEMA
 
         rules_list = []
+        data_model_loaded = None
         if rules and task_vars['role_path'] in rules:
             # Load in-memory data model using iac-validate
             # Perform the load in this if block to avoid loading the data model multiple times when custom enhanced rules are provided
             results['data'] = load_yaml_files([mdata])
+            data_model_loaded = results['data']
 
             # Introduce common directory to the rules list by default once vrf and network rules are updated
             parent_keys = ['vxlan', 'fabric']
@@ -156,11 +158,17 @@ def run(self, tmp=None, task_vars=None):
             # Else block to pickup custom enhanced rules provided by the user
             rules_list.append(f'{rules}')
 
+        syntax_validated = False
         for rules_item in rules_list:
             validator = nac_validate.validator.Validator(schema, rules_item)
-            if schema:
+            if schema and not syntax_validated and validator.schema is not None:
                 validator.validate_syntax([mdata])
+                syntax_validated = True
             if rules_item:
+                if data_model_loaded is None:
+                    data_model_loaded = load_yaml_files([mdata])
+                    results['data'] = data_model_loaded
+                validator.data = data_model_loaded
                 validator.validate_semantics([mdata])
 
             msg = ""

plugins/action/common/read_run_map.py

@@ -40,9 +40,11 @@ def run(self, tmp=None, task_vars=None):
         results['diff_run'] = True
         results['validate_only_run'] = False
 
+        fabric_name = self._task.args.get('fabric_name')
         data_model = self._task.args.get('data_model')
         play_tags = self._task.args.get('play_tags')
-        fabric_name = data_model['vxlan']['fabric']['name']
+        if fabric_name is None:
+            fabric_name = data_model['vxlan']['fabric']['name']
 
         if 'dtc' in task_vars['role_path']:
             common_role_path = os.path.dirname(task_vars['role_path'])

plugins/action/common/run_map.py

@@ -41,10 +41,12 @@ def run(self, tmp=None, task_vars=None):
         results = super(ActionModule, self).run(tmp, task_vars)
         results['failed'] = False
 
+        fabric_name = self._task.args.get('fabric_name')
         data_model = self._task.args.get('data_model')
         stage = self._task.args['stage']
 
-        fabric_name = data_model['vxlan']['fabric']['name']
+        if fabric_name is None:
+            fabric_name = data_model['vxlan']['fabric']['name']
 
         if 'dtc' in task_vars['role_path']:
             common_role_path = os.path.dirname(task_vars['role_path'])