feat: includeTopologies

[nblair2]

feat: includeTopologies

Description

This change introduces the ability to nest and compose topologies by leveraging existing stored topologies or defined in files on disk. This allows for modularity and reusability when defining complex topologies

Ex:

...
  includeTopooliges:
    - /phenix/topologies/enterprise/phenix-configs/topolgy.yml
    - /phenix/topolgies/ot/phenix-configs/topolgy.yml
    - store-topo
  nodes:
...

Related Issue

Tangential to #247
Documentation sandialabs/sceptre-phenix-docs#27

Type of Change

Please select the type of change your pull request introduces:

• Bugfix
• New feature
• Documentation update
• Other (please describe):

Checklist

• This PR conforms to the process detailed in the Contributing Guide.
• I have included no proprietary/sensitive information in my code.
• I have performed a self-review of my code.
• I have commented my code, particularly in hard-to-understand areas.
• I have made corresponding changes to the documentation.
• My changes generate no new warnings.
• I have tested my code.

[nblair2]

@glattercj @activeshadow let me know if you have thoughts. I am still looking at a few things and testing edge cases.

[GhostofGoes]

Make sure to update the documentation with a minimal example 🤠

[nblair2]

This is not working out how I hoped. It helps a little but then we still have to maintain multiple different versions of the same app in different scenario files to correspond to different topos.

I am thinking about a different way to tackle this through the API, allowing multiple topo/scenario pairs to get loaded at once and then "merged" at experiment creation time.

apiVersion: phenix.sandia.gov/v0
kind: Workflow
metadata: {}
spec:
  auto:
    create: ${BRANCH_NAME}
  configs:
  - topology: ${BRANCH_NAME}-IT
    scenario: ${BRANCH_NAME}-IT
  - topology: ${BRANCH_NAME}-OT
    scenario: ${BRANCH_NAME}-OT
...

Then on exp creation:

• each topo / scenario file is validated against each other (as normal)
• topos are merged (look for duplicate nodes, etc)
• scenarios are merged (the same app can be defined in multiple scenarios, so we have to intelligently combine them. This is the most dangerous part)
• one big beautiful experiment is created, proceed as normal

I will leave this open for now.

[nblair2]

Making the API change does not solve any of the problems I encountered. I still want to work toward this idea in the future, but I need some modularity now. I am going to keep the includeTopologies feature (for now), as it solves the immediate problem. In the future I think there are a couple of improvements to be made (#247) before moving forward.

[nblair2]

@GhostofGoes FYSA. I have done some light testing (fetching from store, fetching files on disk, checking for cyclic imports). Working on docs now.

[GhostofGoes]

A few comments explaining what case each of the 3 if blocks in load topology is handling wouldn't be unwelcome. Otherwise LGTM.