Code restructuring for easier maintenance.

[TonyWildish-BH]

The current codebase is structured in a way that could be improved from an operational standpoint by being more granular. There are a number of things which add up.

1. When deploying a TRE for development, I often don't want to deploy the entire thing. It's often enough to have the core TRE, base workspace, Nexus, Guacamole, and a Linux VM. I can write scripts to pick out just the targets I want, but it would be nicer if I could specify them in a list in the config file - and if that config file were not overwritten by an update from upstream.
2. Similarly, we don't want to provide our users with access to all the services in the TRE at this point, for various reasons. Again, it would be nice to be able to customise the list of services we deploy, rather than having to write scripts for it.
3. When working on the core code, it would be useful to be able to proceed stepwise, deploying or tearing down components of the TRE piece by piece, instead of all in one go. E.g., if I want to work on the Servicebus, I have no clean way of destroying and redeploying just the Servicebus, without redeploying the whole thing. This makes for slow development cycles.
4. Piecewise deployment would also help save costs when running development TREs, which we do a lot of. Currently, the only way to save money by stopping the Servicebus is to destroy the entire TRE. Having the option to destroy/re-deploy the Servicebus, for just one example, could help us save money. See also Enable Azure bastion free sku. #86, about the Bastion in the SDE core. We should be able to deploy this only when we need it, and destroy it afterwards.
5. Point 3 would also help when updating from the upstream codebase. Inevitably, we have customisations that are site-specific, and which should not be contributed back upstream. Having more granular deployment/un-deployment targets would help us manage updates in a more controlled manner, knowing that we can work on a limited blast-radius. Currently, we have to go file by file, checking each one, and we have to know which other files are affected by any decisions we make when we merge.

None of those points are deal-breakers in themselves, but together, they suggest either re-organising the codebase, or at least re-organising the deployment procedure in terms of makefile targets. The key principles I'd like to see adopted are:

1. Fine-grained deployment/destruction of resources, so I can pick and mix what I do in development or production. Separate plan/apply for each target, so I can check it out before deploying.
2. Shape the upstream code so that it's easier to merge with a downstream fork that, inevitably, diverges from the upstream in site-specific ways. This means separating out configuration that may validly drift, allowing for pick-n-mix of components etc.

I'd be very interested to hear from other users of the TRE about their thoughts on this.

[West-P]

@TonyWildish-BH The CI/CD method of deployment covers some of the requests here such as being able to specify which bundles to include in the deployment. It also allows for different branches such as a dev branch to be deployed. Allowing development and testing to be conducted outside of a production instance if needed.

I do however see the issue in the overwriting of config files (however the main config.yaml is in .gitignore by default I believe).

I am a user of the repo rather than developer so I can't speak for how difficult it would be to implement some of the suggestions, however I would also welcome some more ways of the code safely being customised as I suspect some alterations will be needed for the current project I am working on.

[marrobi]

@TonyWildish-BH can you break these down into separate issues?

Understand some of the points, but there is far to much here to actually discuss form an implementation perspective.

Might be useful to have a look at: https://medium.com/nyc-planning-digital/writing-a-proper-github-issue-97427d62a20f and https://wiredcraft.com/blog/how-we-write-our-github-issues/ prior to creating the issues so we can better assist. Thanks.

[TonyWildish-BH]

@marrobi I don't want to break these points out into separate tickets because the whole point is to foster broader discussion, rather than just addressing issues with point fixes. Github issues isn't the best medium for that discussion, but that's what I've been asked to do, following the discussions we've had with Carl about improving the scalability.

That's what this and the other tickets I've been filing are all about. There are many ways to solve some of these problems, and we need to step back and look at the bigger picture.