Ability to archive/re-instantiate workspaces

[TonyWildish-BH]

When a project comes to an end, we want to turn off (destroy) a workspace to stop paying for it. However, we want to archive the data and other resources so they remain available in case a researcher needs to go back to verify something. There can be a few reasons for this:

• Some of our projects want the SDE so they can crunch input data which they then export for external analysis. At that point, the w/s is just sitting there costing money, so it would be good to destroy it. However, downstream analysis may reveal something that needs closer investigation. Unless the user has been diligent in recording their activity (!), they may have trouble reproducing their environment. Ideally, we could archive it in some minimal-cost manner and bring it back to life later.
• We may also be required to keep artifacts for regulatory purposes. In fact, that's very likely, depending on the nature and origin of the project.

Resources that we might want to archive include:

• Shared storage
• Any SQL/NoSQL databases
• Any Gitea repositories they've created
• (possibly) VM images, if they've been customised for analysis.
• AzureML models etc?

We would not need to preserve the user access, the actual workspace itself, or anything else. Just code & data.

Has anyone put any thought into this? We wouldn't need a complex solution, something as simple as dumping DB/git/image archives to disk could suffice, reducing it to a problem of archiving the storage. As long as there's a feasible route for resurrection, that's enough, it doesn't have to be overly user-friendly.

[marrobi]

Yes, please see: #2168

I suggest the value of "isEnabled" is used within the porter update step to conditionally modify/deploy the resource via terraform.

For example, disks/storage could be left, compute destroyed. Then if reenabled, it all reappears. We already have functionality so that if a workspace is disabled it cascades to the services etc.

The work needs doing to the terraform for the various bundles.

[TonyWildish-BH]

Thanks. It looks like #2168 is more a discussion around the meaning of the 'Disabled' button, which is not quite the same as an 'archive' feature unless it's intended to include things like the App Service Plan and the private endpoints, all the bits and pieces other than data in its various forms?

[marrobi]

In technical terms, what is the difference between archive and disable?

[TonyWildish-BH]

'Archive', to me, means "remove from the TRE completely", storing only the artifacts. That removes clutter from the UI and eliminates all components in Azure other than long-term storage for the artifacts.

If we have to archive a project for several years, for regulatory purposes for example, we don't want to be paying any more than absolutely necessary. £3/day for the App Service Plan will be too much.

Another way to look at it: A workspace that is Disabled can be re-enabled from the TRE UI. A project that has had its artifacts archived - and the workspace deleted - can only be re-created as an administrative action. A new workspace would be created and populated with the artifacts.

I'm happy to pass on the how of re-populating for now, but a way to archive a workspace artifacts before deleting the workspace would be extremely useful.

[marrobi]

The app service could be deleted when the workspace is disabled. As can other resources. Just a condition in the terraform. Would that help?

Only a TRE Administrator can re-enable a workspace. Archive would be the same? I believe disabled workspaces are hidden from other users. We can always add filters.

Design wise I feel they are both very similar.

[TonyWildish-BH]

The app service could be deleted when the workspace is disabled. As can other resources. Just a condition in the terraform. Would that help?

Yup, that would help a lot. Maybe the private endpoints too, could that work?

Only a TRE Administrator can re-enable a workspace. Archive would be the same?

yes, for my environment, at least.