Add a test for the update scenario (using a previous/'current' N-1 and 'master'/next version)

[guillaumerose]

General information

• OS: Linux / macOS / Windows

Steps to reproduce

1. (latest released version) crc setup
2. (latest released version) crc start
3. (current tested version) crc setup
4. (current tested version) crc start

Expected

It should work

Actual

We don't know and it's likely to be broken.
People are reporting this kind of error: Error loading bundle metadata: oc not found in bundle.
I thought it was with very old version but it could happen with 1.23.0 to 1.24.0.

[gbraad]

We don't know and it's likely to be broken.

Update scenario is not tested. But we might wanna approach this differently too. It would be best to clear out the binaries from the folder to prevent 'old' versions to hang around and cause trouble,

• Do we test if the oc client needs to be updated?
• What if we decide to rename a binary? Fix #2089 Update Windows tray assembly name #2090
• What if we decide to move the installed application? Change install path of crc to /Applications/CodeReady Containers.app #2163

[gbraad]

@jsliacan @adrianriobo Would it be possible to break down these scenarios and prioritize them on the planning?

[jsliacan]

I'll have a look, but as a quick response:

• How do we know if oc client needs updating? Is there an "expected" version we should be using?
• Binary rename is an interesting usecase, will put that into tests.
• Moving installed app... Need to look at this more closely. Seems a possible usecase.

[guillaumerose]

Do we test if the oc client needs to be updated?

oc client is not installed in the PATH. The user is forced to use crc oc-env. There is no issue with oc version update.

Update scenario is not tested. But we might wanna approach this differently too.

We need to reproduce what users do. It can't be different.

It would be best to clear out the binaries from the folder to prevent 'old' versions to hang around and cause trouble,

This is a solution. It is not an excuse to not test upgrade.

[gbraad]

We need to reproduce what users do. It can't be different.

Approach differently by 'removing the files' on a setup of a new version. That is what I am suggesting.
Each version should be self contained. Or else, we should also describe the scenario in which 'older' versions are used... as at the moment we would leave 'junk' behind, but not use it.

It is not an excuse to not test upgrade.

I never challenged we should not test. And as you can see I also described potentially scenarios that should also be considered. As recent changes can led to this kind of problem.

[cfergeau]

Approach differently by 'removing the files' on a setup of a new version. That is what I am suggesting.

This is more or less what we try to achieve (if OpenShift version of the VM in ~/.crc/machines/crc does not match the version the binary expects, we tell the user to run crc delete or something like this). The problem in this case is that we most likely have a bug which prevents us from running this version check, and returns an obscure error instead.

[cfergeau]

People are reporting this kind of error: Error loading bundle metadata: oc not found in bundle.
I thought it was with very old version but it could happen with 1.23.0 to 1.24.0.

oc was added to bundles in 1.19.0~21

But we added recently extra checks when loading bundles 1.22.0~27 and also changed when the oc symlink is created v1.24.0~70

Not sure a N-1 -> N upgrade test would have caught this specific issue.

[gbraad]

The problem in this case is that we most likely have a bug which prevents us from running this version check, and returns an obscure error instead.

Most likely an error, right. But since we do not have great coverage, performing something close to a 'clean install' is not a bad approach to have. Since what you describe is perhaps not even caught with a test I believe we should consider that each install of CRC is self-contained, so why not install it as such. This way we do not have to deal with 'old junk'... besides the VM that is running.

[cfergeau]

Since what you describe is perhaps not even caught with a test I believe we should consider that each install of CRC is self-contained, so why not install it as such.

I don't understand what the suggestion is. But we can't silently run rm -rf ~/.crc on upgrades as this is going to wipe out user data. And I'm not even sure we can detect this is the first time the upgraded version runs without looking at ~/.crc content. Which I suspect is exactly the cause for the issue Guillaume mentioned.

[gbraad]

rm -rf ~/.crc

I am not suggesting to wipe the user's folder, but a large part of the cache; the stored binaries.

Note: This would have forced a caching of the oc binary and likely result in the oc not found in bundle.

[cfergeau]

The stored binaries are not a huge issue as we validate their versions as part of preflights. We could remove them, but I'm not sure this would have made any difference wrt the initial issue