[BUG] crc-admin context not added

[adrianriobo]

When the cluster is started it tries to add the crc-admin context to the kubeconfig file, under some conditions (specially on nested environments) the context is not added, this create some false positives on tests for cluster functionality.

I think the context add has no retry in case of failure. It would be great to check it and add a retry mechanism in there to ensure the context is successfully added.

[gbraad]

I think the context add has no retry in case of failure.

This might be true...

[adrianriobo]

I think then context add has no retry in case of failure.

Based on last conversation with @praveenkumar during QE sync, sorry did not check the code so can not ensure this is the root cause :(

[gbraad]

the root cause :(

It is unlikely the root cause. a retry-mechanism is merely a workaround for another problem; unable to write context. is this due to the the file permissions? what else?

[adrianriobo]

yeah as I did not analyze the code I added the description based on conversations during qe sync, may the root cause for all the problem is not the addition of the context but a sync issue? , meaning right after the cluster has been started we try to use the context, on nested virtualization this may require a delay?

and yeah the workaround solution would not be the root cause of course :)

[cfergeau]

What platform are you seeing this on? I assume linux since you mention nested virt? I assume you don't have logs when this happens?

[gbraad]

meaning right after the cluster has been started we try to use the context, on nested virtualization this may require a delay?

Does the test progress and use the context already? the startup hasn't finished, so it should not just continue, right? Hmmmm...

Perhaps a snip of the actual test might help.

[adrianriobo]

What platform are you seeing this on?

Yeah mostly on linux, but I think I saw it on other platforms as well, can not ensure...but this thing has been around for a really long time. Most of "false" positives on openshift-ci are due to this, as an example:

#3444 failing

and from the logs of this run

time="2022-12-12T07:29:12Z" level=info msg="Adding crc-admin and crc-developer contexts to kubeconfig..."
time="2022-12-12T07:29:12Z" level=error msg="Cannot update kubeconfig: challenger chose not to retry the request"

So at least from logs it seems also there is no retry mechanism

[adrianriobo]

Does the test progress and use the context already? the startup hasn't finished, so it should not just continue, right?

Tests will try to use the context after start is finished and the cluster status is running. But if fails b/c as I showed the context was not added during start command, even when it finished successfully

Also in here I am suggesting to use oc login instead of context for our functionality tests, and create a specific integration one only for testing if the context is added (in case this is a hard requirement), or at least to avoid "false" positives on those test

[gbraad]

Seen only reports around CRC when searching for "Cannot update kubeconfig: challenger chose not to retry the request". discussions I had never seen before (on access).