[image-builder] image build sometimes fails due to `MountVolume.SetUp failed for volume "gitpod-ca-certificate"`

[kylos101]

Bug description

The error was:

MountVolume.SetUp failed for volume "gitpod-ca-certificate" : failed to sync secret cache: timed out waiting for the condition

As a user, I saw:

Logs

Steps to reproduce

n/a

the error is rare and intermittent

Workspace affected

n/a

Expected behavior

We should be able to mount the gitpod CA cert if the workspace container is in a good state, do the image build, and then start a workspace from it.

If we cannot mount the cert, we won't be able to pull images through registry-facade for the imagebuild workspace, and we should log the state of the pod to better understand why.

Example repository

No response

Anything else?

We might need to wait longer when creating a workspace pod, before trying to use its underlying mounts that are dependent on a secret. Example where we had to wait longer for a secret that is created at runtime. In this case, we're unable to mount a secret that was defined at install time, and then owned by cert-manager.

[andreafalzetti]

@kylos101 what's the impact of this occurring? Any action required during on-call?

[kylos101]

Hi @andreafalzetti 👋, what makes you ask? I'm curious how you bumped into the issue. For example, did you just have this problem?

I updated the expected behavior in the issue description to clarify impact, and anything else to offer a hint about how to mitigate in the future. There should be no action required as part of on-call, unless you receive an alert in pager duty, this SLO is suffering, or image builds are entirely failing in one or all regions, etc.

[andreafalzetti]

This recently occurred and it was the reason I got a PD alarm a few minutes ago, found the issue searching for gitpod-ca-certificate around. Thanks for clarifying 🙏

[kylos101]

Can you share a link to the related log @andreafalzetti ? It may have been me testing (an imagebuild of mine failed to start).

[kylos101]

Thanks for raising the question, @andreafalzetti , the underlying metric we use for imagebuild SLO is getting filtered. Submitted a fix here to avoid the metric drop.

[sagor999]

This looks like an issue with api-server (k8s), where it might be getting bogged down with requests?
We don't seem to have a grafana dashboard that would show k8s internals stats though, to see if memory usage or CPU usage of api server during that time might have spiked.

[jenting]

May consider skipping it just like [1] if it's an intermediate state.

[kylos101]

@jenting in the case of this issue, the workspace does not start due to this error. How would skipping help the workspace start?

[sagor999]

Yeah, I think skipping this error is not the best option, since it might hide some api server related problems.

[kylos101]

This looks like an issue with api-server (k8s), where it might be getting bogged down with requests? We don't seem to have a grafana dashboard that would show k8s internals stats though, to see if memory usage or CPU usage of api server during that time might have spiked.

💡 good point, @sagor999 , thank you for helping break this down. 🙏

I imported this dashboard, visible here, its not filterable yet by cluster, but may help? The issue happened twice over the last day...I couldn't find a clear latency trend for both errors, but, haven't inspected their clusters of origin (yet).

What sort of high watermarks do you look for, to check the K8s api-server for pressure?

I recall @easyCZ studying K8s API server in the past with @aledbf when we were troubleshooting (I think etcd pressure at the time). But, do not recall the specific expressions used to do the study.