Pass down resources to CRI #4113

marquiz · 2023-06-28T14:32:01Z

One-line PR description: KEP for extending the CRI API to pass down unmodified resource information from the kubelet to the CRI runtime.

Issue link: Pass down resources to CRI #4112

Other comments:

Co-authored-by: Antti Kervinen <[email protected]>

marquiz · 2023-06-28T14:34:23Z

/cc @haircommander @mikebrow @zvonkok @fidencio @kad

k8s-ci-robot · 2023-06-28T14:34:28Z

@marquiz: GitHub didn't allow me to request PR reviews from the following users: fidencio.

Note that only kubernetes members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @haircommander @mikebrow @zvonkok @fidencio @kad

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

haircommander · 2023-06-28T14:41:25Z

keps/sig-node/4112-passdown-resources-to-cri/README.md

+    map<string, k8s.io.apimachinery.pkg.api.resource.Quantity> requests = 2;
+    map<string, k8s.io.apimachinery.pkg.api.resource.Quantity> limits = 3;


should the keys here be a special type instead of unstructured?

I don't think it's possible to have smth like type ResourceName string in protobuf. Please correct me if I'm wrong

ping @haircommander, are you satisfied with the reply (close as resolved)?

keps/sig-node/4112-passdown-resources-to-cri/README.md

marquiz · 2023-07-18T07:59:40Z

/retitle Pass down resources to CRI

zvonkok · 2023-08-02T12:42:46Z

@marquiz We need to check how this will work with DRA and CDI devices. If we have enough information to know which devices need to be added to the sandbox just by the resource claim name.

zvonkok · 2023-08-02T12:44:17Z

@marquiz There is already some code for sandbox sizing, accumulation of resources CPU and Memory for reference: kubernetes/kubernetes#104886 that we leverage in Kata, what are the plans for this interface, deprecate or keep it?

zvonkok · 2023-08-02T12:44:46Z

@bergwolf @egernst FYI

elezar

Thanks @marquiz.

It would be good to get more concrete details on the use cases that this would enable.
There is also the question of complex devices that are managed by device plugins where there isn't a clear mapping from the resources entry (e.g. vendor.com/xpu: 1) to the resources added to the container, or DRA where the associated resources.requests.claims entry is not mentioned.

elezar · 2023-08-02T13:02:04Z

keps/sig-node/4112-passdown-resources-to-cri/README.md

+
+#### Story 3
+
+As a cluster administrator, I want to install an OCI hook/runc wrapper/NRI


Could you expand on this use case? How does extending the CRI translate to modifications in the OCI runtime specification which is interpreted by runc (or wrappers)?

The CRI changes (in this KEP) would not directly translate to anything in the OCI config. It's just "informational" that a possible hook/wrapper/plugin can then use to tweak the OCI config. Say you want to do customized cpu pinning in your plugin. I'll come up with some more flesh on this section...

@elezar I updated Story 3, PTAL

ping @elezar may I close this as resolved?

elezar · 2023-08-02T13:05:23Z

keps/sig-node/4112-passdown-resources-to-cri/README.md

+          requests:
+            cpu: 100m
+            memory: 100M
+            vendor.com/xpu: 1


For clarification: This does not indicate the properties of the resource that was actually allocated for a container requesting one of these devices?

That's very much true. I think I'll add a note about this in the KEP somewhere

@elezar I added a not about device plugin resources after this example. WDYT?

ping @elezar may I close this as resolved?

keps/sig-node/4112-passdown-resources-to-cri/README.md

aojea · 2023-08-03T08:37:22Z

keps/sig-node/4112-passdown-resources-to-cri/README.md

+     WindowsPodSandboxConfig windows = 9;
+
+    // Kubernetes resource spec of the containers in the pod.
+    PodResourceConfig pod_resources = 10;


@MikeZappa87 since you shared recently something along these lines for the networking capabilities, this KEP also means to interface with NRI

zvonkok · 2023-08-03T09:45:53Z

Another point to consider is how we're going to integrate or not these enhancements with the new containerd Sandbox API.

marquiz · 2023-08-03T13:24:20Z

There is already some code for sandbox sizing, accumulation of resources CPU and Memory for reference: kubernetes/kubernetes#104886 that we leverage in Kata, what are the plans for this interface, deprecate or keep it?

@zvonkok that one is just the native resources and gives the resources in the "obfuscated" form i.e. not telling the actual reqeusts/limits (plus it's for Linux resources only). I think we wouldn't, or even couldn't, touch this, i.e. keep it.

zvonkok · 2023-08-03T14:50:07Z

@moshe010 @adrianchiris @shivamerla @cdesiniotis FYI

zvonkok · 2023-09-01T07:28:34Z

Since the DevicePlugin API supports CDI devices with this KEP: #4011 we should try to add more restrictions and requirements how we want to design this passthrough interface. @marquiz FYI

zvonkok · 2023-09-01T07:31:51Z

Just for reference linking this old KEP here: #3080, looks like there were some comments that CDI may not be the complete solution to some use-cases. We do not want to break anything by relying only on CDI when passing down the devices.

SergeyKanzhelev · 2024-10-09T18:53:31Z

keps/sig-node/4112-passdown-resources-to-cri/README.md

+The UpdatePodSandboxResources CRI message is also updated when/if that is
+introduced by the [KEP-1287][kep-1287] Beta ([PR][kep-1287-beta-pr]).
+
+Information about the Pod-level resources are added when/if the Pod-level


@ndixita can you please confirm it is tracked in your KEP?

Re-phrased this paragraph to

[KEP-2837][kep-2837] Alpha ([PR][kep-2837-alpha-pr]) proposes to add new Pod-level resource requirements field to the PodSpec. This information will be be added to the PodResourceConfig message, similar to the container resource information, if/when KEP-2837 is implemented as proposed.

The intention is to sync with the other KEPs

This is not tracked in Pod Level Resources KEP. While I am trying to understand this KEP, I have a question: Will the VM based runtimes benefit from knowing the overall requirements of all the containers in a pod and not requirements of individual containers? Again, this question is just for my understanding...

SergeyKanzhelev · 2024-10-09T19:00:19Z

keps/sig-node/4112-passdown-resources-to-cri/README.md

+request.
+
+```diff
+ message PodSandboxConfig {


how big this structure will be? What would be the max number of containers we can report in this structure? (what is the message size limit today?)

By quick search the gRPC message size limit is 4MB. This is a lot more than the maximum size of a Pod object in Kubernetes. I believe we should not hit a limit in any practical scenario. WDYT?

4Mb my be ok.

- rename enum REGULAR_CONTAINER -> CONTAINER - update references to other related keps under Design Details

SergeyKanzhelev · 2024-10-09T19:12:12Z

keps/sig-node/4112-passdown-resources-to-cri/README.md

+     WindowsPodSandboxConfig windows = 9;
+
+    // Kubernetes resource spec of the containers in the pod.
+    PodResourceConfig pod_resources = 10;


what runtime suppose to do when this does not match values provided in resources of LinuxPodSandboxConfig?

There should be no overlapping information between these two structures. Thus a conflict should not be possible

I am not sure I understand. Maybe I am missing something here, but you should be able to calculate what is in resources based on pod spec. Is it correct? So some implementation may take dependency on specific way one is calculated. Thus the question, what if they will not be matching at some point. Also if they are inter-dependent, perhaps some of it may be deprecated?

The runtime is not calculating resources of LinuxPodSandboxConfig. They are set by the kubelet

SergeyKanzhelev · 2024-10-09T19:16:29Z

keps/sig-node/4112-passdown-resources-to-cri/README.md

+
+The resources (mounts, devices, CDI devices, Kubernetes resources) in the
+CreateContainer request should be identical to what was (pre-)informed in the
+RunPodSandbox request. If they are different, the CRI runtime may fail the


how would we separate cases when VM was restarted and we don't really know what was originally passed to the sandbox creation any longer? I do not understand this consistency check argument

Yes, this paragraph seems outdated, especially in the light of the in-place pod updates. The paragraph was added as a response to some review feedback (I just couldn't find that comment now)

How about re-phrasing it for example:

In some usage scenarios the container creation may fail if the resources (mounts, devices, CDI devices, Kubernetes resources) in the CreateContainer request do not match what was (pre-)informed in the RunPodSandbox request. This may be the case e.g. in when changes are not allowed after a VM-based Pod has been created.

WDYT @SergeyKanzhelev ?

I believe he means the original pod metadata plus any pod updates must be applied to the pod before the container is created.

SergeyKanzhelev · 2024-10-09T19:18:00Z

keps/sig-node/4112-passdown-resources-to-cri/README.md

+### kubelet
+
+Kubelet code is refactored/modified so that all container resources are known
+before sandbox creation. This mainly consists of preparing all mounts (of all


Is there a possibility that today some mounts can ONLY be created when container is being created (some thing that init container needs to initialize). Will this break this behavior?

This was discussed/investigated in #4113 (comment)

I think there should breakages because of this.

WDYT @SergeyKanzhelev ?

When appropriate, image mounts included in the container config are added at create container time when we build the rootfs.. additionally oci level plugins, nri plugins, cdi, and cri proxy tools may add mounts. But these are not the "extra" mounts passed in from the host level CRI client (kubelet), additionally the OCI image volume mount is added by the container runtime during create container. In the OCI volume case this is why I'm asking for the oci volume image reference..

The Mount structure is exactly the same (with the same information) that what will be passed as part of the CreateContainerRequest/ContainerConfig message

that works..

it actually goes back to this comment: #4113 (comment)

Would it be expected that runtimes will need to mount everything at pod sandbox creation time now? Or the behavior is not defined? Will we require that the old way (without passing all this extra context in sandbox creation) should continue working?

The runtimes do not need to change the way they operate. They don't need to mount everything at sandbox creation.

All of the data is something that the runtime CAN use if it needs it. E.g. this paragraph of the proposal tries to state that. Maybe we need to state that more clearly in more places(?) And certainly comments in the cri proto file.

The CAN part worries me. If something CAN be used the only assumption we can make is that it IS BEING USED. This is why maybe we should have two modes of operations

@SergeyKanzhelev as it was mentioned above, this information is really needed in some scenarios, and OCI runtimes that are dependent on that information will be using it, to fix current problem. Other runtimes that don't have problems now are expected to ignore it.

SergeyKanzhelev · 2024-10-09T20:50:06Z

Besides comments above, I think it is important to clarify and even test as part of critest the requirement for Container Runtime. It goes to this question: #4113 (comment)

Some critest ideas might be:

We must add a test in critest that old way (without all those specs) must continue work? (unless I am missing something and we are deprecating the current way)
We should add tests on all sorts of mismatches and the runtime behavior in those cases.
Test scenarios (whatever we may think) when some resources (paths?) passed in sandbox cannot be mounted yet (permissions? remounting?) and will be OK to mount later when init container completed. (unless again we want to explicitly break this scenario)

I understand the challenge with the VM-based runtimes, but also worried that we are going against the direction of making Pod more dynamic. All the duplication and mismatch of what we thought Pod and contianers were before and what they will be is scary to me.

Another alternative would be to have two modes of runtime. One mode with all information available, and every dynamic change recreates the Pod. And another mode - what we have today. And runtime will need to tell on startup which mode it wants to work in.

keps/sig-node/4112-passdown-resources-to-cri/README.md

zvonkok · 2024-10-10T01:11:41Z

I understand the challenge with the VM-based runtimes, but also worried that we are going against the direction of making Pod more dynamic.

Can you elaborate on the direction of "making Pod more dynamic"?

All the duplication and mismatch of what we thought Pod and contianers were before and what they will be is scary to me.

What are you scared of, or what are you imagining where this is going?

I understand the challenge with the VM-based runtimes

We will have more and more sandboxed environments that will need such information. The use case with Confidential Containers and GPUs (or any other accelerator) is not realizable with the current architecture.

Additionally, there is HW that you cannot hot-plug during runtime, and you need a static configuration at sandbox creation time.

kad · 2024-10-10T13:46:41Z

We must add a test in critest that old way (without all those specs) must continue work? (unless I am missing something and we are deprecating the current way)

This thing is targeted to solve the problem that CoCo VMs have (and impossible to fix without additional info passed down), and not trying to deprecate or change meaning of any other Pod lifecycle messages or events meaning.

We should add tests on all sorts of mismatches and the runtime behavior in those cases.

If there are no bugs in the kubelet code, there should be no mismatches between raw data derived from pod spec with fields calculated to LinuxContainerResources. Kubelet owns it and passing it/calculates form single in-memory object inside kubelet.
Or which mismatches you're talking about?

Test scenarios (whatever we may think) when some resources (paths?) passed in sandbox cannot be mounted yet (permissions? remounting?) and will be OK to mount later when init container completed. (unless again we want to explicitly break this scenario)

Device and mount paths are provided at pod sandbox creation time as additional information. At the time when RunSandbox is issued to CRI, kubelet already resolved and prepared all the paths and devices. And even in situation when exact paths might be not yet available, that is still fine, they will be vaild during creation of container time. However, getting all mount/devices at VM creation time it is really critical for fixing currently broken scenario in CoCo.

I understand the challenge with the VM-based runtimes, but also worried that we are going against the direction of making Pod more dynamic. All the duplication and mismatch of what we thought Pod and contianers were before and what they will be is scary to me.

In fact, we are not going against direction of pod and containers to be dynamic. It is the opposite: in every piece of additional data passed, we are reflecting current state of the object. e.g. "At pod creation we know this much info of workload", on "container creation, something already got changed, so here is current state about that container.". This allows us to do righsizing of VM during creation including all possible heuristics based on information known at that time, as well failing creation of container on (re-)start if something changed since RunPodSandbox and it doesn't fit anymore.

Another alternative would be to have two modes of runtime. One mode with all information available, and every dynamic change recreates the Pod. And another mode - what we have today. And runtime will need to tell on startup which mode it wants to work in.

We don't really need two modes. We just need enough information to make decisions at existing pod lifecycle events, as well proper error handling for scenarios where dynamic resize of the pod or container is not feasible due to some reasons.

mikebrow

LGTM for moving forward with the KEP under the proviso that a follow up KEP update is required to sync with 1287 and 2837, to cover message/api details not yet discovered, and other dangling issues still in open discussion. I also expect us to find needed additions when we have a cc prototype.

k8s-ci-robot · 2024-10-10T22:37:19Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: haircommander, marquiz, mikebrow
Once this PR has been reviewed and has the lgtm label, please assign dchen1107, wojtek-t for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

keps/prod-readiness/OWNERS
keps/sig-node/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mikebrow · 2024-10-10T22:50:13Z

Besides comments above, I think it is important to clarify and even test as part of critest the requirement for Container Runtime. It goes to this question: #4113 (comment)

Some critest ideas might be:

We must add a test in critest that old way (without all those specs) must continue work? (unless I am missing something and we are deprecating the current way)

We should add tests on all sorts of mismatches and the runtime behavior in those cases.

Test scenarios (whatever we may think) when some resources (paths?) passed in sandbox cannot be mounted yet (permissions? remounting?) and will be OK to mount later when init container completed. (unless again we want to explicitly break this scenario)

I understand the challenge with the VM-based runtimes, but also worried that we are going against the direction of making Pod more dynamic. All the duplication and mismatch of what we thought Pod and contianers were before and what they will be is scary to me.

Another alternative would be to have two modes of runtime. One mode with all information available, and every dynamic change recreates the Pod. And another mode - what we have today. And runtime will need to tell on startup which mode it wants to work in.

Interesting points. Mode works to a certain degree and we already have handler as just such a mode selection. I think the kep will definitely need to be updated to handle the dynamic pod resource changes.. (new networks, different resource claims/devices) and when a handler is found to not be dynamic.. we'll just have to cycle the pod by returning error handler does not support this update type and / or tainting the handler type ahead of time.

zvonkok · 2024-10-31T17:27:36Z

@SergeyKanzhelev @mrunalp How to move this forward?

Apokleos · 2024-12-06T06:49:04Z

Hi folks, any updates of this KEP ? Any ideas to help move it forward ?

- update kep.yaml to target v1.33 - update references to kep-1287 and kep-2837

marquiz · 2024-12-10T18:54:52Z

Updated:

target milestone updated to v1.33
references to other KEPs removed from non-goals
references to other KEPs (In-Place Update of Pod Resources #1287, Pod level resources #2837) elsewhere in the proposal updated (e.g. removed the conditionals "if kep-X is accepted as proposed...")

The related changes in other referenced KEPs (#1287, #2837) are now merged which hopefully makes this slightly easier read. Those changes/KEPs (for Kubernetes v1.32) are still mentioned in the corresponding sections.

haircommander · 2025-01-31T15:58:42Z

keps/sig-node/4112-passdown-resources-to-cri/README.md

+Including the KubernetesResources in the ContainerConfig message serves
+multiple purposes:
+
+1. Catch changes that happen between pod sandbox creation and container


in place resize should be writing a message to the runtime that a pod or container had changed resources, right @tallclair ? I think that'd be better than having a runtime interpret diffs in container values...

Does it do so when the container hasn't yet been created. IIRC the thinking behind this case were the kata/coco use cases. The runtime or nri plugin whatever CAN detect changes if they need/want to.

zvonkok · 2025-02-07T12:29:05Z

@SergeyKanzhelev Are all your questions anwsered, are you happy with the current shape of the KEP? If yes, would you mind revoming the hold? Thanks!

johnbelamaric

Previous PRR still looks good.

marquiz · 2025-02-11T15:20:32Z

Previous PRR still looks good.

@SergeyKanzhelev PTAL

tallclair · 2025-02-12T00:49:42Z

keps/sig-node/4112-passdown-resources-to-cri/README.md

+any information about the resources (such as native resources, host devices,
+mounts, CDI devices etc) that will be required by the application. The CRI
+runtime only becomes aware of the resources piece by piece when containers of
+the pod are created (one-by-one).


Which of this information must come from the Kubelet, and which can be read from the k8s API? Resource requests & limits and volumes can all be read directly from the pod object. Do they need to be plumbed through the CRI, or can you just read from the k8s API instead?

The idea is to have clear separation between layers with normal clear top-to-bottom information flow between components. Kubelet knows current state of the object, so it provides whole state of that object down to lower layers. This allows lower layers to use information immediately in consistent, transactional way. This eliminates ways of delaying reply to CRI grpc transaction due to need of creating another grpc call to outside service(s). We want to eliminate scenarios where different projects are using some other backdoors or out-of-band communications to fetch information needed. Examples are different CNI providers that based on Sandbox ID are trying to connect to k8s api server to get whole pod object or similarly, some CNI providers are connecting back to kubelet with "hey, kubelet, you send me CRI message, but didn't provide enough information, but I know in your PodResources API you have that information for that pod, let me fetch it from there" manner.

So, the whole idea to have clean separation between layers:

kubelet is resposible to get higher level object from API server, prepare it to run, and give it to runtimes in full descriptive way.

Runtimes should not be doing calls from bottom layers back to upper layers (kubelet, api server) to get missing pieces.

Adding to that, for mounts we want more/different information than what is available on the Kubernetes API level. At least the host_path that will be mounted to the container.

Additional to the other points, in Kata/Confidential Containers we cannot handle subPath in volumeMounts because subPath is handled completely by the kubelet. https://github.com/kubernetes/kubernetes/blob/d7774fce9a7fcec890d7c0beffacd6ae34152b01/pkg/volume/util/subpath/subpath_linux.go#L175

KEP: Initial version of the Pass down resources to CRI

7de612d

Co-authored-by: Antti Kervinen <[email protected]>

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory labels Jun 28, 2023

k8s-ci-robot requested review from dchen1107 and mrunalp June 28, 2023 14:32

k8s-ci-robot added sig/node Categorizes an issue or PR as relevant to SIG Node. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jun 28, 2023

k8s-ci-robot requested review from haircommander, mikebrow, zvonkok and kad June 28, 2023 14:34

haircommander reviewed Jun 28, 2023

View reviewed changes

keps/sig-node/4112-passdown-resources-to-cri/README.md Outdated Show resolved Hide resolved

marquiz mentioned this pull request Jul 18, 2023

Pass down resources to CRI #4112

Open

4 tasks

k8s-ci-robot changed the title ~~KEP: Initial version of the Pass down resources to CRI~~ Pass down resources to CRI Jul 18, 2023

KEP-4112: address review feedback from haircommander

bc8e299

elezar reviewed Aug 2, 2023

View reviewed changes

aojea reviewed Aug 3, 2023

View reviewed changes

zvonkok mentioned this pull request Sep 18, 2023

Add support Edit Container Mount for kata/runtime-rs using direct volume in K8S/CSI scenario cncf-tags/container-device-interface#162

Closed

SergeyKanzhelev reviewed Oct 9, 2024

View reviewed changes

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 9, 2024

SergeyKanzhelev reviewed Oct 9, 2024

View reviewed changes

KEP-4112: address review feedback

5f066a0

- rename enum REGULAR_CONTAINER -> CONTAINER - update references to other related keps under Design Details

SergeyKanzhelev reviewed Oct 9, 2024

View reviewed changes

marquiz added 2 commits October 10, 2024 00:12

KEP-4112: update

ea20429

KEP-4112: drop other keps from non-goals

dc1d8f9

ndixita reviewed Oct 10, 2024

View reviewed changes

keps/sig-node/4112-passdown-resources-to-cri/README.md Outdated Show resolved Hide resolved

mikebrow approved these changes Oct 10, 2024

View reviewed changes

zvonkok mentioned this pull request Nov 4, 2024

k8s subPath is not supported by Kata kata-containers/kata-containers#10487

Open

Manciukic mentioned this pull request Nov 29, 2024

[PCIe] Community Guidelines and Roadmap firecracker-microvm/firecracker#4894

Draft

KEP-4112: update for v1.33

b881855

- update kep.yaml to target v1.33 - update references to kep-1287 and kep-2837

zvonkok mentioned this pull request Jan 9, 2025

Road to Confidential Containers with GPUs for v<TARGET_VERSION> confidential-containers/confidential-containers#278

Open

60 tasks

haircommander reviewed Jan 31, 2025

View reviewed changes

zvonkok mentioned this pull request Feb 6, 2025

kata-shim: update default maximum memory for gust vm kata-containers/kata-containers#10837

Open

johnbelamaric reviewed Feb 10, 2025

View reviewed changes

tallclair reviewed Feb 12, 2025

View reviewed changes

tallclair mentioned this pull request Mar 27, 2025

Set memory.min or memory.low, without memory.high kubernetes/kubernetes#131077

Open

		+ map<string, k8s.io.apimachinery.pkg.api.resource.Quantity> requests = 2;
		+ map<string, k8s.io.apimachinery.pkg.api.resource.Quantity> limits = 3;


		#### Story 3

		As a cluster administrator, I want to install an OCI hook/runc wrapper/NRI

Pass down resources to CRI #4113

Are you sure you want to change the base?

Pass down resources to CRI #4113

Conversation

marquiz commented Jun 28, 2023

marquiz commented Jun 28, 2023

k8s-ci-robot commented Jun 28, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marquiz commented Jul 18, 2023

zvonkok commented Aug 2, 2023 • edited Loading

zvonkok commented Aug 2, 2023

zvonkok commented Aug 2, 2023

elezar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zvonkok commented Aug 3, 2023

marquiz commented Aug 3, 2023

zvonkok commented Aug 3, 2023

zvonkok commented Sep 1, 2023

zvonkok commented Sep 1, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SergeyKanzhelev Oct 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikebrow Oct 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SergeyKanzhelev commented Oct 9, 2024 • edited Loading

zvonkok commented Oct 10, 2024 • edited Loading

kad commented Oct 10, 2024

mikebrow left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Oct 10, 2024

mikebrow commented Oct 10, 2024

zvonkok commented Oct 31, 2024

Apokleos commented Dec 6, 2024

marquiz commented Dec 10, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zvonkok commented Feb 7, 2025

johnbelamaric left a comment

Choose a reason for hiding this comment

marquiz commented Feb 11, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zvonkok Feb 13, 2025 • edited Loading

Choose a reason for hiding this comment

zvonkok commented Aug 2, 2023 •

edited

Loading

SergeyKanzhelev Oct 9, 2024 •

edited

Loading

mikebrow Oct 9, 2024 •

edited

Loading

SergeyKanzhelev commented Oct 9, 2024 •

edited

Loading

zvonkok commented Oct 10, 2024 •

edited

Loading

zvonkok Feb 13, 2025 •

edited

Loading