Skip to content

Commit 8a83b89

Browse files
a-mccarthya-mccarthy
committed
Init pass at restructuring TOC
Signed-off-by: Abigail McCarthy <20771501+a-mccarthy@users.noreply.github.com>
1 parent e67f717 commit 8a83b89

8 files changed

Lines changed: 728 additions & 395 deletions

‎confidential-containers/confidential-containers-deploy.rst‎

Lines changed: 9 additions & 388 deletions
Large diffs are not rendered by default.

‎confidential-containers/configure-cc-mode.rst‎

Lines changed: 156 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,156 @@
1+
.. license-header
2+
SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
3+
SPDX-License-Identifier: Apache-2.0
4+
5+
Licensed under the Apache License, Version 2.0 (the "License");
6+
you may not use this file except in compliance with the License.
7+
You may obtain a copy of the License at
8+
9+
http://www.apache.org/licenses/LICENSE-2.0
10+
11+
Unless required by applicable law or agreed to in writing, software
12+
distributed under the License is distributed on an "AS IS" BASIS,
13+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
See the License for the specific language governing permissions and
15+
limitations under the License.
16+
17+
.. headings # #, * *, =, -, ^, "
18+
19+
20+
.. _managing-confidential-computing-mode:
21+
22+
*****************************************
23+
Managing the Confidential Computing Mode
24+
*****************************************
25+
26+
You can set the default confidential computing mode of the NVIDIA GPUs by setting the ``ccManager.defaultMode=<on|off>`` option.
27+
The default value of ``ccManager.defaultMode`` is ``on``.
28+
You can set this option when you install NVIDIA GPU Operator or afterward by modifying the cluster-policy instance of the ClusterPolicy object.
29+
30+
When you change the mode, the manager performs the following actions:
31+
32+
* Evicts the other GPU Operator operands from the node.
33+
34+
However, the manager does not drain user workloads. You must make sure that no user workloads are running on the node before you change the mode.
35+
36+
* Unbinds the GPU from the VFIO PCI device driver.
37+
* Changes the mode and resets the GPU.
38+
* Reschedules the other GPU Operator operands.
39+
40+
The supported modes are:
41+
42+
.. list-table::
43+
:widths: 15 55 30
44+
:header-rows: 1
45+
46+
* - Mode
47+
- Description
48+
- Configuration Method
49+
* - ``on``
50+
- Enable Confidential Computing.
51+
- cluster-wide default, node-level override
52+
* - ``off``
53+
- Disable Confidential Computing.
54+
- cluster-wide default, node-level override
55+
* - ``ppcie``
56+
- Enable Confidential Computing on NVIDIA Hopper GPUs.
57+
58+
On the NVIDIA Hopper architecture multi-GPU passthrough uses protected PCIe (PPCIE)
59+
which claims exclusive use of the NVSwitches for a single Confidential Container
60+
virtual machine.
61+
If you are using NVIDIA Hopper GPUs for multi-GPU passthrough,
62+
set the GPU mode to ``ppcie`` mode.
63+
64+
The NVIDIA Blackwell architecture uses NVLink
65+
encryption which places the switches outside of the Trusted Computing Base (TCB),
66+
meaning the ``ppcie`` mode is not required. Use ``on`` mode in this case.
67+
- node-level override
68+
69+
You can set a cluster-wide default mode, and you can set the mode on individual nodes.
70+
The mode that you set on a node has higher precedence than the cluster-wide default mode.
71+
72+
Setting a Cluster-Wide Default Mode
73+
====================================
74+
75+
To set a cluster-wide mode, specify the ``ccManager.defaultMode`` field like the following example:
76+
77+
.. code-block:: console
78+
79+
$ kubectl patch clusterpolicies.nvidia.com/cluster-policy \
80+
--type=merge \
81+
-p '{"spec": {"ccManager": {"defaultMode": "on"}}}'
82+
83+
*Example Output:*
84+
85+
.. code-block:: output
86+
87+
clusterpolicy.nvidia.com/cluster-policy patched
88+
89+
.. note::
90+
91+
The ``ppcie`` mode cannot be set as a cluster-wide default, it can only be set as a node label value.
92+
93+
Setting a Node-Level Mode
94+
==========================
95+
96+
To set a node-level mode, apply the ``nvidia.com/cc.mode=<on|off|ppcie>`` label on the node.
97+
98+
Set the ``NODE_NAME`` environment variable to the name of the node you want to configure:
99+
100+
.. code-block:: console
101+
102+
$ export NODE_NAME="<node-name>"
103+
104+
Then apply the label:
105+
106+
.. code-block:: console
107+
108+
$ kubectl label node $NODE_NAME nvidia.com/cc.mode=on --overwrite
109+
110+
The mode that you set on a node has higher precedence than the cluster-wide default mode.
111+
112+
Verifying a Mode Change
113+
========================
114+
115+
To verify that a mode change was successful, view the ``nvidia.com/cc.mode``,
116+
``nvidia.com/cc.mode.state``, and ``nvidia.com/cc.ready.state`` node labels:
117+
118+
.. code-block:: console
119+
120+
$ kubectl get node $NODE_NAME -o json | \
121+
jq '.metadata.labels | with_entries(select(.key | startswith("nvidia.com/cc")))'
122+
123+
*Example Output (CC mode disabled):*
124+
125+
.. code-block:: json
126+
127+
{
128+
"nvidia.com/cc.mode": "off",
129+
"nvidia.com/cc.mode.state": "off",
130+
"nvidia.com/cc.ready.state": "false"
131+
}
132+
133+
*Example Output (CC mode enabled):*
134+
135+
.. code-block:: json
136+
137+
{
138+
"nvidia.com/cc.mode": "on",
139+
"nvidia.com/cc.mode.state": "on",
140+
"nvidia.com/cc.ready.state": "true"
141+
}
142+
143+
* The ``nvidia.com/cc.mode`` label is the desired state.
144+
145+
* The ``nvidia.com/cc.mode.state`` label reflects the mode that was last successfully applied to the GPU hardware by the Confidential Computing Manager.
146+
Its value mirrors the applied mode ``on``, ``off``, or ``ppcie``, after the transition is complete on the node.
147+
A value of ``failed`` indicates that the last mode transition encountered an error.
148+
149+
* The ``nvidia.com/cc.ready.state`` label indicates whether the node is ready to run Confidential Container workloads.
150+
It is set to ``true`` when ``cc.mode.state`` is ``on`` or ``ppcie``, and ``false`` when ``cc.mode.state`` is ``off``.
151+
152+
.. note::
153+
154+
It can take one to two minutes for GPU state transitions to complete and the labels to be updated.
155+
A mode change is complete and successful when ``nvidia.com/cc.mode`` and
156+
``nvidia.com/cc.mode.state`` have the same value.

‎confidential-containers/configure-image-pull-timeouts.rst‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
.. license-header
2+
SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
3+
SPDX-License-Identifier: Apache-2.0
4+
5+
Licensed under the Apache License, Version 2.0 (the "License");
6+
you may not use this file except in compliance with the License.
7+
You may obtain a copy of the License at
8+
9+
http://www.apache.org/licenses/LICENSE-2.0
10+
11+
Unless required by applicable law or agreed to in writing, software
12+
distributed under the License is distributed on an "AS IS" BASIS,
13+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
See the License for the specific language governing permissions and
15+
limitations under the License.
16+
17+
.. headings # #, * *, =, -, ^, "
18+
19+
20+
.. _configure-image-pull-timeouts:
21+
22+
*****************************
23+
Configure Image Pull Timeouts
24+
*****************************
25+
26+
The guest-pull mechanism pulls images inside the confidential VM, which means large images can take longer to download and delay container start.
27+
Kubelet can de-allocate your pod if the image pull exceeds the configured timeout before the container transitions to the running state.
28+
29+
The timeout chain has three components that you might need to configure:
30+
31+
* **Kubelet** ``runtimeRequestTimeout``: Controls how long kubelet waits for the container runtime to respond. Default: ``2m``.
32+
* **Kata shim** ``create_container_timeout``: Controls how long the NVIDIA shim allows a container to remain in container creating state. Default: ``1200s`` (20 minutes).
33+
* **Kata Agent** ``image_pull_timeout``: Controls the agent-side timeout for guest-image pull. Default: ``1200s`` (20 minutes).
34+
35+
Configure the Kubelet Timeout
36+
==============================
37+
38+
Configure your cluster's ``runtimeRequestTimeout`` in your `kubelet configuration <https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/>`_ with a higher timeout value than the two-minute default.
39+
Set this value to ``20m`` to match the default values for the NVIDIA shim configurations in Kata Containers.
40+
41+
Add or update the ``runtimeRequestTimeout`` field in your kubelet configuration (typically ``/var/lib/kubelet/config.yaml``):
42+
43+
.. code-block:: yaml
44+
:emphasize-lines: 3
45+
46+
apiVersion: kubelet.config.k8s.io/v1beta1
47+
kind: KubeletConfiguration
48+
runtimeRequestTimeout: 20m
49+
50+
Restart the kubelet service to apply the change:
51+
52+
.. code-block:: console
53+
54+
$ sudo systemctl restart kubelet
55+
56+
Configure Timeouts Beyond 20 Minutes
57+
======================================
58+
59+
If you need a timeout of more than 1200 seconds (20 minutes), you must also adjust the Kata Agent Policy's ``image_pull_timeout`` value.
60+
61+
You can set this value either through a pod annotation or by modifying the shim configuration.
62+
63+
Using a Pod Annotation
64+
-----------------------
65+
66+
Add the ``io.katacontainers.config.hypervisor.kernel_params`` annotation to your pod manifest with the desired ``agent.image_pull_timeout`` value in seconds:
67+
68+
.. code-block:: yaml
69+
:emphasize-lines: 7
70+
71+
apiVersion: v1
72+
kind: Pod
73+
metadata:
74+
name: large-model-kata
75+
namespace: default
76+
annotations:
77+
io.katacontainers.config.hypervisor.kernel_params: "agent.image_pull_timeout=1800"
78+
spec:
79+
runtimeClassName: kata-qemu-nvidia-gpu-snp
80+
restartPolicy: Never
81+
containers:
82+
- name: model-server
83+
image: "nvcr.io/nvidia/example-large-model:latest"
84+
resources:
85+
limits:
86+
nvidia.com/pgpu: "1"
87+
memory: 64Gi
88+
89+
In this example, ``agent.image_pull_timeout=1800`` sets the agent-side timeout to 30 minutes (1800 seconds).
90+
91+
Using the Shim Configuration
92+
-----------------------------
93+
94+
To set the timeout globally, add the ``agent.image_pull_timeout`` kernel parameter to your Kata shim configuration file.
95+
The shim configuration files are located in ``/opt/kata/share/defaults/kata-containers/`` on the worker nodes.
96+
97+
Add the parameter to the ``kernel_params`` field in the ``[hypervisor.qemu]`` section:
98+
99+
.. code-block:: toml
100+
:emphasize-lines: 2
101+
102+
[hypervisor.qemu]
103+
kernel_params = "agent.image_pull_timeout=1800"
104+
105+
.. note::
106+
107+
When setting timeouts beyond 20 minutes, ensure that all three timeout values in the chain are consistent:
108+
the kubelet ``runtimeRequestTimeout``, the Kata shim ``create_container_timeout``, and the
109+
agent ``image_pull_timeout`` should all be set to accommodate the expected image pull duration.

0 commit comments

Comments
 (0)