Skip to content

Commit 55e6c9e

Browse files
amadeuszlalechniakCopilotgewarrenBillWagner
authored
Add Kubernetes Resource Monitoring documentation (#54134)
* Add Kubernetes Resource Monitoring documentation * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Update state of metrics * Refactor * Bring back network metrics * Bump K8s package version * Fix default metrics values note * Apply suggestions from code review Co-authored-by: Bill Wagner <wiwagn@microsoft.com> Co-authored-by: Genevieve Warren <24882762+gewarren@users.noreply.github.com> * Fix console block * Fix lint issue --------- Co-authored-by: Amadeusz Lechniak <alechniak@microsoft.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Genevieve Warren <24882762+gewarren@users.noreply.github.com> Co-authored-by: Bill Wagner <wiwagn@microsoft.com>
1 parent 369d33a commit 55e6c9e

4 files changed

Lines changed: 218 additions & 8 deletions

File tree

docs/core/diagnostics/built-in-metrics-diagnostics.md

Lines changed: 18 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -71,31 +71,32 @@ The `Microsoft.Extensions.Diagnostics.ResourceMonitoring` metrics report resourc
7171
- [`container.cpu.request.utilization`](#metric-containercpurequestutilization)
7272
- [`container.cpu.time`](#metric-containercputime)
7373
- [`container.memory.limit.utilization`](#metric-containermemorylimitutilization)
74+
- [`container.memory.request.utilization`](#metric-containermemoryrequestutilization)
7475
- [`container.memory.usage`](#metric-containermemoryusage)
7576
- [`process.cpu.utilization`](#metric-processcpuutilization)
7677
- [`dotnet.process.memory.virtual.utilization`](#metric-dotnetprocessmemoryvirtualutilization)
7778
- [`system.network.connections`](#metric-systemnetworkconnections)
7879

7980
> [!NOTE]
80-
> Metrics emitted by the `Microsoft.Extensions.Diagnostics.ResourceMonitoring` meter are in experimental stage. This means that there could be breaking changes to them.
81+
> Some metrics emitted by the `Microsoft.Extensions.Diagnostics.ResourceMonitoring` meter (such as disk I/O) are in experimental stage and not documented here.
8182
8283
##### Metric: `container.cpu.limit.utilization`
8384

8485
The instrument is only available on a system running on containers both on Windows and Linux.
8586

8687
| Name | Instrument Type | Unit (UCUM) | Description |
8788
| ---- | --------------- | ----------- | ----------- |
88-
| `container.cpu.limit.utilization` | <xref:System.Diagnostics.Metrics.ObservableGauge`1> | `1` | The CPU consumption of the running containerized application relative to resource limit in range `[0, 1]`. |
89+
| `container.cpu.limit.utilization` | <xref:System.Diagnostics.Metrics.ObservableGauge`1> | `1` | The CPU consumption of the running containerized application relative to resource limit. Linux: range `[0, 1]` by default. Windows: range `[0, 100]` by default. See `UseZeroToOneRangeForMetrics` and `UseZeroToOneRangeForLinuxMetrics` options to change the default behavior. |
8990

9091
Available starting in `Microsoft.Extensions.Diagnostics.ResourceMonitoring` 8.8.0.
9192

9293
##### Metric: `container.cpu.request.utilization`
9394

94-
The instrument is only available on a system running on containers on Linux.
95+
The instrument is only available on a system running on containers both on Windows and Linux.
9596

9697
| Name | Instrument Type | Unit (UCUM) | Description |
9798
| ---- | --------------- | ----------- | ----------- |
98-
| `container.cpu.request.utilization` | <xref:System.Diagnostics.Metrics.ObservableGauge`1> | `1` | The CPU consumption of the running containerized application relative to resource request in range `[0, 1]`. |
99+
| `container.cpu.request.utilization` | <xref:System.Diagnostics.Metrics.ObservableGauge`1> | `1` | The CPU consumption of the running containerized application relative to resource request. Linux: range `[0, 1]` by default. Windows: range `[0, 100]` by default. See `UseZeroToOneRangeForMetrics` and `UseZeroToOneRangeForLinuxMetrics` options to change the default behavior. |
99100

100101
Available starting in `Microsoft.Extensions.Diagnostics.ResourceMonitoring` 8.8.0.
101102

@@ -115,10 +116,20 @@ The instrument is only available on a system running on containers both on Windo
115116

116117
| Name | Instrument Type | Unit (UCUM) | Description |
117118
| ---- | --------------- | ----------- | ----------- |
118-
| `container.memory.limit.utilization` | <xref:System.Diagnostics.Metrics.ObservableGauge`1> | `1` | The memory consumption of the running containerized application relative to resource limit in range `[0, 1]`. |
119+
| `container.memory.limit.utilization` | <xref:System.Diagnostics.Metrics.ObservableGauge`1> | `1` | The memory consumption of the running containerized application relative to resource limit. Linux: range `[0, 1]` by default. Windows: range `[0, 100]` by default. See `UseZeroToOneRangeForMetrics` and `UseZeroToOneRangeForLinuxMetrics` options to change the default behavior. |
119120

120121
Available starting in `Microsoft.Extensions.Diagnostics.ResourceMonitoring` 8.8.0.
121122

123+
##### Metric: `container.memory.request.utilization`
124+
125+
The instrument is only available on a system running on containers both on Windows and Linux.
126+
127+
| Name | Instrument Type | Unit (UCUM) | Description |
128+
| ---- | --------------- | ----------- | ----------- |
129+
| `container.memory.request.utilization` | <xref:System.Diagnostics.Metrics.ObservableGauge`1> | `1` | The memory consumption of the running containerized application relative to resource request. Linux: range `[0, 1]` by default. Windows: range `[0, 100]` by default. See `UseZeroToOneRangeForMetrics` and `UseZeroToOneRangeForLinuxMetrics` options to change the default behavior. |
130+
131+
Available starting in `Microsoft.Extensions.Diagnostics.ResourceMonitoring` 9.8.0.
132+
122133
##### Metric: `container.memory.usage`
123134

124135
The instrument is only available on a system running on containers either on Windows or Linux.
@@ -133,15 +144,15 @@ Available starting in `Microsoft.Extensions.Diagnostics.ResourceMonitoring` 9.8.
133144

134145
| Name | Instrument Type | Unit (UCUM) | Description |
135146
| ---- | --------------- | ----------- | ----------- |
136-
| `process.cpu.utilization` | <xref:System.Diagnostics.Metrics.ObservableGauge`1> | `1` | The CPU consumption of the running application in range `[0, 1]`. |
147+
| `process.cpu.utilization` | <xref:System.Diagnostics.Metrics.ObservableGauge`1> | `1` | The CPU consumption of the running application. Linux: range `[0, 1]` by default. Windows: range `[0, 100]` by default. See `UseZeroToOneRangeForMetrics` and `UseZeroToOneRangeForLinuxMetrics` options to change the default behavior. |
137148

138149
Available starting in: .NET 8.
139150

140151
##### Metric: `dotnet.process.memory.virtual.utilization`
141152

142153
| Name | Instrument Type | Unit (UCUM) | Description |
143154
| ---- | --------------- | ----------- | ----------- |
144-
| `dotnet.process.memory.virtual.utilization` | <xref:System.Diagnostics.Metrics.ObservableGauge`1> | `1` | The memory consumption of the running application in range `[0, 1]`. |
155+
| `dotnet.process.memory.virtual.utilization` | <xref:System.Diagnostics.Metrics.ObservableGauge`1> | `1` | The memory consumption of the running application. Linux: range `[0, 1]` by default. Windows: range `[0, 100]` by default. See `UseZeroToOneRangeForMetrics` and `UseZeroToOneRangeForLinuxMetrics` options to change the default behavior. |
145156

146157
Available starting in: .NET 8.
147158

docs/core/diagnostics/diagnostic-resource-monitoring.md

Lines changed: 172 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -98,8 +98,9 @@ For the source code of this example, see the [Resource monitoring sample](https:
9898
Since the <xref:Microsoft.Extensions.Diagnostics.ResourceMonitoring.IResourceMonitor> interface is deprecated, migrate to the metrics-based approach. The `Microsoft.Extensions.Diagnostics.ResourceMonitoring` package provides several metrics that you can use instead, for instance:
9999

100100
- `container.cpu.limit.utilization`: The CPU consumption share of the running containerized application relative to resource limit in range `[0, 1]`. Available for containerized apps on Linux and Windows.
101-
- `container.cpu.request.utilization`: The CPU consumption share of the running containerized application relative to resource request in range `[0, 1]`. Available for containerized apps on Linux.
101+
- `container.cpu.request.utilization`: The CPU consumption share of the running containerized application relative to resource request in range `[0, 1]`. Available for containerized apps on Linux and Windows.
102102
- `container.memory.limit.utilization`: The memory consumption share of the running containerized application relative to resource limit in range `[0, 1]`. Available for containerized apps on Linux and Windows.
103+
- `container.memory.request.utilization`: The memory consumption share of the running containerized application relative to resource request in range `[0, 1]`. Available for containerized apps on Linux and Windows.
103104

104105
For more information about the available metrics, see the [Built-in metrics: Microsoft.Extensions.Diagnostics.ResourceMonitoring](built-in-metrics-diagnostics.md#microsoftextensionsdiagnosticsresourcemonitoring) section.
105106

@@ -124,6 +125,176 @@ The following is an example of the output from the preceding code:
124125

125126
For the complete source code of this example, see the [Resource monitoring with manual metrics sample](https://github.com/dotnet/docs/tree/main/docs/core/diagnostics/snippets/resource-monitoring-with-manual-metrics).
126127

128+
## Kubernetes resource monitoring
129+
130+
When your application runs inside a Kubernetes cluster, you typically configure resource limits and requests in your pod specification. The [Microsoft.Extensions.Diagnostics.ResourceMonitoring.Kubernetes](https://www.nuget.org/packages/Microsoft.Extensions.Diagnostics.ResourceMonitoring.Kubernetes) NuGet package extends the base resource monitoring library to read these values from environment variables exposed by the [Kubernetes Downward API](https://kubernetes.io/docs/concepts/workloads/pods/downward-api/).
131+
132+
This package automatically detects your container's CPU and memory boundaries and emits accurate utilization metrics relative to those values.
133+
134+
> [!TIP]
135+
> If your cluster runs Kubernetes v1.32 or later with cgroup v2, always prefer `AddKubernetesResourceMonitoring()` over `AddResourceMonitoring()` for accurate request-based utilization metrics. The Kubernetes package reads CPU and memory requests directly from environment variables, bypassing the cgroup weight inversion that can produce inaccurate values. See [Known limitations](#known-limitations-on-cgroup-v2) for details.
136+
137+
### How it works
138+
139+
The Kubernetes Downward API can expose pod resource limits and requests as environment variables. The `Microsoft.Extensions.Diagnostics.ResourceMonitoring.Kubernetes` package reads these environment variables at startup and uses them to calculate utilization metrics.
140+
141+
You choose a prefix for your environment variables (for example, `MY_APP_`). The library then looks for the following variables:
142+
143+
| Environment variable | Description |
144+
| --- | --- |
145+
| `<PREFIX>LIMITS_CPU` | CPU limit in millicores (for example, `2000` for 2 cores) |
146+
| `<PREFIX>LIMITS_MEMORY` | Memory limit in bytes |
147+
| `<PREFIX>REQUESTS_CPU` | CPU request in millicores (optional, defaults to limit value) |
148+
| `<PREFIX>REQUESTS_MEMORY` | Memory request in bytes (optional, defaults to limit value) |
149+
150+
At minimum, you must set `<PREFIX>LIMITS_CPU` and `<PREFIX>LIMITS_MEMORY` to non-zero values. If you omit the request variables or set them to zero, the library defaults them to the corresponding limit values.
151+
152+
### Configure the Downward API
153+
154+
To expose resource limits as environment variables, add `resourceFieldRef` entries to your Kubernetes deployment manifest:
155+
156+
```yaml
157+
apiVersion: apps/v1
158+
kind: Deployment
159+
metadata:
160+
name: my-app
161+
spec:
162+
template:
163+
spec:
164+
containers:
165+
- name: my-app
166+
resources:
167+
requests:
168+
cpu: "500m"
169+
memory: "256Mi"
170+
limits:
171+
cpu: "1000m"
172+
memory: "512Mi"
173+
env:
174+
- name: MY_APP_LIMITS_CPU
175+
valueFrom:
176+
resourceFieldRef:
177+
resource: limits.cpu
178+
divisor: "1m"
179+
- name: MY_APP_LIMITS_MEMORY
180+
valueFrom:
181+
resourceFieldRef:
182+
resource: limits.memory
183+
divisor: "1"
184+
- name: MY_APP_REQUESTS_CPU
185+
valueFrom:
186+
resourceFieldRef:
187+
resource: requests.cpu
188+
divisor: "1m"
189+
- name: MY_APP_REQUESTS_MEMORY
190+
valueFrom:
191+
resourceFieldRef:
192+
resource: requests.memory
193+
divisor: "1"
194+
```
195+
196+
The `divisor: "1m"` for CPU fields ensures Kubernetes expresses the value in millicores (for example, a `500m` limit becomes `500`). The `divisor: "1"` for memory fields returns the value in bytes.
197+
198+
### Register Kubernetes resource monitoring
199+
200+
In your application code, call <xref:Microsoft.Extensions.DependencyInjection.KubernetesResourceQuotaServiceCollectionExtensions.AddKubernetesResourceMonitoring*> with the environment variable prefix that matches your Kubernetes manifest:
201+
202+
:::code source="snippets/resource-monitoring-kubernetes/Program.cs":::
203+
204+
The `"MY_APP_"` prefix tells the library to look for environment variables named `MY_APP_LIMITS_CPU`, `MY_APP_LIMITS_MEMORY`, `MY_APP_REQUESTS_CPU`, and `MY_APP_REQUESTS_MEMORY`.
205+
206+
> [!IMPORTANT]
207+
> Don't call `AddResourceMonitoring()` in addition to `AddKubernetesResourceMonitoring()`. The Kubernetes method already registers all necessary base resource monitoring components. Calling both methods can result in conflicting service registrations.
208+
209+
### Emitted metrics
210+
211+
Once registered, the library emits the following metrics under the `Microsoft.Extensions.Diagnostics.ResourceMonitoring` meter. It uses the Kubernetes resource boundaries you configured for its calculations:
212+
213+
| Metric name | Type | Description |
214+
| --- | --- | --- |
215+
| `container.cpu.limit.utilization` | ObservableGauge | CPU usage relative to the configured limit, in the range [0, 1] |
216+
| `container.cpu.request.utilization` | ObservableGauge | CPU usage relative to the configured request, in the range [0, 1] |
217+
| `container.cpu.time` | ObservableCounter | Total CPU time consumed in seconds, with a `cpu.mode` dimension (user/system) |
218+
| `container.memory.limit.utilization` | ObservableGauge | Memory usage relative to the configured limit, in the range [0, 1] |
219+
| `container.memory.request.utilization` | ObservableGauge | Memory usage relative to the configured request, in the range [0, 1] |
220+
| `container.memory.usage` | ObservableUpDownCounter | Memory usage in bytes |
221+
| `process.cpu.utilization` | ObservableGauge | Process CPU usage relative to the CPU limit, in the range [0, 1] |
222+
| `dotnet.process.memory.virtual.utilization` | ObservableGauge | Process memory usage relative to the memory limit, in the range [0, 1] |
223+
224+
For the full list of metrics emitted by the base resource monitoring library, see [.NET extensions metrics: Microsoft.Extensions.Diagnostics.ResourceMonitoring](built-in-metrics-diagnostics.md#microsoftextensionsdiagnosticsresourcemonitoring).
225+
226+
### Example output
227+
228+
When you run your application inside a Kubernetes pod with the environment variables configured, the metrics produce values such as:
229+
230+
```
231+
Instrument: container.cpu.limit.utilization
232+
Value: 0.23
233+
234+
Instrument: container.cpu.request.utilization
235+
Value: 0.46
236+
237+
Instrument: container.memory.limit.utilization
238+
Value: 0.61
239+
240+
Instrument: container.memory.request.utilization
241+
Value: 0.78
242+
243+
Instrument: container.memory.usage
244+
Value: 312475648 (By)
245+
246+
Instrument: container.cpu.time
247+
Value: 142.35 (s), cpu.mode=user
248+
Value: 28.91 (s), cpu.mode=system
249+
```
250+
251+
In this example, the pod uses 23% of its CPU limit and 61% of its memory limit. Because the CPU request is lower than the CPU limit, `container.cpu.request.utilization` shows a higher value (46%) for the same absolute CPU usage.
252+
253+
### Collect metrics with OpenTelemetry
254+
255+
To export these metrics to your observability backend, register the meter with OpenTelemetry:
256+
257+
```csharp
258+
services.AddOpenTelemetry()
259+
.WithMetrics(builder =>
260+
{
261+
builder.AddMeter("Microsoft.Extensions.Diagnostics.ResourceMonitoring");
262+
builder.AddOtlpExporter(); // Or any other metrics exporter
263+
});
264+
```
265+
266+
### Known limitations on cgroup v2
267+
268+
Starting with Kubernetes v1.32 (using runc 1.3.2+ or crun 1.23+), the OCI runtime uses a new quadratic formula to convert cgroup v1 CPU shares to cgroup v2 CPU weight. This change affects the accuracy of the base `AddResourceMonitoring()` method when it attempts to derive CPU request values from cgroup parameters on Linux.
269+
270+
The core issue is that the base resource monitoring library reads `cpu.weight` from the cgroup v2 filesystem and reverse-converts it to estimate the original CPU request in millicores. However, the new conversion formula is **many-to-one**: multiple milliCPU values map to the same `cpu.weight`. For example, milliCPU values from 90 through 109 all produce `cpu.weight = 17`. Reversing this mapping cannot recover the exact original value, which means `container.cpu.request.utilization` may report inaccurate values.
271+
272+
The following metrics are affected:
273+
274+
| Metric | Affected? | Reason |
275+
| --- | --- | --- |
276+
| `container.cpu.request.utilization` | Yes | Relies on inferred CPU request from `cpu.weight` |
277+
| `container.memory.request.utilization` | Yes | Relies on inferred memory request from cgroup parameters |
278+
| `container.cpu.limit.utilization` | No | Uses `cpu.max`, not `cpu.weight` |
279+
| `container.memory.limit.utilization` | No | Uses memory limit directly from cgroup |
280+
281+
**How the Kubernetes package solves this:** `AddKubernetesResourceMonitoring()` reads the actual CPU and memory request values directly from environment variables you configure through the Downward API. It never reverse-converts cgroup parameters, so utilization metrics are always accurate regardless of which OCI runtime conversion formula your cluster uses.
282+
283+
For more information, see:
284+
285+
- [New cgroup v1 to v2 CPU conversion formula (Kubernetes blog)](https://kubernetes.io/blog/2026/01/30/new-cgroup-v1-to-v2-cpu-conversion-formula/)
286+
- [dotnet/extensions issue #7202](https://github.com/dotnet/extensions/issues/7202)
287+
288+
### Best practices
289+
290+
When deploying .NET applications to Kubernetes, consider the following recommendations:
291+
292+
- **Use the Kubernetes package in Kubernetes environments.** Always prefer `AddKubernetesResourceMonitoring()` over `AddResourceMonitoring()` when running in a Kubernetes cluster. It provides accurate resource utilization metrics by reading CPU and memory request values directly from environment variables rather than inferring them from cgroup parameters.
293+
294+
- **Expose resource metadata through the Downward API.** Configure your deployment manifests to expose CPU and memory limits and requests as environment variables. This ensures the library has access to the exact values you specified in your pod spec.
295+
296+
- **Test metric accuracy after cluster upgrades.** When upgrading your Kubernetes cluster or OCI runtime (runc, crun), verify that your resource utilization metrics still report expected values, especially if you use the base `AddResourceMonitoring()` method.
297+
127298
## Kubernetes probes
128299

129300
In addition to resource monitoring, apps that exist within a Kubernetes cluster report their health through diagnostic probes. The [Microsoft.Extensions.Diagnostics.Probes](https://www.nuget.org/packages/Microsoft.Extensions.Diagnostics.Probes) NuGet package provides support for Kubernetes probes. It externalizes various [health checks](diagnostic-health-checks.md) that align with various Kubernetes probes, for example:
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
using Microsoft.Extensions.DependencyInjection;
2+
using Microsoft.Extensions.Hosting;
3+
4+
var app = Host.CreateDefaultBuilder()
5+
.ConfigureServices(services =>
6+
{
7+
services.AddKubernetesResourceMonitoring("MY_APP_");
8+
})
9+
.Build();
10+
11+
await app.RunAsync();
Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
<Project Sdk="Microsoft.NET.Sdk">
2+
3+
<PropertyGroup>
4+
<OutputType>Exe</OutputType>
5+
<TargetFramework>net10.0</TargetFramework>
6+
<ImplicitUsings>enable</ImplicitUsings>
7+
<Nullable>enable</Nullable>
8+
</PropertyGroup>
9+
10+
<ItemGroup>
11+
<PackageReference Include="Microsoft.Extensions.Logging.Console" Version="10.0.8" />
12+
<PackageReference Include="Microsoft.Extensions.DependencyInjection" Version="10.0.8" />
13+
<PackageReference Include="Microsoft.Extensions.Diagnostics.ResourceMonitoring.Kubernetes" Version="10.7.0" />
14+
<PackageReference Include="Microsoft.Extensions.Hosting" Version="10.0.8" />
15+
</ItemGroup>
16+
17+
</Project>

0 commit comments

Comments
 (0)