cpu_utilization: gracefully handle cgroup v2 CPU detection in CONTAINER mode (#47485)

Commit Message:
cpu_utilization: fix CONTAINER mode in the host cgroup namespace

In CONTAINER mode the cpu_utilization resource monitor read the cgroup
v2 files
straight from /sys/fs/cgroup, assuming it is the process's own cgroup,
and
required cpu.stat, cpu.max and cpuset.cpus.effective to all exist. When
Envoy
shares the host cgroup namespace, as privileged containers do,
/sys/fs/cgroup is
the cgroup v2 root: its cpu.stat covers the whole machine and it has no
cpu.max.
Detection failed and config initialization aborted the server.

Resolve the process's own cgroup from /proc/self/cgroup and
/proc/self/mountinfo,
reusing the parsing in CgroupCpuUtil, before probing the cgroup v2
files, so the
monitor reports container rather than host usage. Key cgroup v2
detection on
cpu.stat. An absent cpu.max means no CPU limit and an absent
cpuset.cpus.effective falls back to the CPU affinity count, while a file
that
exists but cannot be read fails the sample. When no supported cgroup CPU
implementation is found, config initialization still fails as before.
HOST mode
is unchanged.

Additional Description:
Measured inside an affected container (privileged, Kubernetes with
containerd,
cgroup v2):

```
/proc/self/cgroup     0::/kubepods.slice/kubepods-pod<uid>.slice/cri-containerd-<id>.scope
/proc/self/mountinfo  ... 0:30 / /sys/fs/cgroup ... - cgroup2 cgroup rw

/sys/fs/cgroup/cpu.stat                  present   usage_usec 17341693371   <- whole machine
/sys/fs/cgroup/cpu.max                   absent                            <- none at the root
/sys/fs/cgroup/cpuset.cpus.effective     present

<own cgroup>/cpu.stat                    present   usage_usec 12356042342   <- this container
<own cgroup>/cpu.max                     present   max 100000
<own cgroup>/cpuset.cpus.effective       present   0-31
```

`CgroupCpuUtil::getCurrentCgroupInfo()` is a new public wrapper around
the
existing `discoverCgroupMount()` and `constructCgroupPath()`, so the
monitor
shares the mount root stripping and `mountinfo` unescaping already used
for
concurrency detection.

Related to #39978. This supersedes #47503.

AI assistance disclosure: this change was prepared with AI assistance.
The
submitter has reviewed and fully understands all of the submitted code,
per the
project's generative AI policy.

Risk Level: Low. Any resolution failure falls back to the previous
`/sys/fs/cgroup` behavior.

Testing: Unit tests in
`test/extensions/resource_monitors/cpu_utilization/` and
`test/server/cgroup_cpu_util_test.cc` cover resolution in the host
cgroup
namespace, `0::/` resolving to the mount point, the fallback when the
resolved
cgroup has no `cpu.stat`, absent versus unreadable optional
files (stat errors other than `ENOENT`, and read failures after a
successful
stat), the affinity fallback, and the config-init error when no cgroup
CPU
support is found. A native macOS build of Envoy was not possible for
unrelated
toolchain reasons, so build and test verification relies on CI.

Docs Changes: N/A

Release Notes: Updated

`changelogs/current/bug_fixes/cpu_utilization__cgroup-v2-graceful-detection.rst`.

Platform Specific Features: Linux cgroup detection only. The affinity
fallback
is compiled on Linux only; other platforms keep
`hardware_concurrency()`.

---------

Signed-off-by: William Dauchy <william.dauchy@datadoghq.com>

Author: wdauchy

changelogs/current/bug_fixes/cpu_utilization__cgroup-v2-graceful-detection.rst

@@ -0,0 +1,7 @@
+Fixed the ``envoy.resource_monitors.cpu_utilization`` monitor in ``CONTAINER`` mode when Envoy shares
+the host cgroup namespace, for example in a privileged container. ``/sys/fs/cgroup`` is then the
+cgroup v2 root, which has no ``cpu.max``, so detection failed and config initialization aborted
+the server. The monitor now resolves its own cgroup from ``/proc/self/cgroup`` and
+``/proc/self/mountinfo`` and reports container rather than host usage. cgroup v2 detection is keyed
+on ``cpu.stat``: an absent ``cpu.max`` means no CPU limit and an absent ``cpuset.cpus.effective``
+falls back to the CPU affinity count, while a file that exists but cannot be read fails the sample.

source/extensions/resource_monitors/cpu_utilization/BUILD

@@ -51,6 +51,8 @@ envoy_cc_library(
         "//source/common/common:assert_lib",
         "//source/common/common:logger_lib",
         "//source/common/common:thread_lib",
+        "//source/server:cgroup_cpu_util_lib",
+        "//source/server:options_base",
         "@abseil-cpp//absl/strings",
         "@envoy_api//envoy/extensions/resource_monitors/cpu_utilization/v3:pkg_cc_proto",
     ],

source/extensions/resource_monitors/cpu_utilization/cpu_paths.h

@@ -5,6 +5,7 @@
 #include "envoy/filesystem/filesystem.h"
 
 #include "absl/strings/str_cat.h"
+#include "absl/strings/string_view.h"
 
 namespace Envoy {
 namespace Extensions {
@@ -39,15 +40,22 @@ struct CpuPaths {
   };
 
   struct V2 {
+    // Returns the cgroup v2 mount point, used when the process's own cgroup cannot be resolved.
+    static absl::string_view getBasePath() { return CGROUP_V2_BASE; }
+
     // Returns the full path to the CPU stat file (cpu.stat).
-    static std::string getStatPath() { return absl::StrCat(CGROUP_V2_BASE, STAT); }
+    static std::string getStatPath(absl::string_view base = CGROUP_V2_BASE) {
+      return absl::StrCat(base, STAT);
+    }
 
     // Returns the full path to the CPU max file (cpu.max).
-    static std::string getMaxPath() { return absl::StrCat(CGROUP_V2_BASE, MAX); }
+    static std::string getMaxPath(absl::string_view base = CGROUP_V2_BASE) {
+      return absl::StrCat(base, MAX);
+    }
 
     // Returns the full path to the effective CPUs file (cpuset.cpus.effective).
-    static std::string getEffectiveCpusPath() {
-      return absl::StrCat(CGROUP_V2_BASE, EFFECTIVE_CPUS);
+    static std::string getEffectiveCpusPath(absl::string_view base = CGROUP_V2_BASE) {
+      return absl::StrCat(base, EFFECTIVE_CPUS);
     }
 
   private:
@@ -57,10 +65,10 @@ struct CpuPaths {
     static constexpr const char* const EFFECTIVE_CPUS = "/cpuset.cpus.effective";
   };
 
-  // Returns whether cgroup v2 CPU subsystem is available.
-  static bool isV2(Filesystem::Instance& fs) {
-    return fs.fileExists(V2::getStatPath()) && fs.fileExists(V2::getMaxPath()) &&
-           fs.fileExists(V2::getEffectiveCpusPath());
+  // Keyed only on cpu.stat: cpu.max and cpuset.cpus.effective are optional (often
+  // absent on Kubernetes v2 pods) and handled with fallbacks by the reader.
+  static bool isV2(Filesystem::Instance& fs, absl::string_view base = V2::getBasePath()) {
+    return fs.fileExists(V2::getStatPath(base));
   }
 
   // Returns whether cgroup v1 CPU subsystem is available.

source/extensions/resource_monitors/cpu_utilization/linux_cpu_stats_reader.cc

@@ -1,8 +1,11 @@
 #include "source/extensions/resource_monitors/cpu_utilization/linux_cpu_stats_reader.h"
 
 #include <algorithm>
+#include <cerrno>
 #include <chrono>
+#include <optional>
 #include <sstream>
+#include <thread>
 #include <vector>
 
 #include "envoy/common/exception.h"
@@ -11,9 +14,16 @@
 #include "source/common/common/assert.h"
 #include "source/common/common/fmt.h"
 #include "source/common/common/thread.h"
+#include "source/server/cgroup_cpu_util.h"
+
+#ifdef __linux__
+#include "source/server/options_impl_platform_linux.h"
+#endif
 
 #include "absl/strings/numbers.h"
+#include "absl/strings/str_cat.h"
 #include "absl/strings/str_split.h"
+#include "absl/strings/string_view.h"
 #include "absl/strings/strip.h"
 
 namespace Envoy {
@@ -26,6 +36,37 @@ constexpr uint64_t NUMBER_OF_CPU_TIMES_TO_PARSE =
 
 namespace {
 
+// Fallback CPU count when cpuset.cpus.effective is absent. Honors the affinity mask
+// and floors at 1 to avoid divide-by-zero in downstream utilization math.
+int defaultCpuCount() {
+  const unsigned int hw_threads = std::max(1u, std::thread::hardware_concurrency());
+#ifdef __linux__
+  return static_cast<int>(OptionsImplPlatformLinux::getCpuAffinityCount(hw_threads));
+#else
+  return static_cast<int>(hw_threads);
+#endif
+}
+
+// Reads an optional cgroup interface file. std::nullopt means the file does not
+// exist; any other failure is an error, so that a permission problem is not
+// mistaken for an absent controller.
+absl::StatusOr<std::optional<std::string>> readOptionalFile(Filesystem::Instance& fs,
+                                                            const std::string& path) {
+  const Api::IoCallResult<Filesystem::FileInfo> info = fs.stat(path);
+  if (!info.ok()) {
+    if (info.err_->getSystemErrorCode() == ENOENT) {
+      return std::optional<std::string>();
+    }
+    return absl::UnavailableError(
+        absl::StrCat("unable to stat ", path, ": ", info.err_->getErrorDetails()));
+  }
+  absl::StatusOr<std::string> contents = fs.fileReadToEnd(path);
+  if (!contents.ok()) {
+    return contents.status();
+  }
+  return std::optional<std::string>(std::move(contents).value());
+}
+
 absl::StatusOr<int> parseEffectiveCpus(absl::string_view effective_cpu_list,
                                        const std::string& effective_path) {
   int cpu_count = 0;
@@ -165,6 +206,18 @@ absl::StatusOr<double> LinuxCpuStatsReader::getUtilization() {
 
 absl::StatusOr<LinuxContainerCpuStatsReader::ContainerStatsReaderPtr>
 LinuxContainerCpuStatsReader::create(Filesystem::Instance& fs, TimeSource& time_source) {
+  // Prefer the process's own cgroup: in the host cgroup namespace the mount point is
+  // the cgroup root, which reports whole-machine usage and has no cpu.max.
+  const std::optional<CgroupInfo> cgroup = CgroupCpuUtil::getCurrentCgroupInfo(fs);
+  if (cgroup.has_value() && cgroup->version == "v2") {
+    const std::string base(absl::StripSuffix(cgroup->full_path, "/"));
+    if (CpuPaths::isV2(fs, base)) {
+      return std::make_unique<CgroupV2CpuStatsReader>(fs, time_source, base);
+    }
+    ENVOY_LOG_MISC(debug, "No cpu.stat in cgroup {}, falling back to {}", base,
+                   CpuPaths::V2::getBasePath());
+  }
+
   if (CpuPaths::isV2(fs)) {
     return std::make_unique<CgroupV2CpuStatsReader>(fs, time_source);
   }
@@ -261,9 +314,14 @@ absl::StatusOr<double> CgroupV1CpuStatsReader::getUtilization() {
 }
 
 CgroupV2CpuStatsReader::CgroupV2CpuStatsReader(Filesystem::Instance& fs, TimeSource& time_source)
-    : LinuxContainerCpuStatsReader(fs, time_source), stat_path_(CpuPaths::V2::getStatPath()),
-      max_path_(CpuPaths::V2::getMaxPath()), effective_path_(CpuPaths::V2::getEffectiveCpusPath()) {
-}
+    : CgroupV2CpuStatsReader(fs, time_source, CpuPaths::V2::getBasePath()) {}
+
+CgroupV2CpuStatsReader::CgroupV2CpuStatsReader(Filesystem::Instance& fs, TimeSource& time_source,
+                                               absl::string_view base_path)
+    : LinuxContainerCpuStatsReader(fs, time_source),
+      stat_path_(CpuPaths::V2::getStatPath(base_path)),
+      max_path_(CpuPaths::V2::getMaxPath(base_path)),
+      effective_path_(CpuPaths::V2::getEffectiveCpusPath(base_path)) {}
 
 CgroupV2CpuStatsReader::CgroupV2CpuStatsReader(Filesystem::Instance& fs, TimeSource& time_source,
                                                const std::string& stat_path,
@@ -306,35 +364,50 @@ CpuTimesV2 CgroupV2CpuStatsReader::getCpuTimes() {
     return {false, 0, 0, 0};
   }
 
-  // Read cpuset.cpus.effective
-  auto effective_result = fs_.fileReadToEnd(effective_path_);
+  // CPU count from cpuset.cpus.effective, falling back to the affinity CPU count
+  // when absent. If present it must be readable and well-formed.
+  // Format can be: "0", "0-3", "0,2,4", "0-2,4", "0-3,5-7", etc.
+  int N = 0;
+  const absl::StatusOr<std::optional<std::string>> effective_result =
+      readOptionalFile(fs_, effective_path_);
   if (!effective_result.ok()) {
-    ENVOY_LOG(error, "Unable to read effective CPUs file at {}", effective_path_);
+    ENVOY_LOG(error, "Unable to read effective CPUs file {}: {}", effective_path_,
+              effective_result.status().message());
     return {false, 0, 0, 0};
   }
-
-  // Parse effective CPUs
-  // Format can be: "0", "0-3", "0,2,4", "0-2,4", "0-3,5-7", etc.
-  absl::StatusOr<int> cpu_count = parseEffectiveCpus(effective_result.value(), effective_path_);
-  if (!cpu_count.ok()) {
-    ENVOY_LOG(error, "Failed to parse effective CPUs file {}: {}", effective_path_,
-              cpu_count.status().message());
-    return {false, 0, 0, 0};
+  if (effective_result->has_value()) {
+    absl::StatusOr<int> cpu_count = parseEffectiveCpus(**effective_result, effective_path_);
+    if (!cpu_count.ok()) {
+      ENVOY_LOG(error, "Failed to parse effective CPUs file {}: {}", effective_path_,
+                cpu_count.status().message());
+      return {false, 0, 0, 0};
+    }
+    N = cpu_count.value();
+  } else {
+    N = defaultCpuCount();
+    ENVOY_LOG(trace, "cpuset.cpus.effective absent at {}, falling back to {} CPU(s)",
+              effective_path_, N);
   }
-  const int N = cpu_count.value();
 
-  // Read cpu.max
-  auto max_result = fs_.fileReadToEnd(max_path_);
+  // Effective core count from cpu.max. An absent file or a "max" quota means no
+  // limit, i.e. all available CPUs; a present file must be readable and well-formed.
+  double effective_cores = static_cast<double>(N);
+  const absl::StatusOr<std::optional<std::string>> max_result = readOptionalFile(fs_, max_path_);
   if (!max_result.ok()) {
-    ENVOY_LOG(error, "Unable to read CPU max file at {}", max_path_);
+    ENVOY_LOG(error, "Unable to read cpu.max file {}: {}", max_path_,
+              max_result.status().message());
     return {false, 0, 0, 0};
   }
-
-  absl::StatusOr<double> effective_cores = parseEffectiveCores(max_result.value(), N);
-  if (!effective_cores.ok()) {
-    ENVOY_LOG(error, "Failed to parse cpu.max file {}: {}", max_path_,
-              effective_cores.status().message());
-    return {false, 0, 0, 0};
+  if (max_result->has_value()) {
+    absl::StatusOr<double> parsed_cores = parseEffectiveCores(**max_result, N);
+    if (!parsed_cores.ok()) {
+      ENVOY_LOG(error, "Failed to parse cpu.max file {}: {}", max_path_,
+                parsed_cores.status().message());
+      return {false, 0, 0, 0};
+    }
+    effective_cores = parsed_cores.value();
+  } else {
+    ENVOY_LOG(trace, "cpu.max absent at {}, assuming no CPU limit ({} core(s))", max_path_, N);
   }
 
   // Convert usage from usec to match our time units
@@ -344,9 +417,9 @@ CpuTimesV2 CgroupV2CpuStatsReader::getCpuTimes() {
                                     .count();
 
   ENVOY_LOG(trace, "cgroupv2 usage_usec: {}, effective_cores: {}, current_time: {}", usage_usec,
-            effective_cores.value(), current_time);
+            effective_cores, current_time);
 
-  return {true, cpu_times_value_us, current_time, effective_cores.value()};
+  return {true, cpu_times_value_us, current_time, effective_cores};
 }
 
 absl::StatusOr<double> CgroupV2CpuStatsReader::getUtilization() {

source/extensions/resource_monitors/cpu_utilization/linux_cpu_stats_reader.h

@@ -98,6 +98,10 @@ class CgroupV2CpuStatsReader : public LinuxContainerCpuStatsReader,
 public:
   explicit CgroupV2CpuStatsReader(Filesystem::Instance& fs, TimeSource& time_source);
 
+  // Reads the interface files of a specific cgroup directory.
+  CgroupV2CpuStatsReader(Filesystem::Instance& fs, TimeSource& time_source,
+                         absl::string_view base_path);
+
   // Test-friendly constructor that accepts custom file paths
   CgroupV2CpuStatsReader(Filesystem::Instance& fs, TimeSource& time_source,
                          const std::string& stat_path, const std::string& max_path,

source/server/cgroup_cpu_util.cc

@@ -225,6 +225,14 @@ std::optional<CgroupPathInfo> CgroupCpuUtil::getCurrentCgroupPath(Filesystem::In
   return CgroupPathInfo{v2_path, "v2"};
 }
 
+std::optional<CgroupInfo> CgroupCpuUtil::getCurrentCgroupInfo(Filesystem::Instance& fs) {
+  const std::optional<CgroupMount> mount = discoverCgroupMount(fs);
+  if (!mount.has_value()) {
+    return std::nullopt;
+  }
+  return constructCgroupPath(*mount, fs);
+}
+
 // Constructs complete cgroup path by combining mount metadata and process assignment.
 std::optional<CgroupInfo> CgroupCpuUtil::constructCgroupPath(const CgroupMount& mount,
                                                              Filesystem::Instance& fs) {

source/server/cgroup_cpu_util.h

@@ -117,6 +117,15 @@ class CgroupCpuUtil {
   static std::optional<uint32_t> getCpuLimit(Filesystem::Instance& fs,
                                              CgroupDetectionDiagnostic* diag = nullptr);
 
+  /**
+   * Resolves the calling process's own `cgroup` directory by combining the `cgroup` mount from
+   * `/proc/self/mountinfo` with the path from `/proc/self/cgroup`. The mount point alone is the
+   * `cgroup` root when the process shares the host `cgroup` namespace.
+   * @param fs Filesystem instance for file operations.
+   * @return CgroupInfo with the directory and version, nullopt if it cannot be determined.
+   */
+  static std::optional<CgroupInfo> getCurrentCgroupInfo(Filesystem::Instance& fs);
+
 private:
   /**
    * Reads CPU limit from specific `cgroup` `v1` paths.

test/extensions/resource_monitors/cpu_utilization/BUILD

@@ -36,6 +36,8 @@ envoy_extension_cc_test(
         "//test/mocks/server:options_mocks",
         "//test/test_common:environment_lib",
         "//test/test_common:status_utility_lib",
+        "@abseil-cpp//absl/status",
+        "@abseil-cpp//absl/strings",
     ],
 )
 

test/extensions/resource_monitors/cpu_utilization/cpu_paths_test.cc

@@ -49,6 +49,17 @@ TEST(CpuPathsV2Test, GetEffectiveCpusPathReturnsCorrectPath) {
   EXPECT_EQ(CpuPaths::V2::getEffectiveCpusPath(), "/sys/fs/cgroup/cpuset.cpus.effective");
 }
 
+TEST(CpuPathsV2Test, GetBasePathReturnsMountPoint) {
+  EXPECT_EQ(CpuPaths::V2::getBasePath(), "/sys/fs/cgroup");
+}
+
+TEST(CpuPathsV2Test, PathsAreRelativeToSuppliedBase) {
+  const std::string base = "/sys/fs/cgroup/kubepods.slice/pod.slice/container.scope";
+  EXPECT_EQ(CpuPaths::V2::getStatPath(base), base + "/cpu.stat");
+  EXPECT_EQ(CpuPaths::V2::getMaxPath(base), base + "/cpu.max");
+  EXPECT_EQ(CpuPaths::V2::getEffectiveCpusPath(base), base + "/cpuset.cpus.effective");
+}
+
 // =============================================================================
 // CpuPaths::isV1() Detection Tests
 // =============================================================================
@@ -90,45 +101,40 @@ TEST(CpuPathsDetectionTest, IsV1ReturnsFalseWhenUsageFileMissing) {
 // CpuPaths::isV2() Detection Tests
 // =============================================================================
 
-TEST(CpuPathsDetectionTest, IsV2ReturnsTrueWhenAllFilesExist) {
+TEST(CpuPathsDetectionTest, IsV2ReturnsTrueWhenStatFileExists) {
   Filesystem::MockInstance mock_fs;
 
-  // Mock all three required V2 files as existing
+  // Keyed only on cpu.stat; the optional v2 files must not be probed.
   EXPECT_CALL(mock_fs, fileExists("/sys/fs/cgroup/cpu.stat")).WillOnce(Return(true));
-  EXPECT_CALL(mock_fs, fileExists("/sys/fs/cgroup/cpu.max")).WillOnce(Return(true));
-  EXPECT_CALL(mock_fs, fileExists("/sys/fs/cgroup/cpuset.cpus.effective")).WillOnce(Return(true));
+
+  EXPECT_TRUE(CpuPaths::isV2(mock_fs));
+}
+
+TEST(CpuPathsDetectionTest, IsV2ReturnsTrueWhenOnlyStatFileExists) {
+  Filesystem::MockInstance mock_fs;
+
+  // A normal v2 pod may expose only cpu.stat; detection must still succeed.
+  EXPECT_CALL(mock_fs, fileExists("/sys/fs/cgroup/cpu.stat")).WillOnce(Return(true));
+  EXPECT_CALL(mock_fs, fileExists("/sys/fs/cgroup/cpu.max")).Times(0);
+  EXPECT_CALL(mock_fs, fileExists("/sys/fs/cgroup/cpuset.cpus.effective")).Times(0);
 
   EXPECT_TRUE(CpuPaths::isV2(mock_fs));
 }
 
 TEST(CpuPathsDetectionTest, IsV2ReturnsFalseWhenStatFileMissing) {
   Filesystem::MockInstance mock_fs;
 
-  // Mock stat file missing
   EXPECT_CALL(mock_fs, fileExists("/sys/fs/cgroup/cpu.stat")).WillOnce(Return(false));
-  // Short-circuit evaluation - other file checks may not be called
-  EXPECT_CALL(mock_fs, fileExists("/sys/fs/cgroup/cpu.max"))
-      .Times(testing::AtMost(1))
-      .WillRepeatedly(Return(true));
-  EXPECT_CALL(mock_fs, fileExists("/sys/fs/cgroup/cpuset.cpus.effective"))
-      .Times(testing::AtMost(1))
-      .WillRepeatedly(Return(true));
 
   EXPECT_FALSE(CpuPaths::isV2(mock_fs));
 }
 
-TEST(CpuPathsDetectionTest, IsV2ReturnsFalseWhenMaxFileMissing) {
+TEST(CpuPathsDetectionTest, IsV2ProbesSuppliedBase) {
   Filesystem::MockInstance mock_fs;
 
-  // Mock max file missing
-  EXPECT_CALL(mock_fs, fileExists("/sys/fs/cgroup/cpu.stat")).WillOnce(Return(true));
-  EXPECT_CALL(mock_fs, fileExists("/sys/fs/cgroup/cpu.max")).WillOnce(Return(false));
-  // Short-circuit evaluation - effective_cpus check may not be called
-  EXPECT_CALL(mock_fs, fileExists("/sys/fs/cgroup/cpuset.cpus.effective"))
-      .Times(testing::AtMost(1))
-      .WillRepeatedly(Return(true));
+  EXPECT_CALL(mock_fs, fileExists("/sys/fs/cgroup/pod.slice/cpu.stat")).WillOnce(Return(true));
 
-  EXPECT_FALSE(CpuPaths::isV2(mock_fs));
+  EXPECT_TRUE(CpuPaths::isV2(mock_fs, "/sys/fs/cgroup/pod.slice"));
 }
 
 } // namespace