feat: enable THP for guest memory

This commit adds THP for the guest memory, with a new value for the
huge_pages option.

Signed-off-by: Marco Marangoni <mamarang@amazon.com>

Author: marco-marangoni

CHANGELOG.md

@@ -10,6 +10,12 @@ and this project adheres to
 
 ### Added
 
+- [#6003](https://github.com/firecracker-microvm/firecracker/pull/6003): Added a
+  new option `Transparent` for the `huge_pages` setting. If set, Firecracker
+  will use transparent huge pages for the guest memory via
+  `madvise(MADV_HUGEPAGE)`. Guest memory must be a multiple of 2MB when using
+  this option.
+
 ### Changed
 
 ### Deprecated

docs/hugepages.md

@@ -1,14 +1,48 @@
 # Backing Guest Memory by Huge Pages
 
-Firecracker supports backing the guest memory of a VM by 2MB hugetlbfs pages.
-This can be enabled by setting the `huge_pages` field of `PUT` or `PATCH`
-requests to the `/machine-config` endpoint to `2M`.
-
-Backing guest memory by huge pages can bring performance improvements for
-specific workloads, due to less TLB contention and less overhead during
-virtual->physical address resolution. It can also help reduce the number of
-KVM_EXITS required to rebuild extended page tables post snapshot restore, as
-well as improve boot times (by up to 50% as measured by Firecracker's
+Firecracker supports three modes for the `huge_pages` field of `PUT` or `PATCH`
+requests to the `/machine-config` endpoint:
+
+- `None` (default): Uses regular 4K pages with no huge page behavior.
+- `Transparent`: Uses `madvise(MADV_HUGEPAGE)` to request transparent huge pages
+  for guest memory. Guest memory size must be a multiple of 2MB.
+- `2M`: Backs guest memory by 2MB hugetlbfs pages.
+
+## Transparent Huge Pages (THP)
+
+Setting `huge_pages` to `Transparent` enables transparent huge pages for guest
+memory via `madvise(MADV_HUGEPAGE)`. This allows the kernel to opportunistically
+back guest memory with huge pages without requiring a pre-allocated hugetlbfs
+pool.
+
+Note that while traditional THP uses PMD-sized pages (2MB on x86_64), the actual
+THP size depends on the CPU architecture. Modern kernels also support
+"multi-size THP" (mTHP), which can allocate pages in various power-of-2 sizes
+between the base page size and PMD size (e.g. 16K, 32K, 64K). Firecracker
+requires guest memory size to be a multiple of 2MB regardless of the THP size
+used by the host kernel.
+
+Limitations:
+
+- When vhost-user-blk devices are in use, guest memory is memfd-backed (shared
+  memory). THP for shared/shmem memory is controlled separately from anonymous
+  memory via `/sys/kernel/mm/transparent_hugepage/shmem_enabled` and may not be
+  enabled by default. Refer to the
+  [Linux Documentation on shmem THP][thp_shmem_docs] for details on how to
+  configure it.
+- THP does not integrate with UFFD; no transparent huge pages will be allocated
+  during userfault-handling while resuming from a snapshot.
+
+Please refer to the [Linux Documentation][thp_docs] for more information.
+
+## Hugetlbfs (2M)
+
+Setting `huge_pages` to `2M` backs guest memory by 2MB hugetlbfs pages. This can
+bring performance improvements for specific workloads, due to less TLB
+contention and less overhead during virtual->physical address resolution. It can
+also help reduce the number of KVM_EXITS required to rebuild extended page
+tables post snapshot restore, as well as improve boot times (by up to 50% as
+measured by Firecracker's
 [boot time performance tests](../tests/integration_tests/performance/test_boottime.py))
 
 Using hugetlbfs requires the host running Firecracker to have a pre-allocated
@@ -19,7 +53,7 @@ not try to reserve sufficient hugetlbfs pages at the time of the `mmap` call,
 trying to claim them from the pool on-demand. For details on how to manage this
 pool, please refer to the [Linux Documentation][hugetlbfs_docs].
 
-## Huge Pages and Snapshotting
+### Huge Pages and Snapshotting
 
 Restoring a Firecracker snapshot of a microVM backed by huge pages will also use
 huge pages to back the restored guest. There is no option to flip between
@@ -43,15 +77,6 @@ the device is unable to reclaim the hugepage backing of the guest and drop RSS.
 However, the balloon can still be inflated and used to restrict memory usage in
 the guest.
 
-## FAQ
-
-### Why does Firecracker not offer a transparent huge pages (THP) setting?
-
-Firecracker's guest memory can be memfd based. Linux (as of 6.1) does not offer
-a way to dynamically enable THP for such memory regions. Additionally, UFFD does
-not integrate with THP (no transparent huge pages will be allocated during
-userfaulting). Please refer to the [Linux Documentation][thp_docs] for more
-information.
-
 [hugetlbfs_docs]: https://docs.kernel.org/admin-guide/mm/hugetlbpage.html
 [thp_docs]: https://www.kernel.org/doc/html/next/admin-guide/mm/transhuge.html#hugepages-in-tmpfs-shmem
+[thp_shmem_docs]: https://www.kernel.org/doc/html/latest/admin-guide/mm/transhuge.html#shmem-internal-tmpfs

src/firecracker/src/api_server/request/machine_configuration.rs

@@ -104,6 +104,7 @@ mod tests {
 
         let huge_pages_cases = [
             ("None", HugePageConfig::None),
+            ("Transparent", HugePageConfig::Transparent),
             ("2M", HugePageConfig::Hugetlbfs2M),
         ];
 

src/firecracker/swagger/firecracker.yaml

@@ -1442,8 +1442,13 @@ definitions:
         type: string
         enum:
           - None
+          - Transparent
           - 2M
-        description: Which huge pages configuration (if any) should be used to back guest memory.
+        default: None
+        description: >-
+          Which huge pages configuration should be used to back guest memory.
+          "None" uses regular 4K pages. "Transparent" enables THP via
+          madvise(MADV_HUGEPAGE). "2M" uses explicit hugetlbfs 2MB pages.
 
   MemoryBackend:
     type: object

src/vmm/src/devices/virtio/vhost_user.rs

@@ -487,6 +487,7 @@ pub(crate) mod tests {
                 libc::MAP_PRIVATE,
                 Some(file),
                 false,
+                libc::MADV_HUGEPAGE,
             )
             .unwrap()
             .into_iter()

src/vmm/src/persist.rs

@@ -449,8 +449,13 @@ pub fn restore_from_snapshot(
                 .into());
             }
             (
-                guest_memory_from_file(mem_backend_path, mem_state, track_dirty_pages)
-                    .map_err(RestoreFromSnapshotGuestMemoryError::File)?,
+                guest_memory_from_file(
+                    mem_backend_path,
+                    mem_state,
+                    track_dirty_pages,
+                    vm_resources.machine_config.huge_pages,
+                )
+                .map_err(RestoreFromSnapshotGuestMemoryError::File)?,
                 None,
             )
         }
@@ -512,9 +517,11 @@ fn guest_memory_from_file(
     mem_file_path: &Path,
     mem_state: &GuestMemoryState,
     track_dirty_pages: bool,
+    huge_pages: HugePageConfig,
 ) -> Result<Vec<GuestRegionMmap>, GuestMemoryFromFileError> {
     let mem_file = File::open(mem_file_path)?;
-    let guest_mem = memory::snapshot_file(mem_file, mem_state.regions(), track_dirty_pages)?;
+    let guest_mem =
+        memory::snapshot_file(mem_file, mem_state.regions(), track_dirty_pages, huge_pages)?;
     Ok(guest_mem)
 }
 

src/vmm/src/resources.rs

@@ -580,6 +580,7 @@ mod tests {
     use crate::vmm_config::RateLimiterConfig;
     use crate::vmm_config::boot_source::{BootConfig, BootSource, BootSourceConfig};
     use crate::vmm_config::drive::{BlockBuilder, BlockDeviceConfig};
+    use crate::vmm_config::machine_config::HugePageConfig::{Hugetlbfs2M, Transparent};
     use crate::vmm_config::machine_config::{HugePageConfig, MachineConfig, MachineConfigError};
     use crate::vmm_config::net::{NetBuilder, NetworkInterfaceConfig};
     use crate::vmm_config::vsock::tests::default_config;
@@ -1476,6 +1477,26 @@ mod tests {
             Err(MachineConfigError::InvalidMemorySize)
         );
 
+        // Odd memory size - not supported by THP/Hugetlbfs
+        aux_vm_config.mem_size_mib = Some(1025);
+        aux_vm_config.huge_pages = Some(Transparent);
+        assert_eq!(
+            vm_resources.update_machine_config(&aux_vm_config),
+            Err(MachineConfigError::InvalidMemorySize)
+        );
+        aux_vm_config.huge_pages = Some(Hugetlbfs2M);
+        assert_eq!(
+            vm_resources.update_machine_config(&aux_vm_config),
+            Err(MachineConfigError::InvalidMemorySize)
+        );
+        // Odd size supported by HugePageConfig::None
+        aux_vm_config.huge_pages = Some(HugePageConfig::None);
+        vm_resources.update_machine_config(&aux_vm_config).unwrap();
+        assert_eq!(
+            MachineConfigUpdate::from(vm_resources.machine_config.clone()),
+            aux_vm_config
+        );
+
         // Incompatible mem_size_mib with balloon size.
         vm_resources.machine_config.mem_size_mib = 128;
         vm_resources

src/vmm/src/vmm_config/machine_config.rs

@@ -34,9 +34,11 @@ pub enum MachineConfigError {
 /// Describes the possible (huge)page configurations for a microVM's memory.
 #[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
 pub enum HugePageConfig {
-    /// Do not use hugepages, e.g. back guest memory by 4K
+    /// Back guest memory by 4K pages, no hugepage behavior
     #[default]
     None,
+    /// Use madvise(MADV_HUGEPAGE) for transparent huge pages
+    Transparent,
     /// Back guest memory by 2MB hugetlbfs pages
     #[serde(rename = "2M")]
     Hugetlbfs2M,
@@ -49,6 +51,10 @@ impl HugePageConfig {
         let divisor = match self {
             // Any integer memory size expressed in MiB will be a multiple of 4096KiB.
             HugePageConfig::None => 1,
+            // Note: THP technically supports memory not multiple of 2MB, however that disables some optimizations done
+            // by the kernel (e.g. automatic memory alignment on Linux 6.4+).
+            // To avoid performance/fragmentation surprises, having a memory multiple of 2MB is wiser.
+            HugePageConfig::Transparent => 2,
             HugePageConfig::Hugetlbfs2M => 2,
         };
 
@@ -59,11 +65,20 @@ impl HugePageConfig {
     /// create a mapping backed by huge pages as described by this [`HugePageConfig`].
     pub fn mmap_flags(&self) -> libc::c_int {
         match self {
-            HugePageConfig::None => 0,
+            HugePageConfig::None | HugePageConfig::Transparent => 0,
             HugePageConfig::Hugetlbfs2M => libc::MAP_HUGETLB | libc::MAP_HUGE_2MB,
         }
     }
 
+    /// Returns the flags required to pass to [libc::madvise], after allocating anonymous guest memory.
+    /// Note: returning [libc::MADV_NORMAL] might skip the call to `madvise` entirely.
+    pub fn madvise_flags(&self) -> libc::c_int {
+        match self {
+            HugePageConfig::Transparent => libc::MADV_HUGEPAGE,
+            HugePageConfig::None | HugePageConfig::Hugetlbfs2M => libc::MADV_NORMAL,
+        }
+    }
+
     /// Returns `true` iff this [`HugePageConfig`] describes a hugetlbfs-based configuration.
     pub fn is_hugetlbfs(&self) -> bool {
         matches!(self, HugePageConfig::Hugetlbfs2M)
@@ -72,7 +87,7 @@ impl HugePageConfig {
     /// Gets the page size in bytes of this [`HugePageConfig`].
     pub fn page_size(&self) -> usize {
         match self {
-            HugePageConfig::None => 4096,
+            HugePageConfig::None | HugePageConfig::Transparent => 4096,
             HugePageConfig::Hugetlbfs2M => 2 * 1024 * 1024,
         }
     }
@@ -81,7 +96,7 @@ impl HugePageConfig {
 impl From<HugePageConfig> for Option<memfd::HugetlbSize> {
     fn from(value: HugePageConfig) -> Self {
         match value {
-            HugePageConfig::None => None,
+            HugePageConfig::None | HugePageConfig::Transparent => None,
             HugePageConfig::Hugetlbfs2M => Some(memfd::HugetlbSize::Huge2MB),
         }
     }