Node memory swap support

[ehashman]

Enhancement Description

• One-line enhancement description (can be used as a release note): Kubernetes nodes support swap memory.
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2400-node-swap
• Discussion Link: https://docs.google.com/document/d/1CZtRtC8W8FwW_VWQLKP9DW_2H-hcLBcH9KBbukie67M/edit# Kubelet/Kubernetes should work with Swap Enabled kubernetes#53533
• Primary contact (assignee): @iholder101 @kannon92 @pacoxu
• Responsible SIGs: Node
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): 1.22
  • Beta1 release target (x.y): 1.28
  • Beta2 release target (x.y): 1.29-1.31
  • Beta3 release target (x.y): 1.33
  • Stable release target (x.y): 1.34
• Alpha
  • KEP (k/enhancements) update PR(s): Node system swap support #2602
  • Code (k/k) update PR(s):
    • CI/test infra changes: Setup new node image with swap enabled test-infra#22602 Add swap periodics for node test-infra#22809
    • CRI and kubelet changes Alpha node swap support kubernetes#102823
  • Docs (k/website) update PR(s): KEP-2400: Alpha node support for swap memory website#28838
• Beta1
  • KEP (k/enhancements) update PR(s):
    • KEP-2400: Update swap KEP for 1.23 beta #2858
    • KEP-2400: Graduate swap to Beta 1 #3957
  • Issues and Code (k/k) update PR(s):
    • Support cgroups v2 with auto-calculated LimitedSwap for Burstable QoS pods
      • [KEP-2400] Enable Swap Support for Cgroup v2 Only kubernetes#119427
      • [KEP-2400] Enable auto-calculated LimitedSwap for Burstable QoS pods kubernetes#119428
      • [KEP-2400] Add full cgroup v2 swap support with automatically calculated swap limit for LimitedSwap and Burstable QoS Pods kubernetes#118764
    • Add Swap stats to /stats/summary and /metrics/resource
      • [KEP-2400] Add swap memory to the Kubelet stats API kubernetes#119425
      • [KEP-2400] Add swap to stats to Summary API and Prometheus endpoints (/stats/summary and /metrics/resource) kubernetes#118865
    • Determine a set of metrics for measuring node performance with swap enabled
      • [KEP-2400] Determine a set of metrics for measuring node performance with swap enabled kubernetes#119424
      • [KEP-2400] Add swap to stats to Summary API and Prometheus endpoints (/stats/summary and /metrics/resource) kubernetes#118865
      • Add a metric for machine swap capacity google/cadvisor#3339
    • Improve test coverage and make sure tests are green
      • [KEP-2400] Swap: improve test coverage and make sure tests are green kubernetes#119430
      • [KEP-2400] Add full cgroup v2 swap support with automatically calculated swap limit for LimitedSwap and Burstable QoS Pods kubernetes#118764
  • Docs (k/website) update(s):
    • Swap graduates to Beta1: add a blog-post website#42009
    • Swap graduates to Beta1: add documentation and update feature gates website#42167
  • Misc
    • Bugfix: MachineInfo Clone() to also clone "SwapCapacity" google/cadvisor#3293
    • Bump cadvisor version to v0.47.3 kubernetes#119225
  • Archive / Outdated
• Beta2
  • KEP (k/enhancements) update PR(s):
    • 1.29 Graduate swap to beta 2 in 1.29 #4275
    • 1.30 [KEP-2400] - Update KEP with findings from beta2 #4401
    • 1.30 [KEP-2400] update latest milestone for swap KEP #4484
  • Issues and Code (k/k) update PR(s):
    • Add swap conformance tests kubernetes#122175
    • Fail Kubelet at startup if swap is configured with cgroup v1 kubernetes#122241
    • 1.30
      • [KEP-2400] add no swap as the default option for swap kubernetes#122745
      • [KEP-2400]: Fail limited swap cgroup v1 kubernetes#123738
    • 1.31
      • [Swap][Tests][KEP-2400] Add swap serial stress tests, improve NodeConformance tests and adapt NoSwap behavior kubernetes#123557
      • [KEP-2400] Avoid logging that swap cgroup controller is missing for every container kubernetes#123749
      • 1.31 [KEP-2400] Mount tmpfs memory-backed volumes with a noswap option if supported kubernetes#124060
      • 1.31 [KEP-2400]: Restrict access to swap for containers in high priority Pods kubernetes#125277
      • 1.31 [KEP-2400]: Swap e2e tests: skip swap stress tests if swap is not provisioned kubernetes#125402
  • Docs (k/website) update(s):
    • 1.30 [KEP-2400] Update swap with beta2 deprecations website#45128
• Beta3
  • KEP (k/enhancements) update PR(s): [KEP-2400] swap updates, beta3 graduation and GA criterias #4701
  • Issues and Code (k/k) update PR(s):
    • [KEP-2400] [failing-test] resource metrics e2e tests: expect swap node and container level stats kubernetes#130053
    • [KEP-2400] [Bugfix]: Ensure container-level swap metrics are collected kubernetes#129486
    • [KEP-2400] Report swap capacity as part of node.status.nodeSystemInfo kubernetes#129954
    • [KEP-2400]: Swap-aware memory eviction (with no additional APIs) kubernetes#129578
    • [KEP-2400] kubectl top: add a --show-swap option kubernetes#129458
  • Docs (k/website) update(s):
    • 2400: draft update for swap KEP website#49945
• GA (related PRs, can be assigned to a specific track later)
  • KEP (k/enhancements) update PR(s):
  • Issues and Code (k/k) update PR(s):
    • [KEP-2400] [Swap] [Discussion]: Add swap as a ResourceName for pod spec resources API kubernetes#120801
    • [KEP-2400] [Swap]: Verify memory pressure behavior with swap enabled kubernetes#120800
    • [KEP-2400] [Swap]: SwapConformance e2e testing kubernetes#120798.
    • [KEP-2400] Address swap impact on memory-backed volumes kubernetes#105978
    • Add a test for OOMKilled for LimitedSwap kubernetes#121820
    • Add the ability to set a system-reserved quantity of swap on a node
      • [KEP-2400] [Swap] Add the ability to set a system-reserved quantity of swap on a node kubernetes#120802
      • [KEP-2400] Add the ability to set a system-reserved quantity of swap on a node kubernetes#105019 dup by 120802
      • add --system-reserved support for swap(cgroup v2 only): only for burstable pods kubernetes#105271
    • Fail Kubelet at startup if swap is configured with cgroup v1 kubernetes#119327
    • [Swap Feature] Investigate what happens if a cluster has some swap enabled nodes and others that are not enabled. kubernetes#122279
  • Docs (k/website) update(s):

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

/sig node

[karan]

@cookieisaac

[ehashman]

/stage alpha
/milestone v1.22

[jrsapi]

Greetings @ehashman!
Enhancement shadow checking in with a few reminders. 1.22 Enhancements Freeze starts at 23:59:59 pst on Thursday, May 13. A few items needing review for this KEP:

• Enhancement README

• PRR approved

• Node memory swap support #2400 updated with current target and alpha information.

Thanks!

[ehashman]

All of this is covered in #2602

[jrsapi]

Greetings @ehashman!
Thanks for the follow-up. After reviewing the KEP and PRR everything looks on target. The enhancement is marked at risk but once merged we can move its status to tracked. A reminder that tomorrow 5/13 the Enhancement freeze starts at 23:59:59 PST.

Thanks!

[ehashman]

Work breakdown for 1.22

• Documentation for enabling swap (TBD - @ehashman?)
• CI environment/test updates (@ike-ma)
  • Build images with swap for 2 Linux distros
  • Add jobs to test-infra that use the images and enable the swap feature flag/kubelet option for node e2e suite
  • See also Test Plan
• CRI and kubelet changes (@ehashman)
  • Complete API changes per KEP
  • Add e2e tests as appropriate

External to k8s but still need to happen:

• Containerd update to use new CRI
• CRI-O update to use new CRI

Once the above CRI updates happen, ensure that CI environment is using latest container runtimes with updated CRI.