Describe the bug
Runner.Worker downloads action repository archives with HttpClient.GetAsync(downloadUrl) using the default HttpCompletionOption.ResponseContentRead, which buffers the entire response body in the managed heap before ReadAsStreamAsync() returns (see code)
The subsequent CopyToAsync into the FileStream therefore copies from a memory buffer, not from the network, so the streaming-to-disk structure of this code never actually streams.
For actions hosted in large repositories, the codeload tarball is a snapshot of the repository's entire source tree at the referenced commit, which adds up to ~73 MB compressed in our case for a monorepo hosting reusable workflows/composite actions, so peak worker memory grows by the full archive size per download. We found out the hard way when we tried to set a low limit on the .NET heap in order to optimize our CI's memory utilization and pack more runners per k8s node. On failure, the retry loop re-buffers the entire body up to 3 times back-to-back (see the "Job Log Output" section below).
Even where it doesn't OOM, this forces every runner in the world to provision memory headroom of baseline + largest action archive per concurrent job, and puts multi-MB allocations on the Large Object Heap on every uncached action download. A typical job references 2–5 actions. Most marketplace actions are small, for example actions/checkout is a ~1MB tarball, so the buffering waste there is a few MB per download, mostly invisible. But worldwide it adds up quickly, and GitHub Actions have been gaining in popularity and over 2B workflow runs/month, so even small changes add up. Say conservatively 3 uncached downloads per run at ~2MB median: that's ~12 petabytes/month of needless transient heap allocation across the fleet. Given that RAM is now a luxury, it would make sense to be more nimble here.
Related reports of runner memory footprint pressure: #747 (request to support 512MB machines), #3796 (listener memory usage) — this is one of the concrete reasons the worker's peak memory is much larger than expected.
To Reproduce
Steps to reproduce the behavior:
- Run a self-hosted runner in a container with a constrained heap, e.g.
DOTNET_GCHeapHardLimit=0x4000000 (64MiB) or a ~256Mi cgroup memory limit.
- Run a workflow with a
uses: step referencing an action that lives in a large repository (tarball >> heap budget).
- The "Download action repository" phase fails with
System.OutOfMemoryException (or, without a heap cap, worker RSS spikes by the archive size).
Expected behavior
The archive is streamed from the network to disk with a fixed-size buffer, keeping peak memory independent of archive size.
Runner Version and Platform
Version of your runner?
2.336.0 (also present at HEAD, 34ef7f2) on Linux, containerized self-hosted runner (ghcr.io/actions/actions-runner base) on Kubernetes.
What's not working?
.NET runtime OOMs during the download phase with small heaps.
Job Log Output
Warning: Failed to download action 'https://codeload.github.com/<org>/<repo>/tar.gz/<sha>'. Error: Exception of type 'System.OutOfMemoryException' was thrown.
Warning: Back off 17.072 seconds before retry.
Warning: Failed to download action 'https://codeload.github.com/<org>/<repo>/tar.gz/<sha>'. Error: Exception of type 'System.OutOfMemoryException' was thrown.
Warning: Back off 20.037 seconds before retry.
Error: Exception of type 'System.OutOfMemoryException' was thrown.
Error: Failed to download archive 'https://codeload.github.com/<org>/<repo>/tar.gz/<sha>' after 3 attempts.
Runner and Worker's Diagnostic Logs
N/A
Describe the bug
Runner.Workerdownloads action repository archives withHttpClient.GetAsync(downloadUrl)using the defaultHttpCompletionOption.ResponseContentRead, which buffers the entire response body in the managed heap beforeReadAsStreamAsync()returns (see code)The subsequent
CopyToAsyncinto theFileStreamtherefore copies from a memory buffer, not from the network, so the streaming-to-disk structure of this code never actually streams.For actions hosted in large repositories, the codeload tarball is a snapshot of the repository's entire source tree at the referenced commit, which adds up to ~73 MB compressed in our case for a monorepo hosting reusable workflows/composite actions, so peak worker memory grows by the full archive size per download. We found out the hard way when we tried to set a low limit on the .NET heap in order to optimize our CI's memory utilization and pack more runners per k8s node. On failure, the retry loop re-buffers the entire body up to 3 times back-to-back (see the "Job Log Output" section below).
Even where it doesn't OOM, this forces every runner in the world to provision memory headroom of baseline + largest action archive per concurrent job, and puts multi-MB allocations on the Large Object Heap on every uncached action download. A typical job references 2–5 actions. Most marketplace actions are small, for example
actions/checkoutis a ~1MB tarball, so the buffering waste there is a few MB per download, mostly invisible. But worldwide it adds up quickly, and GitHub Actions have been gaining in popularity and over 2B workflow runs/month, so even small changes add up. Say conservatively 3 uncached downloads per run at ~2MB median: that's ~12 petabytes/month of needless transient heap allocation across the fleet. Given that RAM is now a luxury, it would make sense to be more nimble here.Related reports of runner memory footprint pressure: #747 (request to support 512MB machines), #3796 (listener memory usage) — this is one of the concrete reasons the worker's peak memory is much larger than expected.
To Reproduce
Steps to reproduce the behavior:
DOTNET_GCHeapHardLimit=0x4000000(64MiB) or a ~256Mi cgroup memory limit.uses:step referencing an action that lives in a large repository (tarball >> heap budget).System.OutOfMemoryException(or, without a heap cap, worker RSS spikes by the archive size).Expected behavior
The archive is streamed from the network to disk with a fixed-size buffer, keeping peak memory independent of archive size.
Runner Version and Platform
Version of your runner?
2.336.0 (also present at HEAD, 34ef7f2) on Linux, containerized self-hosted runner (ghcr.io/actions/actions-runner base) on Kubernetes.
What's not working?
.NET runtime OOMs during the download phase with small heaps.
Job Log Output
Runner and Worker's Diagnostic Logs
N/A