Skip to content

[iris] Remove 8 GB Docker memory cap on build containers #4625

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rjpower merged 2 commits into from
Apr 10, 2026
+3 −2
Merged

[iris] Remove 8 GB Docker memory cap on build containers #4625

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a2e2805
[iris] Remove 8 GB Docker memory cap on build containers
rjpower Apr 10, 2026
209e237
[iris] Bump Docker build memory limit from 8 GB to 32 GB
github-actions[bot] Apr 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 5 additions & 12 deletions lib/iris/src/iris/cluster/runtime/docker.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -348,19 +348,14 @@ def build(self, on_logs: Callable[[list[LogLine]], None] | None = None) -> list[
setup_script = self._generate_setup_script()
self._write_setup_script(setup_script)

# Build containers get max(8 GB, task request) memory
task_memory_bytes = self.config.resources.memory_bytes if self.config.resources else 0
build_memory_bytes = (
max(self._BUILD_MEMORY_LIMIT_BYTES, task_memory_bytes)
if task_memory_bytes
else self._BUILD_MEMORY_LIMIT_BYTES
)
build_memory_mb = build_memory_bytes // (1024 * 1024)

# Build containers run without a Docker memory limit — uv sync on a large
# workspace can exceed any fixed ceiling (confirmed OOM at 8 GB on a host
# with 1.4 TB free; CONSTRAINT_MEMCG in dmesg). The host cgroup still
# bounds usage; we just don't add an artificial container-level cap.
build_container_id = self._docker_create(
command=["bash", "/app/_setup_env.sh"],
label_suffix="_build",
memory_limit_mb=build_memory_mb,
memory_limit_mb=None,
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Skip memory fallback when removing build limits

Passing memory_limit_mb=None here does not actually remove the Docker memory cap: _docker_create() computes effective_memory_mb = memory_limit_mb or config.get_memory_mb() and still sets --memory whenever the task specifies resources.memory_bytes. That means build containers for memory-scoped tasks remain capped (and tasks below 8 GB now get a tighter limit than before), so uv sync can still OOM despite this change’s stated goal of running build containers without an artificial limit.

Useful? React with 👍 / 👎.

)

build_logs: list[LogLine] = []
Expand Down Expand Up @@ -602,8 +597,6 @@ def cleanup(self) -> None:
# Docker CLI helpers
# -------------------------------------------------------------------------

_BUILD_MEMORY_LIMIT_BYTES = 8 * 1024**3

def _docker_create(
self,
command: list[str],
Expand Down
Loading