Enforce CPU+memory limits on Actor containers and stream systemInfo events to runs - #28
Merged
Pijukatel merged 15 commits intoAug 26, 2026
Conversation
…s to Actor runs
Actor containers now get a hard CFS CPU quota derived from the run's memory
grant at the platform ratio (memoryMbytes/4096 of a core, CpuPeriod/CpuQuota
encoding, 1000us protocol floor) alongside the existing memory cap. Requests
exceeding host capacity (per docker.info) warn in the run's own log and are
applied verbatim, never clamped.
The runtime now samples each running container's real CPU/memory once a second
(one-shot docker stats, stopped before container removal) and pushes
platform-shaped systemInfo frames over a new per-run websocket at
GET /actor-runtime/events/:runId on the existing API port (ws, noServer
upgrade). The endpoint is unauthenticated by design; isolation is structural -
each socket subscribes only to its own run's channel, unknown/terminal run ids
get a 1008 close after the upgrade, normal run end closes 1000.
Run containers receive five new env vars (ACTOR_EVENTS_WEBSOCKET_URL /
APIFY_ACTOR_EVENTS_WS_URL, ACTOR_MEMORY_MBYTES / APIFY_MEMORY_MBYTES,
APIFY_DEDICATED_CPUS) so Crawlee's autoscaled pool inside JS and Python Actors
receives platform resource estimation instead of running blind.
POST /actor-runs/:runId/abort gains ?gracefully= mirroring the platform: it
emits one {"name":"aborting","data":{}} frame, waits 30s, then stops the
container; the default abort path is unchanged. persistState is never
server-sent.
Requirements docs (actor-driver.md, api.md, system.md) updated in lockstep;
unit+integration coverage for the CFS math, warn-not-clamp, sampler lifecycle,
envelope shaping, env contract, isolation, close codes, and graceful abort.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeN7rnNX67yNdUWsRMA7Sp
Review fixes on the resource-limits/events-channel work: - Every accepted events websocket now has an error listener, and the upgrade handler contains connection-setup failures to that one socket - previously a single malformed frame from any container crashed the whole runtime process. - Graceful shutdown terminates the events server's own clients before closing the HTTP server: closeAllConnections() never destroys upgraded sockets, so shutdown hung whenever an Actor was connected. The comments claiming otherwise are corrected and shutdown.test.ts now guards this for the events socket like it already did for log streams. - docker-driver's startRun try/finally now also covers container.start(), the sampler creation, and container.logs(), so a throw in that window still stops the sampler and removes the container. - abortRun derives its was-running decision inside the status transition's per-id mutex (new onBeforeTransition hook on transitionJobStatus) instead of a separate unguarded read that raced READY->RUNNING; the redundant registry read on the default abort path is gone. - gracefully=true is now exercised over a real HTTP round trip in the integration suite. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BeN7rnNX67yNdUWsRMA7Sp
…s.ts - Comments that cited ephemeral design/review documents now cite the durable requirements docs (requirements/actor-driver.md, requirements/api.md) or carry the rationale inline - most importantly the events websocket's no-authentication note, which now points at requirements/api.md instead of an artifact that never ships with the repo. - The shutdown-ordering rationale (why the events server must terminate its own clients before closeServer) now lives only in EventsWebSocketServer.close()'s doc comment; the other three sites defer to it in one line. - events-channel exports follow logs.ts's domain-prefixed convention: subscribeEvents / markEventsTerminal; the import alias in runs.ts is gone. No behavior change: comments, import lines, test titles, and renamed identifiers only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BeN7rnNX67yNdUWsRMA7Sp
…ately A repeat abort call while a gracefully-aborted run is still inside its 30-second window used to stop the container immediately, silently defeating the window: the ABORTING status alone cannot distinguish "my transition just landed" from "another caller's window is active". abortRun now captures that distinction atomically (the pre-transition status reported inside the status-transition mutex): a second gracefully=true call joins the active window and returns the current record without re-triggering anything, while a plain abort deliberately escalates to an immediate stop - the same behavior a plain double-abort always had. The first caller's eventual finalization stays a safe no-op under the existing terminal-status guard. Also covers the over-capacity warning's memory-only and CPU-only arms with dedicated unit tests and fixes a stale subscribe() comment reference. requirements/api.md documents the join/escalate contract. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BeN7rnNX67yNdUWsRMA7Sp
Comment-only cleanup: constraints formerly cited as bare criterion or iteration numbers now state the guarantee inline or cite the matching requirements/*.md section (graceful-abort join/escalate contract, the 501-vs-404 spec-table rule, the APIFY_PROXY_PASSWORD contract, the flusher-gated log-stream race). No code, identifiers, assertions, or behavior changed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BeN7rnNX67yNdUWsRMA7Sp
…omments Comment-only: the two test comments that still cited a bare criterion number now cite requirements/api.md's Graceful abort section. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BeN7rnNX67yNdUWsRMA7Sp
…dedupe Final-review fixes on the resource-limits/events work: - sampler.stop() bounds its wait for an in-flight stats() call with SAMPLER_STOP_GRACE_MS (mirroring LOG_DRAIN_GRACE_MS) - an unresponsive daemon could previously stall run finalization forever, leaving the record RUNNING with its timeout timer already cleared. - memCurrentBytes now excludes the page cache the way docker stats does (usage minus inactive file bytes, cgroup v1/v2 aware), so autoscaling sees reclaimable memory as free instead of steadily inflating usage. - The events websocket path regex rejects a trailing slash, matching the documented route exactly. - The memory/CPU ratio helpers move from services/ to src/resources.ts so the driver no longer imports the services layer. - Shared pollUntilTerminal helper replaces the near-duplicate poll loops in the log and events endpoints; the Docker stats test stub moves to test/unit/helpers/docker-stubs.ts. - requirements/api.md's abort entry names the actually-mounted route; test titles describe behavior; previously untested branches (unmatched upgrade path, warning arms, poll edges) gain coverage. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BeN7rnNX67yNdUWsRMA7Sp
…frames A Docker stats body missing memory_stats.usage produced a seven-field systemInfo frame (dropped by the Python SDK's validation) and then poisoned the memAvgBytes running average with NaN for the rest of the run; a body missing cpu_stats threw synchronously into an unhandled rejection. The sampler now checks both readings for presence and finiteness and skips the whole tick when either is unusable - no frame, accumulators and the CPU delta baseline untouched - so one bad sample costs one tick, never the run's telemetry. Also: the graceful-abort integration suites move byte-identical from job-lifecycle.test.ts (1168 -> 802 lines) into graceful-abort.test.ts, the sampler-grace paragraph in requirements/actor-driver.md now states the bounded-wait semantics the code actually implements, and a test comment states its constraint directly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BeN7rnNX67yNdUWsRMA7Sp
Two unit tests now reach the CPU snapshot guard's second clause (valid total_usage, missing or non-finite system_cpu_usage - the cgroup v2 rootless shape), asserting the tick is skipped and the next good sample recovers. Three comments corrected: two now state their constraint directly, one cross-file reference follows the graceful-abort suite to its new home, and the split rationale no longer cites a policy the repo does not have. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BeN7rnNX67yNdUWsRMA7Sp
… test comments Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BeN7rnNX67yNdUWsRMA7Sp
…ullet Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BeN7rnNX67yNdUWsRMA7Sp
…tests Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BeN7rnNX67yNdUWsRMA7Sp
Resolves four conflicts from the pnpm migration (#26) and the Dockerfile-location work (#27): - package-lock.json: accepted its deletion; ws and @types/ws are now carried by pnpm-lock.yaml, regenerated with pnpm install. - package.json: kept both sides' dependency additions (json5, ws). - docker-driver.ts and test-server.ts: kept both sides' imports; the driver-stub type import now names BuildContext and RunResourceSample. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BeN7rnNX67yNdUWsRMA7Sp
`sample_actor_resources_ts` and `sample_actor_resources_py` print the memory and CPU the runtime granted the run, then one line per systemInfo event, so the whole path this branch adds - cgroup limits, the sampler, the events websocket and the injected env vars - is observable from a single `apify call`. The two differ in where the grant comes from, which is a property of the SDKs rather than a choice: the JS SDK re-emits each frame verbatim, so a JS Actor can read `memMaxBytes` off the event, while the Python SDK maps the frame onto a usage-only structure with no total, so the Python sample reads `Actor.configuration.memory_mbytes` instead. Neither sample makes outbound requests, so unlike the crawling samples they run offline once their base image is pulled. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BeN7rnNX67yNdUWsRMA7Sp
The comments this branch added were roughly 40% of its own new lines, against 24-33% in the same files at base; they now keep the constraints that would otherwise be easy to reintroduce (the ws error listener, the close-before-shutdown ordering, the sampler's bounded stop) and drop the retelling around them. The requirements grew by 187 lines describing how and why - moby's validation rules, docker-modem's timeout handling, pydantic's field requirements. They describe what the runtime does instead, in 87. The standalone resource-reporting sample Actors are gone; the existing samples log their granted resources and each systemInfo event, which is the same demonstration without a second pair of Actors to maintain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BeN7rnNX67yNdUWsRMA7Sp
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Actor containers now start under hard cgroup limits for both memory and CPU, derived from the run's
memoryMbytes. The runtime measures each running container's real CPU/memory over the Docker API and pushes platform-shapedsystemInfoevents to the Actor over a websocket, and five new env vars tell the SDK where to dial and what its grant is.POST /actor-runs/:runId/abortgains an opt-in?gracefully=that sends anabortingevent before stopping the container. Two sample Actors, one per SDK, print the whole picture from inside a run.Why
The runtime capped memory and nothing else, and set
APIFY_IS_AT_HOME=1while injecting neither an events-websocket URL nor a memory var. The SDK inside the container therefore picked its platform event manager, found no socket, logged one debug line and emitted nothing — so Crawlee's autoscaled pool ran blind: no CPU signal at all, and a memory ceiling guessed from the host's total RAM rather than the run's grant. The Actor scaled up until the kernel OOM-killed it, which is exactly the failure a developer comes here to avoid reproducing the slow way on the platform. Meanwhile CPU was unbounded, so one runaway Actor could starve the other four the documented scale budget allows.What changed
HostConfig.CpuPeriod: 100000+CpuQuota: round(cores × 100000)at the platform's ownmemoryMbytes / 4096cores ratio, floored at the daemon's 1000µs minimum.NanoCpusis deliberately never used: moby'sverifyPlatformContainerResourcesrejectsNanoCpus > NumCPU × 1e9outright, which would turn "warn, never clamp" into "cannot run at all" for an over-capacity request.apify callshows it), then applies the requested limits verbatim. Capacity that can't be read is treated as unknown — never as zero, which would warn on every run.systemInfoframes:cpuCurrentUsageas percent of one core (the convention apify-sdk-python divides byAPIFY_DEDICATED_CPUS),memMaxBytesas the configured limit rather than an observed peak, memory excluding page cache the waydocker statsdoes, andisCpuOverloadedas a strictusedCores / grantedCores > 0.95. A stats body missing a field skips that tick whole rather than emitting a partial frame — a seven-field frame fails Python-side validation and a NaN would poison the running averages for the rest of the run.GET /actor-runtime/events/:runIdupgrades on the existing API port. It carries no authentication (a local single-operator dev tool); isolation is structural instead — the fan-out module has no broadcast set, so a socket only ever receives frames for the run id in its own path. Unknown or already-terminal run ids complete the upgrade and then close1008rather than refusing the handshake, because apify-sdk-python treats a failed first connection attempt as fatal toActor.init(). Normal run end closes1000.ACTOR_EVENTS_WEBSOCKET_URL/APIFY_ACTOR_EVENTS_WS_URLandACTOR_MEMORY_MBYTES/APIFY_MEMORY_MBYTESare each set byte-identically — the two SDKs resolve those aliases in opposite precedence order, so any divergence would size JS and Python differently from the same run.APIFY_DEDICATED_CPUShas no JS counterpart and exists so Python stops dividing by an assumed1.?gracefully=truepublishes one{"name":"aborting","data":{}}frame, waits a fixed 30s window, then stops the container. Omitted orfalseis byte-for-byte today's behavior. A second graceful abort during an active window joins it rather than cutting it short; an explicit hard abort escalates.persistStateis never server-sent — both SDKs generate it on their own timer.sample_actor_resources_tsandsample_actor_resources_pyprint the grant they were given and then one line persystemInfoevent, making limits → sampler → websocket → env vars observable from a singleapify call. They read the grant from different places, which is a property of the SDKs rather than a choice: the JS SDK re-emits each frame verbatim so a JS Actor readsmemMaxBytesoff the event, while the Python SDK maps the frame onto a usage-only structure with no total, so the Python sample readsActor.configuration.memory_mbytes. Neither makes outbound requests, so unlike the crawling samples they run offline once their base image is pulled.requirements/actor-driver.md,api.mdandsystem.mdupdated in lockstep with the contracts above.Proof it works
On the merged branch,
pnpm run build,pnpm run lint,pnpm run format:checkandpnpm testare green: 523 tests across 42 files. Of those, this branch adds 74 tests across 4 files (the base contributes the rest, 449 across 38 atcda7e9a); no existing test was weakened or removed on either side of the merge.The regression tests worth knowing about, each written against a reproduced failure:
'error'listener on accepted sockets); graceful shutdown used to hang forever while any Actor was connected, becausecloseAllConnections()does not reach upgraded sockets.sampler.stop()'s wait on an in-flight stats call is bounded, so an unresponsive daemon can't strand a run asRUNNINGwith its timeout timer already cleared; both grace-race timers are cleared once settled.> 0.95threshold on both sides,memMaxBytesconstancy, the skip-and-recover path for incomplete stats, and two-concurrent-run isolation in both directions.Both sample Actors were verified against the real SDKs rather than by inspection: the TypeScript one type-checks and builds against
apify@3.7.2, and the Python one imports cleanly againstapify@4.0.1, where feeding this runtime's exact frame through the SDK's ownto_crawlee_formatyieldscpu_info.used_ratio == 0.8and384.0MB for a 1024 MB / 0.25-core run — the numbers the design predicts.Real-daemon behavior (
docker inspectshowing the computed quota, emitted numbers trackingdocker stats) is the one thing this suite can't assert — there's no Docker socket in CI.Worth knowing before merge
Setting
ACTOR_EVENTS_WEBSOCKET_URLmakes this endpoint a hard dependency for Python Actors: a failed first connection raises out ofActor.init(). The1008-after-upgrade policy keeps rejections survivable, and the existing Python sample-Actor e2e run becomes the canary — but the blast radius is real. JS Actors have no client-side reconnect, so the server must not drop a healthy run's socket for any reason short of run end or shutdown.🤖 Generated with Claude Code
https://claude.ai/code/session_01BeN7rnNX67yNdUWsRMA7Sp