Implement local Apify actor runtime (POCv2) - #11
Merged
Pijukatel merged 19 commits intoAug 18, 2026
Conversation
- Add the TypeScript runtime: Apify-compatible REST API over Crawlee v4 storage frontends, docker-based actor build/run driver, server-rendered console, and TS + Python sample actors with an input-driven e2e suite. - Update requirements/*.md to match the delivered behavior, including the documented known differences from the real platform (no request-queue locking, best-effort request listing, unsupported request deletion). - Verified by 196 unit/integration tests through a real apify-client, plus live checks with stock apify-cli 1.8.0; e2e is docker-gated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019FP1P3RXwZSnTxy634oWCE
This was referenced Aug 15, 2026
- One job runs build, lint, format check, and the unit + integration suites; a second job runs the mandatory CLI-only e2e dev-loop suite against the runner's Docker daemon, per requirements/test.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019FP1P3RXwZSnTxy634oWCE
- Run responses now carry options.build, options.diskMbytes, generalAccess, and storage responses carry consoleUrl; build numbers start at .1 — validated against the Apify Python client's pydantic models, which reject the previous shapes. - The driver waits for the container's demuxed log streams to end (not just container.wait) and logs are flushed before the terminal status is written, so a log fetched right after a run finishes is never empty. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019FP1P3RXwZSnTxy634oWCE
- Four integration tests drive the real apify-client log stream against the server across terminal-transition timing sweeps (the CI exit-13 scenario could not be reproduced locally; these pin the settled-stream contract on every path). - On e2e failure, CI now dumps the runtime container's logs so the server-side view of a failed request is visible in the job output. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019FP1P3RXwZSnTxy634oWCE
- docker-modem's demuxStream never ends the destination streams, so awaiting their end held every real run open until the 300s timeout. Log drain is now derived from the source stream's end (destinations ended manually), raced against container exit plus a 5s grace. - Test stubs now model demuxStream faithfully (data-forwarding only); a regression test finalizes a run in under a second against that stub. - The e2e dumps the runtime container's logs before removing it, so CI failures show the server-side view (afterAll ran before the workflow's dump step, which always found the container already gone). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019FP1P3RXwZSnTxy634oWCE
apify-cli writes log content to stderr (stdout carries machine-readable payloads), so the log-content assertion read an empty stdout while the full log was present on stderr. The runtime side was already correct. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019FP1P3RXwZSnTxy634oWCE
Actor SDK loggers emit ANSI SGR sequences; the console showed them as literal [32m... garbage. Log views now convert SGR codes to colored HTML spans (unsupported control sequences stripped, text HTML-escaped); the /v2/logs API keeps serving raw bytes so CLI terminal colors work. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019FP1P3RXwZSnTxy634oWCE
Both samples drop the in-container synthetic server and crawl the live web through the Actor's request queue: startUrl input (default https://crawlee.dev/) plus maxPages bounding the crawl, one {url,title} dataset item per page. The e2e therefore needs outbound network from Actor containers, documented in requirements/test.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019FP1P3RXwZSnTxy634oWCE
… step README and system.md now show docker build + docker run with a ./data bind mount as the canonical start command; docker-compose.yml is removed (with its validity test and the js-yaml devDependencies). The dev loop assumes an already-logged-in apify-cli — the runtime maps any non-empty token to its single local user, per requirements/cli.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019FP1P3RXwZSnTxy634oWCE
With concurrency above 1 the autoscaled pool starts extra requests before the max-requests stop lands and lets in-flight requests finish, overshooting maxPages (the Python sample produced 8 items for maxPages=2 in CI). Both samples now crawl with concurrency 1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019FP1P3RXwZSnTxy634oWCE
Stock apify-cli prints links using console.apify.com's path shapes
(/actors/:id/runs/:runId, /actors/:id/builds/:buildNumber,
/storage/{datasets,key-value-stores,request-queues}/:id); those now 302
to this console's own pages, and storage consoleUrl fields use the
platform's /storage/... shape.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019FP1P3RXwZSnTxy634oWCE
A background write landing mid-removal recreates entries under an already-emptied directory (CI-only ENOTEMPTY flake); fs.rm's built-in retry absorbs it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019FP1P3RXwZSnTxy634oWCE
The tight-loop test scheduled a timer that could fire before the stub driver's resolver was assigned on slow runners; it now synchronizes on the driver's own started signal and clears timers per iteration. Server lifecycles are wrapped in finally so one failure can't poison the process-wide storage singleton for later tests, and the log-drain test polls for RUNNING instead of asserting it at a fixed instant. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019FP1P3RXwZSnTxy634oWCE
…apify api covered - The e2e suite now fails loudly when no Docker daemon is reachable instead of skipping, per requirements/test.md. - apify login is no longer invoked: the suite seeds the CLI's on-disk credential store directly (APIFY_TOKEN env alone only authenticates actor:*/mcp commands in apify-cli 1.8.0, verified in its source). - Added an e2e case driving apify api against the run and its dataset; verified live against the runtime without Docker first. - Prettier pass over the requirements files edited upstream. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019FP1P3RXwZSnTxy634oWCE
On first authenticated contact the runtime resolves the CLI's token against api.apify.com/v2/users/me (3s timeout, cached per token) and adopts the real id, username, and proxy password for its default user; offline or with an unresolvable token it keeps the local identity. users/me now emits proxy.password only when a real value is known (runtime env wins over the harvested one) — the hardcoded placeholder that could overwrite a logged-in CLI's stored proxy password is gone. The e2e authenticates with a genuine apify login inside an isolated HOME instead of writing the credential file by hand. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019FP1P3RXwZSnTxy634oWCE
Any previously-unseen token now mints its own user on first use: identity
is harvested from the real Apify API when the token resolves there
(username, id, proxy password), and fabricated (local-user-{n}) otherwise.
The realId/realUsername overlay is gone - a user record is just what the
API serves. GET /users/:userId returns the full self view only for the
caller's own id and a minimal {id, username} for anyone else; actors,
builds, runs, and storages are owned by and filtered to their creating
user, while the console lists resources across all users with their owner.
No user is created at startup any more, and requirements files are aligned
(plus formatting fixes that were failing CI).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019FP1P3RXwZSnTxy634oWCE
…sole The run detail page (and the runs list's dataset column) rendered defaultDatasetId/defaultKeyValueStoreId/defaultRequestQueueId as plain text; they now link to /datasets/:id, /key-value-stores/:id, and /request-queues/:id respectively. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019FP1P3RXwZSnTxy634oWCE
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The POCv2 local actor runtime described by
requirements/*.md: a TypeScript service exposing an Apify-compatible REST API (port 3333) backed by Crawlee v4 storage frontends (@crawlee/core+@crawlee/fs-storage, exact-pinned4.0.0-beta.133), a dockerode-based actor build/run driver, a server-rendered view-only console (port 3000), TS + Python sample actors that take amaxPagesinput, and a CLI-only e2e suite. Driven by stockapify-cli1.8.0 viaAPIFY_CLIENT_BASE_URL.Why
The branch previously carried only the requirements draft and a sample actor stub — no implementation of the local
apify push→ build → run → inspect loop.Notes
storage.mdgains a "Known differences from the Apify platform" section (no request-queue locking, best-effort request listing, request deletion answers 501), per the agreed simplifications.apify-client, plus live checks with stockapify-cli(login, info, push mechanics, runs ls, datasets info). The e2e suite requires a Docker daemon and skips cleanly without one, so image build/run paths ran against an injectable driver rather than real containers in CI-less verification.Dataset/KeyValueStore/RequestQueuefrontends with the service locator configured withpurgeOnStart: false; the only backend-level call isteardown()at shutdown.Follow-ups
@types/js-yamldevDependency is redundant withjs-yaml@5's bundled types.__FILES__internal registry is currently unused (kept as designed; natural home for the TARBALL source-upload follow-up).apify actors start/abort/resurrect polish, DinD mode, read-only debug mount, Python sample lockfile parity.🤖 Generated with Claude Code
https://claude.ai/code/session_019FP1P3RXwZSnTxy634oWCE
Generated by Claude Code