Skip to content

README.md

Latest commit

 

History

History
259 lines (209 loc) · 13.6 KB

README.md

File metadata and controls

259 lines (209 loc) · 13.6 KB
Raw

actor-runtime

A minimal, self-contained "local Apify platform" in a single Docker image. Start it with one docker run, point the stock apify-cli at it, and run the full Actor dev loop: apify push -> build -> run -> inspect runs, builds and the run's default storages (key-value store, dataset, request queue). The runtime itself needs no outbound network access after the first build/push (see requirements/system.md); the bundled sample Actors crawl the live web, so running them does.

See requirements/*.md for the full behavioural spec (system.md, api.md, storage.md, actor-driver.md, cli.md, console.md, test.md).

Quick start

docker build -t actor-runtime .
mkdir -p data
docker run --rm -p 3333:3333 -p 3000:3000 \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v "$(pwd)/data:/data" \
  actor-runtime

This mounts ./data on the host as the runtime's /data, so every storage, build and run record lands under ./data for easy inspection.

export APIFY_CLIENT_BASE_URL=http://localhost:3333
export APIFY_CONSOLE_URL=http://localhost:3000
npm install -g apify-cli

cd sample_actor_ts
apify push
apify call --input '{"maxPages":3}'

This assumes you're already logged in (apify login, any stored token works - the runtime maps any non-empty token to its single local user). If that token happens to be a real Apify account token and the real platform is reachable, the runtime also adopts that account's real username/id/proxy password the first time it sees the token; fully offline (or with any other non-empty token) it just keeps using the single local user, with no error either way - see requirements/cli.md's User bootstrap section.

Running with Podman instead of Docker

The runtime talks to the container engine only through its Docker-compatible API socket, and Podman serves that same API. Everything works the same on Docker and Podman, rootful or rootless; the only difference is which socket you mount.

sudo systemctl enable --now podman.socket   # one-time: serve Podman's API socket

podman build -t actor-runtime .
mkdir -p data
sudo podman run --rm -p 3333:3333 -p 3000:3000 \
  -v /run/podman/podman.sock:/var/run/docker.sock \
  -v "$(pwd)/data:/data" \
  actor-runtime

Rootless Podman serves the socket at $XDG_RUNTIME_DIR/podman/podman.sock instead (systemctl --user enable --now podman.socket); mount that path and drop the sudo. Rootless Docker works the same way with its $XDG_RUNTIME_DIR/docker.sock. The socket can also be mounted at any other path together with -e DOCKER_HOST=unix:///that/path.

mkdir -p data
podman run --rm -p 3333:3333 -p 3000:3000 \
  -v "$XDG_RUNTIME_DIR/podman/podman.sock:/var/run/docker.sock" \
  -v "$(pwd)/data:/data" \
  actor-runtime

Good to know:

  • Podman 3.4 (Ubuntu 22.04's stock package) and newer work. On Podman 4 and newer, Actors run on the runtime's own apify-local network; on Podman 3.x they run on the engine's default network instead (its user-defined networks are unreliable: Ubuntu 22.04's CNI plugins reject the config Podman writes), and the runtime says so at startup. Whenever the runtime's own container is not on apify-local (Podman 3.x, or rootless Podman, which refuses to attach it), Actors reach the API through the published port 3333, so keep -p 3333:3333 published on all interfaces. Optionally, on Podman 4 and newer, create the network first and add --network apify-local to podman run for the direct route.
  • Podman does not create a missing host directory for a bind mount (Docker does), hence the mkdir -p data before podman run. apify runtime start creates its data directory itself.
  • Actors run on the engine whose socket you mount, so a dev folder registered for the bind-mount dev loop below is a path on the machine that engine runs on (inside the VM for podman machine), and under a rootless engine it must be readable by that user.
  • A short image name in an Actor's FROM line (apify/actor-node:20, python:3.11) means Docker Hub, as on the platform. The runtime qualifies it to docker.io/... before building, so Podman resolves it without any unqualified-search-registries entry in registries.conf. The build log shows the substitution.
  • A rootless engine can only enforce the per-run limits whose cgroup controllers are delegated to your user: on cgroups v1 none are, and Ubuntu 22.04 delegates memory and pids but not cpu. The runtime asks Podman which controllers it has, leaves out the limits it cannot apply, and says so at startup; runs still start. (To get CPU limits under rootless Podman on Ubuntu 22.04, delegate the controller: sudo mkdir -p /etc/systemd/system/user@.service.d && printf '[Service]\nDelegate=cpu cpuset io memory pids\n' | sudo tee /etc/systemd/system/user@.service.d/delegate.conf && sudo systemctl daemon-reload, then log out and in.)
  • If you restart a hand-started podman system service, the socket file mounted into the runtime goes stale; restart the runtime container too. The podman.socket unit does not have this problem.
  • podman images lists the images the runtime builds as actor-runtime/<actor>:<buildId> under the registry prefix Podman adds itself (docker.io/ or localhost/, depending on the version).

Rapid dev loop: bind-mounting your local source (no rebuild per edit)

After the one push+build above, register your Actor's local source folder so every future run picks up local edits without a rebuild:

apify api POST /actor-runtime/dev-folder/<actorId> --body '"/abs/path/to/sample_actor_ts"'

<actorId> is the id apify push --json printed (.actor.id); the path must be absolute and must already exist on the host - the runtime checks this and rejects the call with a clear error if the path can't be confirmed. The check runs again at every run start, so a folder deleted after registration fails the run instead of running against an empty directory. The same thing is also a single-field form on the Actor's page in the console (http://localhost:3000).

From then on:

# edit src/main.ts, then:
npm run build        # recompile locally - tsc, no apify push
apify call --input '{"maxPages":3}'   # picks up the new dist/, no rebuild

Node doesn't hot-reload a running process, so a local recompile is picked up by the next run's container start, not by any run already in progress. node_modules inside the container still comes from the built image - a per-run volume preserves it underneath the bind mount - so a new dependency in package.json still needs a real apify push/build; only source edits skip it. An entrypoint script the image keeps in its working directory (Apify's Playwright images start through an Xvfb wrapper there) stays available too, unless your folder carries its own copy. Clear the registration with an empty body (--body '""') to go back to running purely from the built image. Full mechanics: requirements/actor-driver.md's "Bind mount volumes with Actor source code"; endpoint/console details: requirements/api.md's /actor-runtime/* section and requirements/console.md.

Debugging a run with a real IDE (breakpoints, step-through)

Turn debug mode on for an Actor once, then every apify call against it starts paused, waiting for a debugger to attach - no change to the Actor's own source, Dockerfile, or requirements.txt:

apify api POST /actor-runtime/debug/<actorId> --body '{"enabled": true}'
apify call

The run's log prints one line with everything you need: the resolved language, the debug tool, the listen/publish address, and the attach action for the relevant IDE - PyCharm's Attach to DAP or VS Code's Python: Remote Attach for Python (default port 5678), VS Code's Attach for Node (default port 9229). Connect, and execution proceeds to your own first breakpoint - the runtime never sets one of its own.

Override the language (for an image the auto-detection can't classify) and/or the port:

apify api POST /actor-runtime/debug/<actorId> --body '{"enabled": true, "language": "node", "port": 9230}'

A POST fully replaces the prior toggle state - omitting language/port resets them to their own defaults, it does not keep whatever a previous call set. Clear the toggle to go back to running normally:

apify api POST /actor-runtime/debug/<actorId> --body '{"enabled": false}'

The run's own timeoutSecs (apify call --timeout) is not extended while paused - a session that runs long still needs a larger --timeout passed up front. An image whose CMD can't be debugged this way (e.g. npm start - it would attach to npm, not your Actor) fails the run immediately, before any container is created, with a message naming the fix. This includes any Node Actor pushed without its own Dockerfile: this runtime's injected default Dockerfile inherits its base image's own CMD ["npm", "start", "--silent"], so it's refused the same way. The fix: give the Actor a Dockerfile whose CMD invokes node directly, e.g. CMD ["node", "dist/main.js"]. The same toggle is also a three-field form (enabled/language/port) on the Actor's page in the console. Full mechanics: requirements/actor-driver.md's "Debug mode" section; endpoint/console details: requirements/api.md's /actor-runtime/* section and requirements/console.md.

Watching an Actor's browser

Turn browser view on for an Actor once, and every run of it gets a live view of the display its browser draws on, served by the console:

apify api POST /actor-runtime/browser-view/<actorId> --body '{"enabled": true}'
apify call

The run log prints the viewer URL (http://localhost:3000/runs/<runId>/browser); the run's console page links to it, and the Actor's console page has the same toggle as a form. "interactive": true also sends your mouse and keyboard to the display; {"enabled": false} turns the view off.

The view only reads the display's pixels. The Actor's container, command, environment, network and ports are those of an ordinary run, so neither the browser nor the sites it visits can tell whether anyone is watching. Two things follow: the browser must run headful (Apify's templates default to headless, which shows as a black display - the bundled sample_actor_playwright and sample_actor_playwright_py set headless: false / headless=False), and the image must provide an X display, which the Apify Playwright and Puppeteer base images do. Like Python debug mode, this needs the runtime to run from its own built image.

Publishing the image

Images go to apify/actor-runtime on Docker Hub by default; the target repository is a workflow input, so a one-off build can be pushed elsewhere.

The Release Docker image workflow (.github/workflows/release.yml) is manual only: Actions -> Release Docker image -> Run workflow, pick the branch in Use workflow from, and run it. That is the only branch to choose - the workflow always builds the branch it was dispatched from. Everything else is optional: an extra tag such as v0.1.0, whether to also move :latest, and which platforms to build.

It pushes one multi-arch manifest per tag - linux/amd64 and linux/arm64 by default - so the same tag serves x86_64 and Apple Silicon. Every run publishes <branch>-<short-sha> (immutable) and <branch> (moving), with / in a branch name slugified to -. It pushes as the Apify service account, using the same two repository secrets as apify-actor-docker: APIFY_SERVICE_ACCOUNT_DOCKERHUB_USERNAME and APIFY_SERVICE_ACCOUNT_DOCKERHUB_TOKEN. They are synced into this repository's Actions secrets from Doppler, so they are managed there rather than added by hand.

Development

pnpm install
pnpm run build     # tsc
pnpm test          # unit + integration (no Docker needed)
pnpm run test:e2e  # full CLI-driven dev loop against a built image (requires Docker, or Podman with CONTAINER_CLI=podman; the browser-view case pulls the ~2 GB Playwright base image)
pnpm run dev       # run the server directly against ./data with tsx

pnpm run dev sets ACTOR_RUNTIME_DATA_DIR=./data inline in the script (DEFAULT_DATA_DIR otherwise falls back to the container path /data - see src/config.ts); this only works as written on a POSIX shell (Linux/macOS). On Windows, set the env var separately before running tsx src/index.ts (e.g. in PowerShell: $env:ACTOR_RUNTIME_DATA_DIR="./data"; tsx src/index.ts), or use a cross-platform env-setter like cross-env if you add it as a dependency.

Bumping the pinned Crawlee v4 version

@crawlee/core and @crawlee/fs-storage are pinned to the exact version the npm v4 dist-tag resolves to (both must move in lockstep - @crawlee/fs-storage pins its own native addon, @crawlee/fs-storage-native). To bump:

pnpm view @crawlee/core dist-tags.v4
pnpm view @crawlee/fs-storage dist-tags.v4   # should match
# update both versions in package.json, then:
pnpm install
pnpm run build && pnpm test

While bumping, check whether the pnpm.overrides pin on @crawlee/fs-storage-native in package.json is still needed: it forces the first release with linux-arm64 bindings (0.1.5-beta.19, API-identical to the 0.1.5-beta.18 that released @crawlee/fs-storage versions still depend on). Once the bumped @crawlee/fs-storage depends on >= 0.1.5-beta.19 on its own, delete the override.

Apify Proxy

Set APIFY_PROXY_PASSWORD in the runtime container's own environment (e.g. docker run -e APIFY_PROXY_PASSWORD=your-password ...) to have it forwarded, unscoped, into every Actor container's APIFY_PROXY_PASSWORD. Leave it unset and the variable is simply absent from every Actor container - never a placeholder value.