Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 17 additions & 6 deletions CLAUDE.MD
Original file line number Diff line number Diff line change
@@ -1,25 +1,36 @@
# Using local Actor runtime

This file provides guidance to programming agents when using the local Actor runtime.
Local Actor runtime is an Actor development tool for developing, running, and debugging Actors. It emulates subset of the real Apify API and Apify console to allow actor development locally without the need to do costly rebuilds on Apify platform.

# Set up

- Build the docker image `docker build -t actor-runtime .`
- Run the container `docker run --rm -p 3333:3333 -p 3000:3000 -v /var/run/docker.sock:/var/run/docker.sock -v "$(pwd)/data:/data" actor-runtime`
- `-v "$(pwd)/data:/data"` shared volumes `data` is used to store internal actor runtime data. When exposed it can be directly inspected to determine internal state and storage backend (It is not recommended to manually edit those files. Any edit should be done through http API call).
- `-v "$(pwd)/data:/data"` shared volumes `data` is used to store internal actor runtime data. When exposed it can be directly inspected to determine internal state and storage backend (It is not recommended to manually edit those files. Any edit should be done through http API call).

## Work through CLI

- The actor runtime is best used through `apify cli`: https://docs.apify.com/cli/docs
- Use the cli according to the skill: https://docs.apify.com/cli/docs/agent-skill#install-the-skill
- To use local Actor runtime set environment variable for the Apify CLI:
- `APIFY_CLIENT_BASE_URL=http://localhost:3333`
- `APIFY_CONSOLE_URL=http://localhost:3000`
- `APIFY_PROXY_PASSWORD` (optional if Apify proxy is desired)
- `APIFY_CLIENT_BASE_URL=http://localhost:3333`
- `APIFY_CONSOLE_URL=http://localhost:3000`
- `APIFY_PROXY_PASSWORD` (optional if Apify proxy is desired)
- To redirect Apify CLI back to the original Apify services unset the environment variables:
- `APIFY_CLIENT_BASE_URL`
- `APIFY_CONSOLE_URL`
- `APIFY_CLIENT_BASE_URL`
- `APIFY_CONSOLE_URL`
- Use `apify cli api ...` to send API calls and inspect the Actors, builds, runs, storages and other objects.
- To simulate multiple users use custom token and in additional authorization header. For example: `apify cli api v2/datasets -H '{"authorization": "Bearer TOKEN"}'`
- You can use already authenticated CLI or call `apify login --token TOKEN`
- To iterate on an Actor's source without a rebuild for every change: push and build the Actor once, then
register its local source folder with
`apify api POST /actor-runtime/dev-folder/<actorId> --body '"/abs/path/to/src"'` (this runtime's own
`/actor-runtime/*` endpoint - also reachable at `/v2/actor-runtime/*`, purely because `apify api`
hardcodes a `/v2`-suffixed base URL). From then on, edit locally, recompile locally (`tsc` or the
language-appropriate equivalent), and `apify call` again - no `apify push`/build in between. Submitting
`--body '""'` clears the registration. Dependency changes still need a real rebuild.

## Through direct API calls

- You can also send direct API calls. For example: http://localhost:3333/v2/datasets?token=TOKEN
33 changes: 33 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,39 @@ the real platform is reachable, the runtime also adopts that account's real user
the first time it sees the token; fully offline (or with any other non-empty token) it just keeps using
the single local user, with no error either way - see `requirements/cli.md`'s User bootstrap section.

## Rapid dev loop: bind-mounting your local source (no rebuild per edit)

After the one push+build above, register your Actor's local source folder so every future run picks up
local edits without a rebuild:

```bash
apify api POST /actor-runtime/dev-folder/<actorId> --body '"/abs/path/to/sample_actor_ts"'
```

`<actorId>` is the id `apify push --json` printed (`.actor.id`); the path must be absolute and must
already exist on the **host** - the runtime verifies this by actually trying to mount it, and rejects
the call with a clear error if the Actor has no build tagged `latest` yet (a stock `apify push` always
tags its build `latest`, so this is normally just "build at least once first") or the path can't be
confirmed.
The same thing is also a single-field form on the Actor's page in the console (`http://localhost:3000`).

From then on:

```bash
# edit src/main.ts, then:
npm run build # recompile locally - tsc, no apify push
apify call --input '{"maxPages":3}' # picks up the new dist/, no rebuild
```

Node doesn't hot-reload a running process, so a local recompile is picked up by the **next** run's
container start, not by any run already in progress. `node_modules` inside the container still comes
from the built image - an anonymous volume preserves it underneath the bind mount - so a new dependency
in `package.json` still needs a real `apify push`/build; only source edits skip it. Clear the
registration with an empty body (`--body '""'`) to go back to running purely from the built image. Full
mechanics: `requirements/actor-driver.md`'s "Bind mount volumes with Actor source code";
endpoint/console details: `requirements/api.md`'s `/actor-runtime/*` section and
`requirements/console.md`.

## Development

```bash
Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
"test": "vitest run test/unit test/integration",
"test:unit": "vitest run test/unit",
"test:integration": "vitest run test/integration",
"test:e2e": "vitest run test/e2e",
"test:e2e": "vitest run test/e2e --no-file-parallelism",
"test:watch": "vitest"
},
"dependencies": {
Expand Down
43 changes: 39 additions & 4 deletions requirements/actor-driver.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,10 +39,45 @@
- Actor details are saved in `__ACTORS__` internal storage

# Bind mount volumes with Actor source code
- To enable rapid development of Actors, it is desired to avoid the need to rebuild the Actors. This can be achieved by bind mounting the actor local development folder when starting the container.
- When building an Actor, get the location where the Actor source code is locally located and save it in `__ACTORS__` under `localDevFolder`
- Detect the working directory of docker image `docker inspect -f '{{.Config.WorkingDir}}' {DOCKER_IMAGE}` (replace {DOCKER_IMAGE} by the docker image identifier) and save it to `__ACTORS__` under `imageWorkingDirectory`
- Each Actor stored by the local Actor runtime is started with bind mounted development folder using these additional arguments `-v {localDevFolder}:{imageWorkingDirectory} -v {imageWorkingDirectory}/node_modules`

- To enable rapid development of Actors, it is desired to avoid the need to rebuild the Actors for
every source change. This is achieved by bind mounting the Actor's local development folder over the
built image's working directory when starting the container - edit locally, recompile locally
(`tsc`, or the language-appropriate equivalent), `apify call` again, with no `apify push`/build in
between. A running container never picks up a recompile; only the next run's container start does.
Dependency or environment changes still require a real rebuild.
- `localDevFolder` is **registered explicitly** on a new local-only endpoint,
`POST /actor-runtime/dev-folder/:actorId` (see `api.md`), also exposed as a single-field form on the
console's Actor detail view (`console.md`), sets or clears it. Both surfaces funnel through one
shared validate-and-persist path, so they can never disagree.
- **Registration validates the path in two layers**, not shape alone:
1. A cheap shape check: the submitted value must be an absolute POSIX path.
2. A **host-side existence-and-directory check**. The runtime's own filesystem cannot be trusted to
judge a host path - it is not necessarily the host's filesystem at all - so this must be verified
some other way.
- Submitting the **empty string clears the registration** and never runs either validation layer.
- Every non-success outcome is classified rather than guessed: being unable to verify the path at
all (e.g. Docker is unreachable) is reported as "could not verify", never as "does not exist"; a
path confirmed missing is reported as "path does not exist"; a path that exists but is a file is
reported as "path is not a directory"; anything else unverifiable is a generic "could not verify".
- **Registration has no build-first precondition.** It requires no build of the Actor's own to exist,
succeeded or otherwise - the host-side check needs only something host-present to validate against,
never a build a run would actually use.
- **`imageWorkingDirectory` is captured by the driver itself, right after a successful build, and is
build-specific, not Actor-specific** - it is persisted on that build's own record (see `storage.md`),
never on the Actor. The mount a run applies always reads it off _that run's own resolved build_, never
off any other build the Actor happens to have.
- **The mount is applied only when both a registered dev folder and a known working directory exist**
for the run's resolved build; either missing means the run starts exactly as if the feature did not
exist.
- The registration status the console and API report is the registered folder alone - it never claims a
mount "will apply", since that depends on which build a given run resolves, which an Actor-level status
has no way to know in advance.
- If the registered folder has since been deleted, moved, or made unreadable, the run must **fail
visibly** - never silently mount an empty directory in its place.
- The Actor image's own installed dependencies (e.g. `node_modules`) must remain available to the Actor
despite the mount covering the whole working directory.
- **Registering or clearing a dev folder never bumps the Actor's `modifiedAt`.**

# Networking

Expand Down
26 changes: 23 additions & 3 deletions requirements/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,11 +124,31 @@
implement. Not built here; both paths are in the vendored spec table (`api/spec-table.ts`) as
`implemented: false` so they answer `501`, not `404`.
- All endpoints from the specification that do not have implementation must return response `501 Not Implemented`
- All endpoints not present in specification must return `404 Not Found`
- All endpoints not present in specification must return `404 Not Found` - **except** the `/actor-runtime/*`

# Private API
# Actor runtime API

- Not implemented
- `/actor-runtime/*` is API that control specifics function of the local Actor runtime
- **`POST /actor-runtime/dev-folder/:actorId`** - registers (or clears) the Actor's local dev folder for
the bind-mount feature (`actor-driver.md`). `:actorId` accepts the same forms as the rest of the API
(id, plain name, `username~name`).
- **Authenticated** the same way as every `/v2` route, and scoped to the caller's own Actors.
- **No build-first precondition** - registration works for an Actor that has never been built at all.
- **Request body**: a JSON string - the absolute path to set, or `""` to clear.
- **Response**: on success, `{ data: { localDevFolder } }` - the same value the console detail page
shows (`console.md`), doubling as the read-back this design has no separate `GET` for.
- **Error responses**, by rejection reason:
- `400` `invalid-request` - the body isn't a JSON string, or the string isn't a valid absolute
path.
- `400` `dev-folder-path-not-found` - the path does not exist on the host.
- `400` `dev-folder-not-a-directory` - the path exists but is not a directory.
- `400` `dev-folder-check-failed` - the path could not be verified, for any other reason.
- `503` `dev-folder-check-unavailable` - Docker itself is unreachable.
- `500` `internal-error` - an operational fault unrelated to the submitted path.
- The console's own dev-folder form (`console.md`) does **not** go through this endpoint - it posts to a
console-local, unauthenticated route on the console's own port. Both routes funnel into the same
underlying validate-and-persist path, so the two surfaces can never drift apart in behavior, only in
how they are reached.

## Upstream fallback (opt-in, off by default, all HTTP methods)

Expand Down
23 changes: 19 additions & 4 deletions requirements/console.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,16 @@
# Frontend

- Console frontend is simple view-only page that allows to inspect each user object.
- Console frontend is a page that allows inspecting each user's objects across the whole runtime.
- Server-rendered HTML from the same process that serves the API, on its own fixed port (3000). No
SPA, no bundler, no build step - plain Express routes returning HTML strings. It reads through the
same service layer as the API handlers, so storage/build/run access logic is shared rather than
reimplemented.
- Frontend shows for each object the owner (`userId`).
- The console has no login of its own (it is unauthenticated and view-only), so with multiple users it
lists and shows every user's objects rather than scoping to one - the API's own endpoints stay
strictly scoped to the calling token's user (`storage.md`'s "Users" section).
- The console has no login of its own, so with multiple users it lists and shows every user's objects
rather than scoping to one - the API's own endpoints stay strictly scoped to the calling token's user
(`storage.md`'s "Users" section).
- The console is unauthenticated. Every route is a read except the dev-folder form below, which is the
console's one write - it is no longer strictly view-only.
- There are three types of objects: key-value store, dataset, request queue.
- For each object type there must be exactly one widget for inspection.
- The request-queue widget leads with the authoritative counts from `RequestQueue.getInfo()`
Expand All @@ -34,3 +36,16 @@
list's dataset column), not as plain text.
- Log views render ANSI colors from actor output as HTML, while the `/v2/logs/:id` API keeps serving logs raw (unconverted) for the CLI to render itself.
- The console accepts the real Apify Console's URL shapes (as printed by stock apify-cli, e.g. `/actors/:actorId/runs/:runId`, `/storage/datasets/:id`) via redirects to its own pages.

## Local dev-folder registration form (Actor detail view)

- The Actor detail view shows the Actor's registered local dev folder, or that none is registered - the
same status the API endpoint reports (`api.md`). It never claims a mount "will apply": that depends on
which build a given run resolves, which this Actor-level view has no way to know in advance.
- A single-field form exposes the same registration capability as the API endpoint, with no build-first
precondition either: submitting it sets or clears the dev folder, funnelling into the same
validate-and-persist path, so the two surfaces can never observe or produce different outcomes for the
same input. Submitting an empty value clears the registration, matching the API; a whitespace-only
value is rejected as a malformed path, also matching the API.
- A submission that fails validation redirects back to the same detail page with the classified error
message shown inline, never swallowed by the redirect.
21 changes: 19 additions & 2 deletions requirements/storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,8 +71,21 @@
- `value` is the metadata of the Actor
- owner (`userId`)
- metadata
- localDevFolder
- imageWorkingDirectory
- `localDevFolder` - **optional**. Absent means no dev folder has ever been registered for this
Actor. Set or cleared only through `POST /actor-runtime/dev-folder/:actorId` or the console's
equivalent form (`api.md`, `console.md`), never as a side effect of any other Actor write, and
never bumping `modifiedAt` (`actor-driver.md`). Submitting the empty string clears it - the
field is removed entirely, not stored as an empty string. When present, it is always an
absolute host path that has passed registration validation (`actor-driver.md`).
- There is **no `imageWorkingDirectory` field on the Actor record.** It lives on the `BuildRecord`
instead (`__BUILDS__` below) - build-specific, not Actor-specific: the workdir a run mounts
against must be the one for the build that run itself resolved, never whichever tag happened to
build most recently.
- `localDevFolder`, together with the resolved build's `imageWorkingDirectory`, are **absent (or
empty) meaning no mount**: a run only adds the dev-folder bind mount when both are present and
non-empty (`actor-driver.md`).
- Neither `localDevFolder` nor any build's `imageWorkingDirectory` is ever exposed on the public
`/v2` API.
- The system stores Actor runs in dedicated key-value store called `__RUNS__`:
- `key` is the id of the Actor run `runId`
- `value` is the metadata of the Actor
Expand All @@ -85,6 +98,10 @@
- owner (`userId`)
- Actor (`actorId`)
- metadata
- `imageWorkingDirectory` - **optional**, and specific to this one build. Absent unless this
particular build succeeded and its own image's working directory could be captured (also
absent when the captured value was empty or `/`) - never on any other build, and never derived
from, or copied onto, the Actor record (see `localDevFolder`'s entry above).
- The system stores logs in dedicated key-value store called `__LOGS__`:
- `key` is the id of the Actor build (`logId`)
- `value` is the metadata of the Actor
Expand Down
4 changes: 2 additions & 2 deletions src/api/dto/actors.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import type { ActorRecord, BuildRecord, RunRecord } from '../../storage/entities.js';

/** Matches `services/runs.ts`'s `DEFAULT_BUILD_TAG` - backfilled here only for run records that predate
* `options.build` (directly-seeded test fixtures); every real run always has it set already. */
/** Matches `services/actors.ts`'s `DEFAULT_BUILD_TAG` - backfilled here only for run records that
* predate `options.build` (directly-seeded test fixtures); every real run always has it set already. */
const DEFAULT_RUN_BUILD_TAG = 'latest';
/** Matches `services/runs.ts`'s `DISK_MBYTES_PER_MEMORY_MBYTE` - backfilled here only for run records
* that predate `options.diskMbytes`; every real run always has it set already. */
Expand Down
Loading
Loading