Add opt-in upstream API fallback with two independent toggles

CLAUDE.MD

@@ -30,6 +30,17 @@ Local Actor runtime is an Actor development tool for developing, running, and de
   hardcodes a `/v2`-suffixed base URL). From then on, edit locally, recompile locally (`tsc` or the
   language-appropriate equivalent), and `apify call` again - no `apify push`/build in between. Submitting
   `--body '""'` clears the registration. Dependency changes still need a real rebuild.
+- If a call fails because this runtime doesn't have the Actor/run/build/storage id you're after, or
+  doesn't implement that endpoint at all, you can opt in to having such calls transparently relayed to
+  the real Apify platform instead of failing:
+  `apify api POST /actor-runtime/api-fallback --body '{"fallbackUnimplementedEnabled": true, "fallbackNotFoundEnabled": true}'`
+  (either field alone is also accepted; `apify api GET /actor-runtime/api-fallback` reads the current
+  state). Both default off and reset to off on every restart. Enabling either forwards whatever token you
+  authenticated the failing call with to the real platform, and - since all HTTP methods are eligible -
+  can turn a locally-missing `POST`/`PUT`/`DELETE` into a real write against your real account; only turn
+  this on with a token/account you're comfortable with that. A relayed response carries an
+  `x-actor-runtime-fallback` header naming which platform served it, and an
+  `x-actor-runtime-fallback-trigger` header naming which toggle let it through.
 
 ## Through direct API calls
 

requirements/api.md

@@ -152,4 +152,54 @@
 
 ## Upstream fallback (opt-in, off by default, all HTTP methods)
 
-- Not implemented
+- Two independent booleans, `fallbackUnimplementedEnabled` and `fallbackNotFoundEnabled`, gate whether
+  a request this runtime cannot satisfy locally is instead relayed to the real Apify platform. Both
+  default to `false`, and a restart always brings both back to `false`, regardless of how they were
+  last set. Either can be on without the other; all four combinations are valid.
+- **`GET /actor-runtime/api-fallback`** (also reachable at `/v2/actor-runtime/api-fallback`, like every
+  other endpoint in this namespace) returns
+  `{ "data": { "fallbackUnimplementedEnabled": <bool>, "fallbackNotFoundEnabled": <bool>, "upstreamBaseUrl": <string> } }`.
+  `upstreamBaseUrl` is the platform this runtime would relay to (default `https://api.apify.com`, or the
+  value of `APIFY_UPSTREAM_API_BASE_URL` if set) - reported for visibility, but read-only: no request
+  body can change it.
+- **`POST /actor-runtime/api-fallback`** (same two mounts) accepts a body naming either field, or both;
+  a field the body doesn't mention keeps its current value. The response is the same shape `GET`
+  returns, showing the state immediately after the change.
+    - **Authenticated** the same way as every other route in this namespace: no token is `401`
+      `user-not-authenticated`, with no state change.
+    - **Error responses**: a body that isn't a JSON object (a JSON array, scalar, or `null`), a body
+      present but empty (`{}`), a body containing a key other than the two above, or a body where a
+      present key's value isn't a boolean, is `400` `invalid-request`, with no state change.
+- **Which local outcome each toggle covers** (exhaustive - every other error response is never
+  eligible, under any toggle combination):
+    - `fallbackUnimplementedEnabled` covers a request the runtime does not serve at all: a local `404`
+      or `501` response (see "501 vs 404" above). From the caller's point of view both mean "nothing
+      local answers this", so one toggle covers both.
+    - `fallbackNotFoundEnabled` covers a request that reaches a route this runtime does serve, but
+      whose specific record id doesn't exist locally (`record-not-found`, see "Response envelopes"
+      above).
+    - Every other error type - `invalid-request`, `user-not-authenticated`,
+      `cannot-remove-running-run`, `deleting-unfinished-build`, any `dev-folder-*` type,
+      `internal-error` - is never relayed, regardless of either toggle's state.
+- **All HTTP methods are eligible for both toggles, writes included**: a `POST`/`PUT`/`DELETE` that
+  would otherwise 404/501 locally is relayed exactly like a `GET` when its toggle is on - and, if the
+  platform accepts it, becomes a real write against the caller's real account. This is a deliberate
+  consequence of opting in, not an oversight. An eligible request reaches the platform at most once, so
+  a relayed write is never duplicated.
+- **A successful relay** returns the platform's response status and body to the caller unchanged,
+  marked with two response headers: `x-actor-runtime-fallback: <upstreamBaseUrl>` naming which platform
+  served it, and `x-actor-runtime-fallback-trigger: unimplemented` or `record-not-found` naming which
+  toggle let it through. Only a final `2xx` status counts as successful.
+- **Fail-closed guarantee**: anything else - a non-`2xx` response, a timeout, or the platform being
+  unreachable - reproduces the exact response the caller would have gotten with both toggles off: the
+  original local error, unchanged, with neither marker header present. The platform's own status or
+  body is never surfaced to the caller.
+- **Only the caller's own presented token is ever forwarded.** A relayed request's `Authorization`
+  header is always the exact bearer token the caller themselves sent on that request - never a
+  different or runtime-internal credential, and never sent at all for a request this runtime didn't
+  authenticate. Enabling either toggle therefore means the caller's own Apify token reaches the
+  configured `upstreamBaseUrl` on every eligible request; this is the risk being opted into.
+- **Never enriches a call that already succeeds locally**: a collection/list endpoint (e.g.
+  `GET /v2/datasets`) that already returns `200` from local data never consults either toggle and never
+  gains platform objects. Fallback only ever resolves an otherwise-failing request; it does not make a
+  local listing "complete".

requirements/cli.md

@@ -67,7 +67,9 @@
   `apify-cli` fetches the actor-templates manifest from the internet the first time it needs to create
   an Actor. Every push/call/log-stream/storage-access afterwards, and every build of an
   already-pulled base image, works with no outbound network access (see `system.md`'s offline-after-
-  first-build note).
+  first-build note) - unless the opt-in upstream API fallback (`api.md`'s "Upstream fallback" section)
+  is switched on, in which case any such call that misses locally and is eligible for fallback makes one
+  outbound request to the configured upstream instead of failing offline.
 - The bundled sample Actors crawl the live web (`https://crawlee.dev/` by default), so an `apify call`
   that runs one of them needs outbound network access from the Actor container even though the
   CLI-to-runtime interaction itself does not (see `system.md`).

requirements/console.md

@@ -9,8 +9,12 @@
 - The console has no login of its own, so with multiple users it lists and shows every user's objects
   rather than scoping to one - the API's own endpoints stay strictly scoped to the calling token's user
   (`storage.md`'s "Users" section).
-- The console is unauthenticated. Every route is a read except the dev-folder form below, which is the
-  console's one write - it is no longer strictly view-only.
+- The console is unauthenticated. Every route is a read except the dev-folder form below and the
+  Settings form below, which are the console's only two writes - it is no longer strictly view-only.
+- Both of those two writes reject a submission that identifies itself as cross-site (via the
+  `Sec-Fetch-Site` header) with a plain `403`; a submission that does not is unaffected. This
+  narrows the console's existing "anyone who can reach it" model by one specific vector, it does
+  not add a login.
 - There are three types of objects: key-value store, dataset, request queue.
     - For each object type there must be exactly one widget for inspection.
     - The request-queue widget leads with the authoritative counts from `RequestQueue.getInfo()`
@@ -28,8 +32,11 @@
         - key-value stores
         - datasets
         - request queues
+    - Settings (a single page, not a list/detail pair - see "Settings page" below)
 
 - List view is a list of objects that can be clicked on to open detail view.
+- The Actor builds list, the Actor runs list, and the combined Logs list all show the most recently
+  started build or run first.
 - Detail view of an object is showing only one object with all the available data
 - A run's default storage ids (`defaultDatasetId`, `defaultKeyValueStoreId`, `defaultRequestQueueId`) are
   rendered as links to the corresponding storage detail views (in the run detail view and in the runs
@@ -49,3 +56,20 @@
   value is rejected as a malformed path, also matching the API.
 - A submission that fails validation redirects back to the same detail page with the classified error
   message shown inline, never swallowed by the redirect.
+
+## Settings page
+
+- Every page's header navigation includes a link to `/settings`, the one page for the upstream API
+  fallback toggles (`api.md`'s "Upstream fallback" section). The link itself shows both toggles'
+  current values, independently of each other, so neither toggle can ever be on without being visible
+  from anywhere in the console.
+- `/settings` shows `fallbackUnimplementedEnabled`, `fallbackNotFoundEnabled`, and `upstreamBaseUrl`
+  (the same values the API's toggle endpoint reports), plus a warning that enabling either toggle
+  forwards the caller's own Apify token to that URL.
+- The Settings page lets a caller set both toggles at once. Unlike the API's partial `POST` (`api.md`),
+  submitting the form always sets both toggles explicitly - leaving one unchecked sets it to `false`,
+  never "leave this toggle unchanged". A change made through either surface is immediately visible on
+  the other, and via the API's own `GET`, with no restart needed either way.
+- Since the console has no login of its own, anyone who can reach it can flip either toggle for every
+  caller of the API - the same unauthenticated, cross-user model the rest of the console already has,
+  not a new exposure specific to this page.

requirements/system.md

@@ -64,7 +64,9 @@
   network access because the stock CLI fetches its actor-templates manifest from the internet. Once
   both of those have happened at least once, the runtime itself operates fully offline: repeat builds
   reuse the already-pulled base image and every other push/call/log-stream/storage-access needs no
-  outbound network access at all (see `cli.md`'s offline-capability note).
+  outbound network access at all (see `cli.md`'s offline-capability note) - unless the opt-in upstream
+  API fallback (`api.md`'s "Upstream fallback" section) has been switched on, in which case a local miss
+  that is eligible for it makes one outbound request per such call.
 - The bundled sample Actors (`sample_actor_ts`, `sample_actor_py`) are not offline: they crawl a live
   site (`https://crawlee.dev/` by default). Running them needs outbound network access from the Actor
   container, unlike operating the runtime around them (see `test.md`).

src/api/routes/api-fallback.ts

@@ -0,0 +1,78 @@
+/**
+ * `GET`/`POST /actor-runtime/api-fallback` (`api.md`'s "Upstream fallback" section) - mounted on the
+ * same `/actor-runtime` sub-router `dev-folder.ts` already registers on, so it shares that router's
+ * single `auth()` registration (`server.ts`) rather than adding its own, and is served at both mounts
+ * (`/actor-runtime/api-fallback` and `/v2/actor-runtime/api-fallback`) the same way the dev-folder route
+ * is.
+ *
+ * `GET` reads the current toggle state; `POST` accepts a **partial** body - either
+ * `fallbackUnimplementedEnabled`, `fallbackNotFoundEnabled`, or both - and merges it into the existing
+ * state via `setApiFallbackState`, leaving any field the body didn't mention untouched. `upstreamBaseUrl`
+ * is reported on every response but is never itself a settable field.
+ */
+import type { Router } from 'express';
+
+import { sendData } from '../envelope.js';
+import { invalidRequest } from '../errors.js';
+import { h, jsonBody } from '../handler.js';
+import { getApiFallbackState, setApiFallbackState, type ApiFallbackState } from '../../services/api-fallback.js';
+import { upstreamApiBaseUrl } from '../../services/identity-resolution.js';
+
+const SETTABLE_FIELDS = new Set<keyof ApiFallbackState>(['fallbackUnimplementedEnabled', 'fallbackNotFoundEnabled']);
+
+function respondWithState(): ApiFallbackState & { upstreamBaseUrl: string } {
+	return { ...getApiFallbackState(), upstreamBaseUrl: upstreamApiBaseUrl() };
+}
+
+/** Parses and validates a `POST` body into a `setApiFallbackState` patch, throwing `invalid-request` for
+ * every malformed shape the spec names: not a JSON object (array, string, number, `null`), present but
+ * empty (`{}`), an unknown key, or a present key whose value isn't a boolean. Never partially applies a
+ * rejected body - the caller only ever sees the merged state after every field in the body has passed
+ * this check. */
+function parsePatch(raw: unknown): Partial<ApiFallbackState> {
+	if (typeof raw !== 'object' || raw === null || Array.isArray(raw)) {
+		throw invalidRequest(
+			'Request body must be a JSON object with fallbackUnimplementedEnabled and/or fallbackNotFoundEnabled',
+		);
+	}
+
+	const entries = Object.entries(raw as Record<string, unknown>);
+	if (entries.length === 0) {
+		throw invalidRequest(
+			'Request body must set at least one of fallbackUnimplementedEnabled or fallbackNotFoundEnabled',
+		);
+	}
+
+	const patch: Partial<ApiFallbackState> = {};
+	for (const [key, value] of entries) {
+		if (!SETTABLE_FIELDS.has(key as keyof ApiFallbackState)) {
+			throw invalidRequest(`Unknown field "${key}"`);
+		}
+		if (typeof value !== 'boolean') {
+			throw invalidRequest(`Field "${key}" must be a boolean`);
+		}
+		patch[key as keyof ApiFallbackState] = value;
+	}
+	return patch;
+}
+
+/** Mounts the `/api-fallback` routes onto `router`, matching every other route module's
+ * `mount*(router, ...): void` convention. `router` is expected to already have `auth()` registered on
+ * it by the caller (`server.ts`), same as `mountDevFolder`. */
+export function mountApiFallback(router: Router): void {
+	router.get(
+		'/api-fallback',
+		h(async (_req, res) => {
+			sendData(res, respondWithState());
+		}),
+	);
+
+	router.post(
+		'/api-fallback',
+		h(async (req, res) => {
+			const patch = parsePatch(jsonBody<unknown>(req));
+			setApiFallbackState(patch);
+			sendData(res, respondWithState());
+		}),
+	);
+}

src/api/routes/dev-folder.ts

@@ -1,11 +1,13 @@
 /**
  * `POST /actor-runtime/dev-folder/:actorId` - deliberately outside the emulated `/v2` surface
- * (`api.md`'s `/actor-runtime/*` namespace). `server.ts` creates its own sub-router, calls
- * `mountDevFolder` on it once, and mounts that same router instance at both `/actor-runtime` (canonical)
- * and `/v2/actor-runtime` (an alias existing solely because `apify api` hardcodes a `/v2`-suffixed base
- * URL - see `server.ts`'s doc comment). Neither mount is nested under the `v2` router, so this route
- * needs its own `auth()` rather than inheriting `v2`'s - and since only one of the two mounts ever
- * matches a given request, that `auth()` still runs exactly once per request either way.
+ * (`api.md`'s `/actor-runtime/*` namespace). `server.ts` creates one shared sub-router (with its own
+ * `auth()`, registered once there - not by this module) for the whole `/actor-runtime/*` namespace,
+ * calls this and `mountApiFallback` on it, and mounts that same router instance at both
+ * `/actor-runtime` (canonical) and `/v2/actor-runtime` (an alias existing solely because `apify api`
+ * hardcodes a `/v2`-suffixed base URL - see `server.ts`'s doc comment). Neither mount is nested under
+ * the `v2` router, so this namespace needs its own `auth()` rather than inheriting `v2`'s - and since
+ * only one of the two mounts ever matches a given request, that `auth()` still runs exactly once per
+ * request either way.
  *
  * Canonical body is a JSON string: `'"/abs/path"'` to set, `'""'` to clear (`api.md`). A JSON value that
  * parses but isn't a string is rejected the same way a malformed body is.
@@ -15,7 +17,7 @@
  */
 import type { Router } from 'express';
 
-import { auth, requireUser } from '../auth.js';
+import { requireUser } from '../auth.js';
 import { sendData } from '../envelope.js';
 import { ApiError, invalidRequest, recordNotFound } from '../errors.js';
 import { h, jsonBody } from '../handler.js';
@@ -50,13 +52,10 @@ function toApiError(result: Exclude<SetDevFolderResult, { kind: 'ok' }>): ApiErr
 }
 
 /** Mounts the `/dev-folder/:actorId` route onto `router`, matching every other route module's
- * `mount*(router, deps): void` convention - `server.ts` creates the sub-router, calls this on it, and
- * mounts the result at `/actor-runtime` itself (owning the path prefix the same way it owns `/v2`).
- * Registers its own `auth()` on `router` rather than inheriting `v2`'s, since this route lives outside
- * the `v2` router entirely (`api.md`'s `/actor-runtime/*` namespace). */
+ * `mount*(router, deps): void` convention - `server.ts` creates the sub-router (with its own shared
+ * `auth()`, registered by the caller, not here), calls this on it, and mounts the result at
+ * `/actor-runtime` itself (owning the path prefix the same way it owns `/v2`). */
 export function mountDevFolder(router: Router, deps: ApiServerDeps): void {
-	router.use(auth());
-
 	router.post(
 		'/dev-folder/:actorId',
 		h(async (req, res) => {