Add @tenki compute backend (Tenki Sandbox microVMs)

[AlvaroDeleglise]

PR Type

• Bug fix
• New feature
• Core Runtime change (higher bar -- see CONTRIBUTING.md)
• Docs / tooling
• Refactoring

Summary

Add @tenki, a new compute backend that runs each flow step inside a disposable
Tenki Sandbox Linux microVM — the Tenki
analogue of @kubernetes / @batch. Entirely additive and opt-in; the rest of
Metaflow (datastore, metadata, cards, @retry, @resources, @catch, resume,
@pypi/@conda) is unchanged.

Issue

Fixes #3310.

Reproduction

Not a bug fix — this adds a backend, so "before" = the feature does not exist and
"after" = a step runs on a Tenki microVM. Minimal end-to-end exercise:

Runtime: tenki (new); datastore = S3 (validated with MinIO reached over a
temporary tunnel, so the microVM can read/write it).

Commands to run:

pip install tenki-sandbox
export TENKI_API_KEY=tk_...
export METAFLOW_DEFAULT_DATASTORE=s3
export METAFLOW_DATASTORE_SYSROOT_S3=s3://your-bucket/mf
# self-hosted / non-AWS S3 also needs: export METAFLOW_S3_ENDPOINT_URL=https://...
python tenki_flow.py --datastore=s3 run

# tenki_flow.py
from metaflow import FlowSpec, step, tenki

class TenkiFlow(FlowSpec):
    @tenki(cpu=1, memory=1024)
    @step
    def start(self):
        self.x = 21
        self.next(self.end)

    @tenki(cpu=1, memory=1024)
    @step
    def end(self):
        print("RESULT", self.x * 2)

if __name__ == "__main__":
    TenkiFlow()

Where evidence shows up: parent console (per-step task logs streamed from the
datastore), artifacts via the Client API, and python tenki_flow.py tenki list.

Before (feature absent)

ImportError: cannot import name 'tenki' from 'metaflow'

After (runs on real Tenki microVMs)

start ✓  ... Downloading code package → Code package downloaded → PY_VERSION 3.12.3 → Task finished (exit 0)
end   ✓  ... RESULT 42 → Task finished (exit 0)
Done!

Each step ran in a fresh microVM on the stock Tenki image; the artifact
persisted across steps (self.x = 21 → RESULT 42).

Root Cause

N/A — this is a new feature, not a bug fix. For Core-Runtime context, the only
change to a shared core file, and its causal chain:

• metaflow/plugins/pypi/conda_decorator.py keeps a hardcoded allowlist of remote
  backends for which it leaves the trampoline interpreter alone (interpreter = None). A backend not on the list has its local tenki step trampoline
  swapped into the resolved @pypi/@conda environment, which then cannot import
  the datastore/metadata dependencies — so @pypi + @tenki fails. Adding
  "tenki" (one line, alongside batch/kubernetes/nvidia) gives it identical
  treatment.

Everything else is additive: the new metaflow/plugins/tenki/ package, two
registration lines in plugins/__init__.py, and a TENKI_* block in
metaflow_config.py.

Why This Fix Is Correct

• Reuses the existing @kubernetes remote-execution contract verbatim (the
  bash -c "…" entrypoint + METAFLOW_* env vars, via the same shlex.split
  transformation), so remote behavior matches an established backend; the
  Tenki-specific surface is small (create/exec/teardown + credential forwarding).
• The shared-file change is minimal — one list entry mirroring the existing
  ones; no behavior change for other backends.
• Success is judged by the SDK's own result.ok (exit_code == 0 and not signal), not a partial exit-code heuristic, so a signalled task is never read
  as success.
• Soft dependency — the tenki-sandbox SDK is imported lazily (not in
  install_requires), consistent with how kubernetes/azure-*/google-cloud-*
  are handled, with a >= 0.4.0 version guard.

Failure Modes Considered

1. Concurrency / retries — one fresh sandbox per attempt (unique name so a
   retry never collides with a still-terminating prior VM); @retry is owned by
   the runtime; concurrent foreach (3 parallel microVMs) validated live.
2. Credential leakage — only the active datastore's credentials are
   forwarded into the VM (s3 creds are not leaked into an azure/gs run);
   per-datastore isolation is unit-tested.
3. Orphan / hard-crash cleanup — in-process teardown (finally + atexit,
   which retries on the atexit pass and warns if it still fails), the
   tenki list / tenki kill CLI over tagged sandboxes, and a server-side
   max_duration cap. In-process teardown does not run on a hard SIGKILL; the
   cap / tenki kill cover that. (Note: for project-scoped sandboxes
   max_duration pauses rather than terminates — a Tenki-side terminate-on-
   expiry would be the ideal backstop.)
4. Signalled tasks & result interpretation — a task killed by a signal
   (exit 0 but signalled) is treated as a failure; signal/reason/errno are
   surfaced; timeout arrives as either CommandTimeoutError or a builtin
   TimeoutError and both map to a retryable timeout.
5. Base-image variance — Tenki cannot pull public registries and its stock
   image ships python3 but no python; a startup shim provisions python/pip
   (no-op on images that already have them) and fails with a clear error if the
   image has no python3 at all.
6. Command-string escaping — the entrypoint is built and shlex.split the
   exact way @kubernetes does, so the \"-escaped inner quotes resolve
   correctly through sb.exec(*argv).

Tests

• Unit tests added/updated — test/unit/test_tenki.py, 41 tests, all
  green; no regressions to the existing @kubernetes / resource-merge tests.
• Reproduction script provided — not in CI (see below).
• CI passes — black-clean; no regressions.
• If tests are impractical: explain — the tenki-sandbox SDK is a soft
  dependency, so CI unit tests run against a stubbed SDK (real-SDK
  integration cannot run in CI without a Tenki account/API key). Live validation
  is therefore manual (below).

Unit coverage: resource merge; datastore / @parallel validation;
sandbox-name sanitization; the _command bash contract + runtime-shim ordering +
no-python3 guard; runner behavior with a stubbed SDK (result.ok success,
non-zero / OOM(137) / segfault(139) / unknown / signalled results,
signal/reason/errno diagnostics, both timeout types, session-lost,
permission-denied non-retryable, stderr surfacing); _cleanup retry +
warn-on-final-pass; per-datastore credential isolation + GCP JSON materialization;
secrets/S3-SSE/OTEL forwarding; tag sanitization; list/kill client
credentials; SDK minimum-version enforcement; and a behavioral regression for the
conda_decorator change (drives the real CondaStepDecorator, asserts
interpreter is None for a @tenki step).

Manual / end-to-end (real Tenki microVMs; datastore = MinIO over a tunnel):
happy path (linear + foreach + join) and plain @tenki steps on the stock
image; @retry (fresh sandbox per attempt); concurrency (--max-workers 3);
@timeout; --metadata=service (real Postgres-backed service — integer run-ids +
task/metadatum registration); @catch; resume; @pypi (ran a real package
inside the microVM); and orphan cleanup (tenki list / tenki kill).

Not validated live: Azure/GS (forwarding is implemented + unit-tested; needs a
real account) and a custom TENKI_CONTAINER_IMAGE (Tenki serves images only from
its own registry, so it needs a Metaflow-ready image pushed there).

Non-Goals

• @parallel / gang scheduling, GPU, and persistent volumes/snapshots — deferred.
• Tenki native storage as the Metaflow datastore — it is session-bound and cannot
  meet the datastore contract; a shared blob store (S3/Azure/GS) is still required.
• Changing how compute-backend SDK dependencies are declared — the SDK is kept a
  soft dependency here (matching existing backends like kubernetes/azure-*);
  glad to expose an extras_require={"tenki": ["tenki-sandbox>=0.4.0"]} extra
  instead if maintainers prefer.

AI Tool Usage

• No AI tools were used in this contribution

• AI tools were used (describe below)

• Tool: Claude Code.

• Used for: implementation scaffolding, unit tests, and the live-validation
  harness (MinIO/tunnel setup, e2e probes).

• Reviewed/understood/tested: yes — all code was reviewed and is understood by
  me, and the design decisions, failure modes, and tests reflect my own judgment.

[greptile-apps]

Greptile Summary

This PR adds Tenki Sandbox microVMs as an opt-in Metaflow compute backend. The main changes are:

• New @tenki decorator and remote task runner.
• Sandbox launch, log streaming, result handling, and cleanup.
• Flow-scoped tenki list and tenki kill commands.
• Datastore credentials, metadata settings, secrets configuration, and tracing forwarded to workloads.
• Plugin, configuration, and Conda/PyPI integration.
• Unit coverage for resources, credentials, lifecycle behavior, CLI scope, and failures.

Confidence Score: 5/5

This looks safe to merge.

• Sandbox management is now scoped by flow and defaults to the latest run.
• Cleanup tries each supported teardown method until one succeeds.
• Transient launch failures remain eligible for configured retries.
• No blocking issues were found in the updated code.

Important Files Changed

Filename	Overview
metaflow/plugins/tenki/tenki.py	Adds sandbox launch, remote command execution, environment forwarding, result handling, tagging, and cleanup.
metaflow/plugins/tenki/tenki_cli.py	Adds task execution and flow-scoped commands for listing and terminating Tenki sandboxes.
metaflow/plugins/tenki/tenki_client.py	Adds lazy SDK loading, version checks, project discovery, sandbox operations, and client cleanup.
metaflow/plugins/tenki/tenki_decorator.py	Adds the @tenki decorator, resource merging, runtime validation, and task metadata integration.
metaflow/plugins/init.py	Registers the Tenki trampoline command and step decorator.
metaflow/plugins/pypi/conda_decorator.py	Prevents Conda and PyPI environments from replacing the Tenki trampoline interpreter.
metaflow/metaflow_config.py	Adds configuration for Tenki credentials, endpoint, project, workspace, image, resources, and initialization.
test/unit/test_tenki.py	Adds coverage for resource handling, SDK compatibility, scope resolution, cleanup, credentials, execution results, and retry classification.

_{Reviews (4): Last reviewed commit: "Classify launch failures so transient on..." | Re-trigger Greptile}