remove devtools and create a separate JIRA in backlog

Author: Tingting-Chang

TESTING.md

@@ -13,25 +13,24 @@ pip install pre-commit && pre-commit install
 
 ---
 
-## What to run
+## What to run locally vs in CI
 
 ```
 unit tests  →  core-local  →  open PR  →  CI handles the rest
 ```
 
-| Suite | Command | Needs | Time |
-|-------|---------|-------|------|
-| Unit | `tox` | nothing | ~2 min |
-| Core — local | `tox -c test/core/tox.ini -e core-local` | nothing | ~1 hr |
-| Core — GCS | `tox -c test/core/tox.ini -e core-gcs` | devtools `fake-gcs-server` | ~1 hr |
-| Core — Azure | `tox -c test/core/tox.ini -e core-azure` | devtools `azurite` | ~1 hr |
-| Core — Batch/K8s/Argo/SFN | `tox -c test/core/tox.ini -e core-<name>` | full devstack | 2–3 hr |
-| UX — local | `tox -e ux-local` | nothing | ~30 min |
-| UX — cloud | `tox -e ux-<name>` | full devstack | ~1 hr |
+| Suite | Command | Needs | Run where |
+|-------|---------|-------|-----------|
+| Unit | `tox` | nothing | locally + CI |
+| Core — local | `tox -c test/core/tox.ini -e core-local` | nothing | locally + CI |
+| Core — GCS / Azure | `tox -c test/core/tox.ini -e core-{gcs,azure}` | emulator at known port | **CI** |
+| Core — Batch/K8s/Argo/SFN | `tox -c test/core/tox.ini -e core-<name>` | full devstack | **CI** |
+| UX — local | `tox -e ux-local` | nothing | locally + CI |
+| UX — cloud | `tox -e ux-<name>` | full devstack | **CI** |
 
-Run **unit + core-local** before every PR. Cloud-backend tests (`core-gcs`,
-`core-batch`, …) are only needed if you changed that backend's storage code —
-otherwise let CI run them.
+Run **unit + core-local** before every PR. All cloud-backend tests require
+infrastructure that CI provisions — there is no need to set up emulators or
+Docker containers locally. Open your PR and let CI handle them.
 
 ---
 
@@ -54,7 +53,7 @@ flow definition (~64 classes), runs it as a subprocess, and verifies results
 in-process. This yields ~470 parametrised items per backend, identified as
 `backend/graph/FlowDefinition/executor`.
 
-### core-local
+### core-local — run this locally
 
 ```bash
 tox -c test/core/tox.ini -e core-local
@@ -65,40 +64,22 @@ tox -c test/core/tox.ini -e core-local -- --core-graphs simple-foreach
 tox -c test/core/tox.ini -e core-local -- -n auto   # parallel
 ```
 
-### core-gcs / core-azure (cloud storage emulators)
+### core-gcs / core-azure / core-batch / core-k8s / core-argo / core-sfn — let CI run these
 
-Only needed when you changed GCS or Azure storage code. The emulators are
-part of the devtools stack:
+These backends require external infrastructure (GCS emulator, Azure emulator,
+MinIO, Kubernetes, …). CI provisions all of it automatically. You do not need
+to install Docker or start any services to get these tests to pass.
 
-```bash
-cd devtools
-SERVICES_OVERRIDE=fake-gcs-server make up   # GCS  (port 4443)
-SERVICES_OVERRIDE=azurite make up           # Azure (port 10000)
-```
-
-Then:
-
-```bash
-tox -c test/core/tox.ini -e core-gcs
-tox -c test/core/tox.ini -e core-azure
-```
-
-### core-batch / core-k8s / core-argo / core-sfn
-
-These require the full devstack. For most PRs, **let CI run them**. If you
-need to debug a scheduler-specific failure locally, start only the services
-that backend needs, then run the env:
+If you are debugging a specific backend failure locally and already have the
+required infrastructure running, you can invoke the env directly:
 
 ```bash
-# Required services per backend:
-#   core-batch: minio, postgresql, metadata-service, localbatch
-#   core-k8s:   minio, postgresql, metadata-service
-#   core-argo:  minio, postgresql, metadata-service, argo-workflows
-#   core-sfn:   minio, postgresql, metadata-service, localbatch, ddb-local, sfn-local
-cd devtools && SERVICES_OVERRIDE=minio,postgresql,metadata-service,localbatch make up
+tox -c test/core/tox.ini -e core-gcs    # expects fake-gcs-server at localhost:4443
+tox -c test/core/tox.ini -e core-azure  # expects azurite at localhost:10000
+tox -c test/core/tox.ini -e core-batch  # expects devstack (see devtools/README.md)
 ```
 
-See `devtools/README.md` for the full devstack reference.
+For devstack setup see `devtools/README.md`.
 
 ---
 
@@ -118,7 +99,5 @@ pre-commit run --all-files    # all checks
 | `tox: command not found` | `pip install tox` |
 | `Username 'root' is not allowed` | `export METAFLOW_USER=tester` |
 | `ModuleNotFoundError` during collection | `rm -rf .tox && tox -c test/core/tox.ini -e core-local` |
-| `core-gcs` — `ConnectionRefusedError` | `cd devtools && SERVICES_OVERRIDE=fake-gcs-server make up` |
-| `core-azure` — Azure connection error | `cd devtools && SERVICES_OVERRIDE=azurite make up` |
-| `core-batch/k8s` — metadata service error | Start devstack: `cd devtools && make up` |
-| Tests run slowly | `tox -c test/core/tox.ini -e core-local -- -n auto` (don't use `-n` for cloud envs) |
+| `core-local` slow | `tox -c test/core/tox.ini -e core-local -- -n auto` |
+| Cloud env fails locally | Check that the required emulator/devstack is running — or just open a PR and let CI handle it |

devtools/Tiltfile

@@ -30,7 +30,6 @@ components = {
     "ddb-local": [],
     "sfn-local": ["ddb-local"],
     "airflow": ["postgresql"],
-    "fake-gcs-server": [],
 }
 
 # ---------------------------------------------------------------------------
@@ -94,7 +93,6 @@ load('./tilt/localbatch.tiltfile', 'setup_localbatch')
 load('./tilt/ddb_local.tiltfile', 'setup_ddb_local')
 load('./tilt/sfn_local.tiltfile', 'setup_sfn_local')
 load('./tilt/airflow.tiltfile', 'setup_airflow')
-load('./tilt/fake_gcs_server.tiltfile', 'setup_fake_gcs_server')
 
 _SETUP = {
     "minio": setup_minio,
@@ -106,7 +104,6 @@ _SETUP = {
     "ddb-local": setup_ddb_local,
     "sfn-local": setup_sfn_local,
     "airflow": setup_airflow,
-    "fake-gcs-server": setup_fake_gcs_server,
 }
 
 # ---------------------------------------------------------------------------

devtools/pick_services.sh

@@ -21,7 +21,6 @@ SERVICE_OPTIONS=(
     "ddb-local"
     "sfn-local"
     "airflow"
-    "fake-gcs-server"
 )
 
 gum style "$LOGO" \

devtools/tilt/fake_gcs_server.tiltfile

@@ -87,72 +87,65 @@ def pytest_generate_tests(metafunc: Any) -> None:
     if "flow_triple" not in metafunc.fixturenames:
         return
 
-    try:
-        ok_tests_raw = metafunc.config.getoption("--core-tests", default=None)
-        ok_graphs_raw = metafunc.config.getoption("--core-graphs", default=None)
-        ok_tests = (
-            {t.lower() for t in ok_tests_raw.split(",") if t} if ok_tests_raw else set()
-        )
-        ok_graphs = (
-            {g.lower() for g in ok_graphs_raw.split(",") if g} if ok_graphs_raw else set()
-        )
-
-        # All context configuration comes from the environment (set by tox setenv).
-        marker_name = os.environ.get("METAFLOW_CORE_MARKER", "local")
-        executors = [
-            e for e in os.environ.get("METAFLOW_CORE_EXECUTORS", "cli,api").split(",") if e
-        ]
-        disabled_tests = {
-            t for t in os.environ.get("METAFLOW_CORE_DISABLED_TESTS", "").split(",") if t
-        }
-        enabled_tests = {
-            t for t in os.environ.get("METAFLOW_CORE_ENABLED_TESTS", "").split(",") if t
-        }
-        disable_parallel = os.environ.get("METAFLOW_CORE_DISABLE_PARALLEL", "") == "1"
-
-        mark = getattr(pytest.mark, marker_name)
-        all_tests = sorted(_iter_tests(), key=lambda t: t.PRIORITY)
-        all_graphs = list(_iter_graphs())
-
-        params = []
-        for graph in all_graphs:
-            if ok_graphs and graph["name"].lower() not in ok_graphs:
+    ok_tests_raw = metafunc.config.getoption("--core-tests", default=None)
+    ok_graphs_raw = metafunc.config.getoption("--core-graphs", default=None)
+    ok_tests = (
+        {t.lower() for t in ok_tests_raw.split(",") if t} if ok_tests_raw else set()
+    )
+    ok_graphs = (
+        {g.lower() for g in ok_graphs_raw.split(",") if g} if ok_graphs_raw else set()
+    )
+
+    # All context configuration comes from the environment (set by tox setenv).
+    marker_name = os.environ.get("METAFLOW_CORE_MARKER", "local")
+    executors = [
+        e for e in os.environ.get("METAFLOW_CORE_EXECUTORS", "cli,api").split(",") if e
+    ]
+    disabled_tests = {
+        t for t in os.environ.get("METAFLOW_CORE_DISABLED_TESTS", "").split(",") if t
+    }
+    enabled_tests = {
+        t for t in os.environ.get("METAFLOW_CORE_ENABLED_TESTS", "").split(",") if t
+    }
+    disable_parallel = os.environ.get("METAFLOW_CORE_DISABLE_PARALLEL", "") == "1"
+
+    mark = getattr(pytest.mark, marker_name)
+    all_tests = sorted(_iter_tests(), key=lambda t: t.PRIORITY)
+    all_graphs = list(_iter_graphs())
+
+    params = []
+    for graph in all_graphs:
+        if ok_graphs and graph["name"].lower() not in ok_graphs:
+            continue
+        if disable_parallel and any(
+            "num_parallel" in node for node in graph["graph"].values()
+        ):
+            continue
+
+        for test in all_tests:
+            test_name = test.__class__.__name__
+            if ok_tests and test_name.lower() not in ok_tests:
+                continue
+            if test_name in disabled_tests:
                 continue
-            if disable_parallel and any(
-                "num_parallel" in node for node in graph["graph"].values()
-            ):
+            if enabled_tests and test_name not in enabled_tests:
+                continue
+            if not FlowFormatter(graph, test).valid:
                 continue
 
-            for test in all_tests:
-                test_name = test.__class__.__name__
-                if ok_tests and test_name.lower() not in ok_tests:
-                    continue
-                if test_name in disabled_tests:
-                    continue
-                if enabled_tests and test_name not in enabled_tests:
-                    continue
-                if not FlowFormatter(graph, test).valid:
-                    continue
-
-                for executor in executors:
-                    param_id = "%s/%s/%s/%s" % (
-                        marker_name,
-                        graph["name"],
-                        test_name,
-                        executor,
-                    )
-                    params.append(
-                        pytest.param(
-                            (graph, test, executor),
-                            marks=[mark],
-                            id=param_id,
-                        )
+            for executor in executors:
+                param_id = "%s/%s/%s/%s" % (
+                    marker_name,
+                    graph["name"],
+                    test_name,
+                    executor,
+                )
+                params.append(
+                    pytest.param(
+                        (graph, test, executor),
+                        marks=[mark],
+                        id=param_id,
                     )
+                )
 
-        metafunc.parametrize("flow_triple", params)
-    except Exception as e:
-        import traceback
-
-        print("Warning: could not generate core test combinations: %s" % e)
-        traceback.print_exc()
-        metafunc.parametrize("flow_triple", [])
+    metafunc.parametrize("flow_triple", params)

devtools/tilt/k8s/fake-gcs-secret.yaml

@@ -109,6 +109,7 @@ def _flow_lines(self):
             "from metaflow_test import is_resumed, ResumeFromHere, "
             "TestRetry, try_to_get_card"
         )
+        yield 0, "import pytest"
         if tags:
             yield 0, "from metaflow import %s" % ",".join(tags)