test: keep one mock Cloudflare DNS API, in tools/

There were two. `tools/mock-cf-dns-api` was 882 lines of Flask and gunicorn
behind a published `kvin/mock-cf-dns-api:latest`, and after the gateway suite
switched to a repo-built mock, nothing used it. The survivor is 354 lines of
stdlib with no dependencies, and it is the only one that answers DNS, which
is what lets an ACME server validate rather than be told to skip.

Nothing is lost: certbot's Cloudflare client calls four endpoints -- list
zones, list records, create, delete -- and the survivor implements exactly
those. The dropped extras (PUT, single-record GET, a log buffer, a web UI)
had no caller.

Moved to tools/ because it is no longer one suite's fixture: both build it,
and the gateway suite was reaching into the other suite's directory through
four levels of `..` to find it.

Author: kvinwang

dstack/gateway/test-run/e2e/docker-compose.yml

@@ -36,9 +36,10 @@ services:
   # Mock Cloudflare DNS API. Built from the repo rather than pulled, and the
   # same image the full-stack suite builds, so both suites answer DNS the same
   # way and neither depends on a prebuilt image nobody can rebuild.
+  # Source: tools/mock-cf-dns.
   mock-cf-dns-api:
     build:
-      context: ../../../../test-suites/full-stack-compose/mock-cf-dns
+      context: ../../../../tools/mock-cf-dns
     image: dstack-e2e-mock-cf-dns:local
     container_name: mock-cf-dns-api
     networks:

dstack/gateway/test-run/e2e/run-e2e.sh

@@ -136,6 +136,14 @@ log_info "Starting e2e test environment..."
 
 export GATEWAY_IMAGE=dstack-gateway:test
 
+# Build the mock rather than trusting whatever carries its tag. `image:` plus
+# `build:` means compose reuses a local tag if one exists, and this tag is
+# shared with the full-stack suite, so a stale copy from an older checkout
+# would be picked up silently -- and a mock that does not answer TCP fails
+# every challenge now that Pebble actually validates them.
+log_info "Building the mock DNS/Cloudflare API..."
+docker compose build mock-cf-dns-api
+
 docker compose up -d mock-cf-dns-api pebble
 log_info "Waiting for mock services to be healthy..."
 sleep 5

test-suites/full-stack-compose/compose.yml

@@ -57,7 +57,7 @@ services:
 
   mock-cf-dns-api:
     build:
-      context: ./mock-cf-dns
+      context: ../../tools/mock-cf-dns
     image: dstack-e2e-mock-cf-dns:local
     networks:
       e2e-net:
@@ -76,9 +76,11 @@ services:
       retries: 10
 
   pebble:
-    # Custom Pebble build used by the existing gateway E2E suite. It supports
-    # plain HTTP ACME URLs, which keeps the host-side Gateway bootstrap simple.
-    image: ${DSTACK_E2E_PEBBLE_IMAGE:-kvin/pebble:latest}
+    # Custom Pebble build shared with the gateway E2E suite: upstream plus a
+    # `-http` flag, which keeps the host-side Gateway bootstrap simple.
+    # github.com/kvinwang/pebble branch http-flag. Pinned rather than :latest,
+    # so a rebuild of that tag cannot change what this suite validates against.
+    image: ${DSTACK_E2E_PEBBLE_IMAGE:-kvin/pebble:v2.10.1-http}
     command: ["-http", "-dnsserver", "172.31.0.10:53"]
     networks:
       e2e-net:
@@ -88,7 +90,10 @@ services:
       - "0.0.0.0:${DSTACK_E2E_PEBBLE_MGMT_PORT:-35000}:15000"
     environment:
       PEBBLE_VA_NOSLEEP: "1"
-      PEBBLE_VA_ALWAYS_VALID: "1"
+      # No PEBBLE_VA_ALWAYS_VALID. It was needed because the mock answered only
+      # UDP while Pebble queries a custom resolver over TCP, so nothing it was
+      # pointed at could ever answer; the mock now serves both and challenges
+      # are validated against the records certbot writes.
     depends_on:
       mock-cf-dns-api:
         condition: service_healthy

tools/README.md

@@ -4,7 +4,9 @@ This directory contains developer and operator utilities that do not belong to
 one runtime component or one guest-OS backend.
 
 - `add-spdx-attribution.py` updates SPDX attribution metadata.
-- `mock-cf-dns-api/` provides a local Cloudflare DNS API test double.
+- `mock-cf-dns/` is the Cloudflare DNS API test double both E2E suites build:
+  the API endpoints certbot calls, plus TXT answers on 53 over UDP and TCP so
+  an ACME server can validate against the records it writes.
 - `sca/` builds self-contained application images.
 - `dev-stack.sh` preserves the unsupported legacy all-in-one host, guest, and
   deployment helper.