Skip to content

Commit e0dd019

Browse files
cursoragentMuon
cursoragent
and
Muon
committed
Add CI smoke test for data provider example with remote-data-loader
Adds an end-to-end smoke test for rust/examples/data_provider that: 1. Builds the example data-provider server into a Docker image. 2. Starts MinIO as an S3-compatible cache backend. 3. Starts a remote-data-loader container (public image from GAR) configured to point at the data provider with MinIO caching. 4. Queries the data provider directly: fetches the manifest (validated with jq), extracts the source URL from the manifest, fetches MCAP from that URL, and validates with the mcap CLI (magic bytes, message count, channel name). 5. Queries the remote-data-loader via POST /v1/stream (fresh then cached) and validates the resulting MCAP files identically. The test is strict: required tools (curl, jq, mcap, xxd) are asserted up front; every step uses set -euo pipefail; there are no fallback or degraded-mode code paths. The remote-data-loader API was discovered by probing: it exposes POST /v1/stream with a JSON body containing manifestQueryParams (array of [key, value] tuples) plus start/end timestamps. Known issue: the remote-data-loader returns 401 on POST /v1/stream. The binary validates client authentication via Foxglove OAuth, and without the correct auth env vars configured, Steps 3-4 fail. The correct auth configuration needs to be provided by someone familiar with the remote-data-loader's auth system. Files: - .github/workflows/data_provider_smoke_test.yml - rust/examples/data_provider/smoke-test/Dockerfile.data-provider - rust/examples/data_provider/smoke-test/docker-compose.yml - rust/examples/data_provider/smoke-test/run.sh Co-authored-by: Mak Nazečić-Andrlon <Muon@users.noreply.github.com>
1 parent ac32d0f commit e0dd019

File tree

4 files changed

+427
-0
lines changed

4 files changed

+427
-0
lines changed

‎.github/workflows/data_provider_smoke_test.yml‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
# Smoke test for rust/examples/data_provider with a remote-data-loader and MinIO cache.
2+
#
3+
# This workflow builds the example data-provider, starts it alongside MinIO and
4+
# a remote-data-loader, then runs curl + mcap CLI checks to verify that data is
5+
# served correctly (fresh and from cache).
6+
#
7+
# The remote-data-loader image is public on Google Artifact Registry; no
8+
# credentials are needed to pull it.
9+
10+
name: Data Provider Smoke Test
11+
12+
on:
13+
push:
14+
branches: [main]
15+
paths:
16+
- "rust/foxglove/**"
17+
- "rust/foxglove_derive/**"
18+
- "rust/examples/data_provider/**"
19+
- ".github/workflows/data_provider_smoke_test.yml"
20+
pull_request:
21+
paths:
22+
- "rust/foxglove/**"
23+
- "rust/foxglove_derive/**"
24+
- "rust/examples/data_provider/**"
25+
- ".github/workflows/data_provider_smoke_test.yml"
26+
27+
env:
28+
COMPOSE_FILE: rust/examples/data_provider/smoke-test/docker-compose.yml
29+
MCAP_CLI_VERSION: "0.0.61"
30+
31+
jobs:
32+
smoke-test:
33+
runs-on: ubuntu-latest
34+
timeout-minutes: 30
35+
steps:
36+
- uses: actions/checkout@v6
37+
with:
38+
lfs: true
39+
40+
# -- Install the mcap CLI --------------------------------------------------------------------
41+
- name: Install mcap CLI
42+
run: |
43+
curl -sfL \
44+
"https://github.com/foxglove/mcap/releases/download/releases/mcap-cli/v${MCAP_CLI_VERSION}/mcap-linux-amd64" \
45+
-o /usr/local/bin/mcap
46+
chmod +x /usr/local/bin/mcap
47+
mcap version
48+
49+
# -- Build and start all services ------------------------------------------------------------
50+
- name: Build and start services
51+
run: docker compose up --build -d
52+
53+
- name: Wait for data-provider to be healthy
54+
run: |
55+
# The data-provider has a Docker healthcheck (it has curl inside).
56+
# The remote-data-loader image is minimal and has no in-container
57+
# health tooling; its readiness is checked by run.sh from the host.
58+
echo "Waiting for data-provider ..."
59+
timeout 300 bash -c '
60+
while true; do
61+
health=$(docker compose ps data-provider --format json | jq -r ".Health" 2>/dev/null)
62+
if [ "$health" = "healthy" ]; then
63+
echo "data-provider is healthy."
64+
break
65+
fi
66+
echo " data-provider: $health"
67+
sleep 5
68+
done
69+
'
70+
docker compose ps
71+
72+
# -- Run the smoke test ----------------------------------------------------------------------
73+
- name: Run smoke test
74+
env:
75+
REMOTE_DATA_LOADER_URL: "http://localhost:8080"
76+
DATA_PROVIDER_URL: "http://localhost:8081"
77+
run: rust/examples/data_provider/smoke-test/run.sh
78+
79+
# -- Cleanup (always) ------------------------------------------------------------------------
80+
- name: Dump service logs
81+
if: always()
82+
run: docker compose logs
83+
84+
- name: Tear down services
85+
if: always()
86+
run: docker compose down -v

‎rust/examples/data_provider/smoke-test/Dockerfile.data-provider‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Dockerfile for the example data provider server.
2+
# Build from the workspace root:
3+
# docker build -f rust/examples/data_provider/smoke-test/Dockerfile.data-provider .
4+
5+
FROM rust:1-bookworm AS builder
6+
7+
WORKDIR /app
8+
COPY . .
9+
10+
RUN cargo build --release -p example_data_provider
11+
12+
FROM debian:bookworm-slim
13+
RUN apt-get update && apt-get install -y --no-install-recommends curl && rm -rf /var/lib/apt/lists/*
14+
COPY --from=builder /app/target/release/example_data_provider /usr/local/bin/
15+
EXPOSE 8080
16+
CMD ["example_data_provider"]

‎rust/examples/data_provider/smoke-test/docker-compose.yml‎

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
# Docker Compose for smoke-testing the example data provider with remote-data-loader.
2+
#
3+
# Usage (from the repo root):
4+
# docker compose -f rust/examples/data_provider/smoke-test/docker-compose.yml up --build -d
5+
#
6+
# Services:
7+
# data-provider – The example data provider server (built from this repo).
8+
# minio – S3-compatible object store used as the remote-data-loader cache.
9+
# minio-init – One-shot container that creates the cache bucket in MinIO.
10+
# remote-data-loader – Foxglove remote-data-loader, pointed at the data provider.
11+
12+
services:
13+
data-provider:
14+
build:
15+
context: ../../../..
16+
dockerfile: rust/examples/data_provider/smoke-test/Dockerfile.data-provider
17+
ports:
18+
- "8081:8080"
19+
healthcheck:
20+
test:
21+
[
22+
"CMD",
23+
"curl",
24+
"-sf",
25+
"-H",
26+
"Authorization: Bearer test",
27+
"http://localhost:8080/v1/manifest?flightId=test&startTime=2024-01-01T00:00:00Z&endTime=2024-01-01T00:00:01Z",
28+
]
29+
interval: 5s
30+
timeout: 5s
31+
retries: 30
32+
start_period: 5s
33+
34+
minio:
35+
image: minio/minio:latest
36+
command: ["server", "--console-address", ":9001", "/data"]
37+
environment:
38+
MINIO_ROOT_USER: root
39+
MINIO_ROOT_PASSWORD: password
40+
healthcheck:
41+
test: ["CMD", "curl", "-sf", "http://localhost:9000/minio/health/live"]
42+
interval: 5s
43+
timeout: 5s
44+
retries: 10
45+
46+
minio-init:
47+
image: minio/mc:latest
48+
depends_on:
49+
minio:
50+
condition: service_healthy
51+
restart: on-failure
52+
entrypoint: >
53+
/bin/sh -c "
54+
/usr/bin/mc alias set minio http://minio:9000 root password;
55+
/usr/bin/mc mb --ignore-existing minio/foxglove-cache;
56+
exit 0;
57+
"
58+
59+
remote-data-loader:
60+
# This image is public; no registry credentials are needed.
61+
# Tags are commit hashes from the foxglove/data-platform repo.
62+
# The tag below matches the appVersion in the foxglove/helm-charts
63+
# remote-data-loader chart.
64+
image: ${REMOTE_DATA_LOADER_IMAGE:-us-central1-docker.pkg.dev/foxglove-images/images/remote-data-loader:24e79cafd7f2fbe44e6e5b14b7c96590e2491a74}
65+
depends_on:
66+
data-provider:
67+
condition: service_healthy
68+
minio-init:
69+
condition: service_completed_successfully
70+
ports:
71+
- "8080:8080"
72+
environment:
73+
MANIFEST_ENDPOINT: http://data-provider:8080/v1/manifest
74+
ADDRESS: "0.0.0.0:8080"
75+
REQUEST_CACHE_STORAGE_PROVIDER: s3_compatible
76+
REQUEST_CACHE_BUCKET_NAME: foxglove-cache
77+
S3_COMPATIBLE_ACCESS_KEY_ID: root
78+
S3_COMPATIBLE_SECRET_ACCESS_KEY: password
79+
S3_COMPATIBLE_SERVICE_REGION: us-east-1
80+
S3_COMPATIBLE_SERVICE_URL: http://minio:9000
81+
# Token the remote-data-loader sends to the upstream data provider.
82+
BEARER_TOKEN: test-token
83+
RUST_LOG: info
84+
# No Docker healthcheck: the image is minimal (no curl/wget inside).
85+
# The smoke-test script checks /liveness from the host instead.

0 commit comments

Comments
 (0)