Benchmark agentic query scenarios against the current metadata service

[valayDave]

AI coding agents interacting with Metaflow perform a predictable set of operations: finding failed runs, diagnosing task errors, filtering runs by time window, and searching for artifacts. The metadata service currently handles all of these through generic hierarchical traversal. There is no server-side filtering by status, time range, or artifact properties.

Before any improvements can be made, there needs to be a clear baseline: for each realistic agent scenario, how many API calls are made, how many DB queries are triggered, and how does this scale with the size of the run? The instrumentation introduced in issues #4 and #5 is the measurement layer. This issue is about building the scenarios and running them.

The five scenarios to cover:

1. Failed task discovery — find all failed tasks in a run, at varying parallelism (e.g. 5 / 50 / 200 parallel tasks)
2. Log retrieval — fetch the stderr of a failed task; record total bytes transferred vs. actual error content size
3. Status-filtered run listing — list only failed runs from a flow's history; record how many total runs are fetched to achieve this client-side
4. Time-horizon queries — list all runs and tasks within a specific time window (e.g. last 24 hours); ts_epoch is stored in the DB but is not a filterable parameter at the API level — measure the full traversal cost of doing this today
5. Cross-run artifact search — find which runs produced a specific named artifact; record traversal depth and total requests

Goals

• Write a reproducible benchmark suite (scripts + test data setup) covering all five scenarios
• For each scenario, report: total API calls, total DB queries, total rows returned, wall time — using the instrumentation from issues Instrument the Metaflow Client to trace metadata requests #4 and Instrument the metadata service to trace database queries per request #5
• Show how each metric scales with run/task count where applicable
• Identify which scenarios degrade most severely with scale

Instructions

You will need the dev stack running with test flows at varying scales. Include a setup script or instructions for generating the test data (a flow with N parallel tasks, known failures, predictable artifact names).

This issue is about measurement, not solutions. The output is a results table and a short written analysis of where the worst bottlenecks are.

When you have met the goals, share your benchmark scripts and results table via a fork as a message on this issue thread.

Resources

Dev stack:

• Setting Up the Dev Stack — metaflow-dev up spins up the full environment; metaflow-dev shell drops you into a shell already configured to talk to it — run your test flows from there to generate benchmark data at varying scales
• Swagger UI at http://localhost:8080/api/doc — useful for inspecting raw endpoint responses alongside benchmark output

Key code — client side:

• metaflow/client/core.py#L2541 — Flow.runs(), entry point for scenario 3
• metaflow/client/core.py#L1549 — Task.successful, how failure status is determined today
• metaflow/client/core.py#L1638 — Task.stdout / Task.stderr, relevant for scenario 2
• metaflow/metadata_provider/metadata.py#L352 — get_object(), the instrumentation hook from issue Instrument the Metaflow Client to trace metadata requests #4

Key code — service side:

• services/data/postgres_async_db.py#L248 — execute_sql(), the instrumentation hook from issue Instrument the metadata service to trace database queries per request #5
• services/data/models.py — note ts_epoch on RunRow and TaskRow, relevant for scenario 4 (time-horizon queries)
• services/metadata_service/api/task.py — includes the existing filtered_tasks endpoint; compare its pattern to what scenarios 1 and 3 need

Docs:

• Setting Up the Dev Stack
• Metaflow Client API reference
• Service architecture overview

[Youssef-SH]

@valayDave I built out the benchmark suite for this issue. Instrumented ServiceMetadataProvider._request on the client side and added /__audit/* middleware on the service side for per-phase counter resets, so client and service counts are tracked independently. Ran five of the six scenarios across 10/100/500-task foreach runs with a cold-cache harness and K=5 run history, Postgres volume wiped before each primary dataset.

Summary across all scenarios:

Use case	10 tasks	100 tasks	500 tasks	Dominant scaling factor
Status filter	C=33, S=35	C=33, S=35	C=33, S=33	#runs in history
Time filter	C=3, S=3	C=3, S=3	C=3, S=3	#runs in history (bytes only)
Failed tasks	C=15, S=15	C=105, S=105	C=505, S=505	#tasks, linear
Artifact metadata	C=7, S=7	C=7, S=7	C=7, S=7	Fixed, task scope
Log tail	C=12, S=12	C=102, S=102	C=502, S=502	#tasks, linear
Artifact search	C=10, S=10	C=10, S=10	C=10, S=10	Bounded, max_runs=1

Failed-task detection and log retrieval both scale linearly with task count, and they compound badly when chained: a cold-cache "find and explain the failing run" across 500 tasks runs to C=1040 total requests. The artifact search stays flat only because of a max_runs=1 guard in the harness; without it the traversal is O(runs x steps x tasks) and unbounded.

Full write-up with call-chain traces, per-scenario byte measurements, and detailed scaling analysis is here: https://gist.github.com/Youssef-SH/34679e867ebcf396761ea270b121d330

Happy to walk through any of it or adjust the harness design based on what the real issues end up needing.

[Youssef-SH]

Hey @valayDave, following up on the benchmark/audit I posted earlier.

I implemented a fix for the log-tail case. Right now task.stderr ends up calling load_bytes() unconditionally in task_datastore.py:1038, so it downloads the full log even when only the tail is needed.

This version uses a seek-from-end approach for the local datastore to read only the required tail portion. Cloud backends can follow the same idea via the existing tail readers (S3Tail / AzureTail / GSTail); I have a corresponding implementation for those as well if useful.

Diff:
Youssef-SH/metaflow@db2283ce

Can't open a PR on Netflix/metaflow due to the contribution restriction, happy to share it in whatever way works best.

[Youssef-SH]

Following up on the failed-task scenario from the audit.

I implemented a bounded client-side helper for failed-task discovery. Right now using task.successful ends up resolving _success and reading the blob per task, which is what drives the linear cost measured (for example 505 requests at 500 tasks).

This version just checks for _success descriptor existence via task["_success"] and never touches .data. It also adds simple bounds: max_tasks to cap the scan and stop_after to exit early once enough failures are found.

So instead of O(N) metadata plus O(N) datastore reads, it becomes O(K) metadata requests with no datastore access, where K is bounded.

It stays purely on the client side and assumes a completed run. Tasks without _success are treated as failed under that assumption.

Diff:
Youssef-SH/metaflow@06695ce

[guts27]

While studying the code for issue #6, I noticed that _request(), _version(), and compute_info() in service.py use unbounded
exponential backoff (sleep(2**i)) without jitter.

From my experience with Celery-based systems in production, this pattern can cause issues for agent workloads

The Problems

Thundering Herd: If many agents get a 503 error at the same time, they all wait for the exact same time and retry together. This attacks the recovering server again.

Wasted Sleep at the End:Even though it throws an exception on the last retry anyway, it waits 16 seconds before that.

Benchmark Results

[BEFORE]

• compute_info(): avg 15.55s, spread 0.05s
• _request(): avg 31.05s, spread 0.04s
• _version(): avg 31.06s, spread 0.04s

[AFTER]

(Jitter applied)

• compute_info(): avg 7.51s, spread 4.47s
• _request(): avg 14.60s, spread 8.97s
• _version(): avg 15.41s, spread 10.23s

Changes

Total blocking time: 31.05s → 14.60s (53% faster to fail safely)

Thundering herd spread: 0.04s → 8.97s (Retries are nicely distributed)

Changes Made

Added SERVICE_RETRY_WAIT_MAX (default 30s) in metaflow_config.py.

Added a max cap limit: min(2**i, SERVICE_RETRY_WAIT_MAX).

Added Jitter: Used random.uniform(0.5, 1.5) to spread out the retry times.

Skipped the final sleep: Added i < SERVICE_RETRY_COUNT - 1 so it fails fast without waiting an extra 16 seconds.

Notes

The max cap only works if a user sets the retry count to 6 or more.

random is already imported in this file.

@valayDave I would really appreciate your review on this when you have a moment.
Fork: https://github.com/guts27/metaflow

[Aryan95614]

Youssef's client-side call counts are useful baseline data, especially the linear scaling on failed-task and log-tail scenarios.

I can add the server-side half. I've been running pagination and tag filtering benchmarks against real Postgres with 10K seeded runs. The scenarios where server-side changes make a difference are the collection-level queries:

Scenario 3 (run listing with filters):

Query	Rows	Payload	Latency
Unbounded GET /runs	10,000	2,258 KB	65ms
+ _tags=env:prod (GIN index)	~3,300	~700 KB	28ms
+ _limit=50 (cursor pagination)	50	11 KB	3ms

Scenario 4 (time-horizon):
Cursor on ts_epoch bounds rows and bytes per page. Page 100 at limit=50: 0.9ms, same 11 KB payload as page 1.

Scenarios 1/2 (failed tasks, log tail) are per-task fan-out and don't benefit from pagination. Youssef's client-side bounds (stop_after, seek-from-end) are the right fix there.

My metadata service PRs (saikonen/metaflow-service #9, #10) implement the server-side pagination that bounds these collection responses. Sakari suggested combining both projects into one proposal since the service improvements are a direct dependency.

Question for @valayDave: for agent consumers, should paginated responses use a JSON envelope with cursor metadata, or keep cursors in HTTP headers? Automated clients typically parse bodies more naturally than headers.