Best practices for large-scale workflows with GCS storage and Google Batch executor

[mat10d]

Environment

• Snakemake version: 9.12.0
• Executor: snakemake-executor-plugin-googlebatch 0.6.0 (https://github.com/mboulton-fathom/snakemake-executor-plugin-googlebatch/tree/feat/allow-custom-containers)
• Storage: snakemake-storage-plugin-gcs 1.1.4
• Workflow scale: 14,368 jobs
• All data on GCS: Input and output files at gs://scale1/qinling/

Question

I'm running a large-scale image analysis workflow (14,368 jobs) on Google Batch with all data stored on GCS. I'm experiencing severe overhead from file existence checks during both DAG building and job scheduling phases that makes the workflow impractical to run.

Current observations at full scale (14K jobs):

• DAG building: Takes multiple minutes (estimated~60 minutes minutes based on file check patterns). DAG build takes < 30 seconds on a single VM, if the files are stored on that VM
• Job scheduling overhead: Continuous file existence checks throughout execution. Selection of 500 jobs, for example takes __ minutes.
• Individual jobs: Complete quickly (minutes), but the workflow spends most time on overhead
• All file paths are GCS URLs: Every file check requires a GCS API call (~100-200ms each)
• Thousands of files: Each job has 8-32 input files, all requiring existence verification

The bottleneck: With thousands of jobs each checking multiple GCS files during both DAG construction and job selection, the file existence check overhead dominates workflow execution time.

My configuration:

default-storage-provider: gcs
default-storage-prefix: gs://scale1
not-retrieve-storage: false  # Check GCS during DAG build
shared-fs-usage:
  - persistence
  - source-cache
  - sources

Path structure question: My sample CSV files contain paths like:

qinling/input_ph/plate_1/round_1/0/A1_0_0_Fluorescence_405_nm_-_Penta.tiff

Which get combined with default-storage-prefix to create:

gs://scale1/qinling/input_ph/plate_1/round_1/0/A1_0_0_Fluorescence_405_nm_-_Penta.tiff

Question: Should I instead use:

• CSV paths: input_ph/... (relative from project root)
• Config: default-storage-prefix: gs://scale1/qinling
• Result: Same final paths, but cleaner separation of project prefix and file structure

Does the current approach (including qinling/ in every CSV path) cause any issues or contribute to overhead?

Specific Questions

1. Is not-retrieve-storage: false the right choice for GCS-heavy workflows?

   • Does this cause double-checking (once during DAG build, once during job scheduling)?
   • For workflows with 10K+ jobs, is it better to skip DAG-time checks and fail fast at runtime?

2. Should shared-fs-usage be configured differently for Google Batch?

   • Batch uses ephemeral containers with no shared filesystem between jobs
   • Is listing source-cache and sources causing unnecessary checks?

3. Have others experienced file existence check overhead at scale?

   • With thousands of GCS files, is there a recommended approach to minimize API calls?
   • Are there configuration settings specifically for large-scale remote storage workflows?

4. What are the recommended settings for:

   • --latency-wait
   • --max-inventory-time
   • --max-status-checks-per-second
   • seconds-between-status-checks

   ...when all files are on remote storage?

5. Is my path structure setup correct?

   • CSV files include project prefix: qinling/input_ph/...
   • Config has: default-storage-prefix: gs://scale1
   • Result: gs://scale1/qinling/input_ph/...

   Should I move qinling/ to the config instead (gs://scale1/qinling) and use purely relative paths in CSV files?

[mat10d]

Update: Performance Investigation Results

Test Results Summary

I've conducted extensive testing with a 1,798-job workflow (full well scale) on Google Batch with GCS storage to investigate solutions to the file existence check overhead problem.

Baseline Performance

• Configuration: Standard snakemake-storage-plugin-gcs v1.1.4, unmodified workflow
• Total time: 614 minutes (10 hours 14 minutes)
• Bottleneck: First job selection takes 131+ minutes due to thousands of individual GCS API calls
• Job selections: Only 11 rounds total (very slow polling)

This confirms the overhead is severe and impractical for large-scale workflows.

---

Optimization Approaches Tested

I tested three different optimization strategies:

Approach A: Job Scheduler Patch ✅ (Working Solution)

Configuration: Runtime Python patch that intercepts Snakemake's job selection phase

How it works:

1. Patches JobScheduler.job_selector() to run bulk GCS inventory before first job selection
2. Performs ONE list_blobs() operation to cache ~86K files (48 seconds)
3. Intercepts _IOFile.exists() and _IOFile.size() to check cache first
4. Falls back to GCS API calls for cache misses (newly created files)

Results:

• Total time: 253 minutes (4 hours 13 minutes)
• Improvement: 2.4x faster than baseline
• First job selection: 49.6 seconds (vs 131+ minutes baseline) = ~160x faster
• Job selection rounds: 3,628 rounds (much more responsive polling)
• Cache performance: 98% hit rate
• Status: ✅ Successfully completed all 1,798 jobs

Code: Available at gcs_job_scheduler_patch.py

Pros:

• Proven to work reliably at scale
• Significant performance improvement
• Non-invasive (doesn't modify installed packages)
• Easy to enable/disable via import

Cons:

• Still slower than ideal for very large workflows
• Monkey-patching approach (not integrated into Snakemake)

---

Approach B: Modified GCS Storage Plugin (Standalone)

Configuration: Modified snakemake-storage-plugin-gcs to perform bulk inventory at workflow start and remove negative caching, available at snakemake_storage_plugin_gcs/init.py.

How it works:

1. Modified inventory() method to do bulk list_blobs() at workflow root level
2. Added class-level cache variables to persist across StorageObject instances
3. Removed negative cache to allow discovery of newly created output files
4. Makes fresh API calls for files not in inventory cache

Results:

• Total time: 575 minutes (9 hours 35 minutes)
• Improvement: 1.07x faster than baseline (marginal)
• First job selection: 135+ minutes (similar to baseline)
• Additional API calls: 1,913 cache misses requiring fresh API calls
• Status: ✅ Completed all 1,798 jobs

Why slower?: Removing negative cache means output files are repeatedly checked via API calls until they exist, causing 11x more API calls than Approach A during execution.

---

Approach C: Combined (Job Scheduler Patch + Modified GCS Plugin) ⚠️ (Promising but Crashes)

Configuration: Both the job scheduler patch AND the modified GCS storage plugin enabled simultaneously

Results:

• Total time: 39 minutes for 705 jobs (39% completion)
• Projected completion: ~100 minutes (6x faster than baseline!)
• Status: ❌ CRASHED with error:

  KeyError: convert_phenotype
  TypeError: sequence item 0: expected str instance, Empty found
  

Why it crashes: Unknown - possible conflict between the two caching strategies or race condition. The patch's cache and the modified plugin's cache may interfere with each other.

Observation: Despite the crash, this approach was extremely fast - it completed 39% of the work in just 39 minutes, suggesting the combined approach could achieve ~6x improvement if the compatibility issue is resolved.

---

Performance Comparison Table

Configuration	Total Time	vs Baseline	First Job Selection	Job Selections	Status
Baseline	614 min	—	131+ min	11	✅
Job Scheduler Patch	253 min	2.4x faster	49.6 sec	3,628	✅
Modified GCS Plugin	575 min	1.07x faster	135+ min	12	✅
Combined	39 min (39% done)	~6x faster*	~40 sec	1,648	❌ Crashed

*Projected if it completed successfully

---

Key Insights

1. Job Scheduler Patch is a reliable solution: Provides 2.4x speedup with proven stability at scale

2. Combined approach shows great promise: The 39-minute result for 39% of work suggests potential 6x improvement, but crashes prevent full completion

3. Negative cache trade-off: Removing negative caching allows output file discovery but increases API calls significantly

4. Job selection is the bottleneck: Baseline takes 131+ minutes for first job selection; optimized approaches reduce this to ~50 seconds

[cademirch]

Its confusing to me why and how job selection is the bottleneck. The scheduler should only care about the available resources and if more jobs can be scheduled given those resources. While the solutions above seem like they work well, it doesn't appear to me to address the root cause.

I did a little digging, and it looks like this might be part of the issue:
https://github.com/snakemake/snakemake/blob/804be9f27b5a4e51a0807081833570af0fb52dbc/src/snakemake/scheduling/job_scheduler.py#L198-L210

The scheduler checking the size and existence of inputs does seem problematic, but I'm not sure how and when this is called. @johanneskoester can hopefully chime in.

To me, the ideal solution for this issue should happen in the IOCache (https://github.com/snakemake/snakemake/blob/main/src/snakemake/io/__init__.py). I'm not sure how/if storage plugins interact with the IOCache, though.

[mat10d]

@cademirch will keep discussion going on this here, but I just wanted to confirm, given that you have some experience with the googlebatch plugin, that my configuration (in first comment) isn't wildly off and causing some of these issues. Otherwise, will keep snakemake-related things in the snakemake issue I have opened. Thanks!

[cademirch]

@cademirch will keep discussion going on this here, but I just wanted to confirm, given that you have some experience with the googlebatch plugin, that my configuration (in first comment) isn't wildly off and causing some of these issues. Otherwise, will keep snakemake-related things in the snakemake issue I have opened. Thanks!

Unfortunately I have no real direct experience with Google Batch, plugin or otherwise. I did use Google life sciences a lot, but that was deprecated (by Google) in favor of batch.