Skip to content

Batching enhancements - #2029

Open
lbluque wants to merge 23 commits into
mainfrom
batching-enhancements
Open

Batching enhancements#2029
lbluque wants to merge 23 commits into
mainfrom
batching-enhancements

Conversation

@lbluque

@lbluque lbluque commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Summary

This branch extends the Ray Serve batched inference stack with runtime autobatch probing, broader concurrency backend support, and InferenceBatcher usability tweaks. It aligns the earlier batching-enhancements work with main's refactored batch_server layout, makes OOM splitting opt-in, and adds a configure-at-runtime API.

Main changes

src/fairchem/core/calculate/_batch.py — InferenceBatcher

  • Add RayActorPoolExecutor and extend _get_concurrency_backend to support threads, processes, and ray-actors. Executor choice is now documented per workload type.
  • New auto_configure_batching(probe_data, config) that runs probing and applies the result, plus exposed update_checkpoint, delete, shutdown lifecycle matching main.
  • split_oom_batch defaults to False, with updated docstrings explaining heterogeneous vs homogeneous trade-offs.
  • Expanded class and method docstrings with autobatch usage example.

src/fairchem/core/components/batch_server.py

  • Change default split_oom_batch=False on BatchConfig, BatchPredictServer, and MultiplexedBatchPredictServer; improve OOM error messages with guidance.
  • Add BatchPredictServerMixin.configure_batching(max_batch_size, batch_wait_timeout_s) for runtime tuning.
  • New autobatching API:
    • AutobatchConfig and AutobatchResult dataclasses
    • probe_optimal_batch_size(predict_unit, probe_data, config) — warmup then binary-search GPU memory to find max safe atom batch size, back off by factor, derive timeout from median latency. CPU path returns conservative defaults.
    • _expand_probe_data helper to repeat probe structures to target atom counts.

tests/core/calculate/test_batcher.py

  • Parametrize initialization test for new processes and ray-actors backends.
  • Add test_batched_atomic_data_predictions for direct AtomicData predict path.
  • Add test_autobatch_config_initialization exercising auto_configure on GPU fixture.
  • Add test_probe_optimal_batch_size_cpu verifying CPU fallback behavior.

Out of scope for the committed diff: local untracked examples/batching_inference_example.py and uncommitted config edits in configs/uma/...

lbluque and others added 12 commits January 26, 2026 19:14
- Replace _batch_serve.py with batch_server.py (matches main's rename)
- Move AutobatchConfig, AutobatchResult, and probe_optimal_batch_size
  into batch_server.py
- Add _PredictWorker and MLIPWorkerPredictUnit to predict.py
  (appended after main's BatchServerPredictUnit)
- Restore ProcessPoolExecutor and RayActorPoolExecutor backends
  in _batch.py alongside main's new features (uuid deployment names,
  autoscaling_config, update_checkpoint, delete, shutdown_ray)
- Adapt tests to main's fixture pattern (ray_session, ray_controlled)
  while preserving branch's backend and autobatch tests
# Conflicts:
#	src/fairchem/core/calculate/__init__.py
#	src/fairchem/core/calculate/_batch.py
#	src/fairchem/core/calculate/_batch_server.py
#	src/fairchem/core/components/batch_server.py
#	src/fairchem/core/units/mlip_unit/batch_server.py
#	tests/core/units/mlip_unit/test_inference_serve.py
#	tests/core/units/mlip_unit/test_predict.py
@lbluque
lbluque requested a review from zulissimeta June 10, 2026 04:03
@meta-cla meta-cla Bot added the cla signed label Jun 10, 2026
@lbluque
lbluque requested a review from misko June 10, 2026 04:04
@lbluque lbluque added enhancement New feature or request minor Minor version release labels Jun 10, 2026
@lbluque
lbluque requested a review from rayg1234 August 12, 2026 18:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla signed enhancement New feature or request minor Minor version release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant