Skip to content

Model batching v2 #195

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

adenzler-nvidia
Copy link
Collaborator

replaces #109 - annoying not to keep the discussion there..

This now does the modulo approach with helpers.

@adenzler-nvidia adenzler-nvidia mentioned this pull request Apr 28, 2025
@adenzler-nvidia
Copy link
Collaborator Author

Plan after offline discussion @erikfrey:

  • let's forget about the modulo indexing and just have all the batched fields be size nworlds, and have the user do repeat data for now. But only expands fields to have different data for different model batches.
  • use the kernel analyzer to enforce correctness around what field can be expanded and what can't.
  • wai with this until [WIP] API changes to address multiple issues. #148 is merged.

@adenzler-nvidia
Copy link
Collaborator Author

PR is updated to remove the helper function and just use either stride 0 or an array that has nworld as first dimension, with users responsible for filling that array correctly.

@erikfrey I consider this ready for review - I mostly need input on what fields should be batchable.

@btaba - do you by any chance have an example learning workflow around where I could test-drive this?

My main worry right now is that I don't see a good way uf unittesting this, because we are just replacing arrays at runtime. We would either have to validate shapes at runtime (expensive) or add another layer/help that is mandatory.

@adenzler-nvidia
Copy link
Collaborator Author

closing in favor of #231

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant