Dataloader reinsertion for recursive predictionsrs #20336
leonardcaquot94
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
I'm working with recursive predictions of variable lengths (e.g., sequence predictions) and exploring two approaches for handling backpropagation:
training_step
, I predict the entire sequence recursively, accumulate the losses at each iteration, and return a single loss at the end of the loop. I then perform one update for the entire recursive prediction.training_step
, I predict only one step of the recursive process, allowing each step to trigger an update.To implement this second approach, I need a way to reinsert partially completed predictions back into the dataloader to continue processing them in future batches. Since the sequences I’m predicting have a maximum length, there’s no risk of endlessly reinserting them—they will either reach their max size or the end-of-sequence token will be predicted.
Is there a straightforward way to implement this using PyTorch Lightning?
Beta Was this translation helpful? Give feedback.
All reactions