The problem of changing data shape.

[houjanet]

Hello~ I have a question on different data shape to train convlstm model.

If the original data shape is [20, 10000, 64, 64], with the first 10 frames as input and the last 10 frames as predictions, and I want to change it so that the first 15 frames are input and the last 5 frames are predictions, how should I modify the code?

code:
def main():
###
# Common Params
###
artifact_dir = "./test_label_frame2"
input_seq_length = 15
train_batch_size = 32
validation_bath_size = 16
###
# Setup Pipeline
###
model_params: Seq2SeqParams = {
"input_seq_length": input_seq_length,
"num_layers": 2,
"num_kernels": 64,
"return_sequences": False,
"convlstm_params": {
"in_channels": 1,
"out_channels": 1,
"kernel_size": (3, 3),
"padding": "same",
"activation": "relu",
"frame_size": (64, 64),
"weights_initializer": WeightsInitializer.He,
},
}
erorr:
Traceback (most recent call last):

File ~/anaconda3/envs/convlstm/lib/python3.11/site-packages/spyder_kernels/py3compat.py:356 in compat_exec
exec(code, globals, locals)

File /data/ConvLSTM-test/examples/moving_mnist_convlstm.py:72
main()

File /data/ConvLSTM-test/examples/moving_mnist_convlstm.py:68 in main
experimenter.run()

File /data/ConvLSTM-test/pipelines/experimenter.py:44 in run
self.__evaluate()

File /data/ConvLSTM-test/pipelines/experimenter.py:74 in __evaluate
evaluator.run()

File /data/ConvLSTM-test/pipelines/evaluator.py:34 in run
pred_frames = self.__predict_frames(input, label) # 對輸入預測

File /data/ConvLSTM-test/pipelines/evaluator.py:63 in __predict_frames
pred_frames[:, :, frame_idx] = self.model(

IndexError: index 5 is out of bounds for dimension 2 with size 5

[tsugumi-sys]

@houjanet
I need to add label_seq_length parameter and change models code.
Please wait a few days, I'll fix it:)

[tsugumi-sys]

@houjanet
Now, you can use label_seq_length for Seq2Seq model (ConvLSTM). Please try it :)
I'll also support for the other models.

SA-ConvLSTM-Pytorch/convlstm/seq2seq.py

Line 15 in fd8a460

label_seq_length: NotRequired[Optional[int]]

[houjanet]

error:

Traceback (most recent call last):

File ~/anaconda3/envs/convlstm/lib/python3.11/site-packages/spyder_kernels/py3compat.py:356 in compat_exec
exec(code, globals, locals)

File /data/SA-ConvLSTM-Pytorch-main/examples/moving_mnist_convlstm.py:71
main()

File /data/SA-ConvLSTM-Pytorch-main/examples/moving_mnist_convlstm.py:67 in main
experimenter.run()

File /data/SA-ConvLSTM-Pytorch-main/pipelines/experimenter.py:43 in run
self.__evaluate()

File /data/SA-ConvLSTM-Pytorch-main/pipelines/experimenter.py:70 in __evaluate
evaluator.run()

File /data/SA-ConvLSTM-Pytorch-main/pipelines/evaluator.py:33 in run
pred_frames = self.__predict_frames(input, label)

File /data/SA-ConvLSTM-Pytorch-main/pipelines/evaluator.py:60 in __predict_frames
pred_frames[:, :, frame_idx] = self.model(

IndexError: index 5 is out of bounds for dimension 2 with size 5

[tsugumi-sys]

@houjanet Thanks for your error report. Could you share the whole code so that I can reproduce the same error?

[houjanet]

moving_mnist_convlstm.py code:

import os
os.chdir('/data/SA-ConvLSTM-Pytorch-main')
print(os.getcwd())

from torch import nn
from torch.optim import Adam

from convlstm.seq2seq import Seq2Seq, Seq2SeqParams
from core.constants import WeightsInitializer
from data_loaders.moving_mnist import MovingMNISTDataLoaders
from pipelines.experimenter import Experimenter
from pipelines.trainer import TrainingParams
from pipelines.utils.early_stopping import EarlyStopping


def main():
    ###
    # Common Params
    ###
    artifact_dir = "./test"
    **input_seq_length = 15**
    train_batch_size = 32
    validation_bath_size = 16
    ###
    # Setup Pipeline
    ###
    model_params: Seq2SeqParams = {
        "input_seq_length": input_seq_length,
        "num_layers": 2,
        "num_kernels": 64,
        "return_sequences": False,
        "convlstm_params": {
            "in_channels": 1,
            "out_channels": 1,
            "kernel_size": (3, 3),
            "padding": "same",
            "activation": "relu",
            "frame_size": (64, 64),
            "weights_initializer": WeightsInitializer.He,
        },
    }

    model = Seq2Seq(**model_params)

    training_params: TrainingParams = {
        "epochs": 1,
        "loss_criterion": nn.BCELoss(reduction="sum"),
        "accuracy_criterion": nn.L1Loss(),
        "optimizer": Adam(model.parameters(), lr=1e-4),
        "early_stopping": EarlyStopping(
            patience=30,
            verbose=True,
            delta=0.0001,
        ),
        "metrics_filename": "metrics.csv",
    }

    print("Loading dataset ...")
    data_loaders = MovingMNISTDataLoaders(
        train_batch_size=train_batch_size,
        validation_batch_size=validation_bath_size,
        input_frames=model_params["input_seq_length"],
        label_frames=1,
        split_ratios=[0.7, 0.299, 0.001],
    )

    experimenter = Experimenter(artifact_dir, data_loaders, model, training_params)
    experimenter.run()


if __name__ == "__main__":
    main()

I only changed this part: input_seq_length = 15, and the error would be:

Traceback (most recent call last):

  File ~/anaconda3/envs/convlstm/lib/python3.11/site-packages/spyder_kernels/py3compat.py:356 in compat_exec
    exec(code, globals, locals)

  File /data/SA-ConvLSTM-Pytorch-main/examples/moving_mnist_convlstm.py:72
    main()

  File /data/SA-ConvLSTM-Pytorch-main/examples/moving_mnist_convlstm.py:68 in main
    experimenter.run()

  File /data/SA-ConvLSTM-Pytorch-main/pipelines/experimenter.py:43 in run
    self.__evaluate()

  File /data/SA-ConvLSTM-Pytorch-main/pipelines/experimenter.py:70 in __evaluate
    evaluator.run()

  File /data/SA-ConvLSTM-Pytorch-main/pipelines/evaluator.py:33 in run
    pred_frames = self.__predict_frames(input, label)

  File /data/SA-ConvLSTM-Pytorch-main/pipelines/evaluator.py:60 in __predict_frames
    pred_frames[:, :, frame_idx] = self.model(

IndexError: index 5 is out of bounds for dimension 2 with size 5

[tsugumi-sys]

@houjanet
Thanks! You can try this? I added label_seq_lengh and modified the label_frames argument of the MovingMNIST data loader.

input_seq_length = 15
label_seq_length = 5
model_params: Seq2SeqParams = {
        "input_seq_length": input_seq_length,
        "label_seq_length": label_seq_length,
        "num_layers": 2,
        "num_kernels": 64,
        "return_sequences": False,
        "convlstm_params": {
            "in_channels": 1,
            "out_channels": 1,
            "kernel_size": (3, 3),
            "padding": "same",
            "activation": "relu",
            "frame_size": (64, 64),
            "weights_initializer": WeightsInitializer.He,
        },
    }

    model = Seq2Seq(**model_params)

    training_params: TrainingParams = {
        "epochs": 1,
        "loss_criterion": nn.BCELoss(reduction="sum"),
        "accuracy_criterion": nn.L1Loss(),
        "optimizer": Adam(model.parameters(), lr=1e-4),
        "early_stopping": EarlyStopping(
            patience=30,
            verbose=True,
            delta=0.0001,
        ),
        "metrics_filename": "metrics.csv",
    }

    print("Loading dataset ...")
    data_loaders = MovingMNISTDataLoaders(
        train_batch_size=train_batch_size,
        validation_batch_size=validation_bath_size,
        input_frames=model_params["input_seq_length"],
        label_frames=model_params["input_seq_length"],
        split_ratios=[0.7, 0.299, 0.001],
    )

[houjanet]

I tried this code, and the error changed to:

Traceback (most recent call last):

  File ~/anaconda3/envs/convlstm/lib/python3.11/site-packages/spyder_kernels/py3compat.py:356 in compat_exec
    exec(code, globals, locals)

  File /data/SA-ConvLSTM-Pytorch-main/examples/moving_mnist_convlstm.py:71
    main()

  File /data/SA-ConvLSTM-Pytorch-main/examples/moving_mnist_convlstm.py:67 in main
    experimenter.run()

  File /data/SA-ConvLSTM-Pytorch-main/pipelines/experimenter.py:43 in run
    self.__evaluate()

  File /data/SA-ConvLSTM-Pytorch-main/pipelines/experimenter.py:70 in __evaluate
    evaluator.run()

  File /data/SA-ConvLSTM-Pytorch-main/pipelines/evaluator.py:33 in run
    pred_frames = self.__predict_frames(input, label)

  File /data/SA-ConvLSTM-Pytorch-main/pipelines/evaluator.py:58 in __predict_frames
    pred_frames[:, :, frame_idx] = self.model(input)

RuntimeError: The expanded size of the tensor (1) must match the existing size (5) at non-singleton dimension 1.  Target sizes: [1, 1, 64, 64].  Tensor sizes: [5, 64, 64]

[tsugumi-sys]

@houjanet
hmm, are you using your custom dataset?
If so, I want to know more detail code about it

[houjanet]

No, I just used the original moving_mnist dataset to try input_seq_length = 15 and label_seq_length = 5