Reproduction trouble: how to prepare images_flow

[kazunosuke1108]

Hello, I'm trying to reproduce your research results.

When I download EGTEA Gaze+ dataset, I found that it doesn't contain rgb images & flow images.
So I created them by using denseflow, which is reccommended to use in another discussion.

After creating images in dataset/images_flow or images_rgb, I ran the code below as instructed.
python3 main.py --mode test

However, the error occured and failed to reproduce your result.

datasplit:     1
weight:        weights/i3d_iga_best1_base.pt
mode:          test
test_sparse:   False
loading weight file: weights/i3d_iga_best1_base.pt
loading weight file: weights/i3d_iga_best1_gaze.pt
loading weight file: weights/i3d_iga_best1_attn.pt
run on cuda
[ WARN:0@1.801] global loadsave.cpp:244 findDecoder imread_('dataset/images_flow/P05-R01-PastaSalad-160540-162131-F003848-F003896/u/0048.jpg'): can't open/read file: check file path/integrity
[ WARN:0@1.801] global loadsave.cpp:244 findDecoder imread_('dataset/images_flow/P05-R01-PastaSalad-160540-162131-F003848-F003896/v/0048.jpg'): can't open/read file: check file path/integrity
[ WARN:0@1.811] global loadsave.cpp:244 findDecoder imread_('dataset/images_flow/P05-R01-PastaSalad-169741-171463-F004069-F004120/u/0051.jpg'): can't open/read file: check file path/integrity
[ WARN:0@1.811] global loadsave.cpp:244 findDecoder imread_('dataset/images_flow/P05-R01-PastaSalad-169741-171463-F004069-F004120/v/0051.jpg'): can't open/read file: check file path/integrity
[ WARN:0@1.817] global loadsave.cpp:244 findDecoder imread_('dataset/images_flow/P04-R06-GreekSalad-682170-683940-F016368-F016419/u/0051.jpg'): can't open/read file: check file path/integrity
[ WARN:0@1.817] global loadsave.cpp:244 findDecoder imread_('dataset/images_flow/P04-R06-GreekSalad-682170-683940-F016368-F016419/v/0051.jpg'): can't open/read file: check file path/integrity
[ WARN:0@1.832] global loadsave.cpp:244 findDecoder imread_('dataset/images_flow/P04-R06-GreekSalad-767250-769130-F018410-F018464/u/0054.jpg'): can't open/read file: check file path/integrity
[ WARN:0@1.832] global loadsave.cpp:244 findDecoder imread_('dataset/images_flow/P04-R06-GreekSalad-767250-769130-F018410-F018464/v/0054.jpg'): can't open/read file: check file path/integrity
Traceback (most recent call last):
  File "main.py", line 261, in <module>
    main()
  File "main.py", line 85, in main
    test(test_loader, model_base, model_gaze, model_attn, num_action)
  File "main.py", line 222, in test
    for i, (rgb, flow, label) in enumerate(test_loader, 1):
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/dataloader.py", line 681, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/dataloader.py", line 1376, in _next_data
    return self._process_data(data)
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/dataloader.py", line 1402, in _process_data
    data.reraise()
  File "/usr/local/lib/python3.8/dist-packages/torch/_utils.py", line 461, in reraise
    raise exception
TypeError: Caught TypeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/hayashide/catkin_ws/src/third_party/Gaze-Attention/dataset.py", line 67, in __getitem__
    fimg = np.concatenate((fimgu[..., np.newaxis], fimgv[..., np.newaxis]), -1)
TypeError: 'NoneType' object is not subscriptable

Please give me some advice to resolve this issue.
Thanks,

[kylemin]

The error message is saying that your code did not load the flow images properly because it couldn't find them.
[ WARN:0@1.801] global loadsave.cpp:244 findDecoder imread_('dataset/images_flow/P05-R01-PastaSalad-160540-162131-F003848-F003896/u/0048.jpg'): can't open/read file: check file path/integrity -> It says your code cannot find u/0048.jpg.
TypeError: 'NoneType' object is not subscriptable -> It says fimgu or fimg is None because the code fails to load the files.

Are you sure you put the flow images into the correct directories (dataset/images_flow/clip_name/{u or v})?

[kazunosuke1108]

Thank you for your advice.

Yes I checked that there're opticalflow images. However, there aren't enough images. For example, as for P05-R01-PastaSalad-160540-162131-F003848-F003896/u, 0047.jpg is existing, but 0048.jpg is not although I just used denseflow. If there're someone who created opticalflow by hand and everything goes fine, I'd like to know how they do this.