Allow users to provide a custom video background

[Ahuge]

This PR allows users to bring their own background image/imageSequence/video instead of the default green background.

It creates a new parameter in convert_video called bgr_source: Optional[str] = None that, when passed will attempt to load either an ImageSequenceReader, VideoReader, or an ImageReader.

I have moved the green background into a dummy class called ConstantImage which implements the Dataset interface so that it can be indexed into per frame.

[Ahuge]

I don't know if you are accepting PRs on this repo. I just found this functionality useful and wanted to offer it upstream

[PeterL1n]

Right now I am very busy with other projects. I won't have time to review those pull requests. So I am just gonna leave them open for now.

[Ahuge]

All good, thanks for the awesome research!

[chenying99]

thank you for your share

1. the file inference.py line in 123 125:
   if (output_composition is not None) and (output_type == 'video'):
   if bgr_source is not None and os.path.isfile(bgr_source):
   if os.path.isfile(bgr_source):
   if os.path.splitext(bgr_source)[-1].lower() in [".png", ".jpg"]:
   bgr_raw = ImageReader(bgr_source, transform=transform)
   else:
   bgr_raw = VideoReader(bgr_source, transform)
   else:
   bgr_raw = ImageSequenceReader(bgr_source, transform)
   else:
   bgr_raw = ConstantImage(120, 255, 155, device=device, dtype=dtype)

the bgr' size should resize, make it equal to the source' size

2. line in 148:
   bgr = bgr_raw[index].to(device, dtype, non_blocking=True).unsqueeze(0) # [B, T, C, H, W]

if seq_chunk>1, bgr shuld be batch, the batchsize equal to seq_chunk; and the last batch' batchsize is the remainder

3. len(bgr_raw) should be greater than len(source)，otherwise, it could throw exception
   the file inference_utils.py line 25:
   frame = self.video[idx]

[xiami1234567890]

thank you for your share

1. the file inference.py line in 123 125:
   if (output_composition is not None) and (output_type == 'video'):
   if bgr_source is not None and os.path.isfile(bgr_source):
   if os.path.isfile(bgr_source):
   if os.path.splitext(bgr_source)[-1].lower() in [".png", ".jpg"]:
   bgr_raw = ImageReader(bgr_source, transform=transform)
   else:
   bgr_raw = VideoReader(bgr_source, transform)
   else:
   bgr_raw = ImageSequenceReader(bgr_source, transform)
   else:
   bgr_raw = ConstantImage(120, 255, 155, device=device, dtype=dtype)

the bgr' size should resize, make it equal to the source' size

2. line in 148:
   bgr = bgr_raw[index].to(device, dtype, non_blocking=True).unsqueeze(0) # [B, T, C, H, W]

if seq_chunk>1, bgr shuld be batch, the batchsize equal to seq_chunk; and the last batch' batchsize is the remainder

3. len(bgr_raw) should be greater than len(source)，otherwise, it could throw exception
   the file inference_utils.py line 25:
   frame = self.video[idx]

thank you for your share! ! I have same question in it.When I input a background image, it always doesn't match. Do you have a solution?It would be a big breakthrough if you could help me......

[justforyourdream]

RuntimeError: The size of tensor a (3) must match the size of tensor b (4) at non-singleton dimension 2

[akashAD98]

@Ahuge @justforyourdream @chenying99 have you solved the issue? still not able to get results,
its not matching with input size,fps & duration of video ,how to fix it