Padding for RandomCrop does not match description in documentation

[brando90]

🐛 Describe the bug

RandomCrop does not behave as expected. It should add padding to all sides according to the docs but only puts it at the bottom:

I discovered & confirmed that by doing it on cifar. But note this NOT what the docs say for RandomCrop:

Optional padding on each border of the image. Default is None. If a single int is provided this is used to pad all borders.

it says something very similar to pad:

Padding on each border. If a single int is provided this is used to pad all borders.

def check_padding_random_crop_cifar_pure_torch():
    # -
    import sys
    print(f'python version: {sys.version=}')
    import torch
    print(f'{torch.__version__=}')
    # -
    from uutils.plot.image_visualization import visualize_pytorch_tensor_img
    from torchvision.transforms import RandomCrop

    # - for determinism
    import random
    random.seed(0)
    import torch
    torch.manual_seed(0)
    import numpy as np
    np.random.seed(0)

    # -
    from pathlib import Path
    root = Path('~/data/').expanduser()
    import torch
    import torchvision
    # - test tensor imgs
    from torchvision.transforms import Resize
    from torchvision.transforms import Pad
    from torchvision.transforms import ToTensor
    from torchvision.transforms import Compose

    # -- see if pad doubles length
    print(f'--- test padding doubles length with Pad(...)')
    transform = Compose([Resize((32, 32)), Pad(padding=4), ToTensor()])
    train = torchvision.datasets.CIFAR100(root=root, train=True, download=True,
                                          transform=transform,
                                          target_transform=lambda data: torch.tensor(data, dtype=torch.long))
    transform = Compose([Resize((32, 32)), Pad(padding=8), ToTensor()])
    test = torchvision.datasets.CIFAR100(root=root, train=True, download=True,
                                         transform=transform,
                                         target_transform=lambda data: torch.tensor(data, dtype=torch.long))
    # - test padding doubles length
    from torch.utils.data import DataLoader
    loader = DataLoader(train)
    x, y = next(iter(loader))
    print(f'{x[0].size()=}')
    assert x[0].size(2) == 32 + 4 * 2
    assert x[0].size(2) == 32 + 8
    visualize_pytorch_tensor_img(x[0], show_img_now=True)
    #
    loader = DataLoader(test)
    x, y = next(iter(loader))
    print(f'{x[0].size()=}')
    assert x.size(2) == 32 + 8 * 2
    assert x.size(2) == 32 + 16
    visualize_pytorch_tensor_img(x[0], show_img_now=True)

    # -- see if RandomCrop also puts the pad
    print(f'--- test RandomCrop indeed puts padding')
    transform = Compose([Resize((32, 32)), RandomCrop(28, padding=8), ToTensor()])
    train = torchvision.datasets.CIFAR100(root=root, train=True, download=True,
                                          transform=transform,
                                          target_transform=lambda data: torch.tensor(data, dtype=torch.long))
    transform = Compose([Resize((32, 32)), RandomCrop(28), ToTensor()])
    test = torchvision.datasets.CIFAR100(root=root, train=True, download=True,
                                         transform=transform,
                                         target_transform=lambda data: torch.tensor(data, dtype=torch.long))
    # - test that the padding is there visually
    from torch.utils.data import DataLoader
    loader = DataLoader(train)
    x, y = next(iter(loader))
    print(f'{x[0].size()=}')
    assert x[0].size(2) == 28
    visualize_pytorch_tensor_img(x[0], show_img_now=True)
    #
    loader = DataLoader(test)
    x, y = next(iter(loader))
    print(f'{x[0].size()=}')
    assert x.size(2) == 28
    visualize_pytorch_tensor_img(x[0], show_img_now=True

Versions

(meta_learning) brandomiranda~/diversity-for-predictive-success-of-meta-learning ❯ wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
--2022-11-17 17:00:22--  https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.111.133, 185.199.110.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 17278 (17K) [text/plain]
Saving to: ‘collect_env.py’

collect_env.py                                                                                                             100%[=======================================================================================================================================================================================================================================================================================================================================>]  16.87K  --.-KB/s    in 0s      

2022-11-17 17:00:23 (48.3 MB/s) - ‘collect_env.py’ saved [17278/17278]

zsh: command not found: #
Collecting environment information...
PyTorch version: 1.9.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 12.6 (x86_64)
GCC version: Could not collect
Clang version: 14.0.0 (clang-1400.0.29.102)
CMake version: Could not collect
Libc version: N/A

Python version: 3.9.7 (default, Sep 16 2021, 08:50:36)  [Clang 10.0.0 ] (64-bit runtime)
Python platform: macOS-10.16-x86_64-i386-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.1
[pip3] pytorch-transformers==1.2.0
[pip3] torch==1.9.1
[pip3] torchaudio==0.12.1
[pip3] torchmeta==1.8.0
[pip3] torchtext==0.13.1
[pip3] torchvision==0.10.1
[conda] mkl                       2021.4.0           hecd8cb5_637  
[conda] mkl-service               2.4.0            py39h9ed2024_0  
[conda] nomkl                     3.0                           0  
[conda] numpy                     1.21.4                   pypi_0    pypi
[conda] numpy-base                1.23.1           py39hbda7086_0  
[conda] pytorch-transformers      1.2.0                    pypi_0    pypi
[conda] torch                     1.9.1                    pypi_0    pypi
[conda] torchaudio                0.12.1                   pypi_0    pypi
[conda] torchmeta                 1.8.0                    pypi_0    pypi
[conda] torchtext                 0.13.1                   pypi_0    pypi
[conda] torchvision               0.10.1                   pypi_0    pypi

pytorch forum: https://discuss.pytorch.org/t/why-isnt-randomcrop-inserting-the-padding-in-pytorch/166244
SO: https://stackoverflow.com/questions/74482017/why-isnt-randomcrop-inserting-the-padding-in-pytorch

cc @datumbox @vfdev-5 @pmeier

[ptrblck]

RandomCrop is most likely working as expected. As seen in the code the padding is applied first to the input and the padded tensor/image is cropped afterwards:

        # pad the width if needed
        if self.pad_if_needed and width < self.size[1]:
            padding = [self.size[1] - width, 0]
            img = F.pad(img, padding, self.fill, self.padding_mode)
        # pad the height if needed
        if self.pad_if_needed and height < self.size[0]:
            padding = [0, self.size[0] - height]
            img = F.pad(img, padding, self.fill, self.padding_mode)

        i, j, h, w = self.get_params(img, self.size)

        return F.crop(img, i, j, h, w)

Based on this, it's expected to see the padded region on one side or corner of the output.

Double-posting these issues into the discussion board, here on GitHub, and StackOverflow isn't helpful as already explained a few times, as different users might be checking your claims at the same time wasting their time.

[brando90]

But the docs are super confusing.... Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: ptrblck ***@***.***> Sent: Thursday, November 17, 2022 9:57:58 PM To: pytorch/pytorch ***@***.***> Cc: Brando Miranda ***@***.***>; Author ***@***.***> Subject: Re: [pytorch/pytorch] Padding for RandomCrop does not match description in documentation (Issue pytorch/vision#6967) RandomCrop is most likely working as expected. As seen in the code<https://github.com/pytorch/vision/blob/d710f3d1edc06afa244468cb96603ba6dbd4d9d5/torchvision/transforms/transforms.py#L664-L675> the padding is applied first to the input and the padded tensor/image is cropped afterwards: # pad the width if needed if self.pad_if_needed and width < self.size[1]: padding = [self.size[1] - width, 0] img = F.pad(img, padding, self.fill, self.padding_mode) # pad the height if needed if self.pad_if_needed and height < self.size[0]: padding = [0, self.size[0] - height] img = F.pad(img, padding, self.fill, self.padding_mode) i, j, h, w = self.get_params(img, self.size) return F.crop(img, i, j, h, w) Based on this, it's expected to see the padded region on one side or corner of the output. Double-posting these issues into the discussion board, here on GitHub, and StackOverflow isn't helpful as already explained a few times, as different users might be checking your claims at the same time wasting their time. — Reply to this email directly, view it on GitHub<#6967>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAOE6LRS4ATMAVHHSRZJQCLWI4LGNANCNFSM6AAAAAASD66T7Q>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[ptrblck]

I don't think the docs are confusing and explain which operations are applied internally:

Optional padding on each border of the image.

Which I understand as:

• padding will be applied on each border given the specific padding size
• the input will be randomly cropped afterwards.

If you think the docs can be improved, I'm sure PRs are welcome.

Now to your use case and why you are not seeing the padding on all borders of the output:
take a look at my quick and dirty figure:

• blue: input image
• red: padding on each border
• black square: output

As you can see, given this setup it would be impossible to see all padding borders.
However, you can of course force it by using a small input image compared to the desired output shape and padding region:

crop = transforms.RandomCrop(size=(224, 224), padding=100)
x = transforms.ToPILImage()(torch.randn(1, 124, 124))
out = crop(x)

gives:

[brando90]

I think they are and explain in my Q why they are confusing.

…

On Nov 18, 2022, at 4:45 PM, ptrblck ***@***.***> wrote: I don't think the docs are confusing and explain which operations are applied internally: Optional padding on each border of the image. Which I understand as: padding will be applied on each border given the specific padding size the input will be randomly cropped afterwards. If you think the docs can be improved, I'm sure PRs are welcome. Now to your use case and why you are not seeing the padding on all borders of the output: take a look at my quick and dirty figure: <https://user-images.githubusercontent.com/11662379/202825144-eefb29cb-8f08-464b-9888-a89801b44efe.png> blue: input image red: padding on each border black square: output As you can see, given this setup it would be impossible to see all padding borders. However, you can of course force it by using a small input image compared to the desired output shape and padding region: crop = transforms.RandomCrop(size=(224, 224), padding=100) x = transforms.ToPILImage()(torch.randn(1, 124, 124)) out = crop(x) gives: <https://user-images.githubusercontent.com/11662379/202825219-01bfd27a-b037-404c-a25f-d6f02b52fb13.png> — Reply to this email directly, view it on GitHub <#6967>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOE6LRP6OPWII52ZDDDXU3WJAPLLANCNFSM6AAAAAASD66T7Q>. You are receiving this because you authored the thread.

[ptrblck]

I don't think you've explained the confusion, but posted this statement:

It should add padding to all sides according to the docs but only puts it at the bottom

which is wrong.
As shown in my figure and seen in the corresponding code, padding is added to all borders.
You can rerun this code snippet to see that the crop will be randomly applied and different borders will be padded:

crop = transforms.RandomCrop(size=(224, 224), padding=10)
x = transforms.ToPILImage()(torch.randn(1, 224, 224))
out = crop(x)
out.show()

[pmeier]

If all the confusion stems from the sentence

Optional padding on each border of the image.

feel free to send a PR that changes it to something along the lines

Optional padding on each border of the image before the cropping.

This is in line with the documentation of pad_if_needed:

It will pad the image [...]. Since cropping is done after padding, the padding seems to be done at a random offset.

[brando90]

I don't think you've explained the confusion, but posted this statement:

It should add padding to all sides according to the docs but only puts it at the bottom

which is wrong. As shown in my figure and seen in the corresponding code, padding is added to all borders. You can rerun this code snippet to see that the crop will be randomly applied and different borders will be padded:

crop = transforms.RandomCrop(size=(224, 224), padding=10)
x = transforms.ToPILImage()(torch.randn(1, 224, 224))
out = crop(x)
out.show()

Happy to re explain again. The post I made is long I must admit. But it was done with the intention to help.

I think what is confusing is that it's unclear what the order of operations.

I sincerely expect (for RandomCrop) that first the crop was done then the padding would be added. Thus not seeing pads at all was really confusing. Eventually I noticed a crop on one side which I took some time to notice I hate to admit. I think the first sentence should summarize better what most of the ops are doing or hint at the order. The issues is this sentence (especially the first sentence!):

Crop the given image at a random location. If the image is torch Tensor, it is expected to have […, H, W] shape, where … means an arbitrary number of leading dimensions, but if non-constant padding is used, the input is expected to have at most 2 leading dimensions

perhaps something like:

Padd the image and then crop it at a radom location.

Would be better. But padding wasn't really hinted how it'd be used later. Hope it wasn't my denseness that I missed it. I did read the rest of the options since I didn't know how or when the pad would be added.