Assumption about bounds in floating point images

[bglocker]

Took me a while to figure out while the ColorJitter transform produced strange results (mostly fully saturated images) until I realized that there is a hard coded assumption that images of type float are bounded to max value 1.0.

Might either need a warning in the doc, or even better, the bound could be provided as an argument:

vision/torchvision/transforms/functional_tensor.py

Line 314 in a2f6565

def _blend(img1: Tensor, img2: Tensor, ratio: float) -> Tensor:

[NicolasHug]

From our docs:

The expected range of the values of a tensor image is implicitly defined by the tensor dtype. Tensor images with a float dtype are expected to have values in [0, 1). Tensor images with an integer dtype are expected to have values in [0, MAX_DTYPE] where MAX_DTYPE is the largest value that can be represented in that dtype.

Maybe we should make this a bit more obvious and link to it from every single forward(img) docstring?

[datumbox]

Images of uint8 are assumed to have max value of 255 (no surprise). Images encoded as floats have a max value of 1.0. This is a scaling that happens in the following transforms:

• to_tensor (discouraged from being used. use pil_to_tensor instead)
• convert_image_dtype
• to_pil_image (does the inverse of the above)

The upper bound assumption is used by a number of transforms, such as adjust_sharpness, adjust_brightness, adjust_contrast, adjust_saturation, adjust_gamma (due to the clamping at 0-1), invert and autocontrast.

Note that most models are supposed to get the input normalized (there are exceptions, for example the Detection models).

I do think that the above assumption might not be clear to someone new who comes to the TorchVision project, nevertheless there is some exciting parallel work that might fix this (Datasets and Transforms prototypes). Perhaps it's worth documenting it further to be more obvious. Maybe on its own page?

[bglocker]

Fair enough, my bad. It's pretty clear in the docs which I missed as I jumped straight to the doc of the ColorJitter transform. Maybe a little reminder about the assumed ranges in the doc of relevant transforms might help.

[LLYX]

From our docs:

The expected range of the values of a tensor image is implicitly defined by the tensor dtype. Tensor images with a float dtype are expected to have values in [0, 1). Tensor images with an integer dtype are expected to have values in [0, MAX_DTYPE] where MAX_DTYPE is the largest value that can be represented in that dtype.

Maybe we should make this a bit more obvious and link to it from every single forward(img) docstring?

I'm trying to use the transforms for data outside the range of 0-255 and saw the MAX_DTYPE once I traced the code, and it seems like it should be returning higher bounds depending on the input dtype given, however it's clamping my output at 255 no matter what I give it, is this intentional behavior/is there a workaround? I can always copy and modify the code for my own use but would be nice to be able to use the torchvision functional API.

>>> import torchvision.transforms.functional as ttf
>>> data = np.load(<my-numpy-file>)
>>> ttf.adjust_brightness(torch.from_numpy(data[0]).to(torch.int64), brightness_factor=2).max()
tensor(255)

[bhack]

Same problem with 16bit depth PNG images casted to float32 tensor. Also cause directly using it with float16 has unsupported internal ops:

RuntimeError: "clamp_scalar_cpu" not implemented for 'Half' .

[bhack]

So the point is how we could work with 16bit/HALF images?

[bhack]

See also ivy-llc/ivy#13652

[bhack]

The upstream ticket is now at pytorch/pytorch#99563

Any suggested workaround in the meantime?

[vadimkantorov]

I don't know if this _blend still lives in existing codebase, but for floating point inputs, it can use torch.lerp

Also, for blending integral images like uint8 tensors, it would be nice to have lerp support them natively without extra casts/copies: pytorch/pytorch#54389

[bhack]

@vadimkantorov It seems to introduce also some Nan cases with Torch float32 tensor in the 0.0-1.0 range.