DataAugmentation error with masks

[cirobr]

Cheers,

I’m receiving an error when augmenting masks prior to training a model for semantic segmentation. Consider img as RGB image matrix. Consider mask as a matrix with integer elements from 1 to 11. Code is as follows:

item1 = Image(img)
item2 = MaskMulti(mask)

tfm = Maybe(FlipX{2}()) |> Rotate(15) |> Zoom((1.0, 1.2)) |>
      AdjustBrightness(0.3) |> AdjustContrast(0.3) |>
      CenterCrop((256,256))
img_tf = apply(tfm, (item1, item2))

Error message as follows:

ERROR: MethodError: no method matching length(::DataAugmentation.MaskMulti{2, Int64, Int64})
The function `length` exists, but no method is defined for this combination of argument types.

Closest candidates are:
  length(::LLVM.ArrayType)
   @ LLVM ~/.julia/packages/LLVM/iza6e/src/core/type.jl:343
  length(::DataStructures.EnumerateAll)
   @ DataStructures ~/.julia/packages/DataStructures/IrAJn/src/multi_dict.jl:96
  length(::LLVM.ConstantArray)
   @ LLVM ~/.julia/packages/LLVM/iza6e/src/core/value/constant.jl:421
  ...

The error disappears in the following conditions:

1. If the transformation is restricted to the image only.
2. If the transformation is applied to image and mask, but the command line corresponding to AdjustBrightness and AdjustContrast is commented out.

Thanks for advising.

[cirobr]

Have reproduced the same failure in a much simpler code:

using DataAugmentation
using Images
using Random

mask = MaskMulti(rand(1:3, 100, 100))
pipe = AdjustBrightness(0.4)
img_tf = apply(pipe, mask)

[paulnovo]

It looks like there is no implementation of apply for AdjustBrightness and AdjustContrast with MaskMulti (or MaskBinary). I think in this case, we could implement it as a no-op, since adjusting brightness and contrast of a mask doesn't make much sense (and I believe this is what albumentations and other augmentation libraries do). Does this line up with your intentions here?

[cirobr]

It looks like there is no implementation of apply for AdjustBrightness and AdjustContrast with MaskMulti (or MaskBinary). I think in this case, we could implement it as a no-op, since adjusting brightness and contrast of a mask doesn't make much sense (and I believe this is what albumentations and other augmentation libraries do). Does this line up with your intentions here?

Indeed, it seems to line up with application. The way I was thinking to work is as follows: a) to define a single pipeline of transformations; b) the wrappers Image and MaskMulti would handle the transformations that are applicable to images but not to masks (as it is the case for AdjustBrightness and AdjustContrast).

As an interim fix, would that work if two pipelines are defined respectively for images and masks (this second one omitting the intensity transformations)? How to ensure they would properly execute in "sync"?

[paulnovo]

Probably cleanest to make 3 transforms:

item1 = Image(img)
item2 = MaskMulti(mask)

tfm1 = Maybe(FlipX{2}()) |> Rotate(15) |> Zoom((1.0, 1.2))
tfm2 = AdjustBrightness(0.3) |> AdjustContrast(0.3)
tfm3 = CenterCrop((256,256))

img_tf, mask_tf = apply(tfm1, (item1, item2))
img_tf = apply(tfm2, img_tf)
img_tf, mask_tf = apply(tfm3, (img_tf,  mask_tf))

You can keep two transforms in "sync" by first calculating the randomstate and then passing that to apply for both, but I think that would be a messier solution because you would have to splice out the randomstates that aren't needed for the mask transform.