[WIP] Add RandAugment #154

ashertrockman · 2022-02-16T01:39:39Z

Here's a draft PR to make visible our efforts to add a fast implementation of RandAugment [1] to ffcv.

We currently have committed the following transforms:

Ideally, it would be nice to have tests to ensure that our transforms are similar to some baseline (I've currently chosen PyTorch's torchvision.transforms.functional as this baseline).

Now that the transforms have been implemented, there's a few more things:

Implement actual RandAug logic
Refactor njit to Compiler.compile
Custom np.bincount replacement

[1] https://arxiv.org/abs/1909.13719

ashertrockman · 2022-02-16T20:11:06Z

ffcv/transforms/utils/fast_crop.py

+@njit(parallel=True, fastmath=True, inline='always')
+def equalize(source, scratch, destination):
+    for i in prange(source.shape[-1]):
+        scratch[i] = np.bincount(source[..., i].flatten(), minlength=256)


Unfortunate that np.bincount doesn't have an out argument...

A numba version should be pretty fast and relatively easy to implement no ? (and might even be faster since it would skip the first pass of bincount that checks the min and max values)

Yeah, good idea. I'll try to add that in the near future.

GuillaumeLeclerc · 2022-02-16T23:35:57Z

I see that you are using numba.njit. I think it would be better to use Compiler.compile. This way we have a central location to disable compilation. It's quite convenient to debug issues

GuillaumeLeclerc · 2022-02-17T08:35:36Z

This is moving very fast 🚅 I was hoping to release v1.0.0 by the end of the week. Would you like it to be part of the release ? If yes could you change the target branch to v1.0.0 ?

ashertrockman · 2022-02-17T15:41:01Z

Sure, that sounds good. I'll try to finish a working demo soon.

tests/test_rand_aug.py

ashertrockman · 2022-02-18T00:45:07Z

This still needs some testing, but it looks promising. In a brief experiment on CIFAR-10, this RandAugment implementation added +1% test accuracy and cost about 0.1s/epoch. The first epoch takes substantially longer (assuming for the extra memory allocation), adding about 20s.

tfriedel · 2022-02-18T21:53:09Z

I did install this and added it to a training pipeline I'm currently using. Got this error:

Exception in thread Thread-12:
ValueError: cannot assign slice from input of different size

The above exception was the direct cause of the following exception:

SystemError: _PyEval_EvalFrameDefault returned a result with an error set

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/thomas/conda/envs/datapurchase3/lib/python3.9/threading.py", line 973, in _bootstrap_inner
    self.run()
  File "/home/thomas/git/ffcv/ffcv/loader/epoch_iterator.py", line 84, in run
    result = self.run_pipeline(b_ix, ixes, slot, events[slot])
  File "/home/thomas/git/ffcv/ffcv/loader/epoch_iterator.py", line 143, in run_pipeline
    results = stage_code(**args)
  File "", line 2, in stage_code_0
SystemError: CPUDispatcher(<function RandAugment.generate_code.<locals>.randaug at 0x7fe5f4fe7670>) returned a result with an error set

I then disabled jit by setting the env variable NUMBA_DISABLE_JIT=1 and I get this more helpful error:

  File "/home/thomas/conda/envs/datapurchase3/lib/python3.9/threading.py", line 973, in _bootstrap_inner
    self.run()
  File "/home/thomas/git/ffcv/ffcv/loader/epoch_iterator.py", line 84, in run
    result = self.run_pipeline(b_ix, ixes, slot, events[slot])
  File "/home/thomas/git/ffcv/ffcv/loader/epoch_iterator.py", line 143, in run_pipeline
    results = stage_code(**args)
  File "", line 2, in stage_code_0
  File "/home/thomas/git/ffcv/ffcv/transforms/randaugment.py", line 78, in randaug
    translate(src[i], dst[i], 0, int(mag))
  File "/home/thomas/git/ffcv/ffcv/transforms/utils/fast_crop.py", line 171, in translate
    destination[ty:, :] = source[:-ty, :]
ValueError: could not broadcast input array from shape (156,160,3) into shape (28,32,3)

Seems like this is because I didn't call it with size set to the image size. I'll try that now. However in the imagenet training example the image size is scaled over the epochs. How would this work then?
In any case, it's probably likely that users of this function will get the size wrong and the ouptut will not be helpful. I suggest to add some check for the correct size and a meaningful error message.

ashertrockman · 2022-02-18T23:04:06Z

Thanks for pointing this out. I'll look into automatically adapting the image size.

Did it work after fixing the image size in your example?

tfriedel · 2022-02-18T23:22:34Z

Yes it did work and added only a negligible slow down! Good work!
However it didn't improve the accuracy in the example I tried. I'm going to run some more experiments.

ashertrockman · 2022-02-19T00:26:02Z

Great, thanks! Hopefully your experiments work out.

It looks like the size argument was unnecessary, and it should now work even when changing the image size mid-training.

vchiley · 2022-07-12T21:29:10Z

This is awesome! When will it be merged into master / a release?

ashertrockman · 2022-07-12T22:42:08Z

This is awesome! When will it be merged into master / a release?

Thanks! I'm not sure -- the plan was to merge it with release v1.0.0 (#160), but as far as I know, development on that release has slowed down for the time being.

andrewilyas · 2023-06-19T17:00:11Z

Hi @ashertrockman ! It seems I lost track of this PR a while ago - do you think it's feasible to merge into v1.1.0?

ashertrockman · 2023-06-27T21:35:17Z

Hi @ashertrockman ! It seems I lost track of this PR a while ago - do you think it's feasible to merge into v1.1.0?

Yeah, I think it should be fine to merge.

andrewilyas · 2023-06-28T18:10:15Z

ffcv/pipeline/graph.py

                state_allocation = operation.declare_shared_memory(state)

-                if next_state.device.type != 'cuda' and isinstance(operation,
+                if next_state.device != 'cuda' and isinstance(operation,


I think as of v1.0 the device will be a torch.device in which case we would want next_state.device.type?

Abhinav95 · 2024-07-04T01:24:14Z

Great work on this @ashertrockman! I have had a successful training run with this fork (RandAugment ) with ImageNet-1k val acc > 80 with a ViT-B/16 model.

I think it would be valuable to merge this (and other similar augments like Colorjitter, Grayscale, 3Aug etc) because these are essential for any ViT runs.

ashertrockman · 2024-07-04T03:45:37Z

Great work on this @ashertrockman! I have had a successful training run with this fork (RandAugment ) with ImageNet-1k val acc > 80 with a ViT-B/16 model.

I think it would be valuable to merge this (and other similar augments like Colorjitter, Grayscale, 3Aug etc) because these are essential for any ViT runs.

Glad to hear!

ashertrockman · 2024-07-04T03:47:18Z

Great work on this @ashertrockman! I have had a successful training run with this fork (RandAugment ) with ImageNet-1k val acc > 80 with a ViT-B/16 model.

I think it would be valuable to merge this (and other similar augments like Colorjitter, Grayscale, 3Aug etc) because these are essential for any ViT runs.

By the way, if you're training ViTs, allow me to shamelessly promote my research: https://arxiv.org/abs/2305.09828

ashertrockman added 5 commits February 15, 2022 20:20

Init RandAug work

19f4039

Add posterize

21e2512

Add invert

cc3a99b

Add solarize

08b96b0

Add equalize

5b0359a

ashertrockman commented Feb 16, 2022

View reviewed changes

Add autocontrast

2025013

Add sharpness

6b6432c

Jasonlee1995 mentioned this pull request Feb 17, 2022

Inplace Color Augmentations (RandomBrightness, RandomContrast, RandomSaturation) #162

Merged

ashertrockman added 2 commits February 17, 2022 11:10

Add color (adjust_saturation), fix test params

fc2bdf6

Add translate

56e5cc3

ashertrockman changed the base branch from main to v1.0.0 February 17, 2022 16:49

ashertrockman and others added 6 commits February 17, 2022 11:52

Merge branch 'v1.0.0' into randaug

5da59e8

Fix merge problems/typos

784f3d0

Remove unused imports

e267538

Support translate by negative amount

6fc2362

Ensure rotate, shear amounts can be negative

4d2e1ed

Implement RandAug (WIP)

9e3a2ee

ashertrockman commented Feb 17, 2022

View reviewed changes

tests/test_rand_aug.py Outdated Show resolved Hide resolved

ashertrockman added 2 commits February 17, 2022 18:22

Allow more than one op per image, fixes

f6ba479

Move RandAugment

d6074ba

ashertrockman added 2 commits February 17, 2022 21:41

Fix bug (all 0 image) when translation amount is 0

8a2a9be

Solarize and sharpen need scratch memory

69adc18

Remove size argument

02792e3

GuillaumeLeclerc mentioned this pull request Mar 24, 2022

Compatibility with timm augmentation? #195

Closed

Update graph.py

0fa3d7c

ashertrockman mentioned this pull request Feb 28, 2023

[WIP] V1.0.0 #160

Merged

andrewilyas changed the base branch from v1.0.0 to v1.1.0 June 19, 2023 16:59

andrewilyas marked this pull request as ready for review June 27, 2023 22:41

Merge branch 'v1.1.0' into randaug

2019acc

andrewilyas reviewed Jun 28, 2023

View reviewed changes

[WIP] Add RandAugment #154

Are you sure you want to change the base?

[WIP] Add RandAugment #154

Uh oh!

Conversation

ashertrockman commented Feb 16, 2022 • edited by lengstrom Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ashertrockman Feb 16, 2022

Choose a reason for hiding this comment

Uh oh!

GuillaumeLeclerc Feb 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ashertrockman Feb 17, 2022

Choose a reason for hiding this comment

Uh oh!

GuillaumeLeclerc commented Feb 16, 2022

Uh oh!

GuillaumeLeclerc commented Feb 17, 2022

Uh oh!

ashertrockman commented Feb 17, 2022

Uh oh!

Uh oh!

ashertrockman commented Feb 18, 2022

Uh oh!

tfriedel commented Feb 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ashertrockman commented Feb 18, 2022

Uh oh!

tfriedel commented Feb 18, 2022

Uh oh!

ashertrockman commented Feb 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vchiley commented Jul 12, 2022

Uh oh!

ashertrockman commented Jul 12, 2022

Uh oh!

andrewilyas commented Jun 19, 2023

Uh oh!

ashertrockman commented Jun 27, 2023

Uh oh!

andrewilyas Jun 28, 2023

Choose a reason for hiding this comment

Uh oh!

Abhinav95 commented Jul 4, 2024

Uh oh!

ashertrockman commented Jul 4, 2024

Uh oh!

ashertrockman commented Jul 4, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

ashertrockman commented Feb 16, 2022 •

edited by lengstrom

Loading

GuillaumeLeclerc Feb 16, 2022 •

edited

Loading

tfriedel commented Feb 18, 2022 •

edited

Loading

ashertrockman commented Feb 19, 2022 •

edited

Loading