Should we use torch's SiLU to replace Swish when alpha=1

[yiheng-wang-nv]

Hi @wyli @Nic-Ma @rijobro @ericspod
When I checked our efficientnet implementation, I found it used MONAI's implementation of Switch, it is more generic than pytorch's implementation ( see: SiLU) since we can change the parameter alpha.
However, when using default value 1 for alpha in some cases such as efficientnet, the calculation speed for pytorch's SiLU is faster than Swish. Therefore, shall we modify the implementation and at least use SiLU when alpha=1?

[yiheng-wang-nv]

My local test with jupyter notebook:

GPU mode:

from monai.networks.blocks.activation import Swish
from monai.networks.nets import EfficientNetBN
import torch


swish_monai = Swish()
swish = torch.nn.SiLU()

example = torch.randn([1, 3, 512, 512, 512]).cuda()

%%time
for _ in range(1000):
    output_monai = swish_monai(example)  # used 13.2s

%%time
for _ in range(1000):
   output = swish(example)  # used 6.69s

CPU mode:

example = torch.randn([1, 3, 512, 512, 512])

%%time
for _ in range(10):
    output_monai = swish_monai(example)  # used 14.5s

%%time
for _ in range(10):
   output = swish(example)  # used 6.01s

[rijobro]

If the user wanted to use alpha=1, they should probably just be using pytorch's SiLU directly, shouldn't they?

That said, I have no problem with reverting to SiLU for alpha=1.