Snake Activation For Pytorch
snake is an easy activation function.
- SnakeA is: y = tanh(x) + relu(x)
- SnakeB is: y = tanh(x) + silu(x)
- SnakeC is: y = erf(x) + gelu(x)
SnakeB
SnakeC
I've notice that fashion activations like SiLU, GeLU, ReLU as well as Mish are the kind of self-gated activations. They have one in common which is they are closely zero while the input
is negative. According to the paper Searching for Activation Functions (which Introduced swish activation), they found that most input values of the swish activation are in negative part, which in my opinion
shows that the well-trained net is "eager to learn something nagetive". Other than that, if we follows the units like "conv-bn-act" or "linear-act", the output of the activation will be the input of the next layer's
linear weight, not bias. and the input of value will be a closely zero if the network is "eager to learn something negative", then grad will be hard to flow through this part.
So, Snake activation is an activation more like ELU. More grad flow pass the layer and linear in its next layer can get more information.
picture from Geogerbra
full name Soft maxout Funnel Rectified Linear Unit, which is a soft version of FReLU.
origin FReLU are in: https://arxiv.org/pdf/2007.11824.pdf
code in: https://github.com/megvii-model/FunnelAct
SFReLU is an easy paramed activation funcion.
f(x0, x1) --> x:
temp0 = x0 - x1
temp0 = silu(temp0)
x = temp0 + x1
return x
sfrelu(x) --> x:
y = dwconv(x) # shape same as x
x = f(x, y)
return x
the origin maxout can be described as the function below.
f(x0, x1) --> x:
temp0 = x0 - x1
temp0 = relu(temp0)
x = temp0 + x1
return x
I simply replace relu with silu, which is considered a "soft" version of relu. you can also call it FSiLU
as the name suggest, it's the soft version of "+", which is also called "reslink", who create the empire of real DEEP learning.
and I somehow managed to create a soft version of this.
just like everything else, it can be described as this:
y = silu(x0 + x1) - silu(x0 - x1) + x1
that's all, easy
soft version of operator "*".
when x0 >> x1, return x1
when x0 << x1, return x0
when x0 ~~ x1, return x0 * x1
function:
y = silu(-x0)*silu(-x1) + x0 - silu(x0 - x1)