Is there any major significance having the residual layers structured as SiLU -> BatchNorm -> Conv2d?
I read other UNet implementations, where the Conv2d -> BatchNorm -> SiLU ordering was used. I'm not sure if there might be a strong case for one or the other, besides maybe:
SiLU -> BatchNorm -> Conv2d might be a bit more expressive, because the non-linearity is applied before the normalization, thus the model can utilize a wider range of the non-linear function,Conv2d -> BatchNorm -> SiLU might be a bit more stable during training, because the activations cannot explode so easily.
I'm just speculating now, I wonder if you have any experience with this, or any kind of reasoning about the preference of one over the other?
Is there any major significance having the residual layers structured as
SiLU -> BatchNorm -> Conv2d?I read other UNet implementations, where the
Conv2d -> BatchNorm -> SiLUordering was used. I'm not sure if there might be a strong case for one or the other, besides maybe:SiLU -> BatchNorm -> Conv2dmight be a bit more expressive, because the non-linearity is applied before the normalization, thus the model can utilize a wider range of the non-linear function,Conv2d -> BatchNorm -> SiLUmight be a bit more stable during training, because the activations cannot explode so easily.I'm just speculating now, I wonder if you have any experience with this, or any kind of reasoning about the preference of one over the other?