Scanning theory

[enesdoruk]

Hi, What is the background of scanning half of channel features instead of all channels? The model splits channels by 2, and half of the channels pass through scanning, and the other half use just conv. What if we scan all channels and recalibrate with before scanning features?

[ahatamiz]

Hi @enesdoruk the idea is to have the network learn diverse set of features coming from both SSM and non-SSM branches. The SSM branch encodes an implicit inductive bias for pixel dependency where the network does not have access to the entire tokens. However the non-SSM branch removes all such dependencies. This allows the network to not quickly overfit (e.g. some features are easy guesses for example the head or tail of a bird) and learn more robust feature representations.

Hope it helped

[FlxhSui]

"Hello Author, I have further questions about the symmetric branch design in MambaVisionMixer. In the original Mamba, the symmetric branch adaptively adjusts the SSM output through gating weights, while your approach splits the input dimension into two equal halves: one half processed by the SSM branch, and the other half processed by a branch with MLP, 1D convolution, and SiLU activation to extract local information and enhance nonlinear capabilities, followed by concatenation. You mentioned that this design mitigates SSM's direct dependency on pixel-level features, my understanding is that 1D convolution captures local patterns, and the MLP/SiLU improves channel-wise interactions and nonlinearity, but why does this design reduce SSM's reliance on pixel dependencies?"