Anybody observe slow training speed for mamba compared to transformer model?

[vgthengane]

[d62lu]

yes, I just made the comparison. The speed of the Mamba block is truly slower than Transformer, under the same input dimension

[d62lu]

for both model training and inference. I am not looking into the details of the Mamba block, maybe I missed something in the code....

[LMD0311]

I've merged this issue into the “Discussion” section.

On my machine, with parameter expansion mamba has an advantage over Transformer. I'm not sure if this is environment/GPU related.