The influence of not using the c++ extension which provided in the original implementation of Deformable DETR?

Hi there, it's a great work.
I'd like to know if there is any impact, such as slow training or inference speed, etc, for only using the "CPU version of multi-scale deformable attention".