Compatibility Issues with vattention on A100 and A30 GPUs with CUDA 12.5 and 12.3

[alvi75]

I'm encountering several issues while trying to compile and run the vattention library on both NVIDIA A100 and A30 GPUs with different CUDA versions. The problems seem to arise due to compatibility issues between the vattention code and the specific GPU architectures.

System Configuration:

• A100 GPU:

  • CUDA Version: 12.5
  • PyTorch Version: 2.4.0+cu121
  • Driver Version: 555.42.06
  • Error Message:
    - ImportError: /path/to/site-packages/vattention-0.0.1-py3.10-linux-x86_64.egg/vattention.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZNK3c105Error4whatEv

• A30 GPU:

  • CUDA Version: 12.3

  • PyTorch Version: 2.4.0

  • Error Message:

    • Assertion page_size == 64*KB || page_size == 128*KB || page_size == 256*KB' failed.

  • Attempted Fix: I modified the vattn_demo_async.py and utils.py scripts to reduce batch size, context length, and other parameters to suit the A30 GPU. However, I encountered the following issue:

    • ModuleNotFoundError: No module named 'vattention'

  • Note: The module was supposed to be installed correctly, but the import fails, which suggests there might be deeper compatibility or installation issues.

Additional Notes:

• I've tried modifying the page size and other configurations, but the issues persist.
• The problem seems to be related to specific GPU architectures and CUDA versions.

Request:

Could you please provide guidance on how to resolve these compatibility issues? If the library is currently incompatible with A100 or A30 GPUs, would there be any upcoming updates to address these issues?

Also I was able to run the benchmark scripts.