FBGEMM-GPU v1.5.0 Cannot Run on SM70 and SM75 Architectures

[tiankongdeguiji]

Description

FBGEMM-GPU v1.5.0 does not appear to support SM70 (Volta) and SM75 (Turing) GPU architectures, despite them being listed as supported in the official release documentation.

Steps to Reproduce

Running cuobjdump on the compiled shared library reveals that only sm_80 and sm_90a architectures are included:

/usr/local/cuda-12.6/bin/cuobjdump /opt/conda/lib/python3.11/site-packages/fbgemm_gpu/fbgemm_gpu_tbe_training_forward.so \
  | grep -E "sm_" \
  | sort \
  | uniq

Output:

arch = sm_80
arch = sm_90a

Expected Behavior

According to the official release documentation, SM75 should be supported. The compiled binary should include CUDA kernels targeting these architectures.

Actual Behavior

The shared library only contains kernels compiled for sm_80 and sm_90a, making it impossible to run FBGEMM-GPU on SM75 (e.g., T4) GPUs.

Environment

• CUDA Version: 12.6
• Python Version: 3.11
• Package: torch==2.10.0 fbgemm-gpu==1.5.0 --index-url https://download.pytorch.org/whl/cu126

Additional Notes

Could you clarify whether SM70/SM75 support has been dropped in recent releases? If so, please update the documentation accordingly. If this is unintentional, a fix or recompilation targeting these architectures would be appreciated.

[tiankongdeguiji]

hi @q10 , could you take a look?

[tiankongdeguiji]

hi @gchalump , could you take a look?

[q10]

hi @q10 , could you take a look?

Hi @tiankongdeguiji that is correct, SM70/75 has been dropped as of v1.5.0. This decision was made because the resulting binary was over 600MB, which was well over the PyPI file size limits. It may still be possible to build SM70/75 directly, by setting TORCH_CUDA_ARCH_LIST="7.0;7.5" when invoking the build scripts (we provide a one-line invocation build script in https://github.com/pytorch/FBGEMM/blob/main/ci/integration/fbgemm_gpu_oss_build.bash).

[tiankongdeguiji]

hi @q10 , could you take a look?

Hi @tiankongdeguiji that is correct, SM70/75 has been dropped as of v1.5.0. This decision was made because the resulting binary was over 600MB, which was well over the PyPI file size limits. It may still be possible to build SM70/75 directly, by setting TORCH_CUDA_ARCH_LIST="7.0;7.5" when invoking the build scripts (we provide a one-line invocation build script in https://github.com/pytorch/FBGEMM/blob/main/ci/integration/fbgemm_gpu_oss_build.bash).

hi @q10，I tried running the build script you mentioned, but I encountered a Conda installation error. Do you know how to fix this?

conda install --force-reinstall -n build-py3.11-torch2.10.0-cuda12.6.3 -c conda-forge --override-channels -y cuda=12.6.3
Channels:
 - conda-forge
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: failed

LibMambaUnsatisfiableError: Encountered problems while solving:
  - nothing provides cuda-gdb 12.6.77.* needed by cuda-command-line-tools-12.6.3-ha770c72_0


  # NOTE: Currently, CUDA 12.6 and later cannot be installed using the
Could not solve for environment specs
The following packages are incompatible
└─ cuda =12.6.3 * is not installable because there are no viable options
   ├─ cuda 12.6.3 would require
   │  └─ cuda-toolkit =12.6.3 * but there are no viable options
   │     ├─ cuda-toolkit 12.6.3 would require
   │     │  └─ cuda-tools =12.6.3 *, which requires
   │     │     └─ cuda-command-line-tools =12.6.3 *, which requires
   │     │        └─ cuda-gdb =12.6.77 *, which does not exist (perhaps a missing channel);
   │     └─ cuda-toolkit 12.6.3 would require
   │        └─ __win =* *, which is missing on the system;
   └─ cuda 12.6.3 would require
      └─ __win =* *, which is missing on the system.

[q10]

hi @q10 , could you take a look?

Hi @tiankongdeguiji that is correct, SM70/75 has been dropped as of v1.5.0. This decision was made because the resulting binary was over 600MB, which was well over the PyPI file size limits. It may still be possible to build SM70/75 directly, by setting TORCH_CUDA_ARCH_LIST="7.0;7.5" when invoking the build scripts (we provide a one-line invocation build script in https://github.com/pytorch/FBGEMM/blob/main/ci/integration/fbgemm_gpu_oss_build.bash).

hi @q10，I tried running the build script you mentioned, but I encountered a Conda installation error. Do you know how to fix this?

conda install --force-reinstall -n build-py3.11-torch2.10.0-cuda12.6.3 -c conda-forge --override-channels -y cuda=12.6.3
Channels:
 - conda-forge
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: failed

LibMambaUnsatisfiableError: Encountered problems while solving:
  - nothing provides cuda-gdb 12.6.77.* needed by cuda-command-line-tools-12.6.3-ha770c72_0


  # NOTE: Currently, CUDA 12.6 and later cannot be installed using the
Could not solve for environment specs
The following packages are incompatible
└─ cuda =12.6.3 * is not installable because there are no viable options
   ├─ cuda 12.6.3 would require
   │  └─ cuda-toolkit =12.6.3 * but there are no viable options
   │     ├─ cuda-toolkit 12.6.3 would require
   │     │  └─ cuda-tools =12.6.3 *, which requires
   │     │     └─ cuda-command-line-tools =12.6.3 *, which requires
   │     │        └─ cuda-gdb =12.6.77 *, which does not exist (perhaps a missing channel);
   │     └─ cuda-toolkit 12.6.3 would require
   │        └─ __win =* *, which is missing on the system;
   └─ cuda 12.6.3 would require
      └─ __win =* *, which is missing on the system.

Hmm, Im not sure, what is the environment you're running in? Generally, the build environment we use is amazonlinux:2023 docker image or CentOS.

[tiankongdeguiji]

Hi @q10, we are still encountering the same issue when using amazonlinux:2023. You can reproduce it with the following steps:

docker run -it --gpus all --ipc host --network host -v $(pwd):/home/FBGEMM amazonlinux:2023 bash
yum install git wget which make
PYTHON_VERSION=3.11 PYTORCH_VERSION=2.10.0 BUILD_VARIANT=cuda BUILD_CUDA_VERSION=12.6.3 bash ci/integration/fbgemm_gpu_oss_build.bash