Skip to content

FIXING_H5PY_COMPATIBILITY.md

Latest commit

 

History

History
109 lines (81 loc) · 3.9 KB

FIXING_H5PY_COMPATIBILITY.md

File metadata and controls

109 lines (81 loc) · 3.9 KB

Fixing h5py Python Incompatibility with HDF5 Module

Problem

When users loaded the hpic2deps module and attempted to install h5py via pip, the installation would fail with:

error: Unable to load dependency HDF5, make sure HDF5 is installed properly
Library dirs checked: []
error: libhdf5.so: cannot open shared object file: No such file or directory

This occurred because h5py's build system could not locate the HDF5 installation that was compiled by the hPIC2 scripts.

Root Cause

The modulefiles generated by campuscluster_update.py and campus_cluster_update_2.py were setting only HDF5_ROOT and adding the library path to LD_LIBRARY_PATH. However, h5py's build system requires additional environment variables to properly discover HDF5:

  1. HDF5_DIR - The primary variable h5py checks for HDF5 installation location
  2. HDF5_LIBDIR - Explicit path to the HDF5 library directory
  3. HDF5_INCLUDEDIR - Explicit path to the HDF5 include directory
  4. HDF5_MPI - Flag indicating whether HDF5 is built with MPI support
  5. PKG_CONFIG_PATH - Path for pkg-config-based discovery

Solution

Modified the modulefile generation sections in both Python scripts to set all required environment variables:

Before (campuscluster_update.py lines 325-328):

prepend-path --delim {:} PATH {/path/to/hdf5_dev/install/bin}
prepend-path --delim {:} CMAKE_PREFIX_PATH {/path/to/hdf5_dev/install/.}
append-path --delim {:} LD_LIBRARY_PATH {/path/to/hdf5_dev/install/lib}
setenv HDF5_ROOT {/path/to/hdf5_dev/install}

After (campuscluster_update.py lines 325-333):

prepend-path --delim {:} PATH {/path/to/hdf5_dev/install/bin}
prepend-path --delim {:} CMAKE_PREFIX_PATH {/path/to/hdf5_dev/install/.}
append-path --delim {:} LD_LIBRARY_PATH {/path/to/hdf5_dev/install/lib}
prepend-path --delim {:} PKG_CONFIG_PATH {/path/to/hdf5_dev/install/lib/pkgconfig}
setenv HDF5_ROOT {/path/to/hdf5_dev/install}
setenv HDF5_DIR {/path/to/hdf5_dev/install}
setenv HDF5_LIBDIR {/path/to/hdf5_dev/install/lib}
setenv HDF5_INCLUDEDIR {/path/to/hdf5_dev/install/include}
setenv HDF5_MPI ON

The same changes were applied to campus_cluster_update_2.py.

Files Modified

  1. campuscluster_update.py - Lines 328-333 added new environment variables
  2. campus_cluster_update_2.py - Lines 391-396 added new environment variables
  3. campus_cluster_update_3_fixing_mpi_errors.py - Lines 403-408 added new environment variables
  4. campus_cluster_update_3_hypre_cuda.py - Lines 360-365 added new environment variables

All four update scripts now generate modulefiles with the complete set of HDF5 environment variables required by h5py.

Testing

To verify the fix works:

  1. Run the update script to generate new modulefiles:

    python3 campuscluster_update.py update
# or
python3 campus_cluster_update_2.py update

  2. Load the hpic2deps module:

    module use /path/to/modulefiles
module load hpic2deps/~openmp-cuda-arch-None/Release/latest

  3. Verify environment variables are set:

    echo $HDF5_DIR
echo $HDF5_LIBDIR
echo $HDF5_INCLUDEDIR
echo $HDF5_MPI
echo $PKG_CONFIG_PATH

  4. Install h5py:

    pip install h5py

The installation should now succeed, with h5py finding and linking against the HDF5 libraries.

How h5py Discovers HDF5

h5py uses the following discovery mechanisms in order:

  1. Checks HDF5_DIR environment variable
  2. Checks HDF5_LIBDIR and HDF5_INCLUDEDIR for explicit paths
  3. Uses pkg-config via PKG_CONFIG_PATH if available
  4. Searches standard system library paths via LD_LIBRARY_PATH

By setting all these variables, we ensure h5py can find HDF5 regardless of which discovery method it prefers.

References