crawford_test.sh fails on big endian architecture (s390x)

[pranavkaruvally]

There was this recent bug report in Debian: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1143998

libskbb-dev did build without any problem on s390x. So, I went on to run the tests for unifrac-binaries and saw the following error:

HDF5-DIAG: Error detected in HDF5 (2.2.0) thread 1:
  #000: ./src/H5D.c line 1051 in H5Dread(): can't synchronously read data
    major: Dataset
    minor: Read failed
  #001: ./src/H5D.c line 999 in H5D__read_api_common(): can't read data
    major: Dataset
    minor: Read failed
  #002: ./src/H5VLcallback.c line 2209 in H5VL_dataset_read(): dataset read failed
    major: Virtual Object Layer
    minor: Read failed
  #003: ./src/H5VLcallback.c line 2165 in H5VL__dataset_read(): dataset read failed
    major: Virtual Object Layer
    minor: Read failed
  #004: ./src/H5VLnative_dataset.c line 367 in H5VL__native_dataset_read(): unable to set up file and memory dataspaces
    major: Dataset
    minor: Unable to initialize object
  #005: ./src/H5VLnative_dataset.c line 188 in H5VL__native_dataset_io_setup(): selection + offset not within extent for file dataspace
    major: Dataspace
    minor: Out of range

The error was being thrown from su::biom::get_obs_data_direct (biom.cpp:206)
GDB backtrace:

#0  0x000003fff7bdaf70 in __cxa_throw () from /usr/lib/s390x-linux-gnu/libstdc++.so.6
#1  0x000003fff7a4b7c2 in H5::DataSet::read(void*, H5::DataType const&, H5::DataSpace const&, H5::DataSpace const&, H5::DSetMemXferPropList const&) const () from /usr/lib/s390x-linux-gnu/libhdf5_serial_cpp.so.320
#2  0x000003fff7ea26f2 in su::biom::get_obs_data_direct (this=0x3ffffff69c8, id="197790",
    current_indices_out=@0x3ffffff67e8: 0x3fdcdcbf010, current_data_out=@0x3ffffff67e0: 0x3fdbdcbe010) at biom.cpp:206

Analysing crawford.biom using python, I was able to see that all of the data types are stored in little endian fashion:

I tried applying the following fix and it seems to fix the issue:

diff --git a/src/biom.cpp b/src/biom.cpp
index c2fc274..b262551 100644
--- a/src/biom.cpp
+++ b/src/biom.cpp
@@ -156,7 +156,7 @@ void biom::load_indptr(const char *path, std::vector<uint32_t> &indptr) {
                 long(sizeof(uint32_t) * dims[0]), __FILE__, __LINE__);
         exit(EXIT_FAILURE);
     }
-    ds.read((void*)dataout, dtype);
+    ds.read((void*)dataout, H5::PredType::NATIVE_INT32);

     indptr.reserve(dims[0]);
     for(unsigned int i = 0; i < dims[0]; i++)
@@ -203,8 +203,8 @@ unsigned int biom::get_obs_data_direct(const std::string &id, uint32_t *& curren
         exit(EXIT_FAILURE);
     }

-    obs_indices.read((void*)current_indices_out, indices_dtype, indices_memspace, indices_dataspace);
-    obs_data.read((void*)current_data_out, data_dtype, data_memspace, data_dataspace);
+    obs_indices.read((void*)current_indices_out, H5::PredType::NATIVE_INT32, indices_memspace, indices_dataspace);
+    obs_data.read((void*)current_data_out, H5::PredType::NATIVE_DOUBLE, data_memspace, data_dataspace);

     return count[0];
 }
@@ -248,8 +248,8 @@ unsigned int biom::get_sample_data_direct(const std::string &id, uint32_t *& cur
         exit(EXIT_FAILURE);
     }

-    sample_indices.read((void*)current_indices_out, indices_dtype, indices_memspace, indices_dataspace);
-    sample_data.read((void*)current_data_out, data_dtype, data_memspace, data_dataspace);
+    sample_indices.read((void*)current_indices_out, H5::PredType::NATIVE_INT32, indices_memspace, indices_dataspace);
+    sample_data.read((void*)current_data_out, H5::PredType::NATIVE_DOUBLE, data_memspace, data_dataspace);

     return count[0];
 }

Here I have hard coded the values. But I will try to find a way to make this more generic.

[sfiligoi]

@pranavkaruvally I don't have a big-endian machine to test this.
But if you can open a PR that does not have undesired side-effects on standard (x86) CPUs, I would be happy to merge it in.