h5repack tools fails to uncompress file

[a-shahba]

Describe the bug
I am trying to uncompress an HDF5 file using h5repack tool like

h5repack -f NONE path/to/compressed/input path/to/output

The tool prints "Error occurred while repacking" to the console and the process returns with 1 as exit code.

Expected behavior
Expected decompression to finish with no error

Platform (please complete the following information)

• HDF5 1.14.3
• Linux Ubuntu 20.04.6 LTS

Additional context
The input file is about ~365 MB. Please let me know if you need me to upload it somewhere for you to take a look at it.

[byrnHDF]

What compression was used in the original file.
Just use h5dump to show the headers only.

[epourmal]

Also, use --enable-error-stack flag to show error stack.

…

On Wed, Mar 12, 2025 at 3:01 PM Allen Byrne ***@***.***> wrote: What compression was used in the original file. Just use h5dump to show the headers only. — Reply to this email directly, view it on GitHub <#5369 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADLFT3P4NXKZ7GOJ54XYBK32UCVDZAVCNFSM6AAAAABY4RYNH2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMJZGIZDQNBZGE> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***> [image: byrnHDF]*byrnHDF* left a comment (HDFGroup/hdf5#5369) <#5369 (comment)> What compression was used in the original file. Just use h5dump to show the headers only. — Reply to this email directly, view it on GitHub <#5369 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADLFT3P4NXKZ7GOJ54XYBK32UCVDZAVCNFSM6AAAAABY4RYNH2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMJZGIZDQNBZGE> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[a-shahba]

Thank you @byrnHDF and @epourmal. Your suggestions helped me narrow down the issue. I ran h5repack with stack error reporting enabled. I got the following message

HDF5-DIAG: Error detected in HDF5 (1.14.3) thread 0:
  #000: H5A.c line 235 in H5Acreate2(): unable to synchronously create attribute
    major: Attribute
    minor: Unable to create file
  #001: H5A.c line 185 in H5A__create_api_common(): unable to create attribute
    major: Attribute
    minor: Unable to create file
  #002: H5A.c line 129 in H5A__create_common(): unable to create attribute
    major: Attribute
    minor: Unable to initialize object
  #003: H5VLcallback.c line 1034 in H5VL_attr_create(): attribute create failed
    major: Virtual Object Layer
    minor: Unable to create file
  #004: H5VLcallback.c line 1001 in H5VL__attr_create(): attribute create failed
    major: Virtual Object Layer
    minor: Unable to create file
  #005: H5VLnative_attr.c line 110 in H5VL__native_attr_create(): unable to create attribute
    major: Attribute
    minor: Unable to initialize object
  #006: H5Aint.c line 376 in H5A__create(): unable to create attribute in object header
    major: Attribute
    minor: Unable to insert object
  #007: H5Oattribute.c line 320 in H5O__attr_create(): unable to create new attribute in header
    major: Attribute
    minor: Unable to insert object
  #008: H5Omessage.c line 193 in H5O__msg_append_real(): unable to create new message
    major: Object header
    minor: No space available for allocation
  #009: H5Omessage.c line 1731 in H5O__msg_alloc(): unable to allocate space for message
    major: Object header
    minor: Unable to initialize object
  #010: H5Oalloc.c line 1262 in H5O__alloc(): object header message is too large
    major: Object header
    minor: Unable to initialize object
H5tools-DIAG: Error detected in HDF5:tools (1.14.3) thread 0:
  #000: h5repack_copy.c line 327 in copy_objects(): do_copy_objects from <file.nc> could not copy data to <output.nc>
    major: Failure in tools library
    minor: error in function
  #001: h5repack_copy.c line 748 in do_copy_objects(): copy_attr failed
    major: Failure in tools library
    minor: error in function
#002: h5repack.c line 506 in copy_attr(): H5Acreate2 failed on ,YAML_config>
    major: Failure in tools library
    minor: error in function
Error occurred while repacking

The error report shows that h5repack couldn't handle a global attribute, YAML_config. This attribute is quite big indeed. Here is the header information for this attribute

   ATTRIBUTE "YAML_config" {
      DATATYPE  H5T_STRING {
         STRSIZE 168515;
         STRPAD H5T_STR_NULLTERM;
         CSET H5T_CSET_ASCII;
         CTYPE H5T_C_S1;
      }
      DATASPACE  SCALAR
   }

This attribute is ~170,000 char-long. I can confirm that if I remove this attribute from the file, h5repack can successfully uncompress the entire file.

[epourmal]

Can you run h5dump -pH on the initial file?

…

On Wed, Mar 12, 2025 at 5:57 PM Ahmad Shahba ***@***.***> wrote: Thank you @byrnHDF <https://github.com/byrnHDF> and @epourmal <https://github.com/epourmal>. Your suggestions helped me narrow the issue. I ran h5repack with stack error reporting enabled. I got the following message HDF5-DIAG: Error detected in HDF5 (1.14.3) thread 0: #000: H5A.c line 235 in H5Acreate2(): unable to synchronously create attribute major: Attribute minor: Unable to create file #1: H5A.c line 185 in H5A__create_api_common(): unable to create attribute major: Attribute minor: Unable to create file #2: H5A.c line 129 in H5A__create_common(): unable to create attribute major: Attribute minor: Unable to initialize object #3: H5VLcallback.c line 1034 in H5VL_attr_create(): attribute create failed major: Virtual Object Layer minor: Unable to create file #4: H5VLcallback.c line 1001 in H5VL__attr_create(): attribute create failed major: Virtual Object Layer minor: Unable to create file #5: H5VLnative_attr.c line 110 in H5VL__native_attr_create(): unable to create attribute major: Attribute minor: Unable to initialize object #6: H5Aint.c line 376 in H5A__create(): unable to create attribute in object header major: Attribute minor: Unable to insert object #7: H5Oattribute.c line 320 in H5O__attr_create(): unable to create new attribute in header major: Attribute minor: Unable to insert object #8: H5Omessage.c line 193 in H5O__msg_append_real(): unable to create new message major: Object header minor: No space available for allocation #9: H5Omessage.c line 1731 in H5O__msg_alloc(): unable to allocate space for message major: Object header minor: Unable to initialize object #10: H5Oalloc.c line 1262 in H5O__alloc(): object header message is too large major: Object header minor: Unable to initialize object H5tools-DIAG: Error detected in HDF5:tools (1.14.3) thread 0: #000: h5repack_copy.c line 327 in copy_objects(): do_copy_objects from <file.nc> could not copy data to <output.nc> major: Failure in tools library minor: error in function #1: h5repack_copy.c line 748 in do_copy_objects(): copy_attr failed major: Failure in tools library minor: error in function #2: h5repack.c line 506 in copy_attr(): H5Acreate2 failed on ,YAML_config> major: Failure in tools library minor: error in function Error occurred while repacking The error report shows that h5repack couldn't handle a global attribute, YAML_config. This attribute is quite big indeed. Here is the header information for this attribute ATTRIBUTE "YAML_config" { DATATYPE H5T_STRING { STRSIZE 168515; STRPAD H5T_STR_NULLTERM; CSET H5T_CSET_ASCII; CTYPE H5T_C_S1; } DATASPACE SCALAR } This attribute is ~170,000 char-long. I can confirm that if I remove this attribute from the file, h5repack can successfully uncompress the entire file. — Reply to this email directly, view it on GitHub <#5369 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADLFT3NBCYP6CU5KRY7BIID2UDJV3AVCNFSM6AAAAABY4RYNH2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMJZGQ3TCMJQGM> . You are receiving this because you were mentioned.Message ID: ***@***.***> [image: a-shahba]*a-shahba* left a comment (HDFGroup/hdf5#5369) <#5369 (comment)> Thank you @byrnHDF <https://github.com/byrnHDF> and @epourmal <https://github.com/epourmal>. Your suggestions helped me narrow the issue. I ran h5repack with stack error reporting enabled. I got the following message HDF5-DIAG: Error detected in HDF5 (1.14.3) thread 0: #000: H5A.c line 235 in H5Acreate2(): unable to synchronously create attribute major: Attribute minor: Unable to create file #1: H5A.c line 185 in H5A__create_api_common(): unable to create attribute major: Attribute minor: Unable to create file #2: H5A.c line 129 in H5A__create_common(): unable to create attribute major: Attribute minor: Unable to initialize object #3: H5VLcallback.c line 1034 in H5VL_attr_create(): attribute create failed major: Virtual Object Layer minor: Unable to create file #4: H5VLcallback.c line 1001 in H5VL__attr_create(): attribute create failed major: Virtual Object Layer minor: Unable to create file #5: H5VLnative_attr.c line 110 in H5VL__native_attr_create(): unable to create attribute major: Attribute minor: Unable to initialize object #6: H5Aint.c line 376 in H5A__create(): unable to create attribute in object header major: Attribute minor: Unable to insert object #7: H5Oattribute.c line 320 in H5O__attr_create(): unable to create new attribute in header major: Attribute minor: Unable to insert object #8: H5Omessage.c line 193 in H5O__msg_append_real(): unable to create new message major: Object header minor: No space available for allocation #9: H5Omessage.c line 1731 in H5O__msg_alloc(): unable to allocate space for message major: Object header minor: Unable to initialize object #10: H5Oalloc.c line 1262 in H5O__alloc(): object header message is too large major: Object header minor: Unable to initialize object H5tools-DIAG: Error detected in HDF5:tools (1.14.3) thread 0: #000: h5repack_copy.c line 327 in copy_objects(): do_copy_objects from <file.nc> could not copy data to <output.nc> major: Failure in tools library minor: error in function #1: h5repack_copy.c line 748 in do_copy_objects(): copy_attr failed major: Failure in tools library minor: error in function #2: h5repack.c line 506 in copy_attr(): H5Acreate2 failed on ,YAML_config> major: Failure in tools library minor: error in function Error occurred while repacking The error report shows that h5repack couldn't handle a global attribute, YAML_config. This attribute is quite big indeed. Here is the header information for this attribute ATTRIBUTE "YAML_config" { DATATYPE H5T_STRING { STRSIZE 168515; STRPAD H5T_STR_NULLTERM; CSET H5T_CSET_ASCII; CTYPE H5T_C_S1; } DATASPACE SCALAR } This attribute is ~170,000 char-long. I can confirm that if I remove this attribute from the file, h5repack can successfully uncompress the entire file. — Reply to this email directly, view it on GitHub <#5369 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADLFT3NBCYP6CU5KRY7BIID2UDJV3AVCNFSM6AAAAABY4RYNH2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMJZGQ3TCMJQGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[a-shahba]

Sure! I am attaching the h5dump report.

headerInfo.txt

[epourmal]

Please try the "--latest" flag to use h5repack using the latest file format and see if it works. There was a restriction in HDF5 for the object header sizes that I believe was removed sometime ago. If it works we will need to improve h5repack to detect such use case. Elena

…

On Wed, Mar 12, 2025 at 8:14 PM Ahmad Shahba ***@***.***> wrote: Sure! I am attaching the h5dump report. headerInfo.txt <https://github.com/user-attachments/files/19221326/headerInfo.txt> — Reply to this email directly, view it on GitHub <#5369 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADLFT3ONPRY3GSDVLXVIP2T2UDZXTAVCNFSM6AAAAABY4RYNH2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMJZG4ZDAOJSHE> . You are receiving this because you were mentioned.Message ID: ***@***.***> [image: a-shahba]*a-shahba* left a comment (HDFGroup/hdf5#5369) <#5369 (comment)> Sure! I am attaching the h5dump report. headerInfo.txt <https://github.com/user-attachments/files/19221326/headerInfo.txt> — Reply to this email directly, view it on GitHub <#5369 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADLFT3ONPRY3GSDVLXVIP2T2UDZXTAVCNFSM6AAAAABY4RYNH2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMJZG4ZDAOJSHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[a-shahba]

Passing the —latest flag worked! Thanks, Elena On Thu, Mar 13, 2025 at 12:30 Elena Pourmal ***@***.***> wrote:

…

Please try the &quot;--latest&quot; flag to use h5repack using the latest file <br> format and see if it works. There was a restriction in HDF5 for the object <br> header sizes that I believe was removed sometime ago. If it works we will <br> need to improve h5repack to detect such use case. <br> <br> Elena <br> <br> On Wed, Mar 12, 2025 at 8:14 PM Ahmad Shahba ***@***.***&gt; <br> wrote: <br> <br> &gt; Sure! I am attaching the h5dump report. <br> &gt; <br> &gt; headerInfo.txt <br> &gt; &lt; https://github.com/user-attachments/files/19221326/headerInfo.txt&gt; <br> &gt; <br> &gt; — <br> &gt; Reply to this email directly, view it on GitHub <br> &gt; &lt; #5369 (comment)&gt;, <br> &gt; or unsubscribe <br> &gt; &lt; https://github.com/notifications/unsubscribe-auth/ADLFT3ONPRY3GSDVLXVIP2T2UDZXTAVCNFSM6AAAAABY4RYNH2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMJZG4ZDAOJSHE&gt; <br> &gt; . <br> &gt; You are receiving this because you were mentioned.Message ID: <br> &gt; ***@***.***&gt; <br> &gt; [image: a-shahba]*a-shahba* left a comment (HDFGroup/hdf5#5369) <br> &gt; &lt; #5369 (comment)&gt; <br> &gt; <br> &gt; Sure! I am attaching the h5dump report. <br> &gt; <br> &gt; headerInfo.txt <br> &gt; &lt; https://github.com/user-attachments/files/19221326/headerInfo.txt&gt; <br> &gt; <br> &gt; — <br> &gt; Reply to this email directly, view it on GitHub <br> &gt; &lt; #5369 (comment)&gt;, <br> &gt; or unsubscribe <br> &gt; &lt; https://github.com/notifications/unsubscribe-auth/ADLFT3ONPRY3GSDVLXVIP2T2UDZXTAVCNFSM6AAAAABY4RYNH2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMJZG4ZDAOJSHE&gt; <br> &gt; . <br> &gt; You are receiving this because you were mentioned.Message ID: <br> &gt; ***@***.***&gt; <br> &gt; <br> — Reply to this email directly, view it on GitHub <#5369 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AL6RHEWZP74GAXH3O4GOYPD2UGXDHAVCNFSM6AAAAABY4RYNH2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMRRHEYDCMJTG4> . You are receiving this because you authored the thread.Message ID: ***@***.***> [image: epourmal]*epourmal* left a comment (HDFGroup/hdf5#5369) <#5369 (comment)> Please try the &quot;--latest&quot; flag to use h5repack using the latest file <br> format and see if it works. There was a restriction in HDF5 for the object <br> header sizes that I believe was removed sometime ago. If it works we will <br> need to improve h5repack to detect such use case. <br> <br> Elena <br> <br> On Wed, Mar 12, 2025 at 8:14 PM Ahmad Shahba ***@***.***&gt; <br> wrote: <br> <br> &gt; Sure! I am attaching the h5dump report. <br> &gt; <br> &gt; headerInfo.txt <br> &gt; &lt; https://github.com/user-attachments/files/19221326/headerInfo.txt&gt; <br> &gt; <br> &gt; — <br> &gt; Reply to this email directly, view it on GitHub <br> &gt; &lt; #5369 (comment)&gt;, <br> &gt; or unsubscribe <br> &gt; &lt; https://github.com/notifications/unsubscribe-auth/ADLFT3ONPRY3GSDVLXVIP2T2UDZXTAVCNFSM6AAAAABY4RYNH2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMJZG4ZDAOJSHE&gt; <br> &gt; . <br> &gt; You are receiving this because you were mentioned.Message ID: <br> &gt; ***@***.***&gt; <br> &gt; [image: a-shahba]*a-shahba* left a comment (HDFGroup/hdf5#5369) <br> &gt; &lt; #5369 (comment)&gt; <br> &gt; <br> &gt; Sure! I am attaching the h5dump report. <br> &gt; <br> &gt; headerInfo.txt <br> &gt; &lt; https://github.com/user-attachments/files/19221326/headerInfo.txt&gt; <br> &gt; <br> &gt; — <br> &gt; Reply to this email directly, view it on GitHub <br> &gt; &lt; #5369 (comment)&gt;, <br> &gt; or unsubscribe <br> &gt; &lt; https://github.com/notifications/unsubscribe-auth/ADLFT3ONPRY3GSDVLXVIP2T2UDZXTAVCNFSM6AAAAABY4RYNH2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMJZG4ZDAOJSHE&gt; <br> &gt; . <br> &gt; You are receiving this because you were mentioned.Message ID: <br> &gt; ***@***.***&gt; <br> &gt; <br> — Reply to this email directly, view it on GitHub <#5369 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AL6RHEWZP74GAXH3O4GOYPD2UGXDHAVCNFSM6AAAAABY4RYNH2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMRRHEYDCMJTG4> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[epourmal]

Glad it worked! I think it is a bug in h5repack: it should detect the situation when the earliest file format cannot be used and provide a meaningful error message without forcing the user to investigate the error stack or h5repack should use the earliest appropriate version of the object messages that are needed for the particular feature. (i.e., large attribute) without forcing all objects to use the latest versions.