Skip to content

Support large Zip files for JVM/64-bit#23594

Open
lzhou2025 wants to merge 1 commit intoeclipse-openj9:masterfrom
lzhou2025:zipSizeLimit
Open

Support large Zip files for JVM/64-bit#23594
lzhou2025 wants to merge 1 commit intoeclipse-openj9:masterfrom
lzhou2025:zipSizeLimit

Conversation

@lzhou2025
Copy link
Copy Markdown
Member

@lzhou2025 lzhou2025 commented Mar 27, 2026

The changes are to remove the 4GB size limit of Zip files and support LARGE FILE for JVM/64-bit. This PR doesn't support LARGE FILE for JVM/32-bit.

Fixes: #23441

@lzhou2025 lzhou2025 force-pushed the zipSizeLimit branch 12 times, most recently from 5fc8e1c to 075e85f Compare April 2, 2026 19:30
@lzhou2025 lzhou2025 marked this pull request as ready for review April 6, 2026 12:11
Comment thread runtime/zip/zipsup.c Outdated
Comment thread runtime/zip/zipsup.c Outdated
Comment thread runtime/zip/zipsup.c Outdated
Comment thread runtime/zip/zipsup.c Outdated
Comment thread runtime/zip/zipsup.c Outdated
Comment thread runtime/zip/zipsup.c Outdated
Comment thread runtime/zip/zipsup.c Outdated
Comment thread runtime/zip/zipsup.c Outdated
Comment thread runtime/zip/zipsup.c Outdated
Comment thread runtime/zip/zipsup.c Outdated
Comment thread runtime/zip/zipsup.c Outdated
Comment thread runtime/zip/zipsup.c Outdated
Comment thread runtime/zip/zipsup.c Outdated
Comment thread runtime/zip/zipsup.c Outdated
Comment thread runtime/zip/zipsup.c Outdated
Comment thread runtime/zip/zipsup.c
Comment thread runtime/include/vmizip.h
Comment thread runtime/include/zipsup.h
Comment thread runtime/zip/zipsup.c
Comment thread runtime/zip/zipsup.c Outdated
Comment thread runtime/zip/zipsup.c Outdated
Comment thread runtime/zip/zipsup.c Outdated
Comment thread runtime/zip/zipsup.c Outdated
Comment thread runtime/zip/zipsup.c Outdated
Comment thread runtime/zip/zipsup.c Outdated
Comment thread runtime/zip/zipsup.c Outdated
@lzhou2025
Copy link
Copy Markdown
Member Author

I believe all comments are resolved, please reopen if missed.

Comment thread runtime/zip/zipsup.c
Comment thread runtime/zip/zipsup.c Outdated
Comment thread runtime/zip/zipsup.c Outdated
Comment thread runtime/zip/zipsup.c Outdated
Comment thread runtime/zip/zipsup.c Outdated
Comment thread runtime/zip/zipsup.c Outdated
Comment thread runtime/zip/zipsup.c Outdated
Comment thread runtime/zip/zipsup.c Outdated
Comment thread runtime/zip/zipsup.c Outdated
Comment thread runtime/zip/zipsup.c Outdated
@keithc-ca
Copy link
Copy Markdown
Contributor

The commit message would be improved by explicitly stating that this support is not yet complete for 32-bit platforms.

@lzhou2025 lzhou2025 force-pushed the zipSizeLimit branch 2 times, most recently from dde7099 to 39bab11 Compare April 9, 2026 20:57
@lzhou2025
Copy link
Copy Markdown
Member Author

lzhou2025 commented Apr 10, 2026

I believe all comments are resolved, please reopen if missed.
For the legacy codes written decades ago, signed 32-bit integer (IDATA) is widely used in 32-bit platforms, will be replaced by I_64 in the next PR to support #23458 and LARGE FILE in 32-bit platforms.

Comment thread runtime/zip/zipsup.c Outdated
Comment thread runtime/zip/zipsup.c Outdated
Comment thread runtime/zip/zipsup.c Outdated
Comment thread runtime/zip/zipsup.c Outdated
Comment thread runtime/zip/zipsup.c Outdated
The changes are to remove the 4GB size limit of Zip files and
support LARGE FILE for JVM/64-bit. This PR doesn't support
LARGE FILE for JVM/32-bit.

Fixes: eclipse-openj9#23441
@lzhou2025
Copy link
Copy Markdown
Member Author

lzhou2025 commented Apr 13, 2026

That IDATA is widely used in the zip codes appears because of OMR,

omrfile_read(struct OMRPortLibrary *portLibrary, intptr_t fd, void *buf, intptr_t nbytes)

intptr_t should be uintptr_t or size_t, because the read() calls of Unix like systems or POSIX are size_t, unsigned.

@keithc-ca
Copy link
Copy Markdown
Contributor

omrfile_read(struct OMRPortLibrary *portLibrary, intptr_t fd, void *buf, intptr_t nbytes)

intptr_t should be uintptr_t or size_t, because the read() calls of Unix like systems or POSIX are size_t, unsigned.

Yes, perhaps the OMR API could be improved, but that is unrelated to the concepts involved here. The file format allows for large entries and files, perhaps larger than could exist in memory at any given time. Things could still be made to work if OMR only allowed reading 32kB in a single call (in support of even 32-bit zip files).

@lzhou2025
Copy link
Copy Markdown
Member Author

omrfile_read(struct OMRPortLibrary *portLibrary, intptr_t fd, void *buf, intptr_t nbytes)

intptr_t should be uintptr_t or size_t, because the read() calls of Unix like systems or POSIX are size_t, unsigned.

Yes, perhaps the OMR API could be improved, but that is unrelated to the concepts involved here. The file format allows for large entries and files, perhaps larger than could exist in memory at any given time. Things could still be made to work if OMR only allowed reading 32kB in a single call (in support of even 32-bit zip files).

omrfile_read(struct OMRPortLibrary *portLibrary, intptr_t fd, void *buf, intptr_t nbytes)

intptr_t should be uintptr_t or size_t, because the read() calls of Unix like systems or POSIX are size_t, unsigned.

Yes, perhaps the OMR API could be improved, but that is unrelated to the concepts involved here. The file format allows for large entries and files, perhaps larger than could exist in memory at any given time. Things could still be made to work if OMR only allowed reading 32kB in a single call (in support of even 32-bit zip files).

The ZIP format supports up to 4Gb size of each individual uncompressed file. But depending how to decompress it, if trying to allocate that size of memory buffer in 32-bit system, I agree, that will exceed the maximum memory space.

@lzhou2025
Copy link
Copy Markdown
Member Author

OMR layer doesn't know what size of data is required in applications. Before the zip functions call OMR interfaces, should allocate memory buffers, which is fit to uncompress logic.

@lzhou2025
Copy link
Copy Markdown
Member Author

lzhou2025 commented Apr 15, 2026

ZIP64 & LARGE FILE support in two PRs:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ZIP64: Remove the 4Gb size limit of archive file for JVM/64-bit

3 participants