use ADIOI_Type_create_hindexed_x in place of MPI_Type_indexed in case… #1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
A demonstration of stackable ROMIO drivers implementing log-based i/o.
- no longer need to special-case contiguous types - don't flush a file if it's not open - too aggressive check for reading - promote type sizes to match ROMIO: the 32-to-64 bit promotions from a few years back need to propagate to the (much older) logfs code. - logfs does not support atomicity - long ago we broke out open into two parts - 'name' already implemented elsewhere: a while back we added a "romio_filesystem_type" hint for cases where someone wants a human-readable string for what file system romio found. Remove this feature from logfs (since it's already there) let's try the create-and-reopen open now that permissions are in place, the create on one, open on all might work better
when replaying log, logfs was using hard-coded 1mb buffer. good for aggregating tiny writes. Less helpful if operations are large.
it did a poor job managing types with lb and ub markers, and I am suspicous of its prior handling of partial types
logfs will already proces the type and put the first byte in the right place. I wonder if we even need this for common code, now that we use pwrite/pread? It's been in the view code forever.
a simple blocking version of non-blocking read. We can come back and re-think what non-blocking i/o means in a log-based file system.
some mixup here between etypes and bytes when computing offset
logfs was ignoring flags like EXCL and such. use user-provided flags when creating the file, and report an error if we are unable.
not just the logfs metadata
maybe the right way to implement shared file pointers is to drop down to the target file system somehow. maintaining the hidden shared file pointer file as a logfs log is a silly idea anyway.
sometimes we do need read a write-only file (as when replaying a log)
always open an independent version of the file
Useful for debugging and for demonstrating how the logfs data structures are stored.
the logfs write will take into account the type, but it won't know anything about the displacement unless we update it here
… sortindices is larger than INT_MAX
…ad_lustre_rwcontig.c
…tree walk to generate file view with decreasing offset that caused ad_lustre to crash
roblatham00
force-pushed
the
logfs-in-romio
branch
from
415004a to
a956cca
Compare
roblatham00
pushed a commit
that referenced
this pull request
Nov 10, 2025
Even though there can not be a buffer overflow as the string is properly
sized, noncontig_coll2 fails when built with -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 :
----
FAIL: noncontig_coll2
=====================
Thread 1 "noncontig_coll2" received signal SIGABRT, Aborted.
0x00007ffff709c5fc in __pthread_kill_implementation () from /lib64/libc.so.6
(gdb) bt
#0 0x00007ffff709c5fc in __pthread_kill_implementation ()
from /lib64/libc.so.6
#1 0x00007ffff7042106 in raise () from /lib64/libc.so.6
#2 0x00007ffff702938b in abort () from /lib64/libc.so.6
pmodels#3 0x00007ffff702a3ab in __libc_message_impl.cold () from /lib64/libc.so.6
pmodels#4 0x00007ffff712b4fb in __fortify_fail () from /lib64/libc.so.6
pmodels#5 0x00007ffff712adc6 in __chk_fail () from /lib64/libc.so.6
pmodels#6 0x00007ffff712c8f5 in __snprintf_chk () from /lib64/libc.so.6
pmodels#7 0x000000000040275e in snprintf (__s=0x4aafee "", __n=<optimized out>,
__fmt=0x404077 "%s,") at /usr/include/bits/stdio2.h:68
pmodels#8 default_str (mynod=<optimized out>, len=61, array=0x59fca0,
dest=0x4aafd0 "hostname,")
at src/mpi/romio/test/noncontig_coll2.c:189
pmodels#9 main (argc=<optimized out>, argv=<optimized out>)
at src/mpi/romio/test/noncontig_coll2.c:330
----
This is due to the len parameter of snprintf not being updated as we
advance in the string.
Fix this issue by introducing a remaining len var that contains the exact amount
of bytes left.
Signed-off-by: Nicolas Morey <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When writing large files, it is possible that the indexes in the file view exceed the size of int, resulting in overflow.
By replacing MPI_Type_indexed with ADIOI_Type_create_hindexed_x that uses MPI_Aint and MPI_Count, we can support larger files.