Skip to content

Chunk distribution strategies #824

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 27 commits into from
Aug 8, 2025
Merged

Chunk distribution strategies #824

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
a6eabfb
Add chunk distribution algorithms
franzpoeschel Mar 2, 2023
e2acd49
Python bindings: Chunk distribution algorithms
franzpoeschel Mar 2, 2023
72da398
Use chunk distribtion algorithms in openpmd-pipe
franzpoeschel Mar 2, 2023
539f61f
Testing
franzpoeschel Mar 2, 2023
cda9e76
Add DiscardingStrategy
franzpoeschel Mar 3, 2023
8125286
Make Strategy class extensible from Python
franzpoeschel Mar 15, 2023
a66f656
Make mergeChunks function public
franzpoeschel Mar 14, 2023
8fd934a
Add mergeChunksFromSameSourceID
franzpoeschel Feb 9, 2024
b277169
Add RoundRobinOfSourceRanks strategy
franzpoeschel Feb 8, 2024
79cec41
Add Blocks distribution strategy
franzpoeschel Aug 16, 2024
46213d7
Add BlocksOfSourceRanks strategy
franzpoeschel Aug 16, 2024
7024c3f
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Apr 4, 2025
3ac9533
Add rank info to assign() params
franzpoeschel May 13, 2025
337d21b
Fix the Python trampoline logic
franzpoeschel Jul 15, 2025
14b2559
Use discard strategy as second run, only consider my own hostname
franzpoeschel Jul 14, 2023
941fe81
IncreaseGranularity strategy
franzpoeschel Mar 7, 2023
1e68899
Add blocksofsourcerank to pipe script
franzpoeschel May 7, 2025
2511f6c
CI fixes
franzpoeschel Jul 15, 2025
409b2dd
Move BlockSlicer somewhere else
franzpoeschel Jul 15, 2025
01bcc0f
More thorough documentation
franzpoeschel Jul 15, 2025
4385fc7
Cleanup, documentation, proper use of rank specification
franzpoeschel Jul 17, 2025
e72e1b4
Actual testing
franzpoeschel Jul 17, 2025
10cbdd8
Test and fix ByHostname strategy
franzpoeschel Jul 17, 2025
83efecb
CI fixes
franzpoeschel Jul 17, 2025
206341b
Automate CoreTest
franzpoeschel Jul 18, 2025
d82045c
Guard against unprintableString issue on Windows
franzpoeschel Jul 18, 2025
62e3bf0
Use generic flag instead of -n 2
franzpoeschel Aug 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 25 additions & 25 deletions CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -422,7 +422,7 @@ set(CORE_SOURCE
src/backend/PatchRecord.cpp
src/backend/PatchRecordComponent.cpp
src/backend/Writable.cpp
src/benchmark/mpi/OneDimensionalBlockSlicer.cpp
src/auxiliary/OneDimensionalBlockSlicer.cpp
src/helper/list_series.cpp
src/snapshots/ContainerImpls.cpp
src/snapshots/ContainerTraits.cpp
Expand Down Expand Up @@ -1346,30 +1346,30 @@ if(openPMD_BUILD_TESTING)
)
add_test(NAME CLI.pipe.py
COMMAND sh -c
"${MPI_TEST_EXE} ${Python_EXECUTABLE} \
${openPMD_RUNTIME_OUTPUT_DIRECTORY}/openpmd-pipe \
--infile ../samples/git-sample/data%T.h5 \
--outfile ../samples/git-sample/data%T.bp && \
\
${MPI_TEST_EXE} ${Python_EXECUTABLE} \
${openPMD_RUNTIME_OUTPUT_DIRECTORY}/openpmd-pipe \
--infile ../samples/git-sample/data00000100.h5 \
--outfile \
../samples/git-sample/single_iteration_%T.bp && \
\
${MPI_TEST_EXE} ${Python_EXECUTABLE} \
${openPMD_RUNTIME_OUTPUT_DIRECTORY}/openpmd-pipe \
--infile ../samples/git-sample/thetaMode/data%T.h5 \
--outfile \
../samples/git-sample/thetaMode/data_%T.bp && \
\
${MPI_TEST_EXE} ${Python_EXECUTABLE} \
${openPMD_RUNTIME_OUTPUT_DIRECTORY}/openpmd-pipe \
--infile ../samples/git-sample/thetaMode/data_%T.bp \
--outfile ../samples/git-sample/thetaMode/data%T.json \
--outconfig ' \
json.attribute.mode = \"short\" \n\
json.dataset.mode = \"template_no_warn\"' \
"${MPI_TEST_EXE} ${MPIEXEC_NUMPROC_FLAG} 2 ${Python_EXECUTABLE} \
${openPMD_RUNTIME_OUTPUT_DIRECTORY}/openpmd-pipe \
--infile ../samples/git-sample/data%T.h5 \
--outfile ../samples/git-sample/data%T.bp && \
\
${MPI_TEST_EXE} ${Python_EXECUTABLE} \
${openPMD_RUNTIME_OUTPUT_DIRECTORY}/openpmd-pipe \
--infile ../samples/git-sample/data00000100.h5 \
--outfile \
../samples/git-sample/single_iteration_%T.bp && \
\
${MPI_TEST_EXE} ${MPIEXEC_NUMPROC_FLAG} 2 ${Python_EXECUTABLE} \
${openPMD_RUNTIME_OUTPUT_DIRECTORY}/openpmd-pipe \
--infile ../samples/git-sample/thetaMode/data%T.h5 \
--outfile \
../samples/git-sample/thetaMode/data_%T.bp && \
\
${MPI_TEST_EXE} ${Python_EXECUTABLE} \
${openPMD_RUNTIME_OUTPUT_DIRECTORY}/openpmd-pipe \
--infile ../samples/git-sample/thetaMode/data_%T.bp \
--outfile ../samples/git-sample/thetaMode/data%T.json \
--outconfig ' \
json.attribute.mode = \"short\" \n\
json.dataset.mode = \"template_no_warn\"' \
"
WORKING_DIRECTORY ${openPMD_RUNTIME_OUTPUT_DIRECTORY}
)
Expand Down
Loading
Loading