|
| 1 | +This file contains some basic information on how to use the ADIOS to NetCDF conversion tool for converting SCORPIO output files in the ADIOS BP file format to NetCDF. The conversion tool reads the ADIOS BP files in parallel (using SCORPIO and ADIOS) and converts the file using SCORPIO and PnetCDF or NetCDF libraries. |
| 2 | + |
| 3 | +Running the application |
| 4 | +------------------------ |
| 5 | +If SCORPIO is enabled with support for ADIOS (see the configure log of SCORPIO to see if ADIOS support is enabled), using PIO_IOTYPE_ADIOS (for uncompressed output) or PIO_IOTYPE_ADIOSC (for compressed output) will result in all SCORPIO output being written in the ADIOS BP file format. The output files in the ADIOS BP file format can be converted to NetCDF using the conversion tool in a separate batch job. |
| 6 | + |
| 7 | +The conversion tool is an MPI program and can be run with different number (i.e., fewer) of MPI processes compared to the application that generated the output. |
| 8 | + |
| 9 | + |
| 10 | +--------------------------------------------------------------------------------------- |
| 11 | +EXAMPLES: |
| 12 | +--------------------------------------------------------------------------------------- |
| 13 | + |
| 14 | +# Program options |
| 15 | + |
| 16 | +jayesh@compute-386-03:~/scorpio_build2$ ./tools/adios2pio-nm/adios2pio-nm.exe |
| 17 | +Usage : ./tools/adios2pio-nm/adios2pio-nm.exe --[OPTIONAL ARG1 NAME]=[OPTIONAL ARG1 VALUE] --[OPTIONAL ARG2 NAME]=[OPTIONAL ARG2 VALUE] ... |
| 18 | +Optional Arguments : |
| 19 | +--bp-file: data produced by SCORPIO with ADIOS format |
| 20 | +--idir: Directory containing data output from SCORPIO (in ADIOS format) |
| 21 | +--nc-file: output file name after conversion |
| 22 | +--pio-format: output SCORPIO I/O type. Supported parameters: "pnetcdf", "netcdf", "netcdf4c", "netcdf4p", "nczarr" |
| 23 | +--rearr: SCORPIO rearranger. Supported parameters: "subset", "box", "any". Default "any". |
| 24 | +--rm-bp-file-after-conv: remove the ADIOS BP file after conversion |
| 25 | +--verbose: Turn on verbose info messages |
| 26 | + |
| 27 | + |
| 28 | +# Delete all ADIOS BP files after conversion |
| 29 | +jayesh@compute-386-03:~/scorpio_build2$ ./tools/adios2pio-nm/adios2pio-nm.exe --bp-file=/home/jayesh/scorpio_build2/tests/general/test_pio_file_simple_tests.testfile.bp --rm-bp-file-after-conv |
| 30 | +Converting BP file : "/home/jayesh/scorpio_build2/tests/general/test_pio_file_simple_tests.testfile.bp" |
| 31 | +Deleting BP file : /home/jayesh/scorpio_build2/tests/general/test_pio_file_simple_tests.testfile.bp |
| 32 | + |
| 33 | +# Delete all ADIOS BP files with name "(.*)(test_pio_file_simple)(.*)" after conversion |
| 34 | +jayesh@compute-386-03:~/scorpio_build2$ ./tools/adios2pio-nm/adios2pio-nm.exe --bp-file=/home/jayesh/scorpio_build2/tests/general/test_pio_file_simple_tests.testfile.bp --rm-bp-file-after-conv="(.*)(test_pio_file_simple)(.*)" |
| 35 | +Converting BP file : "/home/jayesh/scorpio_build2/tests/general/test_pio_file_simple_tests.testfile.bp" |
| 36 | +Deleting BP file : /home/jayesh/scorpio_build2/tests/general/test_pio_file_simple_tests.testfile.bp |
| 37 | + |
| 38 | +jayesh@compute-386-03:~/scorpio_build2$ ./tools/adios2pio-nm/adios2pio-nm.exe --bp-file=/home/jayesh/scorpio_build2/tests/general/test_pio_file_simple_tests.testfile.bp --rm-bp-file-after-conv="(.*)(DONT_DELETE_test_pio_file_simple)(.*)" |
| 39 | +Converting BP file : "/home/jayesh/scorpio_build2/tests/general/test_pio_file_simple_tests.testfile.bp" |
| 40 | + |
| 41 | +# Convert ADIOS BP output files in directory, /scratch/jayesh/e3sm/scratch/test_F2010_ne4_oQU240/run, and only delete the EAM ADIOS BP files after conversion |
| 42 | +jayesh@compute-386-03:~/scorpio_build2$ mpiexec -n 8 ./tools/adios2pio-nm/adios2pio-nm.exe --idir=/scratch/jayesh/e3sm/scratch/test_F2010_ne4_oQU240/run --rm-bp-file-after-conv="(.*)(.eam.)(.*)" |
| 43 | +Converting BP file : "/scratch/jayesh/e3sm/scratch/test_F2010_ne4_oQU240/run/test_F2010_ne4_oQU240.eam.rs.0001-01-02-00000.nc.bp" |
| 44 | +Deleting BP file : /scratch/jayesh/e3sm/scratch/test_F2010_ne4_oQU240/run/test_F2010_ne4_oQU240.eam.rs.0001-01-02-00000.nc.bp |
| 45 | +Converting BP file : "/scratch/jayesh/e3sm/scratch/test_F2010_ne4_oQU240/run/test_F2010_ne4_oQU240.mosart.rh0.0001-01-02-00000.nc.bp" |
| 46 | +Converting BP file : "/scratch/jayesh/e3sm/scratch/test_F2010_ne4_oQU240/run/test_F2010_ne4_oQU240.elm.rh0.0001-01-02-00000.nc.bp" |
| 47 | +Converting BP file : "/scratch/jayesh/e3sm/scratch/test_F2010_ne4_oQU240/run/test_F2010_ne4_oQU240.cpl.r.0001-01-02-00000.nc.bp" |
| 48 | +Converting BP file : "/scratch/jayesh/e3sm/scratch/test_F2010_ne4_oQU240/run/test_F2010_ne4_oQU240.elm.r.0001-01-02-00000.nc.bp" |
| 49 | +Converting BP file : "/scratch/jayesh/e3sm/scratch/test_F2010_ne4_oQU240/run/test_F2010_ne4_oQU240.eam.h0.0001-01-01-00000.nc.bp" |
| 50 | +Deleting BP file : /scratch/jayesh/e3sm/scratch/test_F2010_ne4_oQU240/run/test_F2010_ne4_oQU240.eam.h0.0001-01-01-00000.nc.bp |
| 51 | +Converting BP file : "/scratch/jayesh/e3sm/scratch/test_F2010_ne4_oQU240/run/test_F2010_ne4_oQU240.eam.r.0001-01-02-00000.nc.bp" |
| 52 | +Deleting BP file : /scratch/jayesh/e3sm/scratch/test_F2010_ne4_oQU240/run/test_F2010_ne4_oQU240.eam.r.0001-01-02-00000.nc.bp |
| 53 | +Converting BP file : "/scratch/jayesh/e3sm/scratch/test_F2010_ne4_oQU240/run/test_F2010_ne4_oQU240.mosart.r.0001-01-02-00000.nc.bp" |
| 54 | + |
0 commit comments