char variables being scrambled following nc_enddef call

[gsjaardema]

I have a model with several variables including a few text variables:

netcdf result {
dimensions:
        num_qa_rec = 1 ;
        four = 4 ;
        len_string = 33 ;
        num_info = 433 ;
        len_line = 81 ;
        len_name = 101 ;
        num_dim = 3 ;
        time_step = UNLIMITED ; // (0 currently)
        num_nodes = 148977 ;
        num_elem = 796986 ;
        num_el_blk = 3 ;
        num_side_sets = 25 ;
        num_el_in_blk2 = 791921 ;
        num_nod_per_el2 = 4 ;
        num_el_in_blk3 = 5065 ;
        num_nod_per_el3 = 4 ;
        num_side_ss2 = 24564 ;
        num_side_ss5 = 11888 ;
        num_side_ss9 = 72 ;
        num_side_ss12 = 156 ;
        num_side_ss14 = 779 ;
        num_side_ss15 = 515 ;
        num_side_ss17 = 24540 ;
        num_side_ss21 = 36 ;
        num_side_ss25 = 1894 ;
        num_elem_maps = 1 ;
variables:
        char qa_records(num_qa_rec, four, len_string) ;
        char info_records(num_info, len_line) ;
        double time_whole(time_step) ;
        int node_num_map(num_nodes) ;
        int elem_num_map(num_elem) ;
        int eb_status(num_el_blk) ;
        int eb_prop1(num_el_blk) ;
                eb_prop1:name = "ID" ;
        char eb_names(num_el_blk, len_name) ;
        int ss_status(num_side_sets) ;
        int ss_prop1(num_side_sets) ;
                ss_prop1:name = "ID" ;
        char ss_names(num_side_sets, len_name) ;
        double coordx(num_nodes) ;
        double coordy(num_nodes) ;
        double coordz(num_nodes) ;
        char coor_names(num_dim, len_name) ;
        int connect2(num_el_in_blk2, num_nod_per_el2) ;
                connect2:elem_type = "tetra4" ;
        int connect3(num_el_in_blk3, num_nod_per_el3) ;
                connect3:elem_type = "tetra4" ;
        int elem_ss2(num_side_ss2) ;
        int side_ss2(num_side_ss2) ;
        int elem_ss5(num_side_ss5) ;
        int side_ss5(num_side_ss5) ;
        int elem_ss9(num_side_ss9) ;
        int side_ss9(num_side_ss9) ;
        int elem_ss12(num_side_ss12) ;
        int side_ss12(num_side_ss12) ;
        int elem_ss14(num_side_ss14) ;
        int side_ss14(num_side_ss14) ;
        int elem_ss15(num_side_ss15) ;
        int side_ss15(num_side_ss15) ;
        int elem_ss17(num_side_ss17) ;
        int side_ss17(num_side_ss17) ;
        int elem_ss21(num_side_ss21) ;
        int side_ss21(num_side_ss21) ;
        int elem_ss25(num_side_ss25) ;
        int side_ss25(num_side_ss25) ;
        int em_prop1(num_elem_maps) ;
                em_prop1:name = "ID" ;
        char emap_names(num_elem_maps, len_name) ;
                emap_names:_FillValue = "" ;
        int elem_map1(num_elem) ;

The values in the eb_names, ss_names, and coor_names are correct if I ncdump the file following outputting them and prior to a subsequent nc_redef/nc_enddef cycle. Here are some of the values prior to calling nc_enddef

eb_names =
  "block_1",
  "block_1_air",
  "block_1_fluid" ;

 ss_names =
  "surface_1",
  "surface_1_air",
  "surface_1_fluid",
  "surface_2",
  "surface_2_air",
  "surface_2_fluid",
  "surface_3",
  "surface_3_air",
  "surface_3_fluid",
  "surface_4",
  "surface_44",
  "surface_44_air",
  "surface_44_fluid",
  "surface_4_air",
  "surface_4_fluid",
  "surface_5",
  "surface_5_air",
  "surface_5_fluid",
  "surface_6",
  "surface_66",
  "surface_66_air",
  "surface_66_fluid",
  "surface_6_air",
  "surface_6_fluid",
  "surface_air_fluid" ;

The values remain as above up until the next time that nc_enddef is called and then they look like:

 eb_names =
  "block_1\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\001",
  "\000\000\000\000\000\000\001\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\001\000\000\000\001\000\000\000\000\000\000\000\001",
  "block_1_fluid" ;

 ss_names =
  "surface_1",
  "surface_1_air",
  "surface_1_fluid",
  "surface_2",
  "surface_2_air",
  "surface_2_fluid",
  "surface_3",
  "surface_3_air",
  "surface_3_fluid",
  "surface_4",
  "surface_44",
  "surface_44_air",
  "surface_44_f\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000surface_5",
  "\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000surface_5_air",
  "\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000surface_5_flui",
  "d\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000surface_6",
  "\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000surface_66",
  "\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000surface_66_air",
  "\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000surface_66_flu",
  "id\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000surface_6_air",
  "\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000surface_6_flui",
  "d\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000surface_air_fl",
  "uid",
  "",
  "" ;

This is being run on 16 mpi ranks, but I get similar behavior on different rank counts. If I change the netCDF format to be netcdf4, then it runs correctly.

I have tried the following and all give similar results:

• all ranks outputting the same data in collective mode nc_put_var_text
• rank 0 outputs valid data, all other ranks call output function, but output nothing, collective mode, nc_put_vara_text
• rank 0 outputs valid data. Other ranks do not call output function. independent mode

It looks like the issue is due to some bad behavior during the nc_enddef call, but it could also be that I have some option set incorrectly.

This always happens on this particular file, but we have many other instances where the char data is not corrupted, so it seems related to something particular about this data.

This is with PnetCDF-1.13.0, netCDF-4.9.2, Open MPI v4.1.6,

[wkliao]

Before calling ncmpi_put_vara_text_all, did you
fill the trailing space of the write buffer with all '\0'?

Below are the code fragments I would use. Note the calls to memset().

    buf = (char*) malloc(LEN_NAME);

    start[0] = 0;
    start[1] = 0;
    count[0] = 1;
    count[1] = LEN_NAME;

    memset(buf, 0, LEN_NAME);
    strcpy(buf, "block_1");
    ncmpi_put_vara_text_all(ncid, varid, start, count, buf);

    start[0] = 1;
    memset(buf, 0, LEN_NAME);
    strcpy(buf, "block_1_air");
    ncmpi_put_vara_text_all(ncid, varid, start, count, buf);

[gsjaardema]

The memory for the write buffer is obtained via calloc, so would be filled with 0. I am using pnetCDF under netCDF, so I am calling nc_put_var_text to put the entire variable at one time -- char ss_names(num_side_sets, len_name) ;

[gsjaardema]

    size_t name_length = ex_inquire_int(exoid, EX_INQ_DB_MAX_ALLOWED_NAME_LENGTH) + 1;
    char *int_names = calloc(num_names * name_length, 1);

    size_t idx          = 0;
    for (size_t i = 0; i < num_names; i++) {
        ex_copy_string(&int_names[idx], names[i], name_length);
       idx += name_length;
    }

    int status = nc_put_var_text(exoid, varid, int_names);

I have also tried this using nc_put_vara_text with the appropriate count and start arrays and also outputting the names one at a time with nc_put_vara_text....

In all cases, the values look correct on the file until the next nc_redef / nc_enddef set of calls is made at which point several (but not all) of the char arrays are scrambled...

[gsjaardema]

I have also tried with and without the _FillValue specified for the char arrays and that seems to make no difference.

[wkliao]

If you have a reproducer, that will be helpful.
In the meantimes, I am also writing one to see if I can reproduce.

One question, what netCDF APIs are called during each pair of nc_redef / nc_enddef?

[wkliao]

I am using this test program
but could not reproduce the problem.

[gsjaardema]

One question, what netCDF APIs are called during each pair of nc_redef / nc_enddef?

There is a new dimension and variable defined:

dimensions:
       ...
        num_elem_maps = 1 ;

variables:
       ...
        int em_prop1(num_elem_maps) ;
                em_prop1:name = "ID" ;
        char emap_names(num_elem_maps, len_name) ;
                emap_names:_FillValue = "" ;
        int elem_map1(num_elem) ;

Once the nc_enddef is called following that, the ss_names is scrambled...

I've run on different processor counts and some work and some don't. I haven't found any pattern in which work and which dont'...

I also ran with the gcc address sanitizer and although it doesn't find any errors, it does change the results such that only the qa_records char value has it's last entry somewhat scrambled and all other names are good. Not sure what this tells us other than that moving things around in memory affect the results(?).

[wkliao]

I have modified my test program to move those dimension and variables into
the redefine area. I tested it with valgrind and also found no warning.

If you can share your program, I can give it a try. Otherwise,
please fell free to modify my test program.

Maybe try MPICH?

Once the nc_enddef is called following that, the ss_names is scrambled...

Do you mean nc_close is called and application exits after nc_enddef, and then
ncdump utility is called to dump the file contents? I don't know how to run
ncdump to dump a file while the application is still running.

[gsjaardema]

Do you mean nc_close is called and application exits after nc_enddef, and then
ncdump utility is called to dump the file contents?

No, if I run in the debugger, I can stop right after the nc_enddef call and then run ncdump on the still open file. It gives me valid output which matches the output if I do the ncdump after nc_close is called. This doesn't usually work with a netcdf-4 format file, but I have had good success in the past which netcdf-3 and netcdf-5 format files.

I will try some other options and see if they make a difference. I don't know if there are some alignment issues involved since the len_name is not even or power of 2...

[wkliao]

To avoid the file system consistency issue, can you try adding
a call to nc_sync after nc_enddef?

[wkliao]

One question, have you tried to call nc_get APIs to read back those
variables after nc_enddef? I wonder if the contents are correct.

[gsjaardema]

Hold off on this for awhile until I check some more. I updated to the latest 1.14 version (had been running 1.13) and at least one of the tests is no longer failing. It will take a little while to update the other more involved test to 1.14, but will let you know how that turns out...

[gsjaardema]

OK, updating to the latest version seemed to fix the smaller program I was using that had been reproducing the problem, but when tried on the larger application, it stil failed.

I did find that setting export PNETCDF_HINTS="nc_header_align_size=1048576"

Does fix the problem on the large application. Basically, as I understand it, this will eliminate most/all rewrites when doing a nc_redef/nc_enddef cycle as long as the header doesn't exceed that size which it isn't in this case.

This does point to some issue, I think, in the header expansion code, but haven't any idea of why this particular file is hitting it.

[gsjaardema]

If I omit the PNETCDF_HINTS or set it to a small value (1024), then the corruption returns with the exact same executable.

[wkliao]

Is it possible for you to create a reproducer?