Richards and TH demos exhibit problems in DMPlex matrix preallocation #237
Description
In #236, we updated to PETSc v3.17.3, and a couple of demos stopped working, encountering errors in code that was likely changed in that release. Here's what we've found so far.
Richards demo (richards-driver-snes-prob1-np4)
Console output (limited to process rank 3 for simplicity)
$ cd demo/richards
$ make
$ mpiexec -n 4 valgrind --log-file=poo ./richards_driver -malloc 0 -successful_exit_code 0 -dm_plex_simplex 0 -dm_plex_dim 3 -dm_plex_box_faces 2,2,2 -dm_plex_box_lower 0,0,0 -dm_plex_box_upper 1,1,1 -tdy_water_density exponential -tdy_regression_test -tdy_regression_test_num_cells_per_process 1 -tdy_regression_test_filename richards-driver-snes-prob1-np4 -tdy_final_time 3.1536e1 -tdy_dt_max 600. -tdy_dt_growth_factor 1.0 -tdy_init_with_random_field -tdy_time_integration
_method SNES -tdy_dt_init 0.1
+++++++++++++++++ TDycore +++++++++++++++++
Creating TDycore
Beginning Richards Driver simulation.
TDycore setup
Using TDycore backward Euler (and PETSc SNES) for time integration.
Creating Vectors
Creating Jacobian matrix
[3]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[3]PETSC ERROR: Argument out of range
[3]PETSC ERROR: Section point -1 should be in [6, 8)
[3]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[3]PETSC ERROR: Petsc Release Version 3.17.3, unknown
[3]PETSC ERROR: ./richards_driver on a debug named crunchy by jeff Thu Jun 30 09:40:42 2022
[3]PETSC ERROR: #1 PetscSectionSetDof() at /home/jeff/projects/pnnl/petsc/src/vec/is/section/interface/section.c:801
[3]PETSC ERROR: #2 DMPlexCreateAdjacencySection_Static() at /home/jeff/projects/pnnl/petsc/src/dm/impls/plex/plexpreallocate.c:472
[3]PETSC ERROR: #3 DMPlexPreallocateOperator() at /home/jeff/projects/pnnl/petsc/src/dm/impls/plex/plexpreallocate.c:777
[3]PETSC ERROR: #4 DMCreateMatrix_Plex() at /home/jeff/projects/pnnl/petsc/src/dm/impls/plex/plex.c:2544
[3]PETSC ERROR: #5 DMCreateMatrix() at /home/jeff/projects/pnnl/petsc/src/dm/interface/dm.c:1432
[3]PETSC ERROR: #6 TDyDiscretizationCreateJacobianMatrix() at /home/jeff/projects/pnnl/TDycore/src/tdydiscretization.c:166
[3]PETSC ERROR: #7 TDyCreateJacobian() at /home/jeff/projects/pnnl/TDycore/src/tdycore.c:2032
[3]PETSC ERROR: #8 TDyDriverInitializeTDy() at /home/jeff/projects/pnnl/TDycore/src/tdydriver.c:87
[3]PETSC ERROR: #9 main() at /home/jeff/projects/pnnl/TDycore/demo/richards/richards_driver.c:33
[3]PETSC ERROR: PETSc Option Table entries:
[3]PETSC ERROR: -dm_plex_box_faces 2,2,2
[3]PETSC ERROR: -dm_plex_box_lower 0,0,0
[3]PETSC ERROR: -dm_plex_box_upper 1,1,1
[3]PETSC ERROR: -dm_plex_dim 3
[3]PETSC ERROR: -dm_plex_simplex 0
[3]PETSC ERROR: -malloc 0
[3]PETSC ERROR: -successful_exit_code 0
[3]PETSC ERROR: -tdy_dt_growth_factor 1.0
[3]PETSC ERROR: -tdy_dt_init 0.1
[3]PETSC ERROR: -tdy_dt_max 600.
[3]PETSC ERROR: -tdy_final_time 3.1536e1
[3]PETSC ERROR: -tdy_init_with_random_field
[3]PETSC ERROR: -tdy_regression_test
[3]PETSC ERROR: -tdy_regression_test_filename richards-driver-snes-prob1-np4
[3]PETSC ERROR: -tdy_regression_test_num_cells_per_process 1
[3]PETSC ERROR: -tdy_time_integration_method SNES
[3]PETSC ERROR: -tdy_water_density exponential
[3]PETSC ERROR: ----------------End of Error Message -------send entire error message to [email protected]
Valgrind report
...
==2251883== Invalid write of size 2
==2251883== at 0x48529E3: memmove (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2251883== by 0x76D7D6A: PetscMemcpy (petscsys.h:1634)
==2251883== by 0x76DA65D: UnpackAndInsert_PetscInt_1_1 (sfpack.c:374)
==2251883== by 0x784A8F1: PetscSFLinkUnpackLeafData_Private (sfpack.c:1094)
==2251883== by 0x784B77B: PetscSFLinkUnpackLeafData (sfpack.c:1124)
==2251883== by 0x76D3D3D: PetscSFBcastEnd_Basic (sfbasic.c:212)
==2251883== by 0x7690ADA: PetscSFBcastEnd (sf.c:1472)
==2251883== by 0x76B9B04: PetscSFCreateRemoteOffsets (sfutils.c:334)
==2251883== by 0x85E2662: DMPlexPreallocateOperator (plexpreallocate.c:753)
==2251883== by 0x84FB91E: DMCreateMatrix_Plex (plex.c:2544)
==2251883== by 0x82BB145: DMCreateMatrix (dm.c:1432)
==2251883== by 0x49798CE: TDyDiscretizationCreateJacobianMatrix (tdydiscretization.c:166)
==2251883== Address 0x22a10808 is 8 bytes after a block of size 32 alloc'd
==2251883== at 0x484E120: memalign (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2251883== by 0x71417EA: PetscMallocAlign (mal.c:48)
==2251883== by 0x71438F1: PetscMallocA (mal.c:414)
==2251883== by 0x76B99BE: PetscSFCreateRemoteOffsets (sfutils.c:332)
==2251883== by 0x85E2662: DMPlexPreallocateOperator (plexpreallocate.c:753)
==2251883== by 0x84FB91E: DMCreateMatrix_Plex (plex.c:2544)
==2251883== by 0x82BB145: DMCreateMatrix (dm.c:1432)
==2251883== by 0x49798CE: TDyDiscretizationCreateJacobianMatrix (tdydiscretization.c:166)
==2251883== by 0x4977092: TDyCreateJacobian (tdycore.c:2032)
==2251883== by 0x497CC0D: TDyDriverInitializeTDy (tdydriver.c:87)
==2251883== by 0x109B04: main (richards_driver.c:33)
==2251883==
==2251883== Invalid write of size 8
==2251883== at 0x4852990: memmove (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2251883== by 0x76D7D6A: PetscMemcpy (petscsys.h:1634)
==2251883== by 0x76DA65D: UnpackAndInsert_PetscInt_1_1 (sfpack.c:374)
==2251883== by 0x784A8F1: PetscSFLinkUnpackLeafData_Private (sfpack.c:1094)
==2251883== by 0x784B77B: PetscSFLinkUnpackLeafData (sfpack.c:1124)
==2251883== by 0x76D3D3D: PetscSFBcastEnd_Basic (sfbasic.c:212)
==2251883== by 0x7690ADA: PetscSFBcastEnd (sf.c:1472)
==2251883== by 0x76B9B04: PetscSFCreateRemoteOffsets (sfutils.c:334)
==2251883== by 0x85E2662: DMPlexPreallocateOperator (plexpreallocate.c:753)
==2251883== by 0x84FB91E: DMCreateMatrix_Plex (plex.c:2544)
==2251883== by 0x82BB145: DMCreateMatrix (dm.c:1432)
==2251883== by 0x49798CE: TDyDiscretizationCreateJacobianMatrix (tdydiscretization.c:166)
==2251883== Address 0x22a10800 is 0 bytes after a block of size 32 alloc'd
==2251883== at 0x484E120: memalign (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2251883== by 0x71417EA: PetscMallocAlign (mal.c:48)
==2251883== by 0x71438F1: PetscMallocA (mal.c:414)
==2251883== by 0x76B99BE: PetscSFCreateRemoteOffsets (sfutils.c:332)
==2251883== by 0x85E2662: DMPlexPreallocateOperator (plexpreallocate.c:753)
==2251883== by 0x84FB91E: DMCreateMatrix_Plex (plex.c:2544)
==2251883== by 0x82BB145: DMCreateMatrix (dm.c:1432)
==2251883== by 0x49798CE: TDyDiscretizationCreateJacobianMatrix (tdydiscretization.c:166)
==2251883== by 0x4977092: TDyCreateJacobian (tdycore.c:2032)
==2251883== by 0x497CC0D: TDyDriverInitializeTDy (tdydriver.c:87)
==2251883== by 0x109B04: main (richards_driver.c:33)
TH demo (th-driver-ts-prob1-np4)
Console output (all ranks included, with -dm_view_preallocation)
$ mpirun -np 4 valgrind --log-file=poo ./th_driver -dm_plex_simplex 0 -dm_plex_dim 3 -dm_plex_box_faces 2,2,2 -dm_plex_box_lower 0,0,0 -dm_plex_box_upper 1,1,1 -tdy_water_density exponential -tdy_regression_test -tdy_regression_test_num_cells_per_process 1 -tdy_regression_test_filename th-driver-ts-prob1-np4 -tdy_final_time 3.1536e3 -tdy_dt_max 600. -tdy_dt_growth_factor 1.5 -tdy_timers -tdy_init_with_random_field -tdy_time_integration_method TS -dm_view_preallocation
+++++++++++++++++ TDycore +++++++++++++++++
Creating TDycore
Running TH mode.
Beginning TH Driver simulation.
TDycore setup
Using PETSc TS for time integration.
Creating Vectors
Creating Jacobian matrix
Input Section for Preallocation:
PetscSection Object: 4 MPI processes
type not yet set
2 fields
field 0 with 1 components
Process 0:
( 0) dim 1 offset 0
( 1) dim 1 offset 2
( 2) dim 1 offset 4
( 3) dim 1 offset 6
( 4) dim 1 offset 8
( 5) dim 1 offset 10
( 6) dim 1 offset 12
( 7) dim 1 offset 14
Process 1:
( 0) dim 1 offset 0
( 1) dim 1 offset 2
( 2) dim 1 offset 4
( 3) dim 1 offset 6
( 4) dim 1 offset 8
( 5) dim 1 offset 10
( 6) dim 1 offset 12
( 7) dim 1 offset 14
Process 2:
( 0) dim 1 offset 0
( 1) dim 1 offset 2
( 2) dim 1 offset 4
( 3) dim 1 offset 6
( 4) dim 1 offset 8
( 5) dim 1 offset 10
( 6) dim 1 offset 12
( 7) dim 1 offset 14
Process 3:
( 0) dim 1 offset 0
( 1) dim 1 offset 2
( 2) dim 1 offset 4
( 3) dim 1 offset 6
( 4) dim 1 offset 8
( 5) dim 1 offset 10
( 6) dim 1 offset 12
( 7) dim 1 offset 14
field 1 with 1 components
Process 0:
( 0) dim 1 offset 1
( 1) dim 1 offset 3
( 2) dim 1 offset 5
( 3) dim 1 offset 7
( 4) dim 1 offset 9
( 5) dim 1 offset 11
( 6) dim 1 offset 13
( 7) dim 1 offset 15
Process 1:
( 0) dim 1 offset 1
( 1) dim 1 offset 3
( 2) dim 1 offset 5
( 3) dim 1 offset 7
( 4) dim 1 offset 9
( 5) dim 1 offset 11
( 6) dim 1 offset 13
( 7) dim 1 offset 15
Process 2:
( 0) dim 1 offset 1
( 1) dim 1 offset 3
( 2) dim 1 offset 5
( 3) dim 1 offset 7
( 4) dim 1 offset 9
( 5) dim 1 offset 11
( 6) dim 1 offset 13
( 7) dim 1 offset 15
Process 3:
( 0) dim 1 offset 1
( 1) dim 1 offset 3
( 2) dim 1 offset 5
( 3) dim 1 offset 7
( 4) dim 1 offset 9
( 5) dim 1 offset 11
( 6) dim 1 offset 13
( 7) dim 1 offset 15
Input Global Section for Preallocation:
PetscSection Object: 4 MPI processes
type not yet set
2 fields
field 0 with 1 components
Process 0:
( 0) dim 1 offset 0
( 1) dim 1 offset 2
( 2) dim -2 offset -5
( 3) dim -2 offset -7
( 4) dim -2 offset -13
( 5) dim -2 offset -15
( 6) dim -2 offset -9
( 7) dim -2 offset -11
Process 1:
( 0) dim 1 offset 4
( 1) dim 1 offset 6
( 2) dim -2 offset -1
( 3) dim -2 offset -3
( 4) dim -2 offset -13
( 5) dim -2 offset -15
( 6) dim -2 offset -9
( 7) dim -2 offset -11
Process 2:
( 0) dim 1 offset 8
( 1) dim 1 offset 10
( 2) dim -2 offset -1
( 3) dim -2 offset -3
( 4) dim -2 offset -13
( 5) dim -2 offset -15
( 6) dim -2 offset -5
( 7) dim -2 offset -7
Process 3:
( 0) dim 1 offset 12
( 1) dim 1 offset 14
( 2) dim -2 offset -1
( 3) dim -2 offset -3
( 4) dim -2 offset -9
( 5) dim -2 offset -11
( 6) dim -2 offset -5
( 7) dim -2 offset -7
field 1 with 1 components
Process 0:
( 0) dim 1 offset 1
( 1) dim 1 offset 3
( 2) dim -2 offset -6
( 3) dim -2 offset -8
( 4) dim -2 offset -14
( 5) dim -2 offset -16
( 6) dim -2 offset -10
( 7) dim -2 offset -12
Process 1:
( 0) dim 1 offset 5
( 1) dim 1 offset 7
( 2) dim -2 offset -2
( 3) dim -2 offset -4
( 4) dim -2 offset -14
( 5) dim -2 offset -16
( 6) dim -2 offset -10
( 7) dim -2 offset -12
Process 2:
( 0) dim 1 offset 9
( 1) dim 1 offset 11
( 2) dim -2 offset -2
( 3) dim -2 offset -4
( 4) dim -2 offset -14
( 5) dim -2 offset -16
( 6) dim -2 offset -6
( 7) dim -2 offset -8
Process 3:
( 0) dim 1 offset 13
( 1) dim 1 offset 15
( 2) dim -2 offset -2
( 3) dim -2 offset -4
( 4) dim -2 offset -10
( 5) dim -2 offset -12
( 6) dim -2 offset -6
( 7) dim -2 offset -8
Input SF for Preallocation:
PetscSF Object: 4 MPI processes
type: basic
[0] Number of roots=45, leaves=25, remote ranks=3
[0] 5 <- (3,10)
[0] 6 <- (1,2)
[0] 7 <- (3,12)
[0] 8 <- (2,6)
[0] 9 <- (3,2)
[0] 10 <- (2,9)
[0] 11 <- (3,8)
[0] 12 <- (2,4)
[0] 13 <- (2,5)
[0] 21 <- (3,20)
[0] 22 <- (1,14)
[0] 23 <- (2,22)
[0] 24 <- (2,18)
[0] 33 <- (3,38)
[0] 34 <- (3,40)
[0] 35 <- (1,27)
[0] 36 <- (1,29)
[0] 37 <- (3,43)
[0] 38 <- (2,33)
[0] 39 <- (2,35)
[0] 40 <- (2,37)
[0] 41 <- (3,33)
[0] 42 <- (2,30)
[0] 43 <- (2,31)
[0] 44 <- (2,32)
[1] Number of roots=45, leaves=21, remote ranks=2
[1] 4 <- (3,12)
[1] 5 <- (3,13)
[1] 6 <- (3,2)
[1] 7 <- (3,3)
[1] 8 <- (3,4)
[1] 9 <- (3,5)
[1] 10 <- (2,3)
[1] 12 <- (2,5)
[1] 17 <- (3,24)
[1] 19 <- (3,14)
[1] 20 <- (2,15)
[1] 26 <- (3,42)
[1] 31 <- (3,43)
[1] 32 <- (3,44)
[1] 33 <- (3,25)
[1] 34 <- (3,26)
[1] 35 <- (3,27)
[1] 36 <- (3,28)
[1] 38 <- (2,27)
[1] 40 <- (2,29)
[1] 43 <- (2,32)
[2] Number of roots=45, leaves=9, remote ranks=1
[2] 7 <- (3,2)
[2] 11 <- (3,4)
[2] 12 <- (3,6)
[2] 13 <- (3,8)
[2] 24 <- (3,15)
[2] 41 <- (3,27)
[2] 42 <- (3,31)
[2] 43 <- (3,33)
[2] 44 <- (3,35)
[3] Number of roots=45, leaves=0, remote ranks=0
MultiSF sort=rank-order
[0]PETSC ERROR: PetscTrFreeDefault() called from DMPlexPreallocateOperator() at /home/jeff/projects/pnnl/petsc/src/dm/impls/plex/plexpreallocate.c:755
[0]PETSC ERROR: [1]PETSC ERROR: PetscTrFreeDefault() called from DMPlexPreallocateOperator() at /home/jeff/projects/pnnl/petsc/src/dm/impls/plex/plexpreallocate.c:755
[1]PETSC ERROR: Block [id=24084(32)] at address 0xfcdc410 is corrupted (probably write past end of array)
[0]PETSC ERROR: Block allocated in PetscSFCreateRemoteOffsets() at /home/jeff/projects/pnnl/petsc/src/vec/is/sf/utils/sfutils.c:332
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Memory corruption: https://petsc.org/release/faq/#valgrind
[0]PETSC ERROR: Corrupted memory
[0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.17.3, unknown
Block [id=23584(32)] at address 0x224766e0 is corrupted (probably write past end of array)
[0]PETSC ERROR: [1]PETSC ERROR: ./th_driver on a debug named crunchy by jeff Wed Jun 29 12:28:12 2022
Block allocated in PetscSFCreateRemoteOffsets() at /home/jeff/projects/pnnl/petsc/src/vec/is/sf/utils/sfutils.c:332
[0]PETSC ERROR: Configure options --CFLAGS=-g --CXXFLAGS=-g --FFLAGS="-g -Wno-unused-function" --download-exodusii --download-fblaslapack --download-hdf5 --download-metis --download-netcdf --download-parmetis --download-pnetcdf --with-mpi-pkg-config=/usr --with-clanguage=c --with-debugging=1 --with-shared-libraries=1 --with-zlib PETSC_ARCH=debug
[0]PETSC ERROR: #1 PetscTrFreeDefault() at /home/jeff/projects/pnnl/petsc/src/sys/memory/mtr.c:306
[0]PETSC ERROR: #2 DMPlexPreallocateOperator() at /home/jeff/projects/pnnl/petsc/src/dm/impls/plex/plexpreallocate.c:755
[1]PETSC ERROR: [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
#3 DMCreateMatrix_Plex() at /home/jeff/projects/pnnl/petsc/src/dm/impls/plex/plex.c:2544
[0]PETSC ERROR: #4 DMCreateMatrix() at /home/jeff/projects/pnnl/petsc/src/dm/interface/dm.c:1432
[1]PETSC ERROR: Memory corruption: https://petsc.org/release/faq/#valgrind
[0]PETSC ERROR: [1]PETSC ERROR: Corrupted memory
#5 TDyDiscretizationCreateJacobianMatrix() at /home/jeff/projects/pnnl/TDycore/src/tdydiscretization.c:166
[1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[1]PETSC ERROR: Petsc Release Version 3.17.3, unknown
[1]PETSC ERROR: ./th_driver on a debug named crunchy by jeff Wed Jun 29 12:28:12 2022
[1]PETSC ERROR: Configure options --CFLAGS=-g --CXXFLAGS=-g --FFLAGS="-g -Wno-unused-function" --download-exodusii --download-fblaslapack --download-hdf5 --download-metis --download-netcdf --download-parmetis --download-pnetcdf --with-mpi-pkg-config=/usr --with-clanguage=c --with-debugging=1 --with-shared-libraries=1 --with-zlib PETSC_ARCH=debug
[0]PETSC ERROR: [1]PETSC ERROR: #6 TDyCreateJacobian() at /home/jeff/projects/pnnl/TDycore/src/tdycore.c:2032
#1 PetscTrFreeDefault() at /home/jeff/projects/pnnl/petsc/src/sys/memory/mtr.c:306
[0]PETSC ERROR: #7 TDyDriverInitializeTDy() at /home/jeff/projects/pnnl/TDycore/src/tdydriver.c:87
[1]PETSC ERROR: #2 DMPlexPreallocateOperator() at /home/jeff/projects/pnnl/petsc/src/dm/impls/plex/plexpreallocate.c:755
[0]PETSC ERROR: #8 main() at /home/jeff/projects/pnnl/TDycore/demo/th/th_driver.c:33
[1]PETSC ERROR: #3 DMCreateMatrix_Plex() at /home/jeff/projects/pnnl/petsc/src/dm/impls/plex/plex.c:2544
[0]PETSC ERROR: Reached the main program with an out-of-range error code 1. This should never happen
[1]PETSC ERROR: #4 DMCreateMatrix() at /home/jeff/projects/pnnl/petsc/src/dm/interface/dm.c:1432
[1]PETSC ERROR: [0]PETSC ERROR: #5 TDyDiscretizationCreateJacobianMatrix() at /home/jeff/projects/pnnl/TDycore/src/tdydiscretization.c:166
...
Valgrind report
...
==1924118== Invalid write of size 4
==1924118== at 0x76DA7BA: UnpackAndInsert_PetscInt_1_1 (sfpack.c:374)
==1924118== by 0x784A8F1: PetscSFLinkUnpackLeafData_Private (sfpack.c:1094)
==1924118== by 0x784B77B: PetscSFLinkUnpackLeafData (sfpack.c:1124)
==1924118== by 0x76D3D3D: PetscSFBcastEnd_Basic (sfbasic.c:212)
==1924118== by 0x7690ADA: PetscSFBcastEnd (sf.c:1472)
==1924118== by 0x76B9B04: PetscSFCreateRemoteOffsets (sfutils.c:334)
==1924118== by 0x85E2662: DMPlexPreallocateOperator (plexpreallocate.c:753)
==1924118== by 0x84FB91E: DMCreateMatrix_Plex (plex.c:2544)
==1924118== by 0x82BB145: DMCreateMatrix (dm.c:1432)
==1924118== by 0x49798CE: TDyDiscretizationCreateJacobianMatrix (tdydiscreti
==1924118== by 0x4977092: TDyCreateJacobian (tdycore.c:2032)
==1924118== by 0x497CC0D: TDyDriverInitializeTDy (tdydriver.c:87)
==1924118== Address 0x2192f5cc is 8 bytes after a block of size 1,652 alloc'd
==1924118== at 0x484E120: memalign (in /usr/libexec/valgrind/vgpreload_memch
==1924118== by 0x71417EA: PetscMallocAlign (mal.c:48)
==1924118== by 0x7145957: PetscTrMallocDefault (mtr.c:183)
==1924118== by 0x71438F1: PetscMallocA (mal.c:414)
==1924118== by 0x76B99BE: PetscSFCreateRemoteOffsets (sfutils.c:332)
==1924118== by 0x85E2662: DMPlexPreallocateOperator (plexpreallocate.c:753)
==1924118== by 0x84FB91E: DMCreateMatrix_Plex (plex.c:2544)
==1924118== by 0x82BB145: DMCreateMatrix (dm.c:1432)
==1924118== by 0x49798CE: TDyDiscretizationCreateJacobianMatrix (tdydiscreti
==1924118== by 0x4977092: TDyCreateJacobian (tdycore.c:2032)
==1924118== by 0x497CC0D: TDyDriverInitializeTDy (tdydriver.c:87)
==1924118== by 0x109B04: main (th_driver.c:33)
==1924118==
iver.c:33)
==1924117==
==1924117== Invalid write of size 8
==1924117== at 0x4852990: memmove (in /usr/libexec/valgrind/vgpreload_memche
==1924117== by 0x76D7D6A: PetscMemcpy (petscsys.h:1634)
==1924117== by 0x76DA65D: UnpackAndInsert_PetscInt_1_1 (sfpack.c:374)
==1924117== by 0x784A8F1: PetscSFLinkUnpackLeafData_Private (sfpack.c:1094)
==1924117== by 0x784B77B: PetscSFLinkUnpackLeafData (sfpack.c:1124)
==1924117== by 0x76D3D3D: PetscSFBcastEnd_Basic (sfbasic.c:212)
==1924117== by 0x7690ADA: PetscSFBcastEnd (sf.c:1472)
==1924117== by 0x76B9B04: PetscSFCreateRemoteOffsets (sfutils.c:334)
==1924117== by 0x85E2662: DMPlexPreallocateOperator (plexpreallocate.c:753)
==1924117== by 0x84FB91E: DMCreateMatrix_Plex (plex.c:2544)
==1924117== by 0x82BB145: DMCreateMatrix (dm.c:1432)
==1924117== by 0x49798CE: TDyDiscretizationCreateJacobianMatrix (tdydiscreti
==1924117== Address 0x22476d50 is 1,648 bytes inside a block of size 1,652 all
==1924117== at 0x484E120: memalign (in /usr/libexec/valgrind/vgpreload_memch
==1924117== by 0x71417EA: PetscMallocAlign (mal.c:48)
==1924117== by 0x7145957: PetscTrMallocDefault (mtr.c:183)
==1924117== by 0x71438F1: PetscMallocA (mal.c:414)
==1924117== by 0x76B99BE: PetscSFCreateRemoteOffsets (sfutils.c:332)
==1924117== by 0x85E2662: DMPlexPreallocateOperator (plexpreallocate.c:753)
==1924117== by 0x84FB91E: DMCreateMatrix_Plex (plex.c:2544)
==1924117== by 0x82BB145: DMCreateMatrix (dm.c:1432)
==1924117== by 0x49798CE: TDyDiscretizationCreateJacobianMatrix (tdydiscreti
==1924117== by 0x4977092: TDyCreateJacobian (tdycore.c:2032)
==1924117== by 0x497CC0D: TDyDriverInitializeTDy (tdydriver.c:87)
==1924117== by 0x109B04: main (th_driver.c:33)