creating large matched meshes from connectivity and coordinates

[cwsmith]

A large structured hex (eventually mixed) mesh is needed and there are issues getting a mesh generator to easily produce this.

The matchedReader branch contains a driver named matchedNodeElmReader that calls the apf::Construct api to create an mds/pumi mesh from a files containing connectivity and coordinates.

There are no new/special compilation flags needed to build this branch of core and the driver.

Run the driver with the following command:

mpirun -np <numProcs> ./matchedNodeElmReader box_small.cnn box_small.crd NULL outModel.dmg outMesh.smb

NULL is the place holder argument for the file containing the matching info.

The box_small files are attached in a tarball: box_small.tar.gz

The format for the .cnn, .crd, and .match files is:

$ head -5 box_small.cnn
1 1 2 53 52 2602 2603 2654 2653
2 2 3 54 53 2603 2604 2655 2654
3 3 4 55 54 2604 2605 2656 2655
4 4 5 56 55 2605 2606 2657 2656
5 5 6 57 56 2606 2607 2658 2657
….
el# global_node1:8

$ head -5 box_small.crd
1 0.000000 0.000000 0.000000
2 0.000000 0.000000 0.020000
3 0.000000 0.000000 0.040000
4 0.000000 0.000000 0.060000
5 0.000000 0.000000 0.080000
nd# x y z

$ head -5 box_small.match
1 51
2 -1
3 -1
4 -1
5 -1
node# matching node if not -1

@rickybalin

[cwsmith]

Pasting Ken's email below

Cameron and I met today to discuss the plan for this effort. He is close to having something to share.

Remove counter from the first column of all files and assume that line number is the global entity number (with obvious adjustment for multiple topology connectivity):

file 1 coordinates (forgot what we named it)
Coordinates
numVerts 3
x1 x2 x3
(repeat for remainder (numVerts -1) times)

File 2 connectivity (again forgot what we called it)

numElmThisTop nElmNodes
gn1 gn2 …. nElmNodes
(repeat for remainder (numElmThisTop -1) times)
Repeat header and rectangular array for each topology. Could put into different files initially if easier and then cat the files to create File 2.

File 3 Matching
numVerts 1
-1 if unmatched
Node number matched with

File 4 Classification
numVerts 1
0=fully interior of the volume
1-6 =classified on face (not edge or vertex)
11-22 = classified on model edge (not end points which are model vertices)
31-38 = classified on a model vertex.

I suppose on this last one we should agree on a master numbering. Cameron, I know SCOREC used to have canonical numbering for hex elements and if you can find that we should stick with it (it probably also exists in Simmetrix documentation too)?

File 5) Boundary conditions
We should probably recycle our existing flat text smd or spj file format with the only change being that model tag is replaced by the classification tag described in File 4.

Riccardo mentioned in Slack that he has a 1B element case ready for testing and called it his largest so I suspect he has some in between which would be good.

Current plan:

1. Cameron is going to push a version of his code that ignores matching to github as soon as he confirms it works on the already provided 175k element on 1, 2, and 4 target mds parts
2. Riccardo and I will retest 1)
3. Riccardo and I will begin to test progressively larger cases
4. Cameron will move on to add features in the following order:
   a) matching capability
   b) classification ingestion
   c) Boundary condition ingestion and connection a), b) and c) such that Chef has what it would have gotten from smd and mesh previously.
5. Riccardo and I test features as they get added and obviously run PHASTA to verify that all the information is coming through

We are leaving Initial conditions out but Riccardo can correct me if I am wrong, we have already in the codes that he is using a way to overwrite good, spatially varying initial and boundary condition values. The latter may or may require correct BC codes but I think the above will do that.

Let me know if I have forgotten anything or gotten any of if differently than we discussed today.

Best regards,

Ken

[cwsmith]

The parallel reader now works.

[KennethEJansen]

Am I correct in assuming that this is a separate repo from SCOREC/core, rather it is a personal repo. Can you provide some instructions for how best to check this out as a branch "under" our standard repo. I guess I am not git-savy enough to do that since for PHASTA we keep everything under either the public or the next repo which are ultimately both "linked" on github.

[cwsmith]

cd /path/to/existing/core/repo
git remote add cws git@github.com:cwsmith/core.git
git fetch cws
git checkout matchedReader

On my system this looks like:

~/develop/core (develop)$ git remote -v
origin	git@github.com:SCOREC/core.git (fetch)
origin	git@github.com:SCOREC/core.git (push)
~/develop/core (develop)$ git remote add cws git@github.com:cwsmith/core.git
~/develop/core (develop)$ git fetch cws
remote: Enumerating objects: 75, done.
remote: Counting objects: 100% (75/75), done.
remote: Compressing objects: 100% (29/29), done.
remote: Total 75 (delta 53), reused 68 (delta 46), pack-reused 0
Unpacking objects: 100% (75/75), done.
From github.com:cwsmith/core
 * [new branch]        apf_sim_config          -> cws/apf_sim_config
 * [new branch]        chef                    -> cws/chef
 * [new branch]        cws/control_print_stmts -> cws/cws/control_print_stmts
 * [new branch]        dcVtx                   -> cws/dcVtx
 * [new branch]        develop                 -> cws/develop
 * [new branch]        fusion_dev              -> cws/fusion_dev
 * [new branch]        generateAdvanced        -> cws/generateAdvanced
 * [new branch]        liipbmod                -> cws/liipbmod
 * [new branch]        master                  -> cws/master
 * [new branch]        matchedReader           -> cws/matchedReader
 * [new branch]        mis-work                -> cws/mis-work
 * [new branch]        phastaChef_mesh_measure -> cws/phastaChef_mesh_measure
 * [new branch]        sim_getDgCopies         -> cws/sim_getDgCopies
 * [new branch]        ugridhacks              -> cws/ugridhacks
~/develop/core (develop)$ git checkout matchedReader
Branch 'matchedReader' set up to track remote branch 'matchedReader' from 'cws'.
Switched to a new branch 'matchedReader'
~/develop/core (matchedReader)$ 

[KennethEJansen]

Thanks. I have successfully built the code on the viz nodes and repeated the 175k test. I further tested the following larger cases

Parts	Mesh size	Time to Create (s)	Memory Peak (GB)	Time to Write smb (s)
4	8M	65	4x1.7	26
4	125M	999	4x27	350
8	""	1012	8x13.5

(note it is called 150M but it is actually 125M according to the code). As expected this is linear in time and memory for 4 parts so all is looking good at this point. The mesh creation does not seem to benefit from more cores/parts but maybe that is expected?

Thanks for adding the render function and output. I did verify the 175k and 8M cases in paraview

I will have to move to Cooley to test the 1B element case since it is projected to need > 800GB.

[KennethEJansen]

kjansen@cc014: /projects/TRAssembly_2/kjansen/Models/BoeingBump/matchedReaderTest/1B $ mpirun -np 32 --hostfile $COBALT_NODEFILE /projects/TRAssembly_2/SCOREC-core/build-matchedReader/test/matchedNodeElmReader box_WMLES_1B.cnn box_WMLES_1B.crd NULL mod mesh/
numElms 1000000000 numVerts 1003003001
isMatched 0
seconds to create mesh 3589.092
mesh verified in 158.253273 seconds
mesh mesh/ written in 22.195060 seconds
writeVtuFile into buffers: 129.821842 seconds
writeVtuFile buffers to disk: 1.203647 seconds
vtk files rendered written in 136.573262 seconds

Looking in top on Cooley on the above case I had 8 nodes so 4 processes per node. One process seems far ahead of the other 3. It is running at 100% while the other three are at 60% with a Status of D. Presumably they are struggling to read from a single file. At this point all are in the compute phase
Tasks: 364 total, 5 running, 359 sleeping, 0 stopped, 0 zombie
%Cpu(s): 28.2 us, 5.5 sy, 0.0 ni, 66.2 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 39597302+total, 30272150+free, 90494048 used, 2757468 buff/cache
KiB Swap: 5242876 total, 4956576 free, 286300 used. 30331353+avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
49976 kjansen 20 0 23.0g 18.8g 10296 R 100.0 5.0 37:28.15 matchedNodeElmR
49977 kjansen 20 0 19.1g 16.7g 10300 R 100.0 4.4 36:23.76 matchedNodeElmR
49974 kjansen 20 0 31.2g 24.1g 10316 R 96.2 6.4 54:27.47 matchedNodeElmR
49975 kjansen 20 0 23.0g 18.9g 10300 R 96.2 5.0 34:57.68 matchedNodeElmR
7344 kjansen 20 0 177836 3324 2180 R 3.8 0.0 0:18.54 top

but note the 18 minute gap. Earlier the ones at 34 minutes were hovering at 1.6G but now they are done reading I guess and processing elements.

Here are the results from 64 processes

kjansen@cc014: /projects/TRAssembly_2/kjansen/Models/BoeingBump/matchedReaderTest/1B $ mpirun -np 64 --hostfile $COBALT_NODEFILE /projects/TRAssembly_2/SCOREC-core/build-matchedReader/test/matchedNodeElmReader box_WMLES_1B.cnn box_WMLES_1B.crd NULL mod mesh/
numElms 1000000000 numVerts 1003003001
isMatched 0
seconds to create mesh 3433.222
mesh verified in 87.809694 seconds
mesh mesh/ written in 13.419787 seconds
writeVtuFile into buffers: 72.740630 seconds
writeVtuFile buffers to disk: 0.592511 seconds
vtk files rendered written in 76.710356 seconds

[KennethEJansen]

Note all of the above is with NULL passed for matching. Please let me know when you think the code is ready to test matching. Also, it looks like you have not made the jump to the new file formats yet (e.g., your code is still not expecting a header and is reading id as the first column on crd and cnn at least).

[KennethEJansen]

I think we are facing a tradeoff between IO bottlenecks at large process/per node and the scalability of the mds element creation. To test the extreme of this I tried one process per node but as as you can see below, this fails on an INT_MAX limit:

jansen@cc014: /projects/TRAssembly_2/kjansen/Models/BoeingBump/matchedReaderTest/1B $ mpirun -np 8 --hostfile $COBALT_NODEFILE /projects/TRAssembly_2/SCOREC-core/build-matchedReader/test/matchedNodeElmReader box_WMLES_1B.cnn box_WMLES_1B.crd NULL mod mesh/
numElms 1000000000 numVerts 1003003001
isMatched 0
ERROR PCU message size exceeds INT_MAX... exiting
ERROR PCU message size exceeds INT_MAX... exiting
[cc015:mpi_rank_2][error_sighandler] Caught error: Aborted (signal 6)
[cc009:mpi_rank_5][error_sighandler] Caught error: Aborted (signal 6)
ERROR PCU message size exceeds INT_MAX... exiting
[cc003:mpi_rank_6][error_sighandler] Caught error: Aborted (signal 6)
ERROR PCU message size exceeds INT_MAX... exiting
[cc010:mpi_rank_3][error_sighandler] Caught error: Aborted (signal 6)
ERROR PCU message size exceeds INT_MAX... exiting
[cc004:mpi_rank_4][error_sighandler] Caught error: Aborted (signal 6)
ERROR PCU message size exceeds INT_MAX... exiting
[cc005:mpi_rank_1][error_sighandler] Caught error: Aborted (signal 6)
ERROR PCU message size exceeds INT_MAX... exiting
[cc014:mpi_rank_0][error_sighandler] Caught error: Aborted (signal 6)

[cwsmith]

Thanks for testing. Glad to see the tool worked for 1B elements. I've added that result to the table.

Parts	Mesh size	Time to Create (s)	Memory Peak (GB)	Time to Write smb (s)
4	8M	65	4x1.7	26
4	125M	999	4x27	350
8	125M	1012	8x13.5
32	1B	3589	32x31?	22
64	1B	3433	?	13
8	1B	Fail
16x1	1B	2807	?	39
16x2	1B	3472	?	18.7
16x4	1B	2906	?	14

I think we'll have to move to MPI_IO and binary files to get the read times down. This would append another bullet on item 4 from here. Matlab supports binary file writing:
https://www.mathworks.com/help/matlab/ref/fwrite.html

Matching is not supported yet. I'll post an update here when it is ready for testing.

[KennethEJansen]

Since this approach appears promising, I am going to take some time here to plan the remaining steps to putting this into a production run. I will put tentative assignments via initials but of course we can redistribute as needed:
1)(DONE) Preliminary testing for meshes up to 1B . This is nearly complete as noted above by KEJ.
2) (DONE) Adding matching capability (CWS). see #211
3) Switch over to the header plus removal of ID column (CWS).
4) Adding classification capability (CWS).
5) Creation of "real" meshes (current ones are isotropic boxes) with spatially varying 2D boundary layers and vertical filling to top boundary (RB). Retest should be easy since, other than bugs, it is just coordinate changes. After coords and connectivity are checked "mesh generation" code needs to add capability to write the model classification file described above. Should also prepare BC/IC flat text file.
6) (DONE CWS confirms flat text will work) Adding the load of flat text BC and IC description a la spj or smd (CWS or other if needed since I think there is a prospect that what we have will just work as long as 4) is done in a way that is compatible).
7) With 6) complete we should be able to start testing in PHASTA (RB).
8) Push this up to our larger target meshes (RB).
9) Adding Multi-topology (CWS)
10) Test 8) and 9) in PHASTA
11) Adding MPI_IO and binary (CWS). This is a low priority for now. Of course it would be nice but I think we can get to > 10B without it. 1B element case on 16 nodes with one process takes about 1/2 an hour to read the mesh. KEJ has modified the code to measure this directly and will get data on that at 2 and 4 processes per node as time allows but since we were able to get 1 B through on 8 nodes, I suspect we can get 10B through on 80 (or probably much less) which we have in Cooley. I further suspect that it should be reasonably straightforward to do the preliminary element partitioning in the same code that creates the coordinates and connectivity such that the elements can be written to already parted files. It might even be possible too write the coordinates (and other vertex-based data files to separate files. This would likely speed up the data ingestion. That said, I am keeping this as the lowest priority because we are not likely to have to do this enough times to justify a huge effort to make this stage as fast as it could be and I don't want us to spend time (both sides) on that fine tuning until we get the first 10 tasks verified.