EOS models, unit conversion, particle fixups

[brbass]

Summary

Features

Singularity EOS

• Includes GammaLawGas, ExtendedVinet, MieGruneisen, Spiner, and EOSPAC.
• Tested against the ordinary GammaLawGas in spheral.
• Will improve a lot once spack is updated, as the current singularity version is a few years out of date.

Null EOS

• NullEquationOfState is the counterpart of NullStrength.

Unit conversion

• Converts between two instances of PhysicalConstants. Very helpful for things like Singularity above.

Subdirectories in silo

• Support for subdirectories in silo files. Currently used for RK corrections.

Improvements for the smoothing parameter

• Adds a node separation and relaxation method that slots into the initial iterateIdealH calculations. To use it, create relaxation = RelaxNodes(...) and then pass it to the controller: SpheralController(initialHExtraPackages = [relaxation]). This separates nodes that are too close and relaxes nodes using the Voronoi, without which the smoothing parameter may not converge for problems with problematic initial node distributions.
• Adds a check for the initial H distribution to limit the number of initial points to some reasonable value. If a problem passes in nodes that have hundreds of thousands of neighbors or whatever, this sharply limits the number before the iterateIdealH calculations get underway.
• Adds a damping factor for the IdealH calculation that helps it converge when struggling.

Verbose time step control

• Adds a verboseStep parameter to the Integrator. Printing the verbose time step info every cycle makes things difficult to read. With the verbose step set to 100, it is reasonable to keep it on all the time.

Bugfixes

• Removed the old, non-functional volume calculation in SPH and requested Voronoi cells instead.
• Removed circular dependencies for Johnson-Cook to let it run. Johnson-Cook Damage Model dependency #303. This will lag the strain calculation.
• Added allReduceLoc to prevent the time step from being printed once per rank when they all calculate the same time step.
• Added a gradientPairs method that is more robust for points with difficult H distributions (yes, more smoothing parameter issues). Also cleaned up the gradient instantiation.
• Added some missing includes.

ToDo :

• Annotate RELEASE_NOTES.md with notable changes.
• Create LLNLSpheral PR pointing at this branch. (PR#)
• LLNLSpheral PR has passed all tests.

[ldowen]

Help me understand what is going on here. It seems like singularity has a ton of dependencies but I don't understand why we have the Spheral Spack package depending on Singularity's dependencies and why we are bringing them into our CMake. Neither of those things should be happening. It is possible to use a newer Spack recipe for Singularity than what is in the current version of Spack we are using, if that will help you. You might have to tweak it a little.

[brbass]

Help me understand what is going on here. It seems like singularity has a ton of dependencies but I don't understand why we have the Spheral Spack package depending on Singularity's dependencies and why we are bringing them into our CMake. Neither of those things should be happening. It is possible to use a newer Spack recipe for Singularity than what is in the current version of Spack we are using, if that will help you. You might have to tweak it a little.

It is onerous to have to include all of these, yes, but I couldn't get it to work otherwise. The dependencies are explicitly listed because the -fPIC flags don't propagate and the code didn't compile for me without them. Likewise, the code would not compile for me without adding the variables to the cmake cache, as the individual packages would not find each other without these.

Do you have any suggestions on how to make this work in a nicer way? I agree that it would be great to get this down to a single singularity-eos statement!

It would also be great to be on a newer version of singularity-eos, but after fighting with spack for a few days, I gave up and figured that it would be easier once Spheral is updated to Spack v1.0.0 or v1.1.0, which supports the latest Singularity version v1.10.0.

[ldowen]

Did you try adding the +kokkos +kokkos-kernels variants to singularity-eos? Is there any particular reason why you need use kokkos? We don't really support GPUs in Spheral yet anyway. I don't want to add another 6 dependencies unless we really need all of them. So if you don't need the Spiner or EOSpac, you should get rid of those too. Similarly, add ~python if you don't need that functionality

You can get a newer version of a TPL than is in the builtin by just adding a package file in our scripts/spack/packages directory. Look at what we we do with Umpire to get an idea of how to do it. You would also add the cflags and cppflags in that package file.

[brbass]

Did you try adding the +kokkos +kokkos-kernels variants to singularity-eos? Is there any particular reason why you need use kokkos? We don't really support GPUs in Spheral yet anyway. I don't want to add another 6 dependencies unless we really need all of them. So if you don't need the Spiner or EOSpac, you should get rid of those too. Similarly, add ~python if you don't need that functionality

You can get a newer version of a TPL than is in the builtin by just adding a package file in our scripts/spack/packages directory. Look at what we we do with Umpire to get an idea of how to do it. You would also add the cflags and cppflags in that package file.

kokkos is one of the required libraries for singularity. It uses it even off of the GPU. The Spiner EOS is the main one of interest here, since it gives another option for tabulated data. I can try pruning the dependencies and using ~python to see if that works, but I recall it being pretty difficult to get to work in the first place, so I'm not holding my breath.

[ldowen]

Based on the singular-eos Spack package recipe, Kokkos is an optional variant. Unless this is a mistake, it implies that Singularity can work without Kokkos. By not including +kokkos +kokkos-kernel on your depends_on('singularity-eos the version of singularity you are building will have Kokkos turned off. Again, unless there is a mistake in the package recipe in Spack.

[brbass]

Based on the singular-eos Spack package recipe, Kokkos is an optional variant. Unless this is a mistake, it implies that Singularity can work without Kokkos. By not including +kokkos +kokkos-kernel on your depends_on('singularity-eos the version of singularity you are building will have Kokkos turned off. Again, unless there is a mistake in the package recipe in Spack.

From what I remember, kokkos is an optional variant, but is required for several of the EOS options that we want. I will try to build without, but it may not work.

[ldowen]

Ok. If you determine you need it, you have to add the +kokkos +kokkos-kernel to variant to your singularity. That might also help with your building issues with CMake.

[brbass]

I have tried all sorts of configurations to get Singularity to build with fewer changes to the build files, but I am not the expert that @ldowen is with spack. Anytime I remove one of the dependencies, the -fPIC flag goes away and it fails to link correctly. It is really a miracle that I got it working the first time.

I would rather not put too much work into it for now since the Singularity spack package changes a lot with the version compatible with the spack v1.0.0 release. Once that comes out, it might be worth revisiting and seeing if some of the pain points have been resolved.

[brbass]

Have you guys started doing internal tests for this PR?

Working in 1D with the gnuplot files is onerous, so I made the siloPointmeshDump compatible with 1D and made that the default (which can be reverted if anyone is actually using the gnuplot version in production). It depends on the changes to siloMeshDump in this PR. Would you prefer that I add them here or create a separate PR that depends on this one?

[jmikeowen]

There is an existing Spheral1dViz method that outputs the data as a table of numbers. This should be pretty friendly to all sorts of 1D plotting packages -- what's different in your version?

[brbass]

There is an existing Spheral1dViz method that outputs the data as a table of numbers. This should be pretty friendly to all sorts of 1D plotting packages -- what's different in your version?

This outputs it in silo format, so you can load the curve data directly in visit. Then you can create movies or whatever using the same workflow as in 2D/3D. Here is an example:

[jmikeowen]

Ah I see. Personally I don't like using Visit for 1D plots but I can understand this would be useful for folks that do. I'm happy to have this as an option for 1D viz output, but I don't think it should be the default.

[jmikeowen]

Another question on a different change: the RelaxNodes package added for iterateIdealH. It appears to me this duplicates the relaxNodeDistribution functionality we have in NodeGenerators, which is invoked by SpheralController optionally. We might want to rearrange how that's used a bit to fit your use case here? I'd rather not have iterateIdealH also adjust the point positions, which seems like an unexpected side-effect of converging the smoothing scale distribution.

[brbass]

Sure, we don't need to make visit the default in 1D.

The reason RelaxNodes is optionally included in the H iterations is just convenience, since iterateIdealH already includes the Voronoi evaluation and other initializations. From what I can see, prerelaxNodeDistribution doesn't call relaxNodeDistribution but instead calls an hourglass control package. The relaxNodeDistribution function doesn't seem to be tested or used anywhere except testStar2d.py, which isn't in the testing suite. It uses an old mesh object rather than the Voronoi cells. All that said, there's no reason to add RelaxNodes to the repo if you don't think it will be generally useful. It meets some of our needs for separating and differentiating points when generating them from a non-ideal distribution, but that isn't a situation anyone is likely to encounter when using the built-in generators.

It seems like the easiest path forward is:

• Include the visit visualization files for 1D but leave gnuplot as the default
• Remove RelaxNodes, since it may not be generally useful outside of niche applications