Skip to content

Commit 37d72aa

Browse files
authored
Merge branch 'main' into bwibking/lhllc
2 parents 3b9082d + fdb7932 commit 37d72aa

56 files changed

Lines changed: 3179 additions & 294 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.devcontainer/devcontainer.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
// devcontainer.json
22
{
33
"name": "athenapk-dev",
4-
"image": "ghcr.io/parthenon-hpc-lab/cuda11.6-mpi-hdf5-ascent",
4+
"image": "ghcr.io/parthenon-hpc-lab/cuda11.6-noascent",
55
// disable Dockerfile for now
66
//"build": {
77
// // Path is relative to the devcontainer.json file.

.github/workflows/ci.yml

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -42,11 +42,11 @@ jobs:
4242
cd build
4343
# Pick GPU with most available memory
4444
export CUDA_VISIBLE_DEVICES=$(nvidia-smi --query-gpu=memory.free,index --format=csv,nounits,noheader | sort -nr | head -1 | awk '{ print $NF }')
45-
ctest -L ${{ matrix.parallel }}
46-
- uses: actions/upload-artifact@v3
45+
ctest -L ${{ matrix.parallel }} --timeout 3600
46+
- uses: actions/upload-artifact@v4
4747
if: ${{ always() }}
4848
with:
49-
name: regression-output
49+
name: regression-output-${{ matrix.parallel }}
5050
path: |
5151
build/CMakeFiles/CMakeOutput.log
5252
build/CMakeFiles/CMakeOutput.log
@@ -57,6 +57,8 @@ jobs:
5757
build/tst/regression/outputs/cluster_tabular_cooling/convergence.png
5858
build/tst/regression/outputs/aniso_therm_cond_ring_conv/ring_convergence.png
5959
build/tst/regression/outputs/aniso_therm_cond_gauss_conv/cond.png
60+
build/tst/regression/outputs/diffusion/ohm.png
61+
build/tst/regression/outputs/diffusion/visc.png
6062
build/tst/regression/outputs/field_loop/field_loop.png
6163
build/tst/regression/outputs/riemann_hydro/shock_tube.png
6264
build/tst/regression/outputs/turbulence/parthenon.out1.hst

CHANGELOG.md

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,16 +2,35 @@
22

33
## Current develop (i.e., `main` branch)
44

5+
### IMPORTANT
6+
7+
If you pulled from `main` after 11 Nov 24 ([[PR 124]](https://github.com/parthenon-hpc-lab/athenapk/pull/124))
8+
please updated immediate to a version after 18 Mar 24 ([[PR 136]](https://github.com/parthenon-hpc-lab/athenapk/pull/136)).
9+
In between a subtle bug was introduced that resulted in inconsistent divergence cleaning speeds in MHD simulation with mesh
10+
refinement.
11+
512
### Added (new features/APIs/variables/...)
13+
- [[PR 140]](https://github.com/parthenon-hpc-lab/athenapk/pull/140) Add hydro reflecting boundary conditions
14+
- [[PR102]](https://github.com/parthenon-hpc-lab/athenapk/pull/102) Add support for tracer particles
15+
- [[PR 89]](https://github.com/parthenon-hpc-lab/athenapk/pull/89) Add viscosity and resistivity
616
- [[PR 1]](https://github.com/parthenon-hpc-lab/athenapk/pull/1) Add isotropic thermal conduction and RKL2 supertimestepping
717

818
### Changed (changing behavior/API/variables/...)
19+
- [[PR 122]](https://github.com/parthenon-hpc-lab/athenapk/pull/122) Fixed sqrt(4pi) factor in CGS Gauss unit and add unit doc
20+
- [[PR 119]](https://github.com/parthenon-hpc-lab/athenapk/pull/119) Fixed Athena++ paper test case for KHI pgen. Added turbulence pgen doc.
921
- [[PR 97]](https://github.com/parthenon-hpc-lab/athenapk/pull/97) Fixed Schure cooling curve. Removed SD one. Added description of cooling function conventions.
1022
- [[PR 84]](https://github.com/parthenon-hpc-lab/athenapk/pull/84) Bump Parthenon to latest develop (2024-02-15)
1123

1224
### Fixed (not changing behavior/API/variables/...)
25+
- [[PR 136]](https://github.com/parthenon-hpc-lab/athenapk/pull/136) Fix using MPI reduced mindx
26+
- [[PR 128]](https://github.com/parthenon-hpc-lab/athenapk/pull/128) Fixed `dt_diff` in RKL2
1327

1428
### Infrastructure
29+
- [[PR 142]](https://github.com/parthenon-hpc-lab/athenapk/pull/142) Bump Kokkos 4.6.1 and Parthenon 25.05
30+
- [[PR 136]](https://github.com/parthenon-hpc-lab/athenapk/pull/136) Bump Kokkos 4.5.1 (for support of AMD APUs)
31+
- [[PR 129]](https://github.com/parthenon-hpc-lab/athenapk/pull/129) Bump Parthenon to support `dn` based outputs
32+
- [[PR 124]](https://github.com/parthenon-hpc-lab/athenapk/pull/124) Bump Kokkos 4.4.1 (and Parthenon to include view-of-view fix)
33+
- [[PR 117]](https://github.com/parthenon-hpc-lab/athenapk/pull/117) Update devcontainer.json to latest CI container
1534
- [[PR 114]](https://github.com/parthenon-hpc-lab/athenapk/pull/114) Bump Parthenon 24.08 and Kokkos to 4.4.00
1635
- [[PR 112]](https://github.com/parthenon-hpc-lab/athenapk/pull/112) Add dev container configuration
1736
- [[PR 105]](https://github.com/parthenon-hpc-lab/athenapk/pull/105) Bump Parthenon to latest develop (2024-03-13)
@@ -20,6 +39,9 @@
2039
### Removed (removing behavior/API/varaibles/...)
2140

2241
### Incompatibilities (i.e. breaking changes)
42+
- [[PR 142]](https://github.com/parthenon-hpc-lab/athenapk/pull/142) Removed `coords...FA<>()` interface in Parthenon
43+
- [[PR 124]](https://github.com/parthenon-hpc-lab/athenapk/pull/124) Enrolling custom boundary conditions changed
44+
- Boundary conditions can now be enrolled using a string that can be subsequently be used in the input file (see, e.g., cloud problem generator)
2345
- [[PR 114]](https://github.com/parthenon-hpc-lab/athenapk/pull/114) Bump Parthenon 24.08 and Kokkos to 4.4.00
2446
- Changed signature of `UserWorkBeforeOutput` to include `SimTime` as last paramter
2547
- Fixes bitwise idential restarts for AMR simulations (the derefinement counter is now included)

README.md

Lines changed: 11 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -14,9 +14,15 @@ Current features include
1414
- HLLE (hydro and MHD), HLLC (hydro), and HLLD (MHD) Riemann solvers
1515
- adiabatic equation of state
1616
- MHD based on hyperbolic divergence cleaning following Dedner+ 2002
17-
- isotropic and anisotropic thermal conduction
18-
- operator-split, second-order RKL2 supertimestepping for diffusive terms
17+
- diffusion processes
18+
- isotropic and anisotropic thermal conduction
19+
- viscosity
20+
- resistivity
21+
- diffusion integrator
22+
- unsplit
23+
- operator-split, second-order RKL2 supertimestepping
1924
- optically thin cooling based on tabulated cooling tables with either Townsend 2009 exact integration or operator-split subcycling
25+
- tracer particles
2026
- static and adaptive mesh refinement
2127
- problem generators for
2228
- linear waves
@@ -79,7 +85,7 @@ The following examples are a few standard cases.
7985

8086
Most simple configuration (only CPU, no MPI).
8187
The `Kokkos_ARCH_...` parameter should be adjusted to match the target machine where AthenaPK will be executed.
82-
A full list of architecture keywords is available on the [Kokkos wiki](https://kokkos.github.io/kokkos-core-wiki/keywords.html#architecture-keywords).
88+
A full list of architecture keywords is available on the [Kokkos wiki](https://kokkos.org/kokkos-core-wiki/get-started/configuration-guide.html#keywords-arch).
8389

8490
# configure with enabling Intel Broadwell or similar architecture (AVX2) instructions
8591
cmake -S. -Bbuild-host -DKokkos_ARCH_BDW=ON -DPARTHENON_DISABLE_MPI=ON
@@ -126,21 +132,8 @@ the `file_type = hdf5` format, see
126132
[VisIt](https://wci.llnl.gov/simulation/computer-codes/visit/).
127133
In ParaView, select the "XDMF Reader" when prompted.
128134

129-
2. With [yt](https://yt-project.org/) -- though currently through a custom frontend
130-
that is not yet part of the main yt branch and, thus, has to be installed manually, e.g.,
131-
as follows:
132-
```bash
133-
cd ~/src # or any other folder of choice
134-
git clone https://github.com/forrestglines/yt.git
135-
cd yt
136-
git checkout parthenon-frontend
137-
138-
# If you're using conda or virtualenv
139-
pip install -e .
140-
# OR alternatively, if you using the plain Python environment
141-
pip install --user -e .
142-
```
143-
Afterwards, `*.phdf` files can be read as usual with `yt.load()`.
135+
2. With [yt](https://yt-project.org/)
136+
As of versions >=4.4 `*.phdf` files can be read as usual with `yt.load()`.
144137

145138
3. Using [Ascent](https://github.com/Alpine-DAV/ascent) (for in situ visualization and analysis).
146139
This requires Ascent to be installed/available at compile time of AthenaPK.

docs/README.md

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
# AthenaPK documentation
2+
3+
Note that we're aware the rendering of equations in markdown on GitHub for the documentation
4+
is currently not great.
5+
Eventually, the docs will transition to Read the Docs (or similar).
6+
7+
## Overview
8+
9+
The documentation currently includes
10+
11+
- [Configuring solvers in the input file](input.md)
12+
- [An extended overview of various conventions for optically thin cooling implementations](cooling_notes.md)
13+
- [Notebooks to calculate cooling tables from literature](cooling)
14+
- [Brief notes on developing code for AthenaPK](development.md)
15+
- [How to add a custom/user problem generator](pgen.md)
16+
- [Units](units.md)
17+
- Detailed descriptions of more complex problem generators
18+
- [Galaxy Cluster and Cluster-like Problem Setup](cluster.md)
19+
- [Driven turbulence](turbulence.md)
20+
21+
## Tutorial
22+
23+
An AthenaPK tutorial was given as part of the **Towards exascale-ready astrophysics**
24+
workshop https://indico3-jsc.fz-juelich.de/event/169/ taking place 25-27 Sep 2024 online.
25+
26+
The material is currently located at https://github.com/pgrete/athenapk_tutorial
27+
28+
While the instructions for building the code are specific to the workshop environment
29+
the tutorial itself should translate directly to other environments/systems.
30+
31+
## Parthenon documenation
32+
33+
Many paramters/options are directly controlled through the Parthenon framework
34+
(both with regard to building and in the input file).
35+
36+
While the [Parthenon documenation](https://parthenon-hpc-lab.github.io/parthenon) is
37+
more geared towards developers it also contains useful information for users.

docs/img/tracer_example.png

196 KB
Loading

docs/img/turb_acc.png

68.8 KB
Loading

docs/img/turb_evol.png

46.7 KB
Loading

docs/img/turb_spec.png

18.3 KB
Loading

docs/input.md

Lines changed: 185 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,8 +67,65 @@ To control the floors, following parameters can be set in the `<hydro>` block:
6767
*Note* the pressure floor will take precedence over the temperature floor in the
6868
conserved to primitive conversion if both are defined.
6969

70+
#### Units
71+
72+
See(here)[units.md].
73+
7074
#### Diffusive processes
7175

76+
Diffusive processes in AthenaPK can be configured in the `<diffusion>` block of the input file.
77+
```
78+
<diffusion>
79+
integrator = unsplit # alternatively: rkl2 (for rkl2 integrator (operator split integrator)
80+
#rkl2_max_dt_ratio = 100.0 # limits the ratio between the parabolic and hyperbolic timesteps (only used for RKL2 operator split integrator)
81+
#cfl = 1.0 # Additional safety factor applied in the caluclation of the diffusive timestep (used in both unsplit and RKL2 integration schemes). Defaults to hyperbolic cfl.
82+
83+
conduction = anisotropic # none (disabled), or isotropic, or anisotropic
84+
conduction_coeff = fixed # alternative: spitzer
85+
thermal_diff_coeff_code = 0.01 # fixed coefficent in code units (code_length^2/code_time)
86+
#spitzer_cond_in_erg_by_s_K_cm = 4.6e7 # spitzer coefficient in cgs units (requires definition of a unit system)
87+
#conduction_sat_phi = 0.3 # fudge factor to account for uncertainties in saturated fluxes
88+
89+
90+
viscosity = none # none (disabled) or isotropic
91+
viscosity_coeff = fixed
92+
mom_diff_coeff_code = 0.25 # fixed coefficent of the kinetmatic viscosity in code units (code_length^2/code_time)
93+
94+
resistivity = none # none (disabled) or ohmic
95+
resistivity_coeff = fixed
96+
ohm_diff_coeff_code = 0.25 # fixed coefficent of the magnetic (ohmic) diffusivity code units (code_length^2/code_time)
97+
```
98+
(An)isotropic thermal conduction (with fixed or Spitzer coefficient), and isotropic viscosity and
99+
resistivity with fixed coefficient are currently implemented.
100+
They can be integrated in an unsplit manner or operator split using a second-order accurate RKL2
101+
supertimestepping algorithm.
102+
More details are described in the following.
103+
104+
#### Integrators
105+
106+
Diffusive processes can be integrated in either an unsplit
107+
fashion (`diffusion/integrator=unsplit`) or operator split using a second-order accurate
108+
RKL2 super timestepping algorithm (`diffusion/integrator=rkl2`) following [^M+14].
109+
110+
In the unsplit case, the diffusive processes are included at the end of every stage in
111+
the main integration loop and the global timestep is limited accordingly.
112+
A separate CFL can be set for the diffusive processes via `diffusion/cfl=...`, which
113+
defaults to the hyperbolic value if not set.
114+
115+
In the RKL2 case, the global timestep is not limited by the diffusive processes by default.
116+
However, as reported by [^V+17] a large number of stages
117+
($`s \approx \sqrt(\Delta t_{hyp}/\Delta t_{par}) \geq 20`$) in the supertimestepping
118+
(in combination with anisotropic, limited diffusion) may lead to a loss in accuracy, which
119+
is why the difference between hyperbolic and parabolic timesteps can be limited by
120+
`diffusion/rkl2_max_dt_ratio=...` and a warning is shown if the ratio is above 400.
121+
122+
[^M+14]:
123+
C. D. Meyer, D. S. Balsara, and T. D. Aslam, “A stabilized Runge–Kutta–Legendre method for explicit super-time-stepping of parabolic and mixed equations,” Journal of Computational Physics, vol. 257, pp. 594–626, 2014, doi: https://doi.org/10.1016/j.jcp.2013.08.021.
124+
125+
[^V+17]:
126+
B. Vaidya, D. Prasad, A. Mignone, P. Sharma, and L. Rickler, “Scalable explicit implementation of anisotropic diffusion with Runge–Kutta–Legendre super-time stepping,” Monthly Notices of the Royal Astronomical Society, vol. 472, no. 3, pp. 3147–3160, 2017, doi: 10.1093/mnras/stx2176.
127+
128+
72129
##### Isotropic (hydro and MHD) and anisotropic thermal conduction (only MHD)
73130
In the presence of magnetic fields thermal conduction is becoming anisotropic with the flux along
74131
the local magnetic field direction typically being much stronger than the flux perpendicular to the magnetic field.
@@ -140,6 +197,29 @@ Default value corresponds to the typical value used in literature and goes back
140197
[^BM82]:
141198
S. A. Balbus and C. F. McKee, “The evaporation of spherical clouds in a hot gas. III - Suprathermal evaporation,” , vol. 252, pp. 529–552, Jan. 1982, doi: https://doi.org/10.1086/159581
142199

200+
#### Viscosity/Momentum diffusion
201+
202+
Only isotropic viscosity with a (spatially and temporally) fixed coefficient in code units
203+
(`code_length`^2/`code_time`) is currently implemented.
204+
To enable set (in the `<diffusion>` block)
205+
```
206+
viscosity = isotropic
207+
viscosity_coeff = fixed
208+
mom_diff_coeff_code = 0.25 # fixed coefficent of the kinetmatic viscosity in code units (code_length^2/code_time)
209+
```
210+
211+
#### Resistivity/Ohmic diffusion
212+
213+
Only resistivity with a (spatially and temporally) fixed coefficient in code units
214+
(`code_length`^2/`code_time`)is currently implemented.
215+
To enable set (in the `<diffusion>` block)
216+
```
217+
resistivity = ohmic
218+
resistivity_coeff = fixed
219+
ohm_diff_coeff_code = 0.25 # fixed coefficent of the magnetic (ohmic) diffusivity code units (code_length^2/code_time)
220+
```
221+
222+
143223
### Additional MHD options in `<hydro>` block
144224

145225
Parameter: `glmmhd_source` (string)
@@ -243,3 +323,108 @@ d_log_temp_tol = 1e-8 # Tolerance in cooling table between subseque
243323
Finally, a more comprehensive descriptions of various conventions used for cooling
244324
functions (and the chosen implementation in AthenaPK) can be found in [Cooling Notes](cooling_notes.md)
245325
and a notebook comparing various cooling tables (and their conversion) in [cooling/cooling.ipynb](cooling/cooling.ipynb).
326+
327+
### Particles
328+
329+
#### Tracers
330+
331+
Tracer particles can be enabled by setting `tracers/enabled=true` in the input file:
332+
333+
```
334+
<tracers>
335+
enabled = true
336+
337+
initial_seed_method = random_per_block # alternative: user
338+
initial_num_tracers_per_cell = 0.125
339+
### Optional arguments
340+
#initial_rng_seed = INTEGER
341+
```
342+
343+
Two seeding methods are currently supported:
344+
345+
- `initial_seed_method=random_per_block`
346+
- seeds particles at random positions in each block
347+
- NOTE: the random number generator seed uses the unique block id. Therefore, simulations with the mesh decomposition (mesh and meshblock sizes) are identical independent of the number of MPI ranks used, but if the meshblock size is changed for given mesh (and thus the total number of blocks) the initial state will be different.
348+
- `initial_num_tracers_per_cell` determines the number of seeded particles per cell
349+
- `initial_rng_seed` (optional) is used as seed in addition to the block id.
350+
- `initial_seed_method=user`
351+
- Calls a problem specific callback function (`ProblemSeedInitialTracers`), see [tracer callback documenation](https://github.com/parthenon-hpc-lab/athenapk/blob/main/docs/pgen.md#tracers).
352+
353+
By default, swarm fields are written only to restart files.
354+
If they are required for "standard" output files (like single precision `hdf5`),
355+
they need to be added manually to the output block, e.g., (bottom two lines)
356+
```
357+
<parthenon/output2>
358+
file_type = hdf5 # Binary data dump
359+
variables = prim # variables to be output
360+
dt = 0.1 # time increment between outputs
361+
id = prim
362+
single_precision_output = true
363+
364+
swarms = tracers
365+
tracers_variables = id, x, y, z, rho
366+
#write_swarm_xdmf=true # uncomment to create an xdmf output (e.g., for Paraview or Visit)
367+
```
368+
369+
Tracers can be read/processed by Paraview or Visit (via the xdmf file)
370+
or by the `phdf` package shipped with the Parthenon submodule.
371+
A sample plotting script for the latter might look like
372+
```python
373+
import matplotlib.pyplot as plt
374+
import numpy as np
375+
import sys
376+
sys.path.insert(
377+
1,
378+
"/path/to/athenapk/external/parthenon"
379+
+ "/scripts/python/packages/parthenon_tools/parthenon_tools",
380+
)
381+
382+
try:
383+
import phdf
384+
except ModuleNotFoundError:
385+
print("Couldn't find module to read Parthenon hdf5 files.")
386+
387+
data = phdf.phdf("../run-turbulence/parthenon.prim.00010.phdf")
388+
tracers = data.GetSwarm("tracers")
389+
xs = tracers.x
390+
ys = tracers.y
391+
zs = tracers.z
392+
ids = tracers.Get("id")
393+
394+
fig = plt.figure()
395+
ax = fig.add_subplot(projection='3d')
396+
397+
ax.scatter(xs, ys, zs,s=1,c=ids)
398+
```
399+
resulting in the following image:
400+
401+
![image](img/tracer_example.png)
402+
403+
Following "restrictions" apply to the current tracer implementation:
404+
- Only 3D simulations.
405+
- Only one advection method (RK2/Heun's method).
406+
- All primitive fields (`rho`, `pressure`, `vel_x`, `vel_y`, `vel_z`, `B_x`, `B_y`, and `B_x`) are traced by default (independent of whether they're needed or not) in addition to the position (`x`, `y`, `z`) and id (`id`) fields.
407+
- Default tracer values (such as the primitive fields) are only updated right before writing an output file.
408+
- Ids are only unique if tracers are seeded at the beginning at the simulations and no new tracer particles are added dynamically while the simulation is running.
409+
410+
Please get in touch, if you interested in running simulations that require lifting one (or more) of those restrictions.
411+
412+
## Boundary conditions
413+
414+
In addition to enrolling custom boundary conditions, three general options are currently supported by default:
415+
416+
- `periodic`: Work without restrictions (implemented in upstream Parthenon)
417+
- `outflow`: Last layer of active cells is copied into ghost cells. Work (technicaly) without restrictions (implemented in upstream Parthenon). Physically care must be taken when using outflow conditions in subsonic flows (as characteristics can travel into the domain, which is not properly handled by copying the active cell data).
418+
It is recommended to implement custom, problem specific boundary conditions for subsonic outflows, see large body of engineering literature.
419+
For supersonic outflows there is no issue as the flow itself is faster than the fastest characteristic.
420+
- `reflecting`: Work only for cell-centered *hydro* fluid variables
421+
(implemented in AthenaPK currently without support for particles or `Metadata::Fine` fields,
422+
where the latter are currently not being used in AthenaPK).
423+
Note, that we specifically do *not* provide "reflecting" boundary conditions for the magnetic
424+
fields as these kind of configurations are typically highly problem dependent and, thus,
425+
require manual treatment.
426+
427+
They are being used by specifying the name as parameter value for the
428+
`ix1_bc`, `ox1_bc`, `ix2_bc`, `ox2_bc`, `ix3_bc`, and `ox3_bc` paraemters
429+
in the `<parthenon/mesh>` block of the input file for the inner and outer
430+
(say left and right in x1 direction) in each coordinate direction, respectively.

0 commit comments

Comments
 (0)