Chombo-style "incrementing" FluxRegisters?

[BenWibking]

Hi,

I've written a single-level conservation law solver that allocates and stores fluxes grid-by-grid in FArrayBoxes rather than in MultiFabs. From reading the example code in the Advection_AmrCore tutorial, it is not clear how to modify my code to do AMR with refluxing, since it the example appears to assume that I have a MultiFab for each flux component.

Is there a way to increment the flux registers grid-by-grid along the lines of what is possible in Chombo, using Chombo's incrementCoarse and incrementFine functions? Or do I need to re-write my code to use global MultiFabs for each flux component?

[WeiqunZhang]

You could use YAFluxRegister in amrex/Src/Boundary. You can use amrex/Tests/EB/CNS/ as an example. The CNS code uses EBFluxRegister, which is derived from YAFluxRegister. When you look at the CNS code, you only need to look at the block for FabType::regular.

[WeiqunZhang]

I tried your latest version with max_level=2. It ran to step 73 and then hit a conservation error. But I see a similar error with FluxRegister. The issue is not FluxRegister or YAFluxRegister. The issue is your state multifab has 4 ghost cells. You have to use at least a blocking factor of 8, otherwise the ghost cells are not properly filled. For testing, you can reduce the number of ghost cells to 1 and use a blocking factor of 2.

[BenWibking]

Ah, well, that would explain it. I can't use nghost < 3 due to the PPM reconstruction I am using.

Is this mentioned in the documentation anywhere? I did not see any note of this when reading the section on the gridding algorithm or the runtime parameters. This is a pretty significant caveat in choosing the blocking factor. Does the blocking factor have to be a multiple of nghost? Or just >= 2*nghost?

[BenWibking]

What causes this constraint? Does FillPatch only use the validbox on the coarse level?

[WeiqunZhang]

Blocking factor is usually better to be a power of 2. For proper nesting, it needs to be greater than (ceil(nghost,ref_ratio)+ninterp)*ref_ratio, where ninterp (typically 1) depends on the scheme used for coarse/fine interpolation. Yes this is because only coarse valid cells are used in interpolation. Otherwise we would have to go down more levels, and that would not be very efficient. However, amrex does support going down as many levels as needed in Amr/AmrLevel, although not in AmrCore. There is also a check for proper nesting in AmrLevel (https://github.com/AMReX-Codes/amrex/blob/21.07/Src/Amr/AMReX_AmrLevel.cpp#L1030). For AmrCore based code, you can check it yourself using https://github.com/AMReX-Codes/amrex/blob/21.07/Src/AmrCore/AMReX_FillPatchUtil_I.H#L9

Anyway, for performance, one should use a factor of at least 8. For GPU, I would go with 16 or 32 at least

[BenWibking]

I've added a check for proper nesting. Thanks.

[WeiqunZhang]

For proper nesting, it needs to be greater than (ceil(nghost,ref_ratio)+ninterp)*ref_ratio, where ninterp (typically 1) depends on the scheme used for coarse/fine interpolation.

It should be greater than or equal to (ceil(nghost/ref_ratio)+ninterp)*ref_ratio.