Add GPU kernel for calc_sources!

[MarcoArtiano]

I first naively implemented the approach following the other existing kernel. However it was as slow as computing the surface fluxes. Here each thread of the GPU is launched for each quadrature node (i, j, element), which makes the GPU kernel roughly 6 times faster.

Per element kernel
source terms 19.2k 4.89s 12.0% 254μs 143MiB 10.5% 7.62KiB

Per quadrature node kernel
source terms 19.2k 792ms 2.0% 41.2μs 104MiB 8.1% 5.55KiB

ps: it may also be worth trying (nnodes(dg)*nnodes(dg), nelements(dg, cache)) and reconstructing the indices i and j in the kernel.

[github-actions]

Review checklist

This checklist is meant to assist creators of PRs (to let them know what reviewers will typically look for) and reviewers (to guide them in a structured review process). Items do not need to be checked explicitly for a PR to be eligible for merging.

Purpose and scope

• The PR has a single goal that is clear from the PR title and/or description.
• All code changes represent a single set of modifications that logically belong together.
• No more than 500 lines of code are changed or there is no obvious way to split the PR into multiple PRs.

Code quality

• The code can be understood easily.
• Newly introduced names for variables etc. are self-descriptive and consistent with existing naming conventions.
• There are no redundancies that can be removed by simple modularization/refactoring.
• There are no leftover debug statements or commented code sections.
• The code adheres to our conventions and style guide, and to the Julia guidelines.

Documentation

• New functions and types are documented with a docstring or top-level comment.
• Relevant publications are referenced in docstrings (see example for formatting).
• Inline comments are used to document longer or unusual code sections.
• Comments describe intent ("why?") and not just functionality ("what?").
• If the PR introduces a significant change or new feature, it is documented in NEWS.md with its PR number.

Testing

• The PR passes all tests.
• New or modified lines of code are covered by tests.
• New or modified tests run in less then 10 seconds.

Performance

• There are no type instabilities or memory allocations in performance-critical parts.
• If the PR intent is to improve performance, before/after time measurements are posted in the PR.

Verification

• The correctness of the code was verified using appropriate tests.
• If new equations/methods are added, a convergence test has been run and the results
  are posted in the PR.

Created with ❤️ by the Trixi.jl community.

[codecov]

Codecov Report

❌ Patch coverage is 81.69014% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.09%. Comparing base (a70bfa2) to head (8fda9ee).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/solvers/dgsem_p4est/dg_3d_gpu.jl	75.00%	11 Missing ⚠️
src/solvers/dgsem_p4est/dg_2d_gpu.jl	86.67%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3012      +/-   ##
==========================================
- Coverage   97.09%   97.09%   -0.01%     
==========================================
  Files         630      631       +1     
  Lines       48881    48911      +30     
==========================================
+ Hits        47461    47487      +26     
- Misses       1420     1424       +4     

Flag	Coverage Δ
unittests	97.09% <81.69%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ranocha]

ps: it may also be worth trying (nnodes(dg)*nnodes(dg), nelements(dg, cache)) and to reconstruct the indices i and j in the kernel.

Would you expect performance improvements coming from this, @vchuravy?

[MarcoArtiano]

ps: it may also be worth trying (nnodes(dg)*nnodes(dg), nelements(dg, cache)) and to reconstruct the indices i and j in the kernel.

Would you expect performance improvements coming from this, @vchuravy?

Comment Flux Diff GPU here I've tested these two cases and I didn't notice any major differences. I'm not sure if one option has a better scalability over the other.

[ranocha]

The GPU CI job on buildkite fails. Could you please check what is going on there? Please ping me when your PR improving the performance of applying the Jacobian has been merged and this PR is updated accordingly to the new structure.

[MarcoArtiano]

This PR will be ready to be reviewed after #3013 will be merged and the conflicts will be resolved.

[MarcoArtiano]

@ranocha this is again ready for another round of review.

[ranocha]

Can you please adapt the allocation check? https://github.com/trixi-framework/Trixi.jl/actions/runs/26111095627/job/76788248592?pr=3012#step:7:1211

[MarcoArtiano]

Is the code coverage expected to fail, even if we are running KA with CPU backend?

[benegee]

Is the code coverage expected to fail, even if we are running KA with CPU backend?

I think you are good!
The KA tests were explicitly added to get coverage reports for the backend::Backend specializations. However, some parts like @index will still not be detected.

[ranocha]

Thanks!

[JoshuaLampert]

Thanks! I have two questions regarding the tests.

[ranocha]

Thanks! I just have a few suggestions to prepare my PR adapting equations as well.

[ranocha]

Thanks for your patience!

[ranocha]

Codecov upload failed, all tests passed (except known buildkite issues on the AMD server); I will merge this PR.