Unified strong/weak spectral element operator variants via a
FormTypeparameter.Divergence,Gradient, andCurlnow carry a second type parameter (StrongFormorWeakForm), andWeakDivergence,WeakGradient, andWeakCurlare backwards-compatibility aliases for theWeakFormvariants (e.g.WeakDivergence{I} = Divergence{I, WeakForm}), collected insrc/Operators/deprecated.jlfor eventual removal. All operator names and their public constructors are unchanged. Results are bitwise identical to the previous implementations, with one exception: the strong-formDivergenceon GPUs in 2D now keeps one accumulator per dimension (matching the other unified operators) instead of summing both dimensions into a single accumulator, which changes results by accumulation roundoff only (verified identical in exact arithmetic; float differences are a fewepsof the accumulated terms). Two notes for downstream code:isachecks against the strong operator types now also match the weak variants (e.g.WeakDivergence() isa Divergenceistrue), so dispatch that must distinguish forms should use theFormTypeparameter (e.g.Divergence{I, StrongForm}); and the internalDivergence{()}(space)-style constructors used to reset operator axes have been replaced byrebuild_operator(op, space). As a side effect of unifying the CUDAGradientkernels,WeakGradientnow accepts rank-1 tensor fields on GPUs, which previously raised an "Unsupported input type" error even though both forms already supported them on CPUs. 2556- Refactor DataLayouts module 2522
- All data layout types are unified into a single
DataLayouttype, and all loops over data go through two communication primitives (foreach_sliceandreduce_points) that work the same way on CPUs, multi-threaded CPUs, and GPUs. Runtime performance on CPUs and GPUs matches the previous release. - Data in a
Fieldis now indexed in the order[v, i, j, h], and every parent array has a 5-dimensional shape, with one axis used to store struct fields. Code that indexes intoparent(field)arrays must be updated. - Several old names (
AbstractData,IJFH,IJHF) are available as aliases, and[i, j, f, v, h]indexing is still allowed for backward compatibility.
- All data layout types are unified into a single
Fixed multithreaded CPU loops over
ThisThreadPoolreading Julia's process-global threaded-region flag more than once per loop. A task that concurrently entered a threaded loop could flip the flag in between two of those reads, which either raised a spurious"Nested loops over ThisThreadPool are not supported"error or leftnum_threadsandthread_rankdisagreeing about how a loop's indices were divided among threads. Loops now resolve their scope once, up front, withresolve_pool_threads, and a thread's rank comes from task-local storage rather than from global state. 2557Divided the CPU thread pool between multithreaded loops that run at the same time, instead of giving every thread to whichever loop starts first and leaving the rest to run on a single thread each. Broadcasting over
Fields from several tasks at once, asThreadsX.mapreducedoes, is about 2.6x faster with two concurrent tasks on 8 threads, and no slower with one. 2557
- Add a
reverseoption to the column integral operators (column_reduce!andcolumn_accumulate!) so vertical reductions and accumulations can proceed from the top of a column downward. 2531
- Refactor Geometry module 2457
- Bug fix for broadcasting gpu errors 2517. 2518
- Make specialized shared memory-based tridiagonal solver mask aware 2518. 2518
- Add a specialized shared memory-based tridiagonal solver that uses PCR algorithm for CUDA backend. 2486
- Replace RecursiveApply with AutoBroadcaster wrappers. 2417
- Add support for nonuniform data structures 2464
- Add shared memory stencil support in CUDAExt (for spaces with non-periodic topologies and no masks) 2466
- Add support for bilinear interpolation for diagnostics to the Remapper. 2455
- Add
PressureInterpolator2422 - Add Buildkite pipeline to check flagship AMIP performance 2438
- Fix conservation of split divergence 2432
- Add
VerticalMassBorrowingLimiter2383
- Use space-filling curve for CubedSphereGrid and ExtrudedCubedSphereGrid 2429
- Restrict GPUCompiler to < v1.7.6 to avoid GPU errors 2430
- Add split divergence operator 2409
- Revert
to_devicedispatch change from #2375 2403
- Add support for horizontal operators on column spaces 2375
-
Add opt-in CUDA kernel renaming based on stacktrace for broadcasts over ClimaCore Fields 2376
To enable, set the following environment variable before starting Julia:
export CLIMA_NAME_CUDA_KERNELS_FROM_STACK_TRACE=trueWith default kernel names, profiling downstream packages results in names like
knl_copyto_(VIJFH<Float32, (int)63, (int)4, CuDeviceArray<Float32, (int)5, (int)1>>, Broadcasted<VIJFHStyle<(int)63, (int)4..., which are hard to read and often more than 50,000 characters long. With renaming enabled, profilers show names likeset_velocity_quantities__FILE_ClimaAtmos_jl_src_cache_precomputed_quantities_jl_L351, which gives more context about where the kernel was launched from.
- Add
projectfor both sides of 2-tensors (previously only left side was supported). Simplify implementation. 2379
- Store
reverse_modein theIntervalMeshstruct, so that we can access it when writing and reading the mesh for restarts
-
Upgrade compatibility with RootSolvers.jl, KrylovKit.jl, DataStructures.jl. Using the following versions of RootSolvers may result in slightly different
IntervalMesh(es) and results ofcolumn_integral_indefinite!(∂ϕ∂z, ᶠϕ)- < v0.45
- = v0.45
- > v0.45
- Extend level and field_values to work with lazy Fields 2356
- Fix unrolling over field broadcast arguments 2350
- minor fixes/Fix field2array error msg, level func bounds check 2347
- Define FiniteDifferenceSpace for extruded spaces 2344
- Implement a columnwise shmem operator 2328
- Dont reference localgeometry fields unnecessarily 2340
- Add missing methods for column spaces and FieldVectors 2330
- Move Δz_metric_component to Geometry module 2336
- Allow for more flexibly constructed LocalGeometry 2335
- Latitude and longitudes were flipped in the default target coordinates in the remapper, leading to non-sense values.
Add option to disable check on spaces. PR 2322
ClimaCore.DebugOnly now contains a function allow_mismatched_spaces_unsafe. When
this function is overridden to return true, ClimaCore will skip consistency
checks on spaces in broadcasted expressions. This is useful when debugging with
deepcopy. See
documentation
for more information.
- Expanded compat for Krylov to
0.10. 2318.
-
Disabled shared memory on finite difference operators. It was found that it has a severe performance penalty for more complex cases. 2317
-
Fixed missing method for
Topologies.mesh(Topology2D)2288.
-
GPU performance should now be more robust 2296.
-
Remapping is now protected for masked operations 2292.
-
Shmem support for InterpolateC2F was added 2290.
-
Some masked operations were fixed 2285.
-
Internal refactoring should help reduce latency and improve gpu performance 2284.
-
Masks now support restart 2212.
-
Fixed error messages in our DataLayouts 2275.
-
Bug fix in our finite difference shared memory implementation 2282.
-
Added support for shared memory in
GradientC2F2268.
Improvements to Remapping.interpolate PR 2272
PR 2272 introduces
convenience interpolate functions that are optimally suited for plotting.
Assuming field is a 3D Field,
import ClimaCore.Remapping: interpolate
array3D = interpolate(fields)array3D will be a 3D Array with values along the three dimensions.
By default, no interpolation in the vertical direction is performed. Linear
interpolation can be obtained by passing zresolution
array3D_linear_in_z = interpolate(fields; zresolution)Controlling the specific interpolation points is also possible, as shown in this example:
import ClimaCore.Geometry: LatLongPoint, ZPoint
import ClimaCore.Remapping: interpolate
target_hcoords = [LatLongPoint(45., 23.)]
target_zcoords = ZPoint.([100., 200., 1000.])
array3D_specific = interpolate(fields; target_hcoords, target_zcoords)- Fix empty interpolation with certain versions of CUDA+MPI 2256.
- Bugfix in ClimaCore FD stencil code PR 2251.
-
Added support for
map!PR 2248. -
The standard
*symbol now denotes matrix multiplication, and the aliasconst ⋅ = MultiplyColumnwiseBandMatrixField()is being deprecated PR 2232. Please replace use of⋅with*. -
Fields.Field(Bool, ::AbstractSpace)is now supported. PR 2239. -
SpectralElementSpace2Dconstructors now support nodal masks. PR 2201. See its documentation here. Note that it does not yet support restarts. -
Added support for InputOutput with PointSpaces PR 2162.
-
Reading a
FieldwithHDF5Readerwill not calleval ∘ Meta.parseon the rawvalue_typeattribute. Instead, the newread_typefunction is used, which prevents the execution of arbitrary code. PR 2162. -
Improved
showfor spaces, and addedshowfor grids. PR 2202. -
UnrolledFunctionsremoved and replaced withUnrolledUtilitiesPR 1713 -
Unused original implicit solver intereface removed PR 2200
-
Using incorrect boundary conditions will now error PR 2180.
-
Prior to this version,
CommonSpacescould not be created withClimaComms.MPICommsContext. This is now fixed with PR 2176. -
Fixed bug in distributed remapping with CUDA. Sometimes,
ClimaCorewould not properly fill the output arrays with the correct values. This is now fixed. PR 2169 -
Added support for 1D extruded spaces. PR 2172. Closed 1989, and 1383.
- A new
Adaptwrapper was added,to_device, which allows users to adapt datalayouts, spaces, fields, and fieldvectors between the cpu and gpu. PR 2159. - Remap interpolation cuda threading was improved. PR 2159.
center_spaceandface_spaceare now exported fromCommonSpaces. PR 2157.- Limiter debug printing can now be suppressed, and printed at the end of a simulation, using
Limiters.print_convergence_stats(limiter). PR 2152. - Fixed getidx for
GradientC2FwithSetValuebcs. PR 2148. - Added support
data2arrayforDataF(i.e.,PointFields). PR 2143. HDF5Reader/HDF5Writernow supportdo-syntax. PR 2147.
-
Added support for CPU<->GPU conversion of spaces, fields and fieldvectors, via Adapt. PR 2114.
-
Fixed numerics of column integrals for deep atmosphere. PR 2119.
-
Fixed high-resolution gpu space construction. PR 2100.
A new ClimaCore.DebugOnly module was added, which can help users find where
NaNs or Infs come from in a simulation, interactively. Documentation, with a
simple representative example can be found here.
Various improvements to Remapper 2060
The ClimaCore.Remapping module received two improvements. First, Remapper is
now compatible with purely vertical Spaces (performing a linear
interpolation), making it compatible with column setups. Second, a new set of
simplified interpolation functions are provided.
Now, interpolating a Field field is as easy as
import ClimaCore.Remapping: interpolate
output_array = interpolate(field)The target coordinates are automatically determined, but can also be customized. Refer to the documentation for more information.
- Fixed gpu support (adapt) for van-leer limiters. PR 2112.
- Support for new TVD limiters were added, PR [1662] (#1662).
- Fixed writing/reading purely vertical spaces. PR 2102
- Fixed correctness bug in reductions on GPUs. PR 2106
ClimaCore.Spaces now comes with two functions, face_space and
center_space, to convert a Space from being cell-centered to be
face-centered (and viceversa). These functions only work for vertical and
extruded spaces.
- We've added new convenience constructors for spaces PR 2082. Here are links to the new constructors:
-
We've added new convenience constructors for grids PR 1848. Here are links to the new constructors:
-
A
strict = truekeyword was added torcompare, which checks that the types match. Ifstrict = false, thenrcomparewill returntrueforFieldVectors andNamedTuples with the same properties but permuted order. For example:rcompare((;a=1,b=2), (;b=2,a=1); strict = true)will returnfalseandrcompare((;a=1,b=2), (;b=2,a=1); strict = false)will returntrue
-
We've added new datalayouts:
VIJHF,IJHF,IHF,VIHF, to explore their performance compared to our existing datalayouts:VIJFH,IJFH,IFH,VIFH. PR #2055, PR #2052. -
We've refactored some modules to use less internals. PR #2053, PR #2052, #2051, #2049.
-
Some work was done in attempt to reduce specializations and compile time. PR #2042, #2041
ClimaCore had incorrect lower bounds for certain packages. PR
#2078 fixes the lower bounds
and adds a GitHub Action workflow to test it. ClimaCore now requires Julia
1.10 or greater.
- Fixed world-age issue on Julia 1.11 issue Julia#54780, PR #2034.
PR #2034 fixes some undefined
behavior in the DataLayout module. This bug was manifesting itself as a world age error in some applications that are using Julia 1.11.
PR #2033 introduces new
constructors for DataLayouts. Instead of writing
array = rand(FT, Nv, Nij, Nij, 3, Nh)
data = VIJFH{S, Nv, Nij}(array)You can now write
data = VIJFH{S}(ArrayType{FT}, rand; Nv, Nij, Nh)and grab the array with parent(data) (if you need).
Note: These constructors are meant to be used in tests and interactive use, not in performance sensitive modules (due to their lack of inferrability).
- Fixed multiple-field solve for land simulations PR #2025.
- Fixed Julia 1.11 PR #2018.
Nhwas turned back into a dynamic parameter, in order to alleviate compile times PR #2005.- Defined some convenience methods #2012
Due to a bug, == was not recursively checking FieldVectors with different
types, which resulted in false positives. This is now fixed and FieldVectors
with different types are always considered different.
Prior to this change, the ClimaCore.InputOutput module did not save whether a
Space was constructed with deep = true. This meant that restarting a
simulation from a HDF5 file led to inconsistent and incorrect spaces and
Fields. This affected only extruded 3D spectral spaces.
We now expect Spaces read from a file to be bitwise identical to the original
one.
PR #2021.
- Fixed some type instabilities PR #2004
- More fixes to higher resolution column cases for the GPU #1854
-
Extended
create_dss_bufferandweighted_dss!forFieldVectors, rather than justFields. PR #2000. -
Spaces withenable_bubble = true
Prior to this change, the ClimaCore.InputOutput module did not save whether a
Space was constructed with enable_bubble = true. This meant that restarting
a simulation from a HDF5 file led to inconsistent and incorrect spaces and
Fields. This affected only 2D spectral spaces (and extruded ones that have
this type of horizontal space).
We now expect Spaces read from a file to be bitwise identical to the original
one.
PR #1999.
- Added support for mixing extruded and horizontal spaces in GPU kernels. PR #1987.
- Inference was fixed for some broadcast expressions involving columns PR #1984.
-
CUDA kernel launch configurations have been tuned to improve performance, and now allows for high resolution in the vertical direction PR #1969, issue #1854 closed.
-
DSS was refactored, and machine precision changes can be expected. PR #1958.
- Added hyperbolic tangent stretching. PR #1930.
- Support for matrix fields on spectral and point spaces was added, PR #1884.
- Support for 3-component DSS transform was added, PR #1693.
- Support for column-wise "accumulate"/"reduce" operations were added, PR #1903. These abstractions will allow us to group, paralellize and optimize more column-wise work on the GPU.
- A new macro,
Fields.@rprint_diffwas added, which recursively print differences between twoFieldVectors (of the same type) (PR #1886). - Julia 1.11 fixes (PR #1883)
Nhhas been added to the type parameter space, which allows us to more flexibly write performant backend kernels (PR #1894). This was leveraged in PR #1898, and may result in slightly more performant kernels.
- Various performance tweaks (PRs #1840, #1837, #1843, #1839).
- CPU/GPU kernels are now determined by dispatching, instead of specializing, which should (hopefully) have generally fixed GPU dispatching issues (PR #1863).
- Matrix multiplication kernels have been improved (PR #1880).
- Support for the following methods have been deprecated (PR #1821, ):
IntervalTopology(::Mesh)in favor of usingIntervalTopology(::ClimaComms.AbstractDevice, ::Mesh)FaceFiniteDifferenceSpace(::Mesh)in favor of usingFaceFiniteDifferenceSpace(::ClimaComms.AbstractDevice, ::Mesh)CenterFiniteDifferenceSpace(::Mesh)in favor of usingCenterFiniteDifferenceSpace(::ClimaComms.AbstractDevice, ::Mesh)FiniteDifferenceGrid(::Mesh)in favor of usingFiniteDifferenceGrid(::ClimaComms.AbstractDevice, ::Mesh)
copyto!andfill!have been fixed PR #1802.
FieldMatrixWithSolver, a wrapper that helps defining implicit Jacobians. PR #1788
-
array2field(::Field)andfield2array(::Field)convenience functions, to help facilitate use with RRTMGP. PR #1768 -
Nvis now a type parameter in DataLayouts that have vertical levels. As a result, users can useDataLayouts.nlevels(::AbstractData)to obtain a compile-time constant for the number of vertical levels. -
interpolate(field, target_hcoords, target_zcoord)convenience function so that theRemapperdoes not have to be explicitely constructed. PR #1764
run_field_matrix_solver!was fixed for column spaces, and tests were added to ensure it doesn't break in the future. PR #1750band_matrix_solve!, which has improved performance. PR #1735.run_field_matrix_solver!, which results in more efficient kernels being launched. PR #1732.band_matrix_solve!for tridiagonal systems, improving its performance. PR #1731.
unrolled_mapinmultiply_matrix_at_indexto avoid the recursive inference limit when compiling nested matrix operations. PR #1684Remappers can now process multipleFields at the same time if created with somebuffer_lenght > 1. PR (#1669) Machine-precision differences are expected. This change is breaking because remappers now return the same array type as the input field.multiple_field_solvekernels, which should improve performance. PR #1715.Fields.@fused_direct. PR #1641.
interpolate!for MPI runs. PR #1642.
support for many deprecated methods have been dropped PR #1632.
rdivwithrmul. PR (#1496) Machine-precision differences are expected.distributed_remapping. Newdistributed_remappingis non-allocating and up to 1000x faster (on GPUs). Newdistributed_remappingno longer supports thephysical_zargument (this option is still available inRemapping.interpolate_column). Newinterpolate!function is available for remapping in-place. The new preferred way to define aRampperisRemapper(space, target_hcoords, target_zcoords)(instead ofRemapper(target_hcoords, target_zcoords, space)). PR (#1630)