[PLACEHOLDER] ccpp-capgen-ng: a unified successor to ccpp-prebuild and ccpp-capgen

[climbfuji]

Description

This is a placeholder for a forthcoming pull request.

ccpp-capgen-ng is a prototype of a potential next-generation code generator for the CCPP framework. It is not a criticism of the existing tools — both ccpp-prebuild and ccpp-capgen have served their respective communities well and embody years of careful design. This project attempts to synthesize the strengths of both: the performance of ccpp-prebuild and the metadata richness and feature set of ccpp-capgen, while also taking advantage of relaxed code-generator requirements that have emerged over time and lessons learned from operating both tools in development, research, and production. It also aims to resolve several outstanding issues that have been blocking progress toward a long-discussed unification of the two code generators.

Supporting documents — including a redesign analysis, the LLM prompt used to bootstrap development, developer and user documentation, and a migration guide — will be provided as part of the draft pull request.

The code was developed with the assistance of an LLM (Claude, by Anthropic) as a coding partner. Where possible, existing code was reused and repackaged.

A pull request will follow shortly. Feedback and questions are welcome.

Testing

The following tests and features are currently supported by ccpp-capgen-ng:

• Former ccpp-capgen tests:
  • advection
  • advection_auto_clone (support legacy constituents auto-cloning from metadata)
  • capgen_ng
  • ddthost
  • nested_suite
  • var_compat
• Former ccpp-prebuild tests:
  • chunked_data
  • blocked-data not supported in capgen or capgen-ng
  • opt_arg
• Newly added features/tests:
  • multi-instances
  • multi-instances + constituents
  • suite_allocate (exercises suite-owned variables with various types of runtime-known dimensions)
  • constituents_dim (suite- and host-owned variables with dimensions from constituents objects)
• capgen-ng unit testing framework
  • 1300+ unit and docstring tests
• Host model integration
  • CCPP-SCM all suites compile and all manual tests run (a sample of all permutations)
    • DRAFT: Update to capgen-ng ccpp-scm#695 and PRs listed in there
    • DRAFT Update to capgen-ng #761
    • DRAFT [scm/dev] Updates for capgen-ng ccpp-physics#1221
    • DRAFT: Update module_mp_tempo_params.meta for capgen-ng TEMPO#31
  • CAM-SIMA started - Python unit tests pass, system tests in progress
    • https://github.com/climbfuji/CAM-SIMA/tree/feature/capgen-ng
    • Feature/capgen ng climbfuji/CAM-SIMA#1
    • kessler, rrtmgp, and se_cslam (the full cam7 suite) build and run to completion on Derecho under both gnu and intel, with bit-comparable results
  • UFS started (@dustinswales)
  • NEPTUNE complete
    • Draft pull requests open in NRL Enterprise GitHub
    • All 300+ tests pass with the three mandatory compilers (Intel LLVM, GCC, LLVM native)
    • Regular, mid- and high-altitude physics covered (including proprietary NRL physics code)

Issues

A list of issues that will be addressed with this issue will be provided in the associated pull request.

WORK IN PROGRESS

Addressing this issue will also address/close the following issues if/when the participating host models switch to capgen-ng:

• Rethinking horizontal dimensions #760
• Weird Capgen fortran parsing error? #757
• Bug in capgen (test/capgen_test) ? surface_air_pressure outvar in ddt_suite #755
• Document --kind-type in ccpp_capgen.py CLI/help, and consider reversing the syntax for external kind modules #754
• Remove unused metadata attribute state_variable #753
• capgen: Add capability and test for passing mpi communicator as a flat field #751
• capgen: case-sensitivity when comparing metadata to Fortran #746
• Bug in scripts/file_utils.py / move_modified_files routine #744
• Unallocated fields passed into Group Caps #690
• Capgen performance when using many suites #688
• Add support for MPI_Comm (mpi08) object in capgen #686
• Transition UFS weather model to Capgen #682
• Transition CCPP-SCM to Capgen #680
• Transition NEPTUNE to capgen #679
• Enhanced/Expanded Testing  #595
• capgen's Fortran parser does not allow array bounds which are not Fortran identifiers #588
• Add "number of instances" dimension to module-level variables to support multi-instance usage #561
• capgen: check handling of type attribute for ddts #516

To check/may not be implemented yet:

• capgen Fortran parser does not accept internal DDTs as part of a registered DDT #749
• capgen: Invalid character declaration, 'character(len=nf_file_length) ' #748
• capgen: dimension specifier foo(N) is not the same as foo(1:N) in Fortran code #747
• Errors when integer variable is output from one phase and used as an interstitial dimension in another #681
• CCPP Framework to provide API calls for suite inspection by host #380
• Fortran/metadata comparison incorrectly reporting bug when comparing Fortran variable declarations and metadata for derived types #360

Working towards:

• Simplification of ccpp-framework code #737

Supporting Materials

Date and time are all local U.S. Mountain Time. Subsections are sorted from newest to oldest.

The documents below were produced by Claude during development and have not yet been formally reviewed. They are shared for transparency and to give context for the design decisions made. They should be treated as starting points rather than authoritative references.

If documents were reviewed/edited by a human, it is mentioned explicitly.

LLM-Generated Drafts (20260605T1924/latest)

The documents capgen_compat_layer.md and briefing_pm.md were fully reviewed and edited.

briefing.md
briefing_pm.md
capgen_compat_layer.md
constituents.md
constituents_overhaul.md
migration.md
redesign_analysis.md
redesign_prompt.md

LLM-Generated Drafts (20260601T0935)

The document briefing_pm.md was fully reviewed and edited.

briefing.md
briefing_pm.md
constituents_overhaul.md
migration.md
redesign_analysis.md
redesign_prompt.md

LLM-Generated Drafts (20260521T1542)

The document auto_clone_constituents.md was fully reviewed and edited.

auto_clone_constituents.md

Slides from ccpp-framework meeting 2026/05/21

https://docs.google.com/presentation/d/1oG9o3HDJH2RA4swwCxTIc97JS6MxBRlo8qUAJENBEIY/edit?slide=id.g3e3de55f508_0_0#slide=id.g3e3de55f508_0_0

LLM-Generated Drafts (20260519T0903)

The document briefing_pm.md was fully reviewed and edited.

migration_20260519T0905.md
briefing_20260519T0905.md
briefing_pm_20260519T0905.md
constituents_20260519T0905.md
constituents_overhaul_20260519T0905.md

LLM-Generated Drafts (20260513T0733)

constituents_20260513T0733.md
constituents_overhaul_20260513T0733.md
migration_20260513T0733.md
redesign_analysis_20260513T0733.md
redesign_prompt_20260513T0733.md

Comparison: size of code base:

LLM-Generated Drafts (202060505T2044/initial)

redesign_analysis_original_202060505T2044.md
redesign_prompt_original_20260505T2044.md

[bbakernoaa]

@climbfuji I'm really interested to get this moving forward. How can I help move this forward?

[climbfuji]

@climbfuji I'm really interested to get this moving forward. How can I help move this forward?

@bbakernoaa Thank you very much for your offer to help, very much appreciated. There are many moving pieces. So far, this has been tested with CCPP-SCM and with NEPTUNE (with three compilers, Intel LLVM + GCC + LLVM native). The next tests need to cover UFS and CAM-SIMA. For CAM-SIMA, I'll work with the developers @peverwhee and @nusbaume. For UFS, your help would be appreciated, but we should coordinate with @dustinswales.

To prepare for UFS and SCM updates, we need to first fix the Metadata vs Fortran inconsistencies in the NCAR CCPP physics. I created pull requests for the UFS weather model, UFSATM, and CCPP physics ufs/dev for that (b4b identical w/ current baseline). It would be great if we could push these pull requests through quickly. When these changes come back to the authoritative NCAR ccpp-physics main branch, we can update the SCM and NEPTUNE branches. The changes will minimize the differences that we need to make when we transition to the new code generator. Maybe you can help with raising the priority for these PRs. I am also tagging @grantfirl and @dustinswales here.

[bbakernoaa]

@climbfuji I'm really interested to get this moving forward. How can I help move this forward?

@bbakernoaa Thank you very much for your offer to help, very much appreciated. There are many moving pieces. So far, this has been tested with CCPP-SCM and with NEPTUNE (with three compilers, Intel LLVM + GCC + LLVM native). The next tests need to cover UFS and CAM-SIMA. For CAM-SIMA, I'll work with the developers @peverwhee and @nusbaume. For UFS, your help would be appreciated, but we should coordinate with @dustinswales.

To prepare for UFS and SCM updates, we need to first fix the Metadata vs Fortran inconsistencies in the NCAR CCPP physics. I created pull requests for the UFS weather model, UFSATM, and CCPP physics ufs/dev for that (b4b identical w/ current baseline). It would be great if we could push these pull requests through quickly. When these changes come back to the authoritative NCAR ccpp-physics main branch, we can update the SCM and NEPTUNE branches. The changes will minimize the differences that we need to make when we transition to the new code generator. Maybe you can help with raising the priority for these PRs. I am also tagging @grantfirl and @dustinswales here.

@climbfuji Thanks. I would really like to get this moving forward as fast as possible. @dustinswales how can I help?

[climbfuji]

@climbfuji I'm really interested to get this moving forward. How can I help move this forward?

@bbakernoaa Thank you very much for your offer to help, very much appreciated. There are many moving pieces. So far, this has been tested with CCPP-SCM and with NEPTUNE (with three compilers, Intel LLVM + GCC + LLVM native). The next tests need to cover UFS and CAM-SIMA. For CAM-SIMA, I'll work with the developers @peverwhee and @nusbaume. For UFS, your help would be appreciated, but we should coordinate with @dustinswales.
To prepare for UFS and SCM updates, we need to first fix the Metadata vs Fortran inconsistencies in the NCAR CCPP physics. I created pull requests for the UFS weather model, UFSATM, and CCPP physics ufs/dev for that (b4b identical w/ current baseline). It would be great if we could push these pull requests through quickly. When these changes come back to the authoritative NCAR ccpp-physics main branch, we can update the SCM and NEPTUNE branches. The changes will minimize the differences that we need to make when we transition to the new code generator. Maybe you can help with raising the priority for these PRs. I am also tagging @grantfirl and @dustinswales here.

@climbfuji Thanks. I would really like to get this moving forward as fast as possible. @dustinswales how can I help?

Realized that Dustin is out of the office this week. I suggest we work with EPIC to prioritize the metadata-fortran bug fixes this week and early next week start working on dropping capgen-ng in the UFS.

@ligiabernardet FYI