The future of NCL in the ESMValTool

[mattiarighi]

The discussion about the future of NCL support in the ESMValTool has come up quite frequently in the last weeks in various GitHub issues here.

The reason behind this discussion is (should be) well known: NCAR has decided to stop the development of NCL, which has been put into maintenance mode. Graphics routine have been translated to Python and are available via the PyNGL library. This means that NCL is still available and will be available "for the foreseeable future" (as NCAR itself states in its letter to users), but the transition to Python is recommended.

Some general considerations to trigger the discussion:

• stopping NCL support in the ESMValTool right now is not realistic and the upcoming v2.0 will still need to support NCL (for the diagnostics and the CMORizers).
• external diagnostic packages that are (or are going to be) implemented in the ESMValTool and are written in NCL still exist (e.g., CVDP).
• several (how many?) ESMValTool developers can only contribute NCL diagnostics and may not want to contribute code in another language or may not have the resources to do that. A survey across all ESMValTool users and developers might help us to get a better feeling on this.
• according to NCAR: "we want to stress that NCL is not going away. NCL users will be able to download NCL and execute their scripts for the foreseeable future". It would be good to get in touch with the NCL development team to better understand the actual perspectives. Are we talking about months or years? What happens in case of NCL installation issues on new systems? Would support still be provided in such cases? Etc.
• after the release of v2.0, we should start recommending our NCL develoeprs to switch to Python, maybe using the aforementioned PyNGL package to make this transition smooth (a transition guide by DKRZ is also available).
• based on my experience in coordinating the v2.0 development, the coding quality standards of the ESMValTool require a non negligible amount of additional resources (for both developers and reviewers), which might discourage the newbies to switch from NCL to python.
• we should be aware that by reducing the multi-language support, we face the risk of losing a (significant?) part of the ESMValTool community.

@ESMValGroup/esmvaltool-coreteam
@zklaus @hb326 @LisaBock @ruthlorenz (please tag other non-core members who might be interested in this topic).

[bouweandela]

external diagnostic packages that are (or are going to be) implemented in the ESMValTool and are written in NCL still exist (e.g., CVDP).

A bit of good news here is that they are translating this particular example to Python as a pilot, see: NCAR/ncl#64 (comment)

[bouweandela]

From the roadmap it also looks like there will be training available.

[zklaus]

Thanks for picking this up, @mattiarighi !

stopping NCL support in the ESMValTool right now is not realistic and the upcoming v2.0 will still need to support NCL (for the diagnostics and the CMORizers).

I completely agree that this is a process, cannot happen over night, and should have (almost) zero bearing on v2.0 (the exception just being that, all else being equal, I wouldn't prefer NCL for new developments).
I think your suggestion below for recommendations to developers is a good thing, more on that below.

external diagnostic packages that are (or are going to be) implemented in the ESMValTool and are written in NCL still exist (e.g., CVDP).

@bouweandela has that covered. I think therefore CVDP isn't a blocker, but maybe we should collect all such packages that we have in some place (extra ticket, wiki?)?

several (how many?) ESMValTool developers can only contribute NCL diagnostics and may not want to contribute code in another language or may not have the resources to do that. A survey across all ESMValTool users and developers might help us to get a better feeling on this.

That is unfortunate, but adding substantial new NCL code seems to go in the wrong direction. In most cases those developers will have to adapt to the changing environment for reasons entirely outside of ESMValTool.

according to NCAR: "we want to stress that NCL is not going away. NCL users will be able to download NCL and execute their scripts for the foreseeable future". It would be good to get in touch with the NCL development team to better understand the actual perspectives. Are we talking about months or years? What happens in case of NCL installation issues on new systems? Would support still be provided in such cases? Etc.

👍 Though of course, as you know, intentions aside, the limits will always be funding and man-power. Usually, few people with the necessary skill are eager to maintain things after a few years without the mandate to change them.

after the release of v2.0, we should start recommending our NCL develoeprs to switch to Python, maybe using the aforementioned PyNGL package to make this transition smooth (a transition guide by DKRZ is also available).

To formalize this: With the release after v2.0 or with v2.0 itself, we should formally deprecate it. Imho, we can already now give the general recommendation to rather prefer python, making it clear that NCL contributions are still acceptable for now. But if somebody is on the fence on what to choose, they should go for python.

based on my experience in coordinating the v2.0 development, the coding quality standards of the ESMValTool require a non negligible amount of additional resources (for both developers and reviewers), which might discourage the newbies to switch from NCL to python.

Hm, this shouldn't be specific to python. If it seems that the burden on python contributions is higher than on other languages, we should rather see how we can achieve the same high standards in those languages. At the moment we are rather lenient there, probably for lack of automatic qa. As an example, we know that, no matter the language, functions should be short. I find around 45 lines as an upper limit a good guide rule. If in python you make much longer functions, you are likely to get a complexity warning or too-many-local-variables from our code checkers. In NCL this malpractice is still wide-spread.

Bottom line: Quality standards should be equally high for all languages, the effort comparable.

we should be aware that by reducing the multi-language support, we face the risk of losing a (significant?) part of the ESMValTool community.

Interesting question. Looking at the list of contributors (here for the private repository for those who have access), my impression is that most would be comfortable with python. What do you think? Of course, this is just a personal impression. To get a better picture we could do a survey, best limited to those having contributed at least one commit, or try automatic numbercrunching, eg see what fraction of lines added per person where in NCL. I think a survey can be done quickly with (eg) google forms; number crunching would take more effort.

It would be very interesting to hear from NCL developers on this.

[valeriupredoi]

I got 0 NCL lines 🤣

So this will be easier to discuss when we will have split ESMValTool (at least) into core and diagnostics. The core is and will be Python; the diagnostics are a bit more relaxed and the shepherding of the scientists developing them towards Python (and away from NCL) can be done in a bit more of a relxaed way rather than at gun point if it was for the core

[mattiarighi]

DKRZ is still organizing NCL workshops for the users, so they are probably not planning to retire NCL support soon.

[jvegreg]

DKRZ is still organizing NCL workshops for the users, so they are probably not planning to retire NCL support soon.

Just remember that the end of python 2 support seemed to be far far away

[valeriupredoi]

DKRZ is still organizing NCL workshops for the users, so they are probably not planning to retire NCL support soon.

Just remember that the end of python 2 support seemed to be far far away

I am working with Met Office software which is (as expected) Python2 and everytime I try and update Py2 deps I get warnings about the looming dat of January 1st 2020 😁

[mattiarighi]

There are some news about the future of NCL:

http://www.ncl.ucar.edu/Document/Pivot_to_Python/september_2019_update.shtml

"To summarize the status of NCL, the project is "feature frozen". CISL does not have plans at this time to add new features to NCL. We will, however, prepare maintenance releases containing critical bug fixes and user-contributed code on an infrequent basis. Thus, there is no reason for members of the community with substantial investments in NCL to consider porting their NCL code to Python (unless you want to take advantage of the improved scalability afforded by our use of Dask)."

[bouweandela]

A new update from 11 November 2020 by the NCL authors:

In January of 2019 NCAR announced plans to transition away from the NCL language and focus efforts on providing geosciences analysis tools that were based on the Python Scientific Ecosystem. Furthermore, we announced that NCL would be put into “maintenance mode”, and due to limited staffing resources, would no longer be actively developed by NCAR.

I wanted to take this opportunity to elaborate further on what “maintenance mode” means for the community, and hopefully relieve anxiety that some may be feeling. Maintenance mode means that NCAR will not add new features to the NCL language or function library, nor will we fix sub-critical software defects. However, NCAR will for as long as we are reasonably able and there is a sufficient demand - put out periodic bug-fix releases and ensure NCL continues to build on currently supported platforms. In addition, we will do our best to timely address installation questions that are related to the Conda package management system. Furthermore, as part of our embracement of Open Development practices, we will facilitate community contributions of all types (e.g. bug fixes, new functions, etc.). Stay tuned for more information on Open Development.

I fully recognize that many in the community have substantial investments in NCL scripts and I do not expect, nor would I advise, translating thousands of lines of NCL code into Python. That said, I would caution against making substantial new investments in workflows based on NCL. To repeat, it is our intent to ensure that existing NCL scripts will continue to run for as long as reasonably possible, but NCAR’s priority for the future will be on modern Python tools.

[zklaus]

NCL in conda-forge has been a bit neglected for a while. This means that it has not been (re-)build against more recent versions of the usual libraries (libgdal, libnetcdf, proj,...) which makes it hard for us to install recent versions that are required by other parts of the ESMValTool eco system.

I have spent a good chunk of the last week and the beginning of this week trying to improve the situation, but this is a real drain on resources and it is not clear how long we can keep this up.