Explore options for a separate package to develop out of scope CDAT features

[tomvothecoder]

Background

LLNL Climate Program staff who are current or former CDAT users have expressed interest in CDAT features deemed beyond the scope of xCDAT (e.g., statistical APIs). This includes:

• genutil features
  • New features/ideas for future releases (e.g., CDAT, other libraries) #271 (comment)
  • about porting genutil.statistics.linearregression function #395)
• cdutil features (under "Consider implementing")
  • New features/ideas for future releases (e.g., CDAT, other libraries) #271 (comment))

Option 1. New repository hosted by PCMDI and distributed via Anaconda (pcmdi_utils?)

Target Audience

• LLNL Climate Program, PCMDI
• General CDAT users

Requirements

• Implement valuable CDAT features that are not available in other Xarray-based packages
• Geared towards more specialized, programmatic usage at LLNL (but also might be useful for general users)
• Considers unique requirements and algorithms, such as Ben Santer's linear regression algorithm implemented in genutil.statistics.linearregression

Benefits

• Share common functionalities under a common platform, instead of scientists/developers duplicating code in their own scripts/packages
• Faster release cycle for features that might be useful to more specific audiences
• Facilitate and improve knowledge transfer through reproducible code

Challenges

• Not duplicating existing functionalities from other packages
  • CDAT users should explore other Xarray-based packages first before trying to implement features
  • Examples: xskillscore, xarray.polyfit, sklearn-xarray
• Who is willing, eager, AND has the bandwidth to maintain this package?
  • It isn't a good idea to have a single person develop and maintain this repository -- if they leave or stop maintaining the package then it will most likely die.
  • xCDAT budget is limited and members of the core dev team cannot lead this effort
• Managing scope and judging usefulness of features for a few individual vs. many users
  • Maybe this package can be less strict on standards and users should be aware that using it is at they're discretion.
• How should the DevOps, code standards, unit testing be integrated?
  • This package should be robust and maintainable since the accuracy of the results produced by computational functions are critical for analysis work.
  • The idea of learning how to unit test seems discourage folks who aren't interested in learning or think it is difficult.
  • In most cases, software and code is a means to an end for non-software engineers. Contributing to an open-source package is another ball-game.

[tomvothecoder]

Hi @xCDAT/core-developers, I'm getting the conversation started here. This idea is not a high priority, but it sounds useful based on CDAT user interest.