Skip to content

Conversation

@chengzhuzhang
Copy link
Collaborator

@chengzhuzhang chengzhuzhang commented Oct 9, 2025

Summary

This feature is useful in glb time-series task for land, as 300+ variables from land are being processed in parallel, and potentially is useful for ne120 HR processing. According to NCO documentation:

For Background parallelism, job_nbr defaults to the number of variables to be split, so that not specifying job_nbr results in launching var_nbr simultaneous splitter tasks. This scales well to over a hundred variables in our tests [76](https://nco.sourceforge.net/nco.html#FOOT76). In practice, splitting timeseries consumes minimal memory, since ncrcat (which underlies the splitter) only holds one record (timestep) of a variable in memory [Memory Requirements](https://nco.sourceforge.net/nco.html#Memory-Requirements).

With added job_nbr flag for ts task, user can manage RAM use by reducing the number for parallel process.

Issue resolution:

Select one: This pull request is...

  • a bug fix: increment the patch version
  • a small improvement: increment the minor version
  • a new feature: increment the minor version
  • an incompatible (non-backwards compatible) API change: increment the major version

Small Change

  • To merge, I will use "Squash and merge". That is, this change should be a single commit.
  • Logic: I have visually inspected the entire pull request myself.
  • Pre-commit checks: All the pre-commits checks have passed.

@chengzhuzhang
Copy link
Collaborator Author

@czender could you help to review if the implementation makes sense?
@forsyth2 I tested in my .cfg file, it works, but I will need you help to fix the CI/CD, we should aim to include this in rc1 release. Thanks.

@forsyth2
Copy link
Collaborator

forsyth2 commented Oct 9, 2025

help to fix the CI/CD

Sure, I will work on that today.

we should aim to include this in rc1

rc1 is already in-process, as of yesterday. Any development now will have to go in to rc2.

Sorry I meant to say rc2..

@czender
Copy link

czender commented Oct 9, 2025

Sure, happy to help. LMK if you have any specific questions.

@forsyth2 forsyth2 added the semver: small improvement Small improvement (will increment patch version) label Oct 9, 2025
@forsyth2
Copy link
Collaborator

forsyth2 commented Oct 9, 2025

help to fix the CI/CD

Fixed with e27e797.

GitHub Actions pass too for 3.11, 3.12, 3.13.

@chengzhuzhang
Copy link
Collaborator Author

thanks @forsyth2 !
@czender could you review changes here to see if this is reasonable? Thanks!

Copy link

@czender czender left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These changes look appropriate to allow optional specification of --job_nbr in zppy.

@chengzhuzhang
Copy link
Collaborator Author

Thank you, @czender !

@chengzhuzhang chengzhuzhang merged commit 3494c9a into main Oct 10, 2025
6 checks passed
@chengzhuzhang chengzhuzhang deleted the feature/730-expose-job-nbr-for-ts branch October 10, 2025 16:21
@forsyth2 forsyth2 mentioned this pull request Oct 16, 2025
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

semver: small improvement Small improvement (will increment patch version)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Expose--job_nbr option for ncclimo time series task for large data (e.g.)ne120

4 participants