Downloading cart using wget is limited to a 1000 files

[LukasSterckx]

I'm trying to download the results of all available models and versions which are part of CMIP-5. I however only need 8 variables (pr, psl, rsds, tas, tasmax, tasmin, sfcwind, rhs), 4 scenario's (rcp2.6, rcp4.5, rcp6.0 and rcp8.5) and the period 2006-2100.

Since it's not possible to add individual files to the cart, my plan was to add all datasets to the cart, then generate a wget script and then use python to edit the script and remove all variables and time-periods I don't need.

However when generating the wget script, I saw that the maximum amount of files was 1000. Is there a way to work around this? Or a way to directly select which variables and time period you want from the datasets before generating the wget script?

[sashakames]

If you use the ESGF-US site aims2.llnl.gov, the API configured there should have an increased limit to 10000 files. Apologies that this solution is not comprehensive and covers non-US data nodes. Alternatively, we recommend esgpull as it should support variable-level file access for CMIP5, though that is a different approach as you cannot subselect with a "data cart" model.

[LukasSterckx]

Thanks so much for the clarification and the helpful suggestions!