Add NSRDB GOES V4

[williamhobbs]

NSRDB's latest "V4" dataset appears to be available. It would be nice to have this available in iotools.

See NREL/SAM#1920 for some additional info. Quoting @cpaulgilman:

NSRDB V4.0.0 data (1998-2023) is now available from new NREL Developer API endpoints:

https://developer.nrel.gov/docs/solar/nsrdb/nsrdb-GOES-tmy-v4-0-0-download/

https://developer.nrel.gov/docs/solar/nsrdb/nsrdb-GOES-aggregated-v4-0-0-download/

SAM's Advanced Downloads option can access these endpoints, but the direct download from the Location and Resource page still accesses the older V3.2.2 (1998-2022) API endpoint.

Note there is an "aggregated" and "conus" endpoint. The latter seems to be a subset of the former, but need to determine the correct endpoint to use.

Also note that there was an issue with DNI that has been fixed (also in that SAM issue's comments).

[markcampanelli]

I was unsure what to make of the GOES V4 datasets. The descriptions of the various GOES endpoints all seem to be generic “copy pasta” at https://developer.nrel.gov/docs/solar/nsrdb/. Perhaps this should be reported?

[cpaulgilman]

Another source of information about these datasets is the NSRDB website, for example, https://nsrdb.nrel.gov/data-sets/us-data. There's no mention of GOES V4 there (yet), although the GOES V4 endpoints are included in the list of available data from the NSRDB Viewer, albeit with different names:

[williamhobbs]

The descriptions of the various GOES endpoints all seem to be generic “copy pasta” at https://developer.nrel.gov/docs/solar/nsrdb/. Perhaps this should be reported?

Tagging @grantbuster, should there be updates to the docs on the NREL developer website for NSRDB GOES V4? I'll admit that I was expecting "PSM V4", so even the name has me a bit confused.

Also, I thought about tagging Paul Edwards, but this seems like a question for the NSRD team instead of the developer website team, so I'll let someone else tag him if needed...

[markcampanelli]

If a get_psm4 or get_goes4 function is added for this, then I think we should consider updating the names parameter to take a sequence of strings instead of just a single string. I often download the entire set of years of historical data, and I'm wondering if it would be more efficient to allow an array of years to be specified all in one go? (My apologies if this discussion has already been had and settled for reasons not apparent to me. If it must be one year at a time, then keep the same functionality but use the singular name instead of plural names.) UPDATED: CSV-formatted responses do not support multi-year requests.

To better follow semver and avoid surprising users when upgrading pvlib, I would be a fan of adding the full version in the function names, e.g., get_goes_4_0_0, and using NREL's corresponding (immutable) URL to that version. Then, each of these version-specific functions could call the same "URL-flexible" data-provider function under the hood, where this latter function does not specify any default URL(s).

[xieyupku]

I recommend updating pvlib from PSM V3 to PSM V4.
Based on the structure of psm3.py, the new description for PSM V4 could be
Get PSM4 TMY
see https://developer.nrel.gov/docs/solar/nsrdb/nsrdb-GOES-tmy-v4-0-0-download/

Enhancements in PSM V4 compared to PSM V3:

1. Improved cloudy-sky direct normal irradiance (DNI) data using the FARMS-DNI model.
2. Improved snow/ice surface albedo data.
3. Implementation of a physics-guided machine learning procedure to gap-fill missing cloud properties.

[cwhanse]

consider updating the names parameter to take a sequence of strings

Does the NREL API parse that? pvlib is just passing the value of names in the request body. If pvlib has to parse a list of strings or similar, that would complicate matters.

adding the full version in the function names, e.g., get_goes_4_0_0

The url parameter can accept the NREL URL for each version. It could be helpful to provide some kind of dictionary, e.g., {'v400': <url for GOES 4.0.0>} if the URLs are actually immutable, and not something we'd have to correct every year or two.

[williamhobbs]

consider updating the names parameter to take a sequence of strings

Does the NREL API parse that?

I can't seem to get a multi-year request to work. The documentation does make it seem like it should work, though.

[markcampanelli]

I can't seem to get a multi-year request to work. The documentation does make it seem like it should work, though.

I think I was mistaken here. CSV-formatted responses apparently do not support multiple years:

CSV Output Format
Direct streaming of CSV data is supported for single location, single year only.

From https://developer.nrel.gov/docs/solar/nsrdb/psm3-2-2-download/.

[markcampanelli]

consider updating the names parameter to take a sequence of strings

Does the NREL API parse that? pvlib is just passing the value of names in the request body. If pvlib has to parse a list of strings or similar, that would complicate matters.

adding the full version in the function names, e.g., get_goes_4_0_0

The url parameter can accept the NREL URL for each version. It could be helpful to provide some kind of dictionary, e.g., {'v400': <url for GOES 4.0.0>} if the URLs are actually immutable, and not something we'd have to correct every year or two.

@cwhanse So the idea is to grow this dictionary as the NSRDB evolves?

[cwhanse]

So the idea is to grow this dictionary as the NSRDB evolves?

Yes, and keep the current function's signature.

[williamhobbs]

@AdamRJensen @kandersolar @wholmgren, it looks like "horns" issue has been resolved with GOES v4. Add this to the "reasons to add GOES V4" bucket.

To my eye, it looks like some variability has increased, but hard to say if it's "better". It looks worse for this one day, but there are other (more important, IMO) types of variable conditions that I haven't looked at.

For reference (from https://doi.org/10.1109/PVSC57443.2024.10748762):

[williamhobbs]

I also just noticed that GOES V4 now includes 2023, while PSM3 is still only available through 2022.

https://developer.nrel.gov/docs/solar/nsrdb/nsrdb-GOES-aggregated-v4-0-0-download/ vs https://developer.nrel.gov/docs/solar/nsrdb/psm3-2-2-download/.

I'm speculating, but if the NSRDB team is not planning to update PSM3, then that makes adding GOES V4 a higher priority. I've sent an email to that team and will report back if I hear anything.

[grantbuster]

I have stepped back from my role in the NSRDB project so I’m no longer a good resource for what’s going on with the latest and greatest… that being said, this has not been responded to in a while now so I’ll try to help.

I’m pretty sure that new years of data will be released under NSRDB v4, and v3 will continue to be available on S3 but will not be maintained.