Backfill national with metadata

[peterdudfield]

In trying to ingest into our data lake we want to set up a pipeline to pull historical forecasts for backfilling in cases the pipeline has failed and for training purposes. Should it be possible to run something like this?

params = {'include_metadata': True
'historic': True,
'start_datetime_utc': "2024-07-01 00:00:00",
'end_datetime_utc': "2024-07-02 00:00:00"}

url = https://api.quartz.solar/v0/solar/GB/national/forecast
r = requests.get(
url=url,
headers={"Authorization": "Bearer " + access_token}, params=params
)

Running this I get a 500 status (internal server error).

The endpoint works if I set include_metadata to False and specify a forecast_horizon_minutes so I have previously just iterated through forecast_horizons to backfill data for my personal consumption. However this doesn’t work with include_metadata = True (which we would like to keep)

[pwdemars]

Would it also be able to address this for GSP forecasts? E.g.

params = {'include_metadata': True,
           'start_datetime_utc': "2024-07-01 00:00:00",
          'end_datetime_utc': "2024-07-02 00:00:00"}
url = "https://api.quartz.solar/v0/solar/GB/gsp/forecast/all/"
r = requests.get(
    url=url,
    headers={"Authorization": "Bearer " + access_token},
    params=params
)

This gives 200 status code but empty data

[peterdudfield]

1. National: You could try something like https://api.quartz.solar/v0/solar/GB/national/forecast?include_metadata=true&start_datetime_utc=2024-07-01&end_datetime_utc=2024-07-02&creation_limit_utc=2024-07-01, so including a creation_limit_utc, but the does take quite long unfortunately. We have optimized our database to make the recent data very fast to access

2. For GSP: Similar as above creation_limit_utc. This however might take a long time.

I run things in swagger https://api.quartz.solar/swagger, and sometimes it takes longer there than it does in cmd line, or python

[peterdudfield]

I think in general if you are trying to backfill with forecasts then you should use creation_limit_utc otherwise you will just get the latest forecast, which will be more accurate than real time forecasts