We don't seem to clean up properly between loop iterations

[robredpath]

Recently, we had an issue downloading one file for the daily data test. Some information (eg file_type, min/max award dates) was duplicated down from the row above the row that had the error. Presumably this should be blank.

I've saved the data as a named version in the daily data test - https://docs.google.com/spreadsheets/d/1iRH0N07Fi-XM6HcZLSR688EiPA4EQGc5wSP1hJIx3L4/edit#gid=0 -

[robredpath]

See also https://opendataservices.plan.io/issues/21460

[robredpath]

@KDuerden this should fix the Lankelly/LandAid issue in the DDT. @mrshll1001 can you make sure that you update datagetter before the next GN load? Ping me if you'd like a hand with that :)

[KDuerden]

@robredpath thanks for the heads up. The changes seem to have knocked 360Insights offline, as it expects different behaviour (a file size value every time). @drkane is on holiday this week so we need to find a way to keep 360Insights up as much as possible until he's back. I'm rebuilding the registry now, having removed a few Lankelly/Landaid files to see if that gives a clear result.

[robredpath]

@kindly @michaelwood could we (temporarily or permanently) make this give a filesize of 0 to help get Insights working?

[robredpath]

Or, would the correct behaviour be to drop the entry if we can't get the file? @KDuerden ?

[KDuerden]

@robredpath I'm not qualified to say.

[drkane]

Omitting the record entirely is probably the safest quick fix (I'm not sure if there are further properties in the same template that would broke, so just adding a dummy file size isn't guaranteed to fix it).

Other option is just to revert to the previous state and then reapply this patch later on.

When I'm back behind a computer I can change insights to check for the right properties and we can revert to the current state.