Manage Harvester errors from Commerce

[neilmb]

On H2.0, harvest sources from Commerce and Education are failing every time.

We should investigate and figure out if it's a problem with the source file or our code.

How to reproduce

Commerce: https://harvest.data.gov/harvest_source/7660cf65-6dbc-49a7-9ebb-6c9736fcb006

Education: https://harvest.data.gov/harvest_source/5ac48d69-f461-4ead-9dc2-b360eaaea04c

Expected behavior

Harvests should be succeeding

Actual behavior

These are showing up as failed jobs

Sketch

Try to load those JSON files, check the Harvester logs to see what the specific errors are, get @tdlowden to reach out to the agencies if necessary.

[tdlowden]

Both of these jobs run and finish on CKAN harvester, so I am inclined to believe it is not a source issue. Warrants further investigation on our end before we reach out to them.

[tdlowden]

Hey @rshewitt what did you find last week? Do you want to finish this ticket out (i.e. it can be completed todayish) or should we pass to @cmhedrickREI on O&M this week?

[rshewitt]

@tdlowden was heads down with nginx proxy on friday. I'll finish this ticket out today!

[rshewitt]

both current (ckan) catalog and h20 are blocked by these sources. there's something about the request these sources don't like. i can pull both sources locally using our user-agent.

[SueValente]

Sue to reach out and see if can find someone