As referenced in #54, different scripts use GOES data to calculate more advanced data products, but copies of these products are not kept in the original GOES Data directory. Instead, the current approach is to update the plots and web page frequently with independent five-minutely data fetches, and then a daily archiving script with customized text file delimiters, formats and naming for each type of data product.
Current file web fetch / read / write
-
5-minutely: plot_goes_data.py
- web fetch
- differential-protons-3-day.json
- integral-protons-3-day.json
- write
- goes_particles.png
- goes_protons.png
-
5-minutely: update_goes_html_page.py
- web fetch
- differential-protons-1-day.json
- integral-protons-1-day.json
- integral-electrons-1-day.json
- xray-flares-7-day.json
- edited_events.json
- write
- goes_pchan_p.html
- goes_part_p.html
- goes_xray_p.html
- Gp_pchan_5m.txt
- Gp_part_5min.txt
-
5-minutely: alert_hrc.py
- read Gp_pchan_5m.txt
- write
- hrc_proxy.csv
- hrc_proxy_viol.json
-
Daily: collect_goes_long.py
- web fetch differential-protons-7-day.json
- write goes_data_r.txt
This creates the user-end products for the GOES radiation webpages cleanly enough, but it fetches inefficiently and doesn't leave lower-end products available for use by other scripts, causing us to fetch them again frequently. Instead, we should fetch the lower level products and construct higher level complexity products (such as goes fluence used in the Alerts radiation summary page rad_summ.html) all in a singular goes data directory for use by all scripts, then create GOES radiation web products from that data set. Overall, this will result in fewer content requests to SWPC and a uniform resource for MTA.
Proposed
- 5-minutely: construct_goes_products.py
- fetch
- differential-protons-3-day.json
- integral-protons-3-day.json
- integral-electrons-3-day.json
- xray-flares-7-day.json
- edited_events.json (At time or writing, we only use this SWPC data product for GOES, but it contains information of all SWPC events. Option to move to a more general script rather than being GOES specific.)
- write
- goes_orbit_fluence.json (to replace the goes orbit fluence data product calculated in the Alerts script set)
- Gp_pchan_5m.txt (text containing information) (eventual deprecation with below)
- goes_protons_2h.csv (raw ascii table)
- Gp_part_5m.txt (text containing information) (eventual deprecation with below)
- goes_particles_2h.csv (raw ascii table)
- (possibly) hrc_proxy.csv
- 5-minutely: plot_goes_data.py
- write
- goes_particles.png
- goes_protons.png
- 5-minutely: update_goes_html_page.py
- write
- goes_pchan_p.html
- goes_part_p.html
- goes_xray_p.html
- 5-minutely: alert_hrc.py
collect_goes_long.py archiving can persist as normal with the eventual goal of transitioning any usage of archival GOES data (such as in the mta-reports/interruption script set for radiation interruptions)
As referenced in #54, different scripts use GOES data to calculate more advanced data products, but copies of these products are not kept in the original GOES Data directory. Instead, the current approach is to update the plots and web page frequently with independent five-minutely data fetches, and then a daily archiving script with customized text file delimiters, formats and naming for each type of data product.
Current file web fetch / read / write
5-minutely: plot_goes_data.py
5-minutely: update_goes_html_page.py
5-minutely: alert_hrc.py
Daily: collect_goes_long.py
This creates the user-end products for the GOES radiation webpages cleanly enough, but it fetches inefficiently and doesn't leave lower-end products available for use by other scripts, causing us to fetch them again frequently. Instead, we should fetch the lower level products and construct higher level complexity products (such as goes fluence used in the Alerts radiation summary page rad_summ.html) all in a singular goes data directory for use by all scripts, then create GOES radiation web products from that data set. Overall, this will result in fewer content requests to SWPC and a uniform resource for MTA.
Proposed
collect_goes_long.py archiving can persist as normal with the eventual goal of transitioning any usage of archival GOES data (such as in the mta-reports/interruption script set for radiation interruptions)