Improve `create_sunrun` flexibility

[joshua-laughner]

Historically, sites that need to correct values in their .gop sunrun files beyond what the xx_sunrun.dat file supports have had to modify the .gop file manually, modify it with an external script, or create a custom version of create_sunrun. The downside to the first two is that it is difficult to reproduce those changes at another location (to run GGG in these cases, we either need the .gop created by the site or the script they used to modify it - which may rely on external dependencies or data) and the downside to the second is that users have to customize Fortran code.

In the future, I think we can encapsulate a lot of the needed flexibility by extending the xx_sunrun.dat file into a fully fledged TOML file. The existing xx_sunrun.dat information can go in one table:

[static]
# 1 = MkIV, 2 = TCCON, 3 = other
instrument = 2

# 1 = moon, 2 = sun
object = 2

# Site coordinates, south and west are negative. Altitude is in kilometers above sea level
obs_lat = 45.9448
obs_lon = -90.2732
obs_alt = 0.476

# Constant pressure correction. Use only if the ENTIRE record has an offset,
# otherwise, use the time period/spectrum specific tables below. Value must be in hPa
pout_corr = 0.0

frequency_shift_factor = 0.99999999
laser_frequency = 15798.0138
tracker_wavenumber = 9900.0
airmass_independent_path_length = 0.002
telescope_magnification = 1.0

[[static.detectors]]
label = "a"
label_index_start = 16
label_index_end = 16
start_freq = 4000
end_freq = 11000

[[static.detectors]]
label = "b"
label_index_start = 16
label_index_end = 16
start_freq = 11000
end_freq = 15000

[[static.detectors]]
label = "c"
label_index_start = 16
label_index_end = 16
start_freq = 1800
end_freq = 6000

[static.null_values]
tins = 999.0
hins = 999.0
pins = 999.0
tout = 999.0
hout = 999.0
pout = 999.0
fvsi = 999.0
wspd = 999.0
wdir = 999.0

Then we can have two other tables. One defines changes by time range, and possibly lat/lon range:

[[modify_range]]
time_range = ["2026-05-28", "2026-06-01"]

[[modify_range]]
time_range = ["2020-01-01", "2020-02-01"]
lat_range = [-46.0, -45.0]
lon_range = [-90.0, -89.0]

The other would define specific spectra:

[[modify_spectra]]
spectrum = "pa20040721saaaa?.043"

In this example, we show the use of the ? wildcard to match any detector.

For both of these, they would contain a change subtable that defines which columns to change. The keys under change would map to any sunrun column. These would be able to define either a constant value to insert or a linear correction:

[[modify_range]]
time_range = ["2026-05-28", "2026-06-01"]
[modify_range.change]
# Linear correction, x_corr = multiplier * x_orig + offset
pout = {multiplier = 0.95, offset = 5.0}

[[modify_spectra]]
spectrum = "pa20040721saaaa?.043"
[modify_spectra.change]
tout = { value = 25.0 }

[joshua-laughner]

As far as implementation, this would be straightforward in Rust - it has great capabilities for reading TOML files. However, there are options like toml-f if we want to keep the implementation in Fortran.

Currently, create_sunrun decides which XX_sunrun.dat file to read based on the first two characters of the input spectrum list (e.g., pa20250101_20250301.gnd). If we stuck with Fortran, we'd probably keep that convention for simplicity. In Rust, it would be easier to read each XX_sunrun.dat the first time a spectrum starting with XX is found in the list and cache that configuration in memory.

I did consider allowing the user to hook into this with either an external script or FFI through e.g., Python. But the problem with these is that we're back to needing those scripts to work on another system if we want to be able to reproduce the sunrun file. I much prefer the approach where users can generate the TOML offline (even with an automatic script) and add it to a single file.

It might be worth allowing users to create multiple TOML files and point to separate files within the main one, something like:

extra_files = ["pa_2025_sunrun.toml", "/home/ggg/oc/pressure_corrections.toml"]

Including subfiles isn't part of the TOML spec, but I've made it work, and this would help keep the main file from becoming enormous if people need to do a lot of spectrum-specific corrections. My thought is relative paths will be assumed to be relative to $GGGPATH/tccon.

[mbuschmann]

I like that idea, it is practical for changes that need to be done similarly for a bunch of spectra. However, I'm not sure about the use-case where you would have to change values for each spectrum or line specifically.

I'm thinking of including third party surface pressures in case of i2s not including them in pout already, or time corrections specific to certain spectra only.

In case you thought about repeating the block

[[modify_spectra]]
spectrum = "pa20040721saaaa?.043"
[modify_spectra.change]
tout = { value = 25.0 }

for every spectrum that needs to be changed, that seems like a lot of overhead.

Additionally it still leaves ambiguity in which pressure (for example) value to use as PIs need to come up with their own mapping. It might be best to recommend pressure measurements be taken at fixed intervals (1/60Hz or such) , agree on a interpolation scheme and input format and take this away from users - in the default case only of course.