Skip to content

I/O improvements #42

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 32 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
d3931c7
add lma writer
wx4stg Dec 28, 2023
aacf855
use xarray sizes instead of dims (warning silence)
wx4stg Jul 1, 2024
f5bfa96
add original_filename, fix station network location
wx4stg Jul 1, 2024
fe657e4
properly support stations appearing/disappearing/moving
wx4stg Jul 1, 2024
043acd8
preserve event_id == number_of_events (deeplycloudy/xlma-python#43)
wx4stg Jul 1, 2024
25385bb
add station_active flag
wx4stg Jul 1, 2024
1477a42
part 2 to 25385bb, actually commit the cf_netcdf file this time
wx4stg Jul 1, 2024
fb0b3ab
fill in known values when stations are offline
wx4stg Jul 1, 2024
b5555e9
only reduce number_of_files if it exists
wx4stg Jul 5, 2024
fb1a2ca
Merge branch 'master' of https://github.com/deeplycloudy/xlma-python …
wx4stg Jul 18, 2024
7031649
Merge branch 'master' of https://github.com/deeplycloudy/xlma-python …
wx4stg Jul 19, 2024
fc4419a
Merge branch 'writer' of https://github.com/wx4stg/xlma-python into i…
wx4stg Jul 23, 2024
d458c22
identify unique network configurations instead of individual files
wx4stg Jul 24, 2024
fe6225c
start time, end time, and analyzed sec of dataset, warn if mismatch
wx4stg Jul 24, 2024
27b3c10
lma dat header (mostly)
wx4stg Jul 24, 2024
3a4c1b3
make station event fraction a percentage as described by metadata
wx4stg Jul 28, 2024
32e90d7
track data coverage across network configs and gaps in coverage
wx4stg Jul 28, 2024
caaf607
use template cf dataset to restore metadata when combining #11
wx4stg Jul 28, 2024
81502fd
fix cf comparison
wx4stg Jul 28, 2024
9491cc2
that's not an xarray yet
wx4stg Jul 28, 2024
dffb0fb
recalculate station mask after combine
wx4stg Jul 28, 2024
8ad6f09
try to make station IDs unique, fix writing, dont recalc center unles…
wx4stg Jul 28, 2024
1e739f6
reading multifile test more comprehensive
wx4stg Jul 28, 2024
e72b3b4
vectorize station mask calculation
wx4stg Jul 28, 2024
3e76283
read station name, board revision, delay, and rec_ch
wx4stg Jul 29, 2024
0cb0d9c
this doesn't appear to be used for anything...
wx4stg Jul 29, 2024
17e8910
make fill values more obvious
wx4stg Jul 29, 2024
55c29e1
remove newlines from attrs
wx4stg Jul 29, 2024
0cc3204
add new properties
wx4stg Jul 29, 2024
0defdea
better handle combining new attrs
wx4stg Jul 29, 2024
d2b28e3
dat.gz write support!
wx4stg Jul 29, 2024
6c1311e
Merge branch 'master' of https://github.com/deeplycloudy/xlma-python …
wx4stg Aug 9, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 26 additions & 1 deletion pyxlma/lmalib/io/cf_netcdf.py
Original file line number Diff line number Diff line change
Expand Up @@ -301,6 +301,10 @@ def new_template_dataset():
'attrs': {'_FillValue': ' '*32, 'long_name': 'LMA network'},
'dtype': '|S32',
},
'station_name': {'dims': ('number_of_stations',),# 'string32'),
'attrs': {'_FillValue': ' '*32, 'long_name': 'Name of receiving station'},
'dtype': '|S32',
},
'station_latitude': {'dims': ('number_of_stations',),
'attrs': {'_FillValue': np.nan,
'valid_range': [-90.0, 90.0],
Expand All @@ -323,6 +327,22 @@ def new_template_dataset():
'positive': 'up'},
'dtype': 'float32',
},
'station_delay': {'dims': ('number_of_stations',),
'attrs': {'_FillValue': np.nan,
'long_name': "Configured delay time for a signal to travel from the antenna to the receiver",
'units': 'nanoseconds'},
'dtype': 'float32',
},
'station_board_revision': {'dims': ('number_of_stations',),
'attrs': {'_FillValue': 0,
'long_name': "Station board revision number"},
'dtype': 'uint8',
},
'station_receive_channels': {'dims': ('number_of_stations',),
'attrs': {'_FillValue': 0,
'long_name': "Number of channels on the station's receiver"},
'dtype': 'uint8',
},
'station_event_fraction': {'dims': ('number_of_stations',),
'attrs': {'_FillValue': np.nan,
'long_name': "Fraction of events to which this station contributed",
Expand All @@ -335,6 +355,11 @@ def new_template_dataset():
'long_name': "<P/P_m>"},
'dtype': 'float32',
},
'station_active': {'dims': ('number_of_stations',),
'attrs': {'_FillValue': np.nan,
'long_name': "Station active flag"},
'dtype': '|S2',
},

}}
return __template_dataset
Expand Down Expand Up @@ -461,7 +486,7 @@ def compare_attributes(ds):
""" Compare the attributes of all data variables in ds to the CF spec
and print a report of any differences.
"""
dst = __template_dataset['data_vars']
dst = new_template_dataset()['data_vars']
dsd = ds.to_dict()['data_vars']
for k in dsd.keys():
if dst[k]['attrs'] != dsd[k]['attrs']:
Expand Down
Loading
Loading