Merge remote-tracking branch 'refs/remotes/origin/develop'

Author: m-stclair

README.md

@@ -86,6 +86,7 @@ us first, so we can expect your contributions and make sure they will be within
 scope.
 
 If you need general support you can find us on [OpenPlanetary Slack](https://app.slack.com/client/T04CWPQL9/C04CWPQM5)
+(available to [OpenPlanetary members](https://www.openplanetary.org/join))
 or feel free to [email](mailto:[email protected]) the team.
 
 ---

docs/supported_datasets.md

@@ -86,11 +86,12 @@ kernels in combination with JPL Horizons).
   - M3 L0, L1B, and L2 images and ancillary files 
   (*note: L0 line prefix tables are not currently supported*)
 #### Clementine
-  - Map-projected Basemap, HiRes, NIR, and UVVIS mosaics
-    - These have been migrated to PDS4, pending archive approval.
+  - Imaging Bundle (PDS4 at IMG)
+    - EDRs (HiRes, LWIR, NIR, UVVIS and star trackers)
+    - Map-projected Basemap, HiRes, NIR, and UVVIS mosaics
   - Gravity and topography derived products
   - LWIR RDRs
-  - RSS bistatic radar RDRs
+  - RSS bistatic radar RDRs (except a subset of image products; see 'Known Unsupported' below)
   - LIDAR data *note: This is a saved PDS data set, not a regular PDS archive, 
     but it can be opened with `pdr`*
 #### Comet D/Shoemaker-Levy 9/Jupiter Impact Observing Campaign
@@ -109,7 +110,7 @@ kernels in combination with JPL Horizons).
   - Galileo Orbiter
     - NIMS tables
     - PPR EDRs
-    - SSI REDRs (except line prefix tables)
+    - SSI REDRs
     - UVS EDRs and RDRs
 #### Dawn
   - Framing Camera: EDRs, RDRs, mosaics, shape models, and most calibration images
@@ -129,6 +130,7 @@ kernels in combination with JPL Horizons).
 #### Deep Space 1
   - IDS RDRs
   - MICAS DEM, images, and matrices
+  - MICAS EDRs and RDRs, and most PEPE EDRs (*note: these datasets are 'safed', use with caution*)
 #### EPOXI
   - HRI-IR raw and calibrated spectra
   - HRI-VIS raw and calibrated images
@@ -151,7 +153,7 @@ kernels in combination with JPL Horizons).
   - SSD: derived electron flux data
   - SSI: 
     - Calibration images
-    - REDRs (except line prefix tables)
+    - REDRs
     - Ida/Gaspra specific data products at SBN
   - UV: UVS and EUV *note: They open correctly from their PDS4 labels, but the 
     PDS3 labels are currently unsupported.*
@@ -230,9 +232,8 @@ kernels in combination with JPL Horizons).
       of the file in the incorrect letter case)
     - L2 and L3 GDR images/backplanes
     - L4 tables
-  - LAMP 
-    - EDR and GDR
-    - RDR (except acquisition list and pixel list tables)
+  - LAMP EDR, RDR, and GDR (*note: The RDR calibrated histogram images are 4-D arrays,
+    so `pdr`'s convenience display functions, e.g. `Data.show()`, won't work on them.*)
   - LEND EDR and RDR
   - LOLA RDR (*note: other LOLA datasets have been migrated to PDS4 and are 
     notionally supported*)
@@ -255,9 +256,6 @@ kernels in combination with JPL Horizons).
 #### Magellan
   - FMAPs
   - Stereo-Derived Topography
-  - C-BIDR images *note: `pdr` reads these correctly according to the 
-    specifications in the labels, but there are unaccounted-for offsets in the
-    images.*
   - F-MIDR and C-MIDR
   - GxDR: GTDR, GEDR, GSDR, and GREDR
   - GVDR
@@ -585,6 +583,8 @@ kernels in combination with JPL Horizons).
     - Pancam color mosaics
 #### MSL
   - CheMin Film EDRs (have efm in filename)
+#### New Horizons
+  - Pluto encounter derived data: ALICE stellar occultation data
 #### Phoenix
   - TEGA LED EDRs
 #### Rosetta
@@ -599,27 +599,26 @@ kernels in combination with JPL Horizons).
 ## Known Unsupported Datasets
 #### Cassini
   - CIRS variable-length '.VAR' tables (support not planned)
-  - ISS prefix tables (support planned)
+  - ISS volume coiss_0011 v4.3 at the IMG JPL and ATM nodes (support not 
+    planned; v3.0 at IMG USGS is supported)
   - MAG error count housekeeping data
   - RPWS telemetry data
   - RSS ancillary products: TDF, TLM, PD1, PD2, TNF, EOP, ION, and TRO (support not planned)
   - VIMS EDR cubes (PDS3 labels; support not planned--open with PDS4 labels)
   - Huygens Probe DISR IR tables (support planned; low priority )
 #### Clementine
-  - Imaging EDRs (basemap, HiRes, NIR, and UVVIS) (support not planned)
-  - LWIR EDRs (support not planned)
   - RSS EDRs (support not planned)
-#### Comet D/Shoemaker-Levy 9/Jupiter Impact Observing Campaign
-  - Galileo SSI REDR line-prefix tables (support planned)
+  - RSS bistatic radar RDR images in the CL_2108 volume's 'sort' directory (support planned)
 #### Dawn
   - Framing camera: a handful of calibration images
 #### Deep Impact
   - Movies of 9P/Tempel 1 approach and encounter (support not planned; not an 
     archive-compliant format)
+#### Deep Space 1
+  - PEPE EDR binary housekeeping files (support not planned; 'safed' dataset)
 #### Galileo
   - MAG summary tables (other tables supported, support not planned)
   - NIMS Jupiter EDRs and cubes (support planned)
-  - SSI REDR line-prefix tables (support planned)
   - PLS EDRs (support not planned)
   - PWS EDRs, binary REDRs, and REFDRs (support planned)
 #### Giotto
@@ -640,7 +639,6 @@ kernels in combination with JPL Horizons).
   - IRSN Halley v1.0 spectroscopy and filter response curve tables (support planned)
   - LSPN compressed images in the Halley V1.0 and Giacobini-Zinner datasets (support not planned)
   - SSN 2D spectral products with SPECTRAL_IMAGE_QUBE pointers (support planned)
-  - SSN 2D spectral products with typos in their filenames (support not planned)
   - RSN UV visibility products (support planned)
 #### Juno
   - JADE Ion sensor housekeeping data prior to arrival at Jupiter (support not planned)
@@ -652,11 +650,12 @@ kernels in combination with JPL Horizons).
   - Level 0: sun pulse and merged telemetry (support planned; low priority)
 #### LRO
   - CRaTER EDR primary science data (support not planned)
-  - LAMP RDR acquisition list and pixel list tables (support planned)
-  - LAMP RDR calibrated histogram images read correctly, but they crash data.show() (support planned)
   - Mini-RF raw SAR products (support not planned)
 #### Magellan
-  - C-BIDR ancillary tables (support not planned)
+  - C-BIDR (support not planned) 
+    - *note: `pdr` reads the image products correctly according to the specifications
+       in the labels, but there are offsets in the images due to underdefined header 
+       rows at the start of each data record.*
   - F-BIDR (support not planned)
   - ALT-EDR (support not planned)
   - ARCDR (support planned)
@@ -697,8 +696,8 @@ kernels in combination with JPL Horizons).
     PDS3-style files exists in the archive.*)
 #### MRO
   - RSS .tnf (support not planned)
+  - RSS products in the 'ancillary' directory (support not planned)
   - MCS DDRs (support not planned)
-  - HiRISE EDR line prefix tables (support planned)
 #### MSL
   - Malin Space Science Systems (MSSS) Camera EDRs: "Raw" (EDR) data from the
     Mars Science Laboratory's MSSS-produced cameras (Mastcam, MAHLI, and
@@ -711,8 +710,6 @@ kernels in combination with JPL Horizons).
     compressed in this way.
   - ChemCam LIBS EDR tables (support not planned)
   - RAD EDRs and RDRs (support planned)
-#### New Horizons
-  - Pluto encounter derived data: ALICE stellar occultation data
 #### Pioneer Venus Orbiter
   - ORPA raw data (support not planned)
   - ORSE ODRs (support not planned)

docs/version_history.md

@@ -1,4 +1,48 @@
 # Version History
+## [1.4.0] - 2025-03-13
+### Added
+
+#### Features
+- Support for line prefix tables
+- Useful messages for missing files in debug mode
+
+#### Dataset Support
+- additional LRO LAMP RDR table/image pointers
+- raw images of 9P/TEMPEL 1 from the Keck Observatory
+- Deep Space 1 safed MICAS and PEPE datasets
+- Lunar Prospector ancillary ascii tables
+- VG1/VG2 safed MAG solar wind and POS cruise phase tables
+- safed Stardust datasets from the mission's early cruise phase
+- New Horizons Pluto encounter ALICE stellar occultation data (updated from
+  'known unsupported' to 'notionally supported')
+- MGS RSS USO tables
+- PVO ORPA and ORSE misc ancillary tables
+- JUNO Waves EDR housekeeping tables
+- Earth-based ring occultation (EBROCC_0001) ancillary data
+- Voyager ring profiles ancillary data
+- ground-based MAHL images
+- CRISM ancillary tables
+- Clementine BSR RDR sorted/aggregated tables (CL_2108)
+- Line prefix tables
+  - Galileo SSI REDR
+  - Cassini ISS
+  - HiRISE EDR
+
+### Changed
+
+
+### Fixed
+- numpy >= 2.0 no longer automatically upcasts integer dtypes in cases like
+  `np.int16(0) + 1000000`. In some unusual cases, this can cause an 
+  `OverflowError` when applying scaling factors / offsets (MOLA MEGDRs formed
+  the motivating example). `Data.get_scaled()` will now catch these errors and
+  automatically typecast arrays in response.
+- glitch reading headers from PDS4-labeled FITS files without unique local 
+  references for said headers
+
+### Removed
+
+
 
 ## [1.3.0] - 2024-10-18
 ### Added

pdr/__init__.py

@@ -8,7 +8,7 @@
 if TYPE_CHECKING:
     from pathlib import Path
 
-__version__ = "1.3.0"
+__version__ = "1.4.0"
 
 pkg_dir = _osp.abspath(_osp.dirname(__file__))
 

pdr/_scaling.py

@@ -1,5 +1,8 @@
-from numbers import Number
-from typing import Optional, Sequence
+import warnings
+from functools import wraps
+from itertools import product
+from numbers import Integral, Number, Real
+from typing import Optional, Sequence, Union
 
 import numpy as np
 
@@ -59,6 +62,73 @@ def mask_specials(obj, specials):
     return obj
 
 
+def fit_to_scale(
+    arr: np.ndarray,
+    scale: Union[Integral, Real],
+    offset: Union[Integral, Real]
+) -> np.ndarray:
+    """
+    Return a version of `arr` cast to the minimum dtype that will hold its
+    range of values after multiplying by `offset` and adding `scale`.
+
+    Supports:
+
+    float32, float64, uint8, int8, uint16, int16, uint32, int32, uint64, int64.
+    """
+    if arr.dtype.char not in 'bBhHiIlLqQnNpPf':
+        raise TypeError(f"This function does not support {arr.dtype.name}")
+    if arr.dtype.char in 'fd' or int(scale + offset) != scale + offset:
+        bases, widths, infofunc = ('f',), (4, 8), np.finfo
+    else:
+        bases, widths, infofunc = ('u', 'i'), (1, 2, 4, 8), np.iinfo
+    amin, amax = map(int, (arr.min(), arr.max()))
+    smin, smax = amin * scale + offset, amax * scale + offset
+    for base, width in product(bases, widths):
+        candidate = np.dtype(f'{base}{width}')
+        cinfo = infofunc(candidate)
+        if smin >= cinfo.min and smax <= cinfo.max:
+            return arr.astype(candidate)
+    raise TypeError("Unable to find a suitable data type for scaling.")
+
+
+def overflow_wrap(array_func):
+    @wraps(array_func)
+    def with_upcasting(arr, scale, offset, *args, **kwargs):
+        with warnings.catch_warnings():
+            warnings.filterwarnings("error", message=".*overflow enc.*")
+            try:
+                return array_func(arr, scale, offset, *args, **kwargs)
+            except (OverflowError, RuntimeWarning):
+                arr = fit_to_scale(arr, scale, offset)
+                return array_func(arr, scale, offset, *args, **kwargs)
+
+    return with_upcasting
+
+
+def _copy_scale(obj, offset, scale):
+    try:
+        # TODO: we should also be doing this per-plane scaling in inplace case
+        if len(obj) == len(scale) == len(offset) > 1:
+            planes = [
+                obj[ix] * scale[ix] + offset[ix] for ix in range(len(scale))
+            ]
+            stacked = np.rollaxis(np.ma.dstack(planes), 2)
+            return stacked
+    except TypeError:
+        pass  # len() is not usable on a float object
+    return obj * scale + offset
+
+
+def _inplace_scale(obj, offset, scale):
+    if len(obj) == len(scale) == len(offset) > 1:
+        for ix, _ in enumerate(scale):
+            obj[ix] = obj[ix] * scale[ix] + offset[ix]
+    else:
+        obj *= scale
+        obj += offset
+    return obj
+
+
 def scale_array(
     meta: PDRLike,
     obj: np.ndarray,
@@ -86,28 +156,13 @@ def scale_array(
     # we're casting to float, we can't
     # TODO: detect rollover cases, etc.
     if inplace is True and not casting_to_float(obj, scale, offset):
-        if len(obj) == len(scale) == len(offset) > 1:
-            for ix, _ in enumerate(scale):
-                obj[ix] = obj[ix] * scale[ix] + offset[ix]
-        else:
-            obj *= scale
-            obj += offset
-        return obj
+        return overflow_wrap(_inplace_scale)(obj, offset, scale)
     # if we're casting to float, permit specification of dtype
     # prior to operation (float64 is numpy's default and often excessive)
     if casting_to_float(obj, scale, offset):
         if float_dtype is not None:
             obj = obj.astype(float_dtype)
-    try:
-        if len(obj) == len(scale) == len(offset) > 1:
-            planes = [
-                obj[ix] * scale[ix] + offset[ix] for ix in range(len(scale))
-            ]
-            stacked = np.rollaxis(np.ma.dstack(planes), 2)
-            return stacked
-    except TypeError:
-        pass  # len() is not usable on a float object
-    return obj * scale + offset
+    return overflow_wrap(_copy_scale)(obj, offset, scale)
 
 
 # TODO: shake this out much more vigorously

pdr/formats/__init__.py

@@ -18,6 +18,7 @@
 import pdr.formats.ihw as ihw
 import pdr.formats.iue as iue
 import pdr.formats.juno as juno
+import pdr.formats.lp as lp
 import pdr.formats.lroc as lroc
 import pdr.formats.lro as lro
 import pdr.formats.mariner as mariner
@@ -31,6 +32,7 @@
 import pdr.formats.msl_ccam as msl_ccam
 import pdr.formats.msl_places as msl_places
 import pdr.formats.msl_rems as msl_rems
+import pdr.formats.msx as msx
 import pdr.formats.nh as nh
 import pdr.formats.odyssey as odyssey
 import pdr.formats.phoenix as phoenix