Skip to content

Suggestion of replacing np.float128 with np.float64 in pyirf to support IRF interpolation on macOS #306

New issue
New issue
Open
#307
Open
Suggestion of replacing np.float128 with np.float64 in pyirf to support IRF interpolation on macOS#306
#307
Assignees
maxnoe
@sogoitoh

Description

@sogoitoh
sogoitoh
opened on Aug 8, 2025

[Moved from] cta-observatory/cta-lstchain#1382

IRF interpolation fails due to numpy.float128, which is not supported on macOS
When I run lstchain_create_dl3_file with interpolation, I get the following error:
Caught unexpected exception: module 'numpuy' has no attribute 'float 128'

Environment

  • lstchain version 0.10.20.dev0
    I installed lstchain on my computer following the instructions provided by the developer on github.
  • https://github.com/cta-observatory/cta-lstchain
  • Python version 3.13
  • Numpy version 2.2.5
  • MacBook Air M4, 2025, mac OS 15.

I downgraded NumPy to a version below 1.26.4 by editing the environment file, but the same error occurred.
I also downgraded Python to 3.12 by editing the environment.yml file, but the error still persisted.

Reproduce

 lstchain_create_dl3_file \
 --input-dl2 /Users/sito/cta/DL2/dl2_LST-1.Run06894.h5 \
 --input-irf-path /Users/sito/cta/data/DL2_to_DL3/IRF/AllSky/20250212_v0.10.17_allsky_interp_dl2_irfs_nsb_0.22/TestingDataset/Gamma/dec_2276 \
 --irf-file-pattern 'node_theta_*_az_*/irf_20240918_v0.10.12_allsky_nsb_tuning_0.22_Gamma_dec_2276_node_theta_*_az_*_.fits.gz' \
 --output-dl3-path /Users/sito/cta/DL3 \
 --source-name Crab \
 --source-ra '83.6324 deg' \
 --source-dec '22.0174 deg' \
 --config 
```/Users/sito/cta/data/DL2_to_DL3/IRF/AllSky/20250212_v0.10.17_allsky_interp_dl2_irfs_nsb_0.22/TestingDataset/Gamma/dec_2276/lstchain_config_nsb0.22.json \
 --debug –overwrite

Result

/opt/anaconda3/envs/baka/lib/python3.13/site-packages/ctaplot/io/dataset.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
 import pkg_resources
2025-07-20 15:15:41,241 INFO [lstchain.DataReductionFITSWriter] (tool.initialize): Loading config from '[PosixPath('/Users/sito/cta/data/DL2_to_DL3/IRF/AllSky/20250212_v0.10.17_allsky_interp_dl2_irfs_nsb_0.22/TestingDataset/Gamma/dec_2276/lstchain_config_nsb0.22.json')]'
2025-07-20 15:15:41,241 DEBUG [lstchain.DataReductionFITSWriter] (application._load_config_files): Looking for /Users/sito/cta/data/DL2_to_DL3/IRF/AllSky/20250212_v0.10.17_allsky_interp_dl2_irfs_nsb_0.22/TestingDataset/Gamma/dec_2276/lstchain_config_nsb0.22 in /Users/sito
2025-07-20 15:15:41,242 DEBUG [lstchain.DataReductionFITSWriter] (application._load_config_files): Loaded config file: /Users/sito/cta/data/DL2_to_DL3/IRF/AllSky/20250212_v0.10.17_allsky_interp_dl2_irfs_nsb_0.22/TestingDataset/Gamma/dec_2276/lstchain_config_nsb0.22.json
2025-07-20 15:15:41,243 INFO [lstchain.DataReductionFITSWriter] (tool.initialize): ctapipe version 0.25.1
The metadata are comparable
The other parameter axes data are comparable
2025-07-20 15:15:42,595 DEBUG [lstchain.DataReductionFITSWriter] (lstchain_create_dl3_file.setup): Output DL3 file: /Users/sito/cta/DL3/dl3_LST-1.Run06894.fits
2025-07-20 15:15:42,595 DEBUG [lstchain.DataReductionFITSWriter] (tool.run): CONFIG: {'DataReductionFITSWriter': {'config_files': [PosixPath('/Users/sito/cta/data/DL2_to_DL3/IRF/AllSky/20250212_v0.10.17_allsky_interp_dl2_irfs_nsb_0.22/TestingDataset/Gamma/dec_2276/lstchain_config_nsb0.22.json')], 'gzip': False, 'input_dl2': PosixPath('/Users/sito/cta/DL2/dl2_LST-1.Run06894.h5'), 'input_irf_path': PosixPath('/Users/sito/cta/data/DL2_to_DL3/IRF/AllSky/20250212_v0.10.17_allsky_interp_dl2_irfs_nsb_0.22/TestingDataset/Gamma/dec_2276'), 'interp_method': 'linear', 'irf_file_pattern': 'node_theta_*_az_*/irf_20240918_v0.10.12_allsky_nsb_tuning_0.22_Gamma_dec_2276_node_theta_*_az_*_.fits.gz', 'keep_duplicated_events': False, 'log_config': {}, 'log_datefmt': '%Y-%m-%d %H:%M:%S', 'log_file': None, 'log_file_level': 'INFO', 'log_format': '[%(name)s]%(highlevel)s %(message)s', 'log_level': 10, 'logging_config': {}, 'output_dl3_path': PosixPath('/Users/sito/cta/DL3'), 'overwrite': True, 'provenance_log': PosixPath('/Users/sito/cta/DL3/DataReductionFITSWriter.provenance.log'), 'quiet': False, 'show_config': False, 'show_config_json': False, 'source_dec': '22.0174 deg', 'source_dep': False, 'source_name': 'Crab', 'source_ra': '83.6324 deg', 'use_nearest_irf_node': False, 'EventSelector': {'filters': {'intensity': [50, inf], 'width': [0, inf], 'length': [0, inf], 'r': [0, 1], 'wl': [0.01, 1], 'leakage_intensity_width_2': [0, 1], 'event_type': [32, 32]}, 'finite_params': ['intensity', 'length', 'width']}, 'DL3Cuts': {'allowed_tels': [1], 'alpha_containment': 0.7, 'fill_alpha_cut': 20.0, 'fill_theta_cut': 0.32, 'gh_efficiency': 0.7, 'global_alpha_cut': 10.0, 'global_gh_cut': 0.7, 'global_theta_cut': 0.2, 'max_alpha_cut': 20.0, 'max_gh_cut': 0.98, 'max_theta_cut': 0.32, 'min_alpha_cut': 1.0, 'min_event_p_en_bin': 100.0, 'min_gh_cut': 0.1, 'min_theta_cut': 0.1, 'theta_containment': 0.7}}}
/opt/anaconda3/envs/baka/lib/python3.13/site-packages/tables/group.py:1220: UserWarning: problems loading leaf ``/provenance/dl1_to_dl2``:: variable length strings are not supported yetThe leaf will become an ``UnImplemented`` node.
 warnings.warn(
2025-07-20 15:15:50,118 INFO [lstchain.DataReductionFITSWriter] (lstchain_create_dl3_file.interp_irfs): Paths of IRFs used for interpolation: [PosixPath('/Users/sito/cta/data/DL2_to_DL3/IRF/AllSky/20250212_v0.10.17_allsky_interp_dl2_irfs_nsb_0.22/TestingDataset/Gamma/dec_2276/node_theta_37.814_az_270_/irf_20240918_v0.10.12_allsky_nsb_tuning_0.22_Gamma_dec_2276_node_theta_37.814_az_270_.fits.gz'), PosixPath('/Users/sito/cta/data/DL2_to_DL3/IRF/AllSky/20250212_v0.10.17_allsky_interp_dl2_irfs_nsb_0.22/TestingDataset/Gamma/dec_2276/node_theta_37.814_az_90_/irf_20240918_v0.10.12_allsky_nsb_tuning_0.22_Gamma_dec_2276_node_theta_37.814_az_90_.fits.gz'), PosixPath('/Users/sito/cta/data/DL2_to_DL3/IRF/AllSky/20250212_v0.10.17_allsky_interp_dl2_irfs_nsb_0.22/TestingDataset/Gamma/dec_2276/node_theta_23.630_az_259.265_/irf_20240918_v0.10.12_allsky_nsb_tuning_0.22_Gamma_dec_2276_node_theta_23.630_az_259.265_.fits.gz')]
2025-07-20 15:15:50,131 ERROR [lstchain.DataReductionFITSWriter] (tool.run): Caught unexpected exception: module 'numpy' has no attribute 'float128'
Traceback (most recent call last):
 File "/opt/anaconda3/envs/baka/lib/python3.13/site-packages/ctapipe/core/tool.py", line 445, in run
  self.start()
  ~~~~~~~~~~^^
 File "/Users/sito/cta/cta-lstchain/lstchain/tools/lstchain_create_dl3_file.py", line 454, in start
  self.interp_irfs()
  ~~~~~~~~~~~~~~~~^^
 File "/Users/sito/cta/cta-lstchain/lstchain/tools/lstchain_create_dl3_file.py", line 322, in interp_irfs
  self.irf_final_hdu = interpolate_irf(
             ~~~~~~~~~~~~~~~^
    self.irf_list, self.data_params, self.interp_method
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  )
  ^
 File "/Users/sito/cta/cta-lstchain/lstchain/high_level/interpolate.py", line 497, in interpolate_irf
  aeff_estimator = EffectiveAreaEstimator(
    grid_points=irf_pars_sel,
    effective_area=irf_list,
    interpolator_kwargs={"method": interp_method},
  )
 File "/opt/anaconda3/envs/baka/lib/python3.13/site-packages/astropy/units/decorators.py", line 313, in wrapper
  return_ = wrapped_function(*func_args, **func_kwargs)
 File "/opt/anaconda3/envs/baka/lib/python3.13/site-packages/pyirf/interpolation/component_estimators.py", line 439, in __init__
  super().__init__(
  ~~~~~~~~~~~~~~~~^
    grid_points=grid_points,
    ^^^^^^^^^^^^^^^^^^^^^^^^
  ...<4 lines>...
    extrapolator_kwargs=extrapolator_kwargs,
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  )
  ^
 File "/opt/anaconda3/envs/baka/lib/python3.13/site-packages/pyirf/interpolation/component_estimators.py", line 331, in __init__
  super().__init__(
  ~~~~~~~~~~~~~~~~^
    grid_points,
    ^^^^^^^^^^^^
  )
  ^
 File "/opt/anaconda3/envs/baka/lib/python3.13/site-packages/pyirf/interpolation/component_estimators.py", line 64, in __init__
  if not np.can_cast(grid_points.dtype, np.float128):
                     ^^^^^^^^^^^
 File "/opt/anaconda3/envs/baka/lib/python3.13/site-packages/numpy/__init__.py", line 414, in __getattr__
  raise AttributeError("module {!r} has no attribute "
             "{!r}".format(__name__, attr))
AttributeError: module 'numpy' has no attribute 'float128'. Did you mean: 'float16'?
2025-07-20 15:15:50,136 INFO [lstchain.DataReductionFITSWriter] (tool.write_provenance): Output:
2025-07-20 15:15:50,136 DEBUG [lstchain.DataReductionFITSWriter] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /Users/sito/cta/DL3/DataReductionFITSWriter.provenance.log'
2025-07-20 15:15:50,137 INFO [lstchain.DataReductionFITSWriter] (tool.run): Finished DataReductionFITSWriter
2025-07-20 15:15:50,137 DEBUG [lstchain.DataReductionFITSWriter] (application.exit): Exiting application: DataReductionFITSWriter

Expected behavior

The IRF interpolation should work.
Actual behavior
The process crashes due to an attempt to use numpy.float 128, which was not supported on Mac. As a result, IRF interpolation fails, and DL3 creation cannot proceed.
The following line causes this error

pyirf/pyirf/interpolation/component_estimators.py

Line 64 in 886c942

if not np.can_cast(grid_points.dtype, np.float128):

where, it is unnecessarily checked if grid_points can be casted to np.float128, resulting in TypeError on macOS.

However, using np.float128 is unnecessary in this context—np.float64 or float should be sufficient because scipy.spatial.Delaunay does not require np.float128.
https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.Delaunay.html
Please consider updating the code to cast to float instead.

Workaround tried

When I added
np.float128 = np.float64
inside pyirf/interpolation/component_estimators.py and ran lstchain_create_dl3_file`, the error did not occur.

Reproduce

lstchain_create_dl3_file \
 --input-dl2 /Users/sito/cta/DL2/dl2_LST-1.Run06894.h5 \
 --input-irf-path /Users/sito/cta/data/DL2_to_DL3/IRF/AllSky/20250212_v0.10.17_allsky_interp_dl2_irfs_nsb_0.22/TestingDataset/Gamma/dec_2276 \
 --irf-file-pattern 'node_theta_*_az_*/irf_20240918_v0.10.12_allsky_nsb_tuning_0.22_Gamma_dec_2276_node_theta_*_az_*_.fits.gz' \
 --output-dl3-path /Users/sito/cta/DL3 \
 --source-name Crab \
 --source-ra '83.6324 deg' \
 --source-dec '22.0174 deg' \
 --config /Users/sito/cta/data/DL2_to_DL3/IRF/AllSky/20250212_v0.10.17_allsky_interp_dl2_irfs_nsb_0.22/TestingDataset/Gamma/dec_2276/lstchain_config_nsb0.22.json \
 --debug –overwrite
  • Result
/opt/anaconda3/envs/baka/lib/python3.13/site-packages/ctaplot/io/dataset.py
UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
 import pkg_resources
2025-07-22 14:06:44,839 INFO [lstchain.DataReductionFITSWriter] (tool.initialize): Loading config from '[PosixPath('/Users/sito/cta/data/DL2_to_DL3/IRF/AllSky/20250212_v0.10.17_allsky_interp_dl2_irfs_nsb_0.22/TestingDataset/Gamma/dec_2276/lstchain_config_nsb0.22.json')]'
2025-07-22 14:06:44,839 DEBUG [lstchain.DataReductionFITSWriter] (application._load_config_files): Looking for /Users/sito/cta/data/DL2_to_DL3/IRF/AllSky/20250212_v0.10.17_allsky_interp_dl2_irfs_nsb_0.22/TestingDataset/Gamma/dec_2276/lstchain_config_nsb0.22 in /Users/sito/cta/cta-lstchain
2025-07-22 14:06:44,841 DEBUG [lstchain.DataReductionFITSWriter] (application._load_config_files): Loaded config file: /Users/sito/cta/data/DL2_to_DL3/IRF/AllSky/20250212_v0.10.17_allsky_interp_dl2_irfs_nsb_0.22/TestingDataset/Gamma/dec_2276/lstchain_config_nsb0.22.json
2025-07-22 14:06:44,841 INFO [lstchain.DataReductionFITSWriter] (tool.initialize): ctapipe version 0.25.1
The metadata are comparable
The other parameter axes data are comparable
2025-07-22 14:06:46,201 DEBUG [lstchain.DataReductionFITSWriter] (lstchain_create_dl3_file.setup): Output DL3 file: /Users/sito/cta/DL3/dl3_LST-1.Run06894.fits
2025-07-22 14:06:46,201 DEBUG [lstchain.DataReductionFITSWriter] (tool.run): CONFIG: {'DataReductionFITSWriter': {'config_files': [PosixPath('/Users/sito/cta/data/DL2_to_DL3/IRF/AllSky/20250212_v0.10.17_allsky_interp_dl2_irfs_nsb_0.22/TestingDataset/Gamma/dec_2276/lstchain_config_nsb0.22.json')], 'gzip': False, 'input_dl2': PosixPath('/Users/sito/cta/DL2/dl2_LST-1.Run06894.h5'), 'input_irf_path': PosixPath('/Users/sito/cta/data/DL2_to_DL3/IRF/AllSky/20250212_v0.10.17_allsky_interp_dl2_irfs_nsb_0.22/TestingDataset/Gamma/dec_2276'), 'interp_method': 'linear', 'irf_file_pattern': 'node_theta_*_az_*/irf_20240918_v0.10.12_allsky_nsb_tuning_0.22_Gamma_dec_2276_node_theta_*_az_*_.fits.gz', 'keep_duplicated_events': False, 'log_config': {}, 'log_datefmt': '%Y-%m-%d %H:%M:%S', 'log_file': None, 'log_file_level': 'INFO', 'log_format': '[%(name)s]%(highlevel)s %(message)s', 'log_level': 10, 'logging_config': {}, 'output_dl3_path': PosixPath('/Users/sito/cta/DL3'), 'overwrite': True, 'provenance_log': PosixPath('/Users/sito/cta/DL3/DataReductionFITSWriter.provenance.log'), 'quiet': False, 'show_config': False, 'show_config_json': False, 'source_dec': '22.0174 deg', 'source_dep': False, 'source_name': 'Crab', 'source_ra': '83.6324 deg', 'use_nearest_irf_node': False, 'EventSelector': {'filters': {'intensity': [50, inf], 'width': [0, inf], 'length': [0, inf], 'r': [0, 1], 'wl': [0.01, 1], 'leakage_intensity_width_2': [0, 1], 'event_type': [32, 32]}, 'finite_params': ['intensity', 'length', 'width']}, 'DL3Cuts': {'allowed_tels': [1], 'alpha_containment': 0.7, 'fill_alpha_cut': 20.0, 'fill_theta_cut': 0.32, 'gh_efficiency': 0.7, 'global_alpha_cut': 10.0, 'global_gh_cut': 0.7, 'global_theta_cut': 0.2, 'max_alpha_cut': 20.0, 'max_gh_cut': 0.98, 'max_theta_cut': 0.32, 'min_alpha_cut': 1.0, 'min_event_p_en_bin': 100.0, 'min_gh_cut': 0.1, 'min_theta_cut': 0.1, 'theta_containment': 0.7}}}
/opt/anaconda3/envs/baka/lib/python3.13/site-packages/tables/group.py:1220: UserWarning: problems loading leaf ``/provenance/dl1_to_dl2``:: variable length strings are not supported yetThe leaf will become an ``UnImplemented`` node.
 warnings.warn(
2025-07-22 14:06:53,725 INFO [lstchain.DataReductionFITSWriter] (lstchain_create_dl3_file.interp_irfs): Paths of IRFs used for interpolation: [PosixPath('/Users/sito/cta/data/DL2_to_DL3/IRF/AllSky/20250212_v0.10.17_allsky_interp_dl2_irfs_nsb_0.22/TestingDataset/Gamma/dec_2276/node_theta_37.814_az_270_/irf_20240918_v0.10.12_allsky_nsb_tuning_0.22_Gamma_dec_2276_node_theta_37.814_az_270_.fits.gz'), PosixPath('/Users/sito/cta/data/DL2_to_DL3/IRF/AllSky/20250212_v0.10.17_allsky_interp_dl2_irfs_nsb_0.22/TestingDataset/Gamma/dec_2276/node_theta_37.814_az_90_/irf_20240918_v0.10.12_allsky_nsb_tuning_0.22_Gamma_dec_2276_node_theta_37.814_az_90_.fits.gz'), PosixPath('/Users/sito/cta/data/DL2_to_DL3/IRF/AllSky/20250212_v0.10.17_allsky_interp_dl2_irfs_nsb_0.22/TestingDataset/Gamma/dec_2276/node_theta_23.630_az_259.265_/irf_20240918_v0.10.12_allsky_nsb_tuning_0.22_Gamma_dec_2276_node_theta_23.630_az_259.265_.fits.gz')]
WARNING: VerifyWarning: Keyword name 'ETRUE_SCALE' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created. [[astropy.io](http://astropy.io/).fits.card]
2025-07-22 14:06:57,214 INFO [lstchain.DataReductionFITSWriter] (lstchain_create_dl3_file.apply_srcindep_gh_cut): Using gamma efficiency of 0.7
2025-07-22 14:06:57,214 INFO [lstchain.DataReductionFITSWriter] (lstchain_create_dl3_file.start): Generating event list
2025-07-22 14:06:59,418 INFO [lstchain.high_level.hdu_table] (hdu_table.get_pointing_params): Mean pointing RA: 83.285 deg, Dec: 21.787 deg
2025-07-22 14:06:59,418 INFO [lstchain.high_level.hdu_table] (hdu_table.get_pointing_params): Fraction of events with pointing beyond 0.050 deg of mean pointing: 0.000000
2025-07-22 14:06:59,418 INFO [lstchain.high_level.hdu_table] (hdu_table.get_pointing_params): Angle between source direction and mean telescope pointing is 0.396 deg
2025-07-22 14:06:59,836 INFO [lstchain.DataReductionFITSWriter] (lstchain_create_dl3_file.start): Target parameters for interpolation: {'ZEN_PNT': <Quantity 25.70045 deg>, 'AZ_PNT': <Quantity 260.5483 deg>, 'B_DELTA': <Quantity 54.8949 deg>}
2025-07-22 14:06:59,836 INFO [lstchain.DataReductionFITSWriter] (lstchain_create_dl3_file.start): Adding IRF HDUs
2025-07-22 14:06:59,836 INFO [lstchain.DataReductionFITSWriter] (lstchain_create_dl3_file.start): Zenith pointing of MC at 25.700 deg
2025-07-22 14:06:59,836 INFO [lstchain.DataReductionFITSWriter] (lstchain_create_dl3_file.start): Azimuth pointing of MC at 260.548 deg
2025-07-22 14:06:59,836 INFO [lstchain.DataReductionFITSWriter] (lstchain_create_dl3_file.start): Geomagnetic delta for the MC is 54.895 deg
2025-07-22 14:06:59,925 INFO [lstchain.DataReductionFITSWriter] (tool.write_provenance): Output: /Users/sito/cta/DL3/dl3_LST-1.Run06894.fits
2025-07-22 14:06:59,925 DEBUG [lstchain.DataReductionFITSWriter] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /Users/sito/cta/DL3/DataReductionFITSWriter.provenance.log'
2025-07-22 14:06:59,936 INFO [lstchain.DataReductionFITSWriter] (tool.run): Finished DataReductionFITSWriter
2025-07-22 14:06:59,936 DEBUG [lstchain.DataReductionFITSWriter] (application.exit): Exiting application: DataReductionFITSWriter

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions