Skip to content

Namespace clean-up#2093

Merged
kbwestfall merged 49 commits intopypeit:developfrom
kbwestfall:pkg_cycles
Mar 31, 2026
Merged

Namespace clean-up#2093
kbwestfall merged 49 commits intopypeit:developfrom
kbwestfall:pkg_cycles

Conversation

@kbwestfall
Copy link
Copy Markdown
Collaborator

Based on some playing around with pydeps, the goal of the PR is to avoid "cycles" (i.e., circular imports) in the repo. This led to three main changes:

  1. All the modules that are effectively only needed by the pypeit/__init__.py file are moved into a new pypeit/pkg directory. This includes the logging code, the exceptions, and the cache system. Anything in this directory should only import from other modules in this directory, not from other modules in the repo.
  2. By importing all spectrograph modules in pypeit/spectrographs/__init__.py, we were polluting the namespace with a lot of cycles. This is cleaned up a bit here by moving these imports out of pypeit/spectrographs/__init__.py and instead defining the __all__ list, and moving code from pypeit/spectrographs/__init__.py to pypeit/spectrographs/util.py.
  3. Essentially the same refactor performed for pypeit/spectrographs/__init__.py is also done for pypeit/scripts/__init__.py.

A minor thing was that I moved some functions from pypeit/par/util.py to pypeit/utils.py because they're used outside of pypeit/par.

There are some associated doc updates and a lot of changes to imports.

@kbwestfall
Copy link
Copy Markdown
Collaborator Author

PS - This also includes all the changes to develop required to sync it with release.

@kbwestfall
Copy link
Copy Markdown
Collaborator Author

Tests pass

Test Summary
--------------------------------------------------------
--- PYTEST PYPEIT UNIT TESTS PASSED  330 passed, 448 warnings in 245.57s (0:04:05) ---
--- PYTEST UNIT TESTS FAILED  1 failed, 156 passed, 1449 warnings in 648.12s (0:10:48) ---
--- PYTEST VET TESTS PASSED  74 passed, 1664 warnings in 4260.14s (1:11:00) ---
--- PYPEIT DEVELOPMENT SUITE PASSED 292/292 TESTS  ---
Total disk usage: 332.294 GiB
Testing Started at 2026-03-24T20:43:19.395103
Testing Completed at 2026-03-25T08:52:32.808205
Total Time: 12:09:13.413102

The one unit test failure is because of the number of files in the RAW_DATA/ldt_deveny/DV1 directory. I suggest we continue to let this one fail until we finalize pypeit/PypeIt-development-suite#397 .

Comment on lines -1609 to +1614
raise PypeItError(f'CODING ERROR: All {frametype} frames in group {calib_ID} '
'are not all associated with the same subset of calibration '
'groups; calib for the first file is '
f'{fitstbl["calib"][indx][0]}.')
log_str = f'All {frametype} frames in group {calib_ID} '
log_str += 'are not all associated with the same subset of calibration '
log_str += 'groups; calib for the first file is '
log_str += f'{fitstbl["calib"][indx][0]}.'
log.warning(log_str)
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a correct demotion from an error to a warning?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is actually part of the changes that are currently in the release branch. This line was causing problems for APF/Levy reductions, so Brad and I talked through just demoting it. We, as the developers, need to keep an eye out that this isn't getting tripped, though.

Comment on lines -1699 to -1733
def save_pickle(fname, obj):
"""Save an object to a python pickle file

Parameters
----------
fname : :class:`str`
Filename
obj : :class:`object`
An object suitable for pickle serialization.
"""
if fname.split(".")[-1] != 'pkl':
fname += '.pkl'
with open(fname, 'wb') as f:
pickle.dump(obj, f, pickle.HIGHEST_PROTOCOL)
log.info('File saved: {0:s}'.format(fname))


def load_pickle(fname):
"""Load a python pickle file

Parameters
----------
fname : :class:`str`
Filename

Returns
-------
:class:`object`
An object suitable for pickle serialization.
"""
log.info('Loading file: {0:s}'.format(fname))
with open(fname, 'rb') as f:
return pickle.load(f)


Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gasp no more pickles???!?

@kbwestfall kbwestfall merged commit 166235b into pypeit:develop Mar 31, 2026
23 checks passed
@kbwestfall kbwestfall deleted the pkg_cycles branch March 31, 2026 18:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants