Merge pull request #2058 from pypeit/deprecate_linetools

Deprecate linetools

Author: kbwestfall

.github/workflows/ci_cron.yml

@@ -16,7 +16,7 @@ jobs:
       matrix:
         os: [ubuntu-latest]
         python: ['3.11', '3.12', '3.13']
-        toxenv: [test, test-alldeps-cov, test-numpydev, test-linetoolsdev, test-gingadev, test-astropydev]
+        toxenv: [test, test-alldeps-cov, test-numpydev, test-gingadev, test-astropydev]
     steps:
     - name: Check out repository
       uses: actions/checkout@v3

.github/workflows/ci_tests.yml

@@ -18,7 +18,7 @@ jobs:
       matrix:
         os: [ubuntu-latest]
         python: ['3.11', '3.12', '3.13']
-        toxenv: [test, test-alldeps-cov, test-numpydev, test-linetoolsdev, test-gingadev, test-astropydev]
+        toxenv: [test, test-alldeps-cov, test-numpydev, test-gingadev, test-astropydev]
     steps:
     - name: Check out repository
       uses: actions/checkout@v3

CLAUDE.md

@@ -0,0 +1,99 @@
+# PypeIt
+
+PypeIt is a pure Python package for processing raw spectroscopic data from
+astronomical telescopes into calibrated spectra for scientific analysis.
+
+## Development Setup
+
+- Create and activate a fresh python environment
+
+- Clone your fork of the main pypeit repo, hosted at
+  [GitHub](https://github.com/pypeit/PypeIt), and add a remote git connection to
+  the main repo called `upstream`.
+
+- Install pypeit using the zshell (or bash equivalent) command `pip install -e ".[dev]"`.
+
+- Create a new git branch, where simple bug fixes branch from `release` and all
+  other development branches from `develop`.
+
+- Frequently fetch changes to the upstream base branch (`release` or `develop`)
+  and rebase, as necessary.
+
+- Additional development guidelines is provided in `doc/dev/development.rst`.
+
+## Key Architectural Components
+
+- PypeIt implements three primary processing paths, depending on the format of
+  the spectrograph used to collect the raw data:
+
+    - **MultiSlit**: Standard long-slit and multi-slit spectrographs
+    - **Echelle**: Cross-dispersed echelle spectrographs
+    - **SlicerIFU**: Slicer-based integral-field units
+
+- Instrument specifications and data-processing considerations that are specific
+  to each spectrograph (including the processing path that should be used) are
+  isolated in their respective spectrograph classes, all of which inherit from
+  `pypeit.spectrographs.spectrograph.Spectrograph`.
+
+- Users primarily modify the code performance via user-level parameters held by
+  the `pypeit.par.pypeitpar.PypeItPar` class, which packages the hierarchy of
+  parameter sets used throughout the code.
+
+- Core processing modules, particularly for calibrations, produce FITS files
+  that are saved to disk and reused as necessary.  These modules commonly use
+  `pypeit.datamodel.DataContainer` as a base class to enforce strict adherence
+  to a well-defined datamodel and to provide a common IO interface.
+
+- Significant portions of the `pypeit/data` directory are not included in the
+  package distribution of the code, but rely on a cache system that downloads
+  files as needed for processing.
+
+## Testing
+
+- All tests are collected in the `pypeit/tests` directory.
+
+- Tests in `pypeit/tests` should be limited to unit tests that do not require
+  the use of large data files.
+
+- Tests should be deterministic; i.e., all random-number generators should use a
+  fixed seed.
+
+- Test coverage is supplemented by the PypeIt development suite, hosted at
+  https://github.com/pypeit/PypeIt-development-suite, which requires data files
+  hosted on Google Drive; see README.rst for the link.
+
+## Documentation
+
+- All functions and classes, except for tests in `pypeit/tests` should include
+  docstrings that explain their purpose, input arguments, and output objects.
+
+- The docstring style is currently not consistent within the repository, but
+  Numpy style docstrings are preferred; see
+  https://www.sphinx-doc.org/en/master/usage/extensions/example_numpy.html
+
+- Package documentation for users is held in the `doc` directory, which is built
+  using Sphinx and hosted on ReadTheDocs at https://pypeit.readthedocs.io/.
+
+- A complete rebuild of the documentation is performed by executing the bash
+  command `cd doc; make clean ; make html`.  This requires access to the
+  internet and the `PYPEIT_DEV` environmental variable, which points to the
+  directory containing containing the `RAW_DATA` directory copied from the
+  PypeIt development suite Google Drive (see "Testing" above).  If these
+  requirements are not met, a limited rebuild of the documentation can be
+  achieved by executing `cd doc ; make htmlonly`.
+
+- Documentation should be updated with each GitHub pull request.
+
+## Usage
+
+- Users interact with the code base via execution of the command-line scripts
+  found in `pypeit/scripts`.
+
+- The primary data-processing script is `run_pypeit.py`, which primarily
+  instantiates the `pypeit.pypeit.PypeIt` class and runs its methods.
+
+## External Resources
+
+- **Documentation**: https://pypeit.readthedocs.io/
+- **Development Suite**: https://github.com/pypeit/PypeIt-development-suite
+

doc/api/pypeit.core.rst

@@ -42,6 +42,7 @@ Submodules
    pypeit.core.pydl
    pypeit.core.qa
    pypeit.core.scattlight
+   pypeit.core.skyspec
    pypeit.core.skysub
    pypeit.core.slitdesign_matching
    pypeit.core.spectrum

doc/api/pypeit.core.skyspec.rst

@@ -0,0 +1,8 @@
+pypeit.core.skyspec module
+==========================
+
+.. automodule:: pypeit.core.skyspec
+   :members:
+   :private-members:
+   :show-inheritance:
+   :undoc-members:

doc/claude/shane_kast_blue.md

@@ -0,0 +1,164 @@
+# PypeIt Reduction with Claude: Shane Kast Blue Example
+
+This document describes a workflow using Claude to reduce spectroscopic data from the Shane Kast Blue spectrograph using PypeIt.
+
+## Initial Prompt
+
+Here was the prompt given to Claude:
+
+> In `/home/xavier/Projects/PypeIt/PypeIt-development-suite/RAW_DATA/shane_kast_blue/600_4310_d55` is a set of raw FITS files for the shane_kast_blue spectrograph. Please use the PypeIt code base to generate a PypeIt file. Do your work in `/home/xavier/Projects/PypeIt/PypeIt_Redux/Claude/shane_kast_blue`. If you run python or any scripts, use the "pypeit" environment in conda.
+
+---
+
+## Step 1: PypeIt File Generation
+
+### Summary
+
+Claude successfully generated a PypeIt file for the shane_kast_blue spectrograph data.
+
+### Location
+
+The PypeIt file is located at: `shane_kast_blue_A.pypeit`
+
+### Configuration Details
+
+- **Setup:** A (600/4310 grating with d55 dichroic)
+- **Total files:** 25 raw FITS files
+
+### File Classification
+
+The setup identified:
+
+| Frame Type | Files | Description |
+|------------|-------|-------------|
+| Arc | b1.fits.gz | For wavelength calibration |
+| Bias | b14-b23.fits.gz | 10 bias frames |
+| Flat | b3-b13.fits.gz | 10 frames (pixel flat, illumination flat, and trace) |
+| Science | b27-b28.fits.gz | 2 frames (target: J1217p3905, 1200s exposures) |
+| Standard | b24.fits.gz | Feige 66, 30s exposure |
+
+---
+
+## Step 2: Running the Reduction
+
+### Prompt
+
+> Please go ahead and execute the reduction
+
+### Command
+
+```bash
+conda run -n pypeit run_pypeit shane_kast_blue_A.pypeit
+```
+
+### Result
+
+**Reduction Complete!** The PypeIt reduction finished successfully in 1 minute 38 seconds.
+
+---
+
+## Reduction Output Summary
+
+### Science Products (`Science/`)
+
+#### 1D Extracted Spectra
+
+- `spec1d_b24-Feige66_KASTb_20150520T041246.960.fits` - Standard star (Feige 66)
+- `spec1d_b27-J1217p3905_KASTb_20150520T045733.560.fits` - Science target (first exposure)
+- `spec1d_b28-J1217p3905_KASTb_20150520T051801.470.fits` - Science target (second exposure)
+
+#### 2D Reduced Spectra
+
+- `spec2d_b24-Feige66_KASTb_20150520T041246.960.fits` (27 MB)
+- `spec2d_b27-J1217p3905_KASTb_20150520T045733.560.fits` (27 MB)
+- `spec2d_b28-J1217p3905_KASTb_20150520T051801.470.fits` (27 MB)
+
+### Calibration Files (`Calibrations/`)
+
+| File | Description |
+|------|-------------|
+| `Arc_A_0_DET01.fits` | Arc line image for wavelength calibration |
+| `Bias_A_0_DET01.fits` | Master bias frame |
+| `Flat_A_0_DET01.fits` | Master flat field |
+| `Edges_A_0_DET01.fits.gz` | Slit edge traces |
+| `Slits_A_0_DET01.fits.gz` | Slit information |
+| `Tilts_A_0_DET01.fits` | Spectral tilt solution |
+| `WaveCalib_A_0_DET01.fits` | Wavelength calibration solution |
+
+### Quality Assurance (`QA/`)
+
+- HTML reports for each exposure (b24, b27, b28)
+- Master Frame QA report (`MF_A.html`)
+- PNG diagnostic plots in the `QA/PNGs/` directory
+
+### Key Features
+
+- Successfully extracted 1 object from each science frame
+- Applied heliocentric correction (-22.46 km/s)
+- Performed spectral flexure correction (-0.023 pixels)
+- All spectra are wavelength calibrated and sky subtracted
+
+You can view the QA HTML files in a browser to inspect the quality of the reduction.
+
+---
+
+## Step 3: Analysis Notebook
+
+### Prompt
+
+> Now generate a Jupyter Notebook describing the outputs and comment on the quality of the reduction.
+
+### Result
+
+Claude created a comprehensive Jupyter notebook that analyzes the PypeIt reduction outputs.
+
+### Notebook Location
+
+`shane_kast_blue_analysis.ipynb`
+
+### Contents
+
+#### 1. Introduction
+- Overview of the observation details and configuration
+- Summary of calibration frames used
+
+#### 2. Calibration Quality Assessment
+- Master bias frame analysis (statistics and visualization)
+- Normalized pixel flat examination
+- Slit edge detection results
+- Wavelength calibration coverage
+
+#### 3. Two-Dimensional Spectra
+- Standard star (Feige 66) 2D spectrum display
+- Science target (J1217+3905) 2D spectra for both exposures
+- Sky model visualizations
+- Spatial and spectral profile plots
+
+#### 4. One-Dimensional Extracted Spectra
+- Standard star 1D spectrum with S/N analysis
+- Science target spectra for both exposures
+- Comparison between exposures
+- Sky subtraction quality assessment
+
+#### 5. Quality Assessment Summary
+- Automated quality metrics calculation
+- Detailed assessment of:
+  - Calibration quality (bias, flat, wavelength)
+  - Science data quality (S/N ratios)
+  - Sky subtraction performance
+  - Consistency between exposures
+- Overall reduction quality rating
+
+#### 6. Conclusions and Next Steps
+- Summary of reduction quality
+- Recommended next steps (flux calibration, coaddition)
+- List of ready-to-use data products
+
+### To Run the Notebook
+
+```bash
+cd /home/xavier/Projects/PypeIt/PypeIt_Redux/Claude/shane_kast_blue
+conda run -n pypeit jupyter notebook shane_kast_blue_analysis.ipynb
+```
+
+The notebook includes detailed visualizations and quantitative assessments that will help you understand the quality of the reduction and identify any potential issues. All analysis is ready to run and will provide comprehensive diagnostics of your Shane Kast Blue spectroscopic data reduction.

doc/claude/shane_kast_blue_analysis.ipynb

@@ -295,13 +295,7 @@ that its file extensions are:
 
 .. include:: include/datamodel_onespec.rst
 
-You view the spectrum using the ``lt_xspec`` script 
-(``pypeit_show_1dspec`` will not work), which loads the data
-and launches a GUI from the `linetools`_ package. e.g.:
-
-.. code-block:: console
-
-    lt_xspec J1217p3905_coadd.fits
+To view the coadded spectrum, using :ref:`pypeit_show_1dspec`.
 
 UVES_popler coaddition
 ======================

doc/coadd1d.rst

@@ -502,8 +502,8 @@ can do:
     cd $PYPEIT_DIR
     tox -e py312-test-astropydev
 
-Similar ``dev`` dependencies are configured for ``numpy``, ``ginga``, and
-``linetools``, as well.
+Similar ``dev`` dependencies are configured for ``numpy`` and ``ginga``, as
+well.
 
 Unit tests included in the main PypeIt repo should *not* require large data
 files.  Some files are kept in the repo for this purpose (see the