Merge pull request #2049 from kbwestfall/versioning

Adds description of new versioning approach

Author: kbwestfall

README.rst

@@ -52,24 +52,41 @@ PypeIt |forks| |stars|
 
 PypeIt (pronounced "pipe it") is a Python package for semi-automated reduction
 of astronomical spectroscopic data. Its algorithms build on decades-long
-development of previous data reduction pipelines by the developers. The
-reduction procedure - including a complete list of the input parameters and
-available functionality - is provided by our `online documentation
+development of previous data reduction pipelines by the developers.
+
+**For a complete description of PypeIt**, please see our `online documentation
 <https://pypeit.readthedocs.io/en/stable/>`__.
 
+----
+
+.. _readme_usage:
+
+Installation and Usage
+++++++++++++++++++++++
+
+Detailed installation instructions can be found `here
+<https://pypeit.readthedocs.io/en/stable/installing.html>`__.  Briefly, after
+creating a fresh python environment, simply type:
+
+.. code-block:: bash
+
+    pip install pypeit
+
 PypeIt is designed to be used by both advanced spectroscopists with prior data
 reduction expertise and astronomers with no prior experience of data reduction.
 It is highly configurable and designed to be applied to any standard
 slit-imaging spectrograph, including long-slit, multi-slit, as well as
-cross-dispersed echelle spectra.  **The spectrographs that PypeIt can be used
-with are listed** `here
+cross-dispersed echelle spectra.
+
+**The spectrographs that PypeIt supports are listed** `here
 <https://pypeit.readthedocs.io/en/stable/spectrographs/spectrographs.html>`__.
 Specifically, look `here
 <https://pypeit.readthedocs.io/en/stable/spectrographs/spectrographs.html#instrument-specific-details>`__
 for useful information about reducing data with certain instruments.
 
-In addition to our primary code base, we maintain an extensive `development
-suite <https://github.com/pypeit/PypeIt-development-suite>`__ primarily used to
+**Example data sets**: In addition to our primary code base, we maintain an
+extensive `development suite
+<https://github.com/pypeit/PypeIt-development-suite>`__ primarily used to
 perform multiple layers of code testing, from basic unit tests to full
 end-to-end tests of all our command-line scripts.  **If you are new to PypeIt**,
 you are encouraged to learn how to use the code by finding and experimenting
@@ -81,22 +98,40 @@ by instrument and configuration) of `this shared Google Drive folder
 
 .. _community:
 
-Community
-+++++++++
+Community and Communication
++++++++++++++++++++++++++++
 
-As a project, PypeIt is committed to fostering a welcoming, diverse, and
-inclusive community.  As a member of this community you are expected to read and
-follow our `Code of Conduct
+**Code of Conduct**: As a project, PypeIt is committed to fostering a
+welcoming, diverse, and inclusive community.  As a member of this community you
+are expected to read and follow our `Code of Conduct
 <https://pypeit.readthedocs.io/en/stable/codeconduct.html>`__.
 
-Along with our extensive `online documentation
-<https://pypeit.readthedocs.io/en/stable/>`__, we encourage the PypeIt user
-base to communicate via our `PypeIt Users Slack <https://pypeit-users.slack.com>`__.
-All are welcome to join using `this invitation link <https://join.slack.com/t/pypeit-users/shared_invite/zt-3cderhn4g-XgFzv6mBWqxtVXKudc8W0w>`__.
-
-If you find a bug (particularly one that is experienced by others in the Users
-Slack) or have a feature request, please `submit a GitHub issue
-<https://github.com/pypeit/PypeIt/issues>`__.
+**Documentation**: We maintain extensive `online documentation
+<https://pypeit.readthedocs.io/en/stable/>`__ that provides usage tutorials and
+describes the PypeIt code, reduction and processing procedures, and output data
+models.
+
+**Real-time Communication**: We strongly encourage users to join 
+our `PypeIt Users Slack <https://pypeit-users.slack.com>`__.  Developers are
+available to answer questions and help troubleshoot issues.  All are welcome to
+join using `this invitation link
+<https://join.slack.com/t/pypeit-users/shared_invite/zt-3cderhn4g-XgFzv6mBWqxtVXKudc8W0w>`__;
+please let us know (by submitting a GitHub issue) if the invitation link has
+expired.  Please make sure to read the `guidelines
+<https://docs.google.com/document/d/1Jo83RKstE8DaHxDCsGxsM3DC7sl153Hq9UGN3Piapuc/edit?usp=sharing>`__
+before posting your question; the guidelines are also linked in the description
+of the ``#guidelines`` channel.
+
+**Bug Reports and Feature Requests**: If you encounter a bug, please first check
+if this could be caused by user error; be sure to consult the online
+documentation and post about your issue in the Users Slack (see above).  If the
+problem persists (particularly if it is also experienced by other users),
+please `submit a GitHub issue <https://github.com/pypeit/PypeIt/issues>`__.
+When submitting the issue, please provide as much information as possible.  If
+you haven't already, we may ask you to join the Users Slack for more efficient
+communication about the issue.  Finally, you are also encouraged to `submit a
+GitHub issue <https://github.com/pypeit/PypeIt/issues>`__ if you have a specific
+feature request.
 
 ----
 
@@ -133,16 +168,20 @@ Team <https://pypeit.readthedocs.io/en/stable/team.html>`__ listing.
 Citation
 ++++++++
 
-If you use PypeIt in your research, please cite the following publications
-(we provide the relevant `BibTeX entries
-<https://pypeit.readthedocs.io/en/stable/index.html#pypeit-bibtex-entries>`__
-for your convenience):
+If you use PypeIt in your research, please cite the following two publications:
 
  - Prochaska et al. (2020, JOSS): `arXiv <https://ui.adsabs.harvard.edu/abs/2020arXiv200506505P/abstract>`__, `JOSS <https://joss.theoj.org/papers/10.21105/joss.02308>`__
- - Prochaska et al. (2020, Zenodo): `Zenodo <https://ui.adsabs.harvard.edu/abs/2020zndo...3743493P/abstract>`__
+ - Prochaska et al. (2020, Zenodo): `Zenodo <https://zenodo.org/records/17244617>`__
+
+We provide the relevant `BibTeX entries
+<https://pypeit.readthedocs.io/en/stable/index.html#pypeit-bibtex-entries>`__
+for your convenience.  Note that the Zenodo BibTex entry is for version 1.0.0 of
+the code; however, the Zenodo DOI is updated with each release of PypeIt.  When
+you cite the Zenodo entry, you are encouraged to cite the specific PypeIt
+version you used!  You are also encouraged to explicitly note the specific
+PypeIt version used (e.g., 1.17.3) in the article text.
 
-You are also encouraged to note the specific version of the code you have used
-(e.g., 1.17.3).  If there is no place to include the relevant citations in the
+If there is no place to include the relevant citations in the
 text of the publication, please include the following acknowledgement (provided
 in latex and using the provided `BibTeX entries
 <https://pypeit.readthedocs.io/en/stable/index.html#pypeit-bibtex-entries>`__):

doc/index.rst

@@ -9,24 +9,6 @@ PypeIt BibTeX Entries
 
 .. code-block:: latex
 
-    @ARTICLE{pypeit:joss_arXiv,
-           author = {{Prochaska}, J. Xavier and {Hennawi}, Joseph F. and {Westfall}, Kyle B. and
-             {Cooke}, Ryan J. and {Wang}, Feige and {Hsyu}, Tiffany and
-             {Davies}, Frederick B. and {Farina}, Emanuele Paolo},
-            title = "{PypeIt: The Python Spectroscopic Data Reduction Pipeline}",
-          journal = {arXiv e-prints},
-         keywords = {Astrophysics - Instrumentation and Methods for Astrophysics},
-             year = 2020,
-            month = may,
-              eid = {arXiv:2005.06505},
-            pages = {arXiv:2005.06505},
-    archivePrefix = {arXiv},
-           eprint = {2005.06505},
-     primaryClass = {astro-ph.IM},
-           adsurl = {https://ui.adsabs.harvard.edu/abs/2020arXiv200506505P},
-          adsnote = {Provided by the SAO/NASA Astrophysics Data System}
-    }
-
     @article{pypeit:joss_pub,
         doi = {10.21105/joss.02308},
         url = {https://doi.org/10.21105/joss.02308},

doc/installing.rst

@@ -182,44 +182,8 @@ PypeIt, upgrading the package should simply be a matter of executing:
 
 If this causes problems (*e.g.*, a new PypeIt script is unavailable or you
 encounter script errors), first try uninstalling (``pip uninstall pypeit``) and
-then reinstalling.  There are two important things to keep in mind when
-upgrading:
-
- - **PypeIt datamodels are not necessarily backwards-compatible.**  This means
-   that, *e.g.*, ``pypeit_show_2dspec`` may fault when trying to view
-   ``spec2d*`` files produced with your existing PypeIt version after upgrading
-   to a new version.  **The best approach is to always re-reduce data you're
-   still working with anytime you update PypeIt.**
-
- - **Cached files are version-specific.**  Every time you upgrade pypeit, we
-   recommend deleting your existing cache and starting fresh!  See
-   :ref:`view-cache`.  The only caveat to this is if you are actively using
-   multiple versions of PypeIt, meaning you will still be using old versions of
-   the cached files.  Otherwise, you will end up with multiple versions of the
-   same file on disk.  **Importantly**, the code also considers local files you
-   have installed (using, e.g., ``pypeit_install_linelist``) to be version
-   specific.  If you have installed such files, you will need to re-install them
-   *after* upgrading.
-
-If you have locally installed files, your upgrade may look something like this:
-
-.. code-block:: console
-
-    # Check the cache contents
-    pypeit_clean_cache -l
-    # Delete everything
-    pypeit_clean_cache --clear
-    # Upgrade pypeit
-    pip install pypeit --upgrade
-    # Reinstall your local line lists
-    pypeit_install_linelist /path/to/my/linelists/*_lines.dat
-
-.. note::
-
-    If you find particular data files useful for your reductions, please
-    consider issuing a PR to include them file in the PypeIt repository.  This
-    helps the community, and it means you'll avoid these upgrading
-    complications.
+then reinstalling.  Also note that not all PypeIt versions are
+backwards-compatible; see :ref:`versioning`.
 
 .. _m1_macs:
 
@@ -236,7 +200,7 @@ installation options are welcome; please `Submit an issue`_.
 Installation on Windows
 -----------------------
 
-Generally speaking, we encounter most installation issues on Windows users.
+Generally speaking, we encounter most installation issues for Windows users.
 
 An alternative for running under Windows is to install the `Windows Subsystem
 for Linux (WSL) <https://learn.microsoft.com/en-us/windows/wsl/install>`_.  This
@@ -267,6 +231,69 @@ options for the ``python.exe`` and ``python3.exe`` executables.
 
 ----
 
+.. _versioning:
+
+Versioning
+==========
+
+Traditionally, we provided no guarantee that *any* PypeIt version was backwards
+compatible.  However, as of version 2.0.0, PypeIt uses `Semantic Versioning
+<https://packaging.python.org/en/latest/discussions/versioning/>`__.  This
+approach uses three version categories --- *major*, *minor*, and *patch* ---
+where releases that increment the *major* version number are *not* backwards
+compatible.  We expect the most common reason for incrementing the *major*
+version number will be because of a backwards-incompatible change to either the
+input configuration files (like the ``*.pypeit`` file) or the data models of the
+primary output products.
+
+.. important::
+
+    When possible, we **always** recommend you use the most recent version of
+    PypeIt and reprocess data as necessary.  The code is always improving, not
+    just in functionality but also in robustness of data reduction and
+    processing.
+
+Beyond this, we emphasize two important considerations regarding PypeIt versioning:
+
+- **Backwards-incompatible changes to datamodels can break simple viewing
+  scripts.**  For example, ``pypeit_show_2dspec`` may fault when trying to view
+  ``spec2d*`` files produced using a version of PypeIt that is not backwards
+  compatible with your current version.  You can always maintain multiple python
+  environments with different PypeIt versions installed or reprocess data with
+  your currently installed PypeIt version.
+
+- **Cached files are version-specific.**  Every time you upgrade PypeIt, we
+  recommend deleting your existing cache and starting fresh!  See
+  :ref:`view-cache`.  The only caveat to this is if you are actively using
+  multiple versions of PypeIt (in different environments), meaning you will
+  still be using old versions of the cached files.  Otherwise, you will end up
+  with multiple versions of the same file on disk.  **Importantly**, the code
+  also considers local files you have installed (using, e.g.,
+  ``pypeit_install_linelist``) to be version specific.  If you have installed
+  such files, you will need to re-install them *after* upgrading.
+
+  If you have locally installed files, your upgrade may look something like
+  this:
+
+  .. code-block:: console
+
+    # Check the cache contents
+    pypeit_clean_cache -l
+    # Delete everything
+    pypeit_clean_cache --clear
+    # Upgrade pypeit
+    pip install pypeit --upgrade
+    # Reinstall your local line lists
+    pypeit_install_linelist /path/to/my/linelists/*_lines.dat
+
+  .. note::
+
+    If you find particular data files useful for your reductions, please
+    consider issuing a PR to include them in the PypeIt repository.  This helps
+    the community, and it means you'll avoid these upgrading complications.
+
+----
+
 .. _data_installation:
 
 Additional Data and the PypeIt Cache
@@ -322,7 +349,7 @@ cache.  To list the cache contents, use the ``-l`` option:
          github               1.15.1                tests solution_arrays.npz
 
 Note that the files hosted on GitHub will be specific to a branch or version of
-PypeIt.  **Every time you upgrade pypeit, we recommend deleting your existing
+PypeIt.  **Every time you upgrade PypeIt, we recommend deleting your existing
 cache and starting fresh!**
 
 **Local files** that have been installed into the cache (e.g., using
@@ -428,7 +455,7 @@ can be included in the PypeIt repository.
 
     Because PypeIt uses the cache system to manage the local files, it will
     associate each file with the version of the code used to install it in the
-    cache.  Every time you upgrade your pypeit version, you should delete the
+    cache.  Every time you upgrade your PypeIt version, you should delete the
     local files from the cache (this will not remove the local file itself) and
     re-install them using the upgraded version of PypeIt.  See :ref:`view-cache`
     and :ref:`upgrade`.
@@ -510,7 +537,7 @@ Python (see :ref:`dependencies`):
 * `PySide <https://wiki.qt.io/Qt_for_Python>`_
 
 At least one of those bindings must be installed for the interactive GUIs to
-work. By default ``pypeit`` will install ``pyqt6``. Other backends can be used
+work. By default PypeIt will install ``pyqt6``. Other backends can be used
 by installing them manually via ``pip`` or ``conda`` and then setting the ``QT_API``
 environment variable. See the `QtPy documentation <https://github.com/spyder-ide/qtpy>`_
 for more details.
@@ -603,7 +630,7 @@ spectrum from the Shane/KAST spectrograph at Lick Observatory.
 Troubleshooting
 ===============
 
-If you have trouble installing pypeit, you're encouraged to join our `PypeIt
+If you have trouble installing PypeIt, you're encouraged to join our `PypeIt
 Users Slack <https://pypeit-users.slack.com>`__ using `this invitation
 link <invite_>`_ and post your issue to the ``#installing`` channel.  Below is an
 incomplete list of issues that users have reported in the past.  In addition to
@@ -612,7 +639,7 @@ let us know if these suggestions do not work for you.*
 
 ----
 
-**I am trying to install pypeit for the first time and it fails!**:  The root
+**I am trying to install PypeIt for the first time and it fails!**:  The root
 problem of this can be system dependent:
 
  - First, *always* make sure you install the code into a fresh environment.
@@ -629,23 +656,23 @@ problem of this can be system dependent:
 
 ----
 
-**I am trying to upgrade pypeit and it fails!**:  First try uninstalling your
-current pypeit version:
+**I am trying to upgrade PypeIt and it fails!**:  First try uninstalling your
+current PypeIt version:
 
 .. code-block:: bash
 
     pip uninstall pypeit
 
 Then reinstall it.  If that also fails, try creating a fresh environment and
-reinstalling pypeit in that new environment.
+reinstalling PypeIt in that new environment.
 
 ----
 
 **The installation process succeeded, but the code is faulting!**:  This could
 be for a few reasons:
 
- - Recall that pypeit isn't necessarily backwards compatible.  If you've
-   upgraded pypeit and tried to use it with data that was reduced by a previous
+ - Recall that PypeIt isn't necessarily backwards compatible.  If you've
+   upgraded PypeIt and tried to use it with data that was reduced by a previous
    version, the fault may because of changes between versions.  You will either
    need to revert to your previous version or reprocess the data.
 
@@ -680,6 +707,6 @@ Slack.  However, here are a few things to note and/or try:
  - And, of course, the code will have bugs.  If you find one, the more
    information you provide the developers, the easier it will be for us to track
    down the issue.  Valuable information includes your OS, OS version, python
-   version, pypeit version, and the full Traceback provided with the error.  QA
+   version, PypeIt version, and the full Traceback provided with the error.  QA
    plots and ``ginga`` screen grabs that illustrate the issue are also very
    helpful!