MAD-NG Features and Compatibility Modes (#135)

Author: fsoubelet

.github/workflows/coverage.yml

@@ -12,5 +12,5 @@ jobs:
       uses: pylhc/.github/.github/workflows/coverage.yml@master
       with:
         src-dir: tfs
-        pytest-options: -m "not cern_network" --cov-report term-missing  
+        pytest-options: --cov-report term-missing
       secrets: inherit

.github/workflows/cron.yml

@@ -1,8 +1,7 @@
 # Runs all tests on master on Mondays at 3 am (UTC time)
 name: Cron Testing
 
-
-on: 
+on:
   schedule:
     - cron:  '* 3 * * mon'
 

.zenodo.json

@@ -16,7 +16,7 @@
         },
         {
             "name": "Felix Soubelet",
-            "affiliation": "University of Liverpool & CERN",
+            "affiliation": "CERN",
             "orcid": "0000-0001-8012-1440"
         },
         {

CHANGELOG.md

@@ -1,5 +1,37 @@
 # TFS-Pandas Changelog
 
+## Version 4.0.0
+
+Version `4.0` is a major release bringing compatibility with `MAD-NG` features in **TFS** files and tables, apart from the more exotic ones.
+We also bring new documentation pages regarding the **TFS** format, code compatibilities, and the new features.
+Please have a look at the documentation should you intent to use `tfs-pandas` 4.0, as there are a few (now documented) caveats.
+
+- Important:
+  - Support for `Python 3.9` has been dropped. The minimum required Python version is now `3.10`.
+  - DataFrame validation is now OFF by default, both when reading from and writing to file. It is up to the user to ask for a given validation mode.
+  - Minimum supported `MAD-NG` version is `1.0.0`, due to synchronized development of some feature details. The corresponding `pymadng` version is `0.6.0`.
+
+- Added:
+  - Handling of boolean-type and complex-type header values (`MAD-NG` feature).
+  - Handling of bolean-type and complex-type columns (`MAD-NG` feature).
+  - Handling of nullable-type `nil` values in headers and columns (`MAD-NG` feature).
+  - Compatibility modes for dataframe validation. The `tfs.frame.validate` function now expects this, and valid choices are `MADX`, `MAD-X`, `MADNG` and `MAD-NG` (case-insensitive, see API documentation).
+  - Many new exceptions have been created for raised errors, which will be more specific. They all inherit from the previously raised `TfsFormatError`, so user code that was catching it will still work.
+
+- Changed:
+  - By default, `TfsDataFrame` validation is now skipped on reading.
+  - By default, `TfsDataFrame` validation is now skipped on writing.
+  - By default, `TfsDataFrame` validation in `MAD-X` requires headers to be present in the dataframe.
+
+- Fixed:
+  - Writing a dataframe with no headers (not empty headers), e.g. a `pandas.DataFrame` - now works correctly.
+
+- Documentation:
+  - The documentation has been updated to reflect the new features and changes.
+  - The documentation now includes a new page on the `TFS` format itself.
+  - The documentation now includes a new page on compatibility modes for `TfsDataFrame` validation.
+  - A great deal of internal documentation has been added to the codebase.
+
 ## Version 3.9.0
 
 - Added:
@@ -93,7 +125,7 @@ Minor API changes to the `TFSCollections`:
 ## Version 3.5.2
 
 - Changed:
-  - The dependency on `pandas` has been pinned to `<2.0` to guarantee the proper functionning of the compability `append` and `join` methods in `TfsDataFrames`. These will be removed with the next release of `tfs-pandas` and users should use the `tfs.frame.concat` compatibility function instead.
+  - The dependency on `pandas` has been pinned to `<2.0` to guarantee the proper functionning of the compatibility `append` and `join` methods in `TfsDataFrames`. These will be removed with the next release of `tfs-pandas` and users should use the `tfs.frame.concat` compatibility function instead.
 
 ## Version 3.5.1
 

README.md

@@ -9,8 +9,8 @@
 [![Conda-forge Version](https://img.shields.io/conda/vn/conda-forge/tfs-pandas?color=orange&logo=anaconda)](https://anaconda.org/conda-forge/tfs-pandas)
 [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.5070986.svg)](https://doi.org/10.5281/zenodo.5070986)
 
-This package provides reading and writing functionality for [**Table Format System (TFS)** files](http://mad.web.cern.ch/mad/madx.old/Introduction/tfs.html).
-Files are read into a `TfsDataFrame`, a class built on top of the famous `pandas.DataFrame`, which in addition to the normal behavior attaches a dictionary of headers to the `DataFrame`.
+This package provides reading and writing functionality for [**Table Format System (TFS)**](https://pylhc.github.io/tfs/tfsformat.html) files.
+Files are read into a `TfsDataFrame`, a class built on top of the `pandas.DataFrame`, which in addition to the normal behavior attaches a dictionary of headers to the `DataFrame`.
 
 See the [API documentation](https://pylhc.github.io/tfs/) for details.
 
@@ -45,14 +45,19 @@ data_frame.headers["NEW_KEY"] = some_variable
 # Manipulate data as you do with pandas DataFrames
 data_frame["NEWCOL"] = data_frame.COL_A * data_frame.COL_B
 
-# You can check the validity of a TfsDataFrame, and choose the behavior in case of errors
-tfs.frame.validate(data_frame, non_unique_behavior="raise")  # or choose "warn"
+# You can check the validity of a TfsDataFrame, speficying the
+# compatibility mode as well as the behavior in case of errors
+tfs.frame.validate(
+    data_frame,
+    non_unique_behavior="raise",  # or choose "warn"
+    compatibility="mad-x",  # or choose "mad-ng"
+)
 
 # Writing out to disk is simple too
 tfs.write("path_to_output.tfs", data_frame, save_index="index_column")
 ```
 
-Reading and writing compressed files is also supported, and done automatically based on the provided file extension:
+Compression is automatically supported, based on the provided file extension (for supported formats):
 
 ```python
 import tfs

doc/compatibility.rst

@@ -0,0 +1,101 @@
+MAD-X and MAD-NG Compatibility
+==============================
+
+As `tfs-pandas` allows one to write `TfsDataFrames` as files in the **TFS** format, which are typically output by simulations codes, compatibility of these files with said codes is crucial.
+Specifically, `tfs-pandas` aims to ensure the files it writes to disk are accepted as input for `MAD-X <https://madx.web.cern.ch/>`_ and `MAD-NG <https://madx.web.cern.ch/releases/madng/html/>`_.
+
+However, as ``MAD-NG`` is the successor to ``MAD-X``, it includes new features regarding the **TFS** format, and files including these features will not be accepted by ``MAD-X``.
+To circumvent this issue, `tfs-pandas` offers functionality - named validation - to ensure compatibility with either code.
+
+TfsDataFrame Validation
+-----------------------
+
+It is possible to perform automatic validation of a `TfsDataFrame` both when reading and writing, or to validate them at any time using the `tfs.frame.validate` function.
+Validation enforces the rules described in the :ref:`caveats section <tfs-pandas caveats>`, both to guarantee the integrity of the dataframe and compatibility of written files with simulation codes.
+
+.. admonition:: When Does Validation Happen?
+
+    Validation is **optional**, and is by default turned off at both read-time and write-time.
+
+Validation is done by providing a `TfsDataFrame` and a compatibility mode to `tfs.frame.validate` (see the :ref:`API reference <modules/index:frame>`).
+It goes as:
+
+.. code-block:: python
+
+    import tfs
+    from tfs.frame import validate
+
+    df = tfs.read("path/to/file.tfs")
+
+    # To validate with MAD-X compatibility
+    validate(df, compatibility="mad-x")  # or use "madx"
+
+    # To validate with MAD-NG compatibility
+    validate(df, compatibility="mad-ng")  # or use "madng"
+
+In case of compatibility issue, an exception is raised which will point to the specific incompatible element.
+All exceptions inherit from the `TfsFormatError`, which one can `except` as a catch-all for this package.
+
+.. _common rules:
+
+Common Validation Rules
+-----------------------
+
+In either compatibility mode, some common rules are enforced.
+These rules are listed and described in the :ref:`API reference <modules/index:frame>` for the `tfs.frame.validate` function.
+
+When validating a `TfsDataFrame`, the behavior in case one of these rules is violated depends on the value of the `non_unique_behavior` parameter.
+These rules are *always* checked against when validating a `TfsDataFrame`.
+Additional checks can be performed by setting the `compatibility` parameter, as described in the :ref:`MAD-NG <madng mode>` and :ref:`MAD-X <madx mode>` below.
+
+.. _madng mode:
+
+MAD-NG Compatibility
+--------------------
+
+.. admonition:: Supported Versions
+
+    The compatibility with ``MAD-NG`` is guaranteed starting with version `1.0.0`.
+    See below for details and caveats.
+
+Since ``MAD-NG`` implements and accepts more features into its **TFS** files, its compatibility mode is naturally less restrictive.
+Namely, the following are accepted by ``MAD-NG`` and ``MAD-NG`` only:
+
+- Boolean dtype for header parameters and table columns.
+- Complex dtype for header parameters and table columns.
+- Nullable dtype for header parameters and table columns (value `nil`).
+
+.. admonition:: Complex Number Representation
+
+    In Python, the imaginary part of a complex number is represented by the letter ``j``, as in `1.4 + 2.6j`.
+    When writing complex values to file, `tfs-pandas` will instead use the ``MAD-NG`` (read `Lua`) representation, and uses the letter ``i``, as in `1.4 + 2.6i`, so that ``MAD-NG`` can properly read such a file.
+    Both of these representations will be correctly read by `tfs-pandas` (including when ``MAD-NG`` uses ``I`` for special complex numbers).
+
+.. admonition:: Handling of Nullable Types
+
+    ``MAD-NG`` uses the nullable `nil`, which is accepted by `tfs-pandas` with the following caveats:
+
+        - When reading from file, `nil` values in the headers are converted to `None` while those in the dataframe are cast to `NaN`, except in string-dtyped columns where they are converted to `None`.
+        - When writing to file, `None` values anywhere are written as `nil`, and `NaN` values in the dataframe are written as `NaN` (remember that setting a `None` in a numeric `pandas.DataFrame` column automatically casts it as `NaN`).
+
+.. attention::
+
+    The exotic "features" of ``MAD-NG`` such as the ``Lua`` operator overloading for ranges and tables, and their inclusion in **TFS** files are not supported by `tfs-pandas`.
+    Should one need to use these features, it is recommended to go through the `pymadng <https://pymadng.readthedocs.io/en/latest/>`_ package to handle them in-memory.
+
+.. _madx mode:
+
+MAD-X Compatibility
+-------------------
+
+The ``MAD-X`` compatibility mode is more restrictive, and enforces that none of the features listed in the :ref:`MAD-NG section <madng mode>` appear in the `TfsDataFrame`.
+
+Additionally, ``MAD-X`` will refuse to read into a table any **TFS** file that does not include a `TYPE` entry in the headers (which should be a string).
+As such, when checking for compatibility with ``MAD-X``:
+
+ - If the dataframe has no headers, an error will be raised indicating the dataframe should have headers.
+ - If the dataframe has headers but no `TYPE` entry is not found, `tfs-pandas` will log a warning and add one with the value `"Added by tfs-pandas for MAD-X compatibility"`
+
+.. admonition:: Default mode
+
+    The default compatibility mode enforced by the `tfs.frame.validate` function is ``MAD-X``.

doc/conf.py

@@ -50,6 +50,7 @@ def about_package(init_posixpath: pathlib.Path) -> dict:
 # ones.
 extensions = [
     "sphinx.ext.autodoc",  # Include documentation from docstrings
+    "sphinx.ext.autosectionlabel",  # Create explicit doc targets for each section
     "sphinx.ext.coverage",  # Collect doc coverage stats
     "sphinx.ext.doctest",  # Test snippets in the documentation
     "sphinx.ext.githubpages",  # Publish HTML docs in GitHub Pages
@@ -64,7 +65,7 @@ def about_package(init_posixpath: pathlib.Path) -> dict:
 ]
 
 # Config for autosectionlabel extension
-autosectionlabel_prefix_document = True
+autosectionlabel_prefix_document = True  # Make sure the target is unique
 autosectionlabel_maxdepth = 2
 
 # Config for the napoleon extension
@@ -138,7 +139,7 @@ def about_package(init_posixpath: pathlib.Path) -> dict:
 
 html_theme_options = {
     "collapse_navigation": False,
-    "display_version": True,
+    "version_selector": True,  # replaces 'display_version' since sphinx-rtd-theme 3.0 but only works on ReadTheDocs
     "logo_only": True,
     "navigation_depth": 3,
 }

doc/index.rst

@@ -1,29 +1,30 @@
 Welcome to tfs-pandas' documentation!
 =====================================
 
-``tfs-pandas`` is a library for reading and writing capabilities for `TFS files <http://mad.web.cern.ch/mad/madx.old/Introduction/tfs.html>`_ used at `CERN <https://home.cern/>`_.
+``tfs-pandas`` is a library for reading and writing capabilities for **TFS** files used at `CERN <https://home.cern/>`_, namely by codes such as `MAD-X <https://madx.web.cern.ch/>`_ and `MAD-NG <https://madx.web.cern.ch/releases/madng/html/>`_.
 
-It provides functionality through a ``TfsDataFrame`` object, an extension of the popular **pandas** ``DataFrame``, which in addition to the normal behaviour attaches a dictionary of headers to the ``DataFrame``.
+It provides functionality through a ``TfsDataFrame`` object, an extension of the popular **pandas** ``DataFrame``, which in addition to the normal behaviour attaches a dictionary of headers to the dataframe.
 Functions are also exported that handle reading and writing of **TFS** files to and from ``TfsDataFrames`` as well as merging and validating for ``TfsDataFrames``.
 
 .. admonition:: **Package Scope**
 
    The package only has as a goal to provide a simple and easy to use interface from **TFS** files to a familiar object build upon the `pandas.DataFrame`.
    It is not meant to implement various calculations on `TfsDataFrames`.
 
-   Tools relative to the **TFS** format are provided, such as validating a `~.TfsDataFrame` and its headers; or lazily manage a collection of **TFS** files.
+   Tools relative to the **TFS** format are provided, such as validating a `TfsDataFrame` and its headers; or lazily managing a collection of **TFS** files.
 
 
 Installation
 ============
 
-Installation is easily done via `pip`:
+The package is published on `PyPI` and installation is easily done via `pip`:
 
 .. code-block:: bash
 
    python -m pip install tfs-pandas
 
-One can also install in a `conda`/`mamba` environment via the `conda-forge` channel with:
+There is also a maintained version of the package on `conda-forge`.
+One can install in a `conda`/`mamba` environment via the `conda-forge` channel with:
 
 .. code-block:: bash
 
@@ -40,6 +41,8 @@ Contents
    :maxdepth: 2
 
    quickstart
+   tfsformat
+   compatibility
    modules/index
 
 

doc/quickstart.rst

@@ -4,10 +4,6 @@
 Yes, 2 minutes.
 That's how little it takes!
 
-.. hint::
-
-   You can click the function names in the code examples below to go directly to their documentation.
-
 Basic Usage
 -----------
 
@@ -44,7 +40,7 @@ Compression
 
 A **TFS** file being text-based, it benefits heavily from compression.
 Thankfully, `tfs-pandas` supports automatic reading and writing of various compression formats.
-Just use the API as you would normally, and the compression will be handled automatically:
+Just use the API as you would normally, and the compression will be handled automatically based on the extension:
 
 .. autolink-preface:: import tfs
 .. code-block:: python
@@ -60,7 +56,7 @@ First though, one needs to install the package with the `hdf5` extra requirement
 
 .. code-block:: bash
 
-   python -m pip install --upgrade tfs-pandas[hdf5]
+   python -m pip install --upgrade "tfs-pandas[hdf5]"
 
 Then, access the functionality from `tfs.hdf`.
 
@@ -70,13 +66,22 @@ Then, access the functionality from `tfs.hdf`.
    from tfs.hdf import read_hdf, write_hdf
    
    # Read a TfsDataFrame from an HDF5 file
-   df = tfs.hdf.read("path_to_input.hdf5", key="key_in_hdf5_file")
+   df = tfs.hdf.read_hdf("path_to_input.hdf5", key="key_in_hdf5_file")
 
    # Write a TfsDataFrame to an HDF5 file
-   tfs.hdf.write("path_to_output.hdf5", df, key="key_in_hdf5_file")
+   tfs.hdf.write_hdf("path_to_output.hdf5", df, key="key_in_hdf5_file")
+
+Validation
+----------
+
+As **TFS** files typically come from the output of simulations codes, validation modes are available to ensure compatibility with said codes.
+This is done through the `tfs.frame.validate` function, or relevant arguments in both the reader and writer functions.
+
+As validation modes and compatibility details are complex, validation warrants its own documentation page.
+Please refer to the :doc:`compatibility and validation guide <compatibility>` for more information.
 
-Compatibility
--------------
+Function Replacements
+---------------------
 
 Finally, some replacement functions are provided for some `pandas` operations which, if used, would return a `pandas.DataFrame` instead of a `~.TfsDataFrame`.
 
@@ -89,9 +94,9 @@ Finally, some replacement functions are provided for some `pandas` operations wh
    # This returns a pandas.DataFrame and makes you lose the headers
    result = pd.concat([df1, df2])
 
-   # Instead, use our own
+   # Instead, use our own wrapper
    result = tfs.frame.concat([df1, df2])  # you can choose how to merge headers too
    assert isinstance(result, tfs.TfsDataFrame)  # that's ok!
+   assert getattr(result, "headers", None) is not None  # headers are not lost
 
 That's it!
-Happy using :)