3.0: Merging functions and reorganized structure (#87)

Author: fsoubelet

README.md

@@ -1,61 +1,49 @@
 # TFS-Pandas
+
 [![Cron Testing](https://github.com/pylhc/tfs/workflows/Cron%20Testing/badge.svg)](https://github.com/pylhc/tfs/actions?query=workflow%3A%22Cron+Testing%22)
 [![Code Climate coverage](https://img.shields.io/codeclimate/coverage/pylhc/tfs.svg?style=popout)](https://codeclimate.com/github/pylhc/tfs)
 [![Code Climate maintainability (percentage)](https://img.shields.io/codeclimate/maintainability-percentage/pylhc/tfs.svg?style=popout)](https://codeclimate.com/github/pylhc/tfs)
 [![GitHub last commit](https://img.shields.io/github/last-commit/pylhc/tfs.svg?style=popout)](https://github.com/pylhc/tfs/)
 [![GitHub release](https://img.shields.io/github/release/pylhc/tfs.svg?style=popout)](https://github.com/pylhc/tfs/)
 [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.5070986.svg)](https://doi.org/10.5281/zenodo.5070986)
 
-This package provides reading and writing functionality for **table format system (tfs)** files. 
-
-## Getting Started
-
-### Prerequisites
+This package provides reading and writing functionality for [**table format system (tfs)** files](http://mad.web.cern.ch/mad/madx.old/Introduction/tfs.html). 
+Files are read into a `TfsDataFrame`, a class built on top of the famous `pandas.DataFrame`, which in addition to the normal behaviour attaches an `OrderedDict` of headers to the `DataFrame`.
 
-The package depends heavily on `pandas` and also on `numpy`, so these packages need
-to be installed in your python environment.
+## Installing
 
-### Installing
-
-Installation is easily done via `pip`. The package is then used as `tfs`.
+Installation is easily done via `pip`:
 
 ```
 pip install tfs-pandas
 ```
 
-Example:
+## Example Usage
 
+ The package is imported as `tfs`, and exports top-level functions for reading and writing:
 ```python
 import tfs
 
-data_frame = tfs.read('path_to_input.tfs', index="index_column")
-tfs.write('path_to_output.tfs', data_frame, save_index="index_column")
-```
-## Description
+# Loading a TFS file is simple
+data_frame = tfs.read("path_to_input.tfs", index="index_column")
 
-Reading and writing capabilities for [tfs-files](http://mad.web.cern.ch/mad/madx.old/Introduction/tfs.html)
-are provided by this package. The files are read into special `pandas` `Dataframes`, called `TfsDataFrames`,
-which in addition to the normal behaviour attach an `OrderedDict` of headers to the `DataFrame`.
+# You can access and modify the headers with the .headers attribute
+useful_variable = data_frame.headers["SOME_KEY"]
+data_frame.headers["NEW_KEY"] = some_variable
 
-### Changelog
+# Manipulate data as you do with pandas DataFrames
+data_frame["NEWCOL"] = data_frame.COL_A * data_frame.COL_B
 
-See the [CHANGELOG](CHANGELOG.md) file.
+# You can check the validity of a TfsDataFrame, and choose the behavior in case of errors
+tfs.frame.validate(data_frame, non_unique_behavior="raise")  # or choose "warn"
 
-## Known Issues
-
-- Creating a new `DataFrame` by combining multiple `TfsDataFrame`,
-for example via `pandas.DataFrame.append()` or `pandas.concat()`, 
-will convert the `TfsDataFrame` back to a `DataFrame` and lose therefore the headers.
-
-## Authors
-
-* **Jaime** - [jaimecp89](https://github.com/jaimecp89)
-* **Lukáš** - [lmalina](https://github.com/lmalina)
-* **Josch** - [JoschD](https://github.com/JoschD)
-* **Felix** - [fsoubelet](https://github.com/fsoubelet)
-* **pyLHC/OMC-Team** - *Working Group* - [pyLHC](https://github.com/orgs/pylhc/teams/omc-team)
+# Writing out to disk is simple too
+tfs.write("path_to_output.tfs", data_frame, save_index="index_column")
+```
 
+It also provides some tools to validate and manipulate `TfsDataFrames` and their headers; or lazily manage a collection of TFS files.
+See the [API documentation](https://pylhc.github.io/tfs/) for details.
 
 ## License
 
-This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+This project is licensed under the `MIT License` - see the [LICENSE](LICENSE) file for details.

doc/conf.py

@@ -79,9 +79,14 @@ def about_package(init_posixpath: pathlib.Path) -> dict:
 
 # General information about the project.
 project = ABOUT_TFS["__title__"]
-copyright_ = "2018-2020, pyLHC/OMC-TEAM"
+copyright_ = "2018-2021, pyLHC/OMC-TEAM"
 author = ABOUT_TFS["__author__"]
 
+# Override link in 'Edit on Github'
+rst_prolog = f"""
+:github_url: {ABOUT_TFS['__url__']}
+"""
+
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the
 # built documents.
@@ -117,15 +122,22 @@ def about_package(init_posixpath: pathlib.Path) -> dict:
 html_theme = "sphinx_rtd_theme"
 
 html_theme_options = {
-    'collapse_navigation': False,
-    'display_version': True,
-    'logo_only': True,
-    'navigation_depth': 2,
+    "collapse_navigation": False,
+    "display_version": True,
+    "logo_only": True,
+    "navigation_depth": 2,
 }
 
-html_logo = '_static/img/omc_logo.svg'
-html_static_path = ['_static']
-html_context = {'css_files': ['_static/css/custom.css']}
+html_logo = "_static/img/omc_logo.svg"
+html_static_path = ["_static"]
+html_context = {
+    "css_files": ["_static/css/custom.css"],
+    "display_github": True,
+    # the following are only needed if :github_url: is not set
+    "github_user": author,
+    "github_repo": project,
+    "github_version": "master/doc/",
+}
 
 smartquotes_action = "qe"  # renders only quotes and ellipses (...) but not dashes (option: D)
 

doc/index.rst

@@ -3,8 +3,8 @@ Welcome to tfs-pandas' documentation!
 
 ``tfs-pandas`` is a library for reading and writing capabilities for `TFS files <http://mad.web.cern.ch/mad/madx.old/Introduction/tfs.html>`_ used at `CERN <https://home.cern/>`_.
 
-It provides an extension of the popular ``pandas`` `Dataframes`, called `TfsDataFrames` to do so, which in addition to the normal behaviour attach an `OrderedDict` of headers to the `DataFrame`.
-
+It provides functionality through a ``TfsDataFrame`` object, an extension of the popular **pandas** ``DataFrame``, which in addition to the normal behaviour attaches a dictionary of headers to the ``DataFrame``.
+Functions are also exported that handle reading and writing of **TFS** files to and from ``TfsDataFrames`` as well as merging and validating for ``TfsDataFrames``.
 
 Package Reference
 =================

doc/modules/index.rst

@@ -5,10 +5,26 @@ TFS-Pandas Modules
     :members:
 
 
-.. automodule:: tfs.handler
+.. automodule:: tfs.constants
+    :members:
+
+
+.. automodule:: tfs.errors
+    :members:
+
+
+.. automodule:: tfs.frame
+    :members:
+
+
+.. automodule:: tfs.reader
     :members:
 
 
 .. automodule:: tfs.tools
     :members:
 
+
+.. automodule:: tfs.writer
+    :members:
+

setup.py

@@ -27,7 +27,6 @@ def about_package(init_posixpath: pathlib.Path) -> dict:
 with README.open("r") as docs:
     long_description = docs.read()
 
-
 # Dependencies for the package itself
 DEPENDENCIES = [
     "numpy>=1.19.0",
@@ -36,12 +35,10 @@ def about_package(init_posixpath: pathlib.Path) -> dict:
 
 # Extra dependencies
 EXTRA_DEPENDENCIES = {
-    "test": ["pytest>=5.2", "pytest-cov>=2.9"],
+    "test": ["pytest>=5.2", "pytest-cov>=2.9", "cpymad>=1.8.1"],
     "doc": ["sphinx", "sphinx_rtd_theme"],
 }
-EXTRA_DEPENDENCIES.update(
-    {"all": [elem for list_ in EXTRA_DEPENDENCIES.values() for elem in list_]}
-)
+EXTRA_DEPENDENCIES.update({"all": [elem for list_ in EXTRA_DEPENDENCIES.values() for elem in list_]})
 
 setuptools.setup(
     name=ABOUT_TFS["__title__"],

tests/inputs/file_x.tfs

@@ -1,5 +1,5 @@
-@ TITLE %s "Title of tfs file"
-@ DPP %le 0.0
+@ TITLE %s "Title of your tfs file"
+@ DPP %le 1.0
 @ Q1 %le 0.269974877952
 @ Q1RMS %le 1.75642567736e-07
 @ NATQ1 %le 0.280041400187

tests/inputs/file_y.tfs

@@ -1,10 +1,10 @@
-@ TITLE %s "Title of tfs file"
+@ TITLE %s "Title of this tfs file"
 @ DPP %le 0.0
 @ Q1 %le 0.289974877952
 @ Q1RMS %le 1.35642567736e-07
 @ NATQ1 %le 0.680041400187
-@ NATQ1RMS %le 0.00102479265802
-@ BPMCOUNT %d 9
+@ NATQ1RMS %le 0.00107479265802
+@ BPMCOUNT %d 12
 # another comment line here
 *            NAME                 S            NUMBER                CO             CORMS           BPM_RES
 $              %s               %le                %d               %le               %le               %le

tests/test_collection.py

@@ -1,13 +1,12 @@
 import os
 import pathlib
-import tempfile
 
 import pytest
 from pandas.testing import assert_frame_equal
 
 from tfs import read_tfs
 from tfs.collection import Tfs, TfsCollection
-from tfs.handler import TfsDataFrame
+from tfs.frame import TfsDataFrame
 
 CURRENT_DIR = pathlib.Path(__file__).parent
 
@@ -27,9 +26,7 @@ def get_filename(self, template, plane=""):
         return template.format(plane)
 
 
-def test_read_pathlib_input(
-    _input_dir_pathlib: pathlib.Path, _tfs_x: TfsDataFrame, _tfs_y: TfsDataFrame
-):
+def test_read_pathlib_input(_input_dir_pathlib: pathlib.Path, _tfs_x: TfsDataFrame, _tfs_y: TfsDataFrame):
     c = CollectionTest(_input_dir_pathlib)
     assert_frame_equal(_tfs_x, c.file_x)
     assert_frame_equal(_tfs_x, c.filex)
@@ -53,9 +50,9 @@ def test_read_str_input(_input_dir_str: str, _tfs_x: TfsDataFrame, _tfs_y: TfsDa
     assert c.value == 10
 
 
-def test_write(_tfs_x: TfsDataFrame, _tfs_y: TfsDataFrame, _output_dir: str):
-    c = CollectionTest(_output_dir)
-    file_x_path = pathlib.Path(_output_dir) / "nofile_x.tfs"
+def test_write(_tfs_x: TfsDataFrame, _tfs_y: TfsDataFrame, tmp_path):
+    c = CollectionTest(tmp_path)
+    file_x_path = tmp_path / "nofile_x.tfs"
     assert not file_x_path.is_file()
 
     c.nofile_x = _tfs_x  # will not throw error, but does nothing
@@ -67,7 +64,7 @@ def test_write(_tfs_x: TfsDataFrame, _tfs_y: TfsDataFrame, _output_dir: str):
     assert_frame_equal(_tfs_x, c.nofile_x)
 
     c.nofile["y"] = _tfs_y
-    file_y_path = pathlib.Path(_output_dir) / "nofile_y.tfs"
+    file_y_path = tmp_path / "nofile_y.tfs"
     assert file_y_path.is_file()
     assert_frame_equal(_tfs_y, c.nofile["y"])
     assert_frame_equal(_tfs_y, c.nofile["Y"])
@@ -119,12 +116,6 @@ def _input_dir_str() -> str:
     return os.path.join(os.path.dirname(__file__), "inputs")
 
 
-@pytest.fixture()
-def _output_dir() -> str:
-    with tempfile.TemporaryDirectory() as cwd:
-        yield cwd
-
-
 @pytest.fixture()
 def _dummy_collection() -> TfsCollection:
     return TfsCollection("")