Release 4.0.0 - MAD-NG Compatibility

Version 4.0 of tfs-pandas is a major release bringing compatibility with MAD-NG features in TFS files and tables, apart from the most exotic ones (more on that later).

We also bring new documentation pages regarding the TFS format, code compatibilities, and the new features.
Please have a look at the documentation should you intent to use tfs-pandas 4.0, as there are a few (now documented) caveats.

Important:

• Support for Python 3.9 has been dropped. The minimum required Python version is now 3.10.
• DataFrame validation is now OFF by default, both when reading from and writing to file. It is up to the user to ask for a given validation mode.
• Minimum supported MAD-NG version is 1.0.0, due to synchronized development of some feature details. The corresponding pymadng version is 0.6.0.

Added:

• Handling of boolean-type and complex-type header values (MAD-NG feature).
• Handling of bolean-type and complex-type columns (MAD-NG feature).
• Handling of nullable-type nil values in headers and columns (MAD-NG feature).
• Compatibility modes for dataframe validation. The tfs.frame.validate function now expects this, and valid choices are MADX, MAD-X, MADNG and MAD-NG (case-insensitive, see API documentation).
• Many new exceptions have been created for raised errors, which will be more specific. They all inherit from the previously raised TfsFormatError, so user code that was catching it will still work.

Changed:

• By default, TfsDataFrame validation is now skipped on reading.
• By default, TfsDataFrame validation is now skipped on writing.
• By default, TfsDataFrame validation in MAD-X requires headers to be present in the dataframe.

Fixed:

• Writing a dataframe with no headers (not empty headers), e.g. a pandas.DataFrame - now works correctly.

Documentation:

• The documentation has been updated to reflect the new features and changes.
• The documentation now includes a new page on the TFS format itself.
• The documentation now includes a new page on compatibility modes for TfsDataFrame validation.
• A great deal of internal documentation has been added to the codebase.

Relevant Changes

• MAD-NG Features and Compatibility Modes by @fsoubelet in #135

Full Changelog: v3.9.0...v4.0.0

Release 3.9.0

Release 3.9.0 brings a helper function into the public API for testing purposes.

Additions

• A module, tfs.testing, has been added and made publicly available. It provides an assert function to compare TfsDataFrame similar to that provided by pandas, destined for unit tests.

What's Changed

• Unit Testing Helper by @fsoubelet in #136

Full Changelog: v3.8.2...v3.9.0

Release 3.8.2

Release 3.8.2 brings two minor changes, which are fully transparent to the user.

Changes

• The headers of a TfsDataFrame are now stored as a dict and no longer an OrderedDict. This should save some memory.
• Removed a workaround function for some pandas behaviour which is no longer necessary due to the higher minimum pandas version.

What's Changed

• Remove unnecessary function and lots of linting by @fsoubelet in #132
• Use dicts instead of OrderedDicts by @fsoubelet in #133

Full Changelog: v3.8.1...v3.8.2

Release 3.8.1

Release 3.8.1 is a patch version fixing compatibility with numpy 2.x. Please note there is a caveat (read below) on Python 3.9 when installing tfs-pandas with the hdf5 extra.

Important Changes:

• The minimum required numpy version is now numpy 1.24.

Fixes

• The package's HDF functionality is fully compatible with numpy 2.x on Python >= 3.10 thanks to a pytables compatibility release.
• The package's HDF functionality limits to numpy < 2 on Python 3.9 due to the lack of compatibility from pytables on this versions.

What's Changed

• Migrate to pyproject.toml by @fsoubelet in #131

Full Changelog: v3.8.0...v3.8.1

Release 3.8.0

Release 3.8.0 is a new minor version fixing warnings, updating to newer versions of our dependencies, and changing some minimum requirements.

Important Changes:

• The minimum required pandas version is now pandas 2.1.
• Support for Python 3.8 has been dropped. The minimum required Python version is now 3.9.

Fixes

• Solved a DeprecationWarning appearing when writing a TfsDataFrame to disk due to the use of .applymap, byusing the now recommended .map method.
• Solved a DeprecationWarning appearing when reading a file from disk due to the use of delim_whitespace in our reader, by using the now recommended sep option.
• Solved a FutureWarning appearing when validating a TfsDataFrame due to the use of the pd.option_context('mode.use_inf_as_na', True) context manager during validation by explicitely casting infinite values to NaNs.
• Solved a FutureWarning appearing when validating a TfsDataFrame due to object downcasting happening during validation by explicitely infering dtypes first.

What's Changed

• Deprecations and FutureWarnings Fixes by @fsoubelet in #128

Full Changelog: v3.7.3...v3.8.0

Release 3.7.3

This is a patch release which fixes a regression introduced in a previous release.

• Fixed:
  • Fixed a regression where the writing of a pd.Series-like object to disk was raising an error. It is now possible again.

What's Changed

• CI Fix by @fsoubelet in #127
• [Fix] Series-like writing by @fsoubelet in #130

Full Changelog: v3.7.2...v3.7.3

Release 3.7.2

This is a patch release, assuring compatibility again with pandas >= 2.1.0.

• Fixed:
  • fixing the issues with pandas >= v2.1.0 (see tfs-pandas v3.7.1) by overwriting the _constructor_from_mgr function.

What's Changed

• Fixing pandas v2.1.0 compatibility by @JoschD in #126

Full Changelog: v3.7.1...v3.7.2

Release 3.7.1

Release 3.7.1 is a patch release to circumvent an issue with the latest pandas release, 2.1.0, which would cause an infinite recursion . This version and above are now prohibited in the dependencies while we investigate with the pandas devs.

What's Changed

• Migrate to common workflows by @fsoubelet in #123
• Pandas dependency version restriction by @fsoubelet in #125

Full Changelog: v3.7.0...v3.7.1

Release 3.7.0

Release 3.7.0 adds some functionality with a reworked TfsCollection class.

• Changed:

  • The old write_to and get_filename of the TfsCollection class have been renamed to _write_to and _get_filename as they
    could only be accessed internally (due to the input parameters not available to the user).
  • The column which is set as index can now also be defined manually by overwriting the INDEX attribute, which defaults to "NAME".
  • The define properties functions have been moved into the Tfs-attribute marker class.
  • The MaybeCall class returns None in case of attribute not found, instead of previously an empty function.

• Added:

  • A get_filename(name) function to TfsCollection which returns the associated filename to the property with name name.
  • A get_path(name) function to TfsCollection which returns the actual file path of the property name.
  • A flush() function to TfsCollection which writes the current state of the TfsDataFrames into their respective files.
  • A write_tfs(filename, data_frame) function to TfsCollection which writes the data_frame to self.directory with the given filename.
  • A defined_properties property to TfsCollection which returns a tuple of strings of the defined properties on this instance.
  • A filenames property to TfsCollection which is a convenience wrapper for get_filename():

For details see the API documentation.

What's Changed

• TFSCollections cleaned and with paths by @JoschD in #122

Full Changelog: v3.6.0...v3.7.0

Release 3.6.0

Release 3.6.0 is a version compatibility release with pandas 2.0.

• Removed:

  • The append and join methods of TfsDataFrame have been removed.

• Changed:

  • The dependency version on pandas has been restored to >=1.0.0 as the above removal restores compatibility with pandas 2.0.

What's Changed

• Release 3.6.0 and removal of TfsDataFrame methods by @fsoubelet in #120

Full Changelog: v3.5.3...v3.6.0