Merge pull request #412 from seperman/dev

6.4.0

Author: seperman

CHANGELOG.md

@@ -1,5 +1,11 @@
 # DeepDiff Change log
 
+- v6-4-0
+    - [Add Ignore List Order Option to DeepHash](https://github.com/seperman/deepdiff/pull/403) by 
+[Bobby Morck](https://github.com/bmorck)
+    - [pyyaml to 6.0.1 to fix cython build problems](https://github.com/seperman/deepdiff/pull/406) by [Robert Bo Davis](https://github.com/robert-bo-davis)
+    - [Precompiled regex simple diff](https://github.com/seperman/deepdiff/pull/413) by [cohml](https://github.com/cohml)
+    - New flag: `zip_ordered_iterables` for forcing iterable items to be compared one by one. 
 - v6-3-1
     - Bugfix deephash for paths by [maggelus](https://github.com/maggelus)
     - Bugfix deephash compiled regex [maggelus](https://github.com/maggelus)

README.md

@@ -1,4 +1,4 @@
-# DeepDiff v 6.3.1
+# DeepDiff v 6.4.0
 
 ![Downloads](https://img.shields.io/pypi/dm/deepdiff.svg?style=flat)
 ![Python Versions](https://img.shields.io/pypi/pyversions/deepdiff.svg?style=flat)
@@ -17,31 +17,20 @@
 
 Tested on Python 3.7+ and PyPy3.
 
-- **[Documentation](https://zepworks.com/deepdiff/6.3.1/)**
+- **[Documentation](https://zepworks.com/deepdiff/6.4.0/)**
 
 ## What is new?
 
 Please check the [ChangeLog](CHANGELOG.md) file for the detailed information.
 
-DeepDiff 6-3-1
+DeepDiff 6-4-0
 
-This release includes many bug fixes.
+- [Add Ignore List Order Option to DeepHash](https://github.com/seperman/deepdiff/pull/403) by 
+[Bobby Morck](https://github.com/bmorck)
+- [pyyaml to 6.0.1 to fix cython build problems](https://github.com/seperman/deepdiff/pull/406) by [Robert Bo Davis](https://github.com/robert-bo-davis)
+- [Precompiled regex simple diff](https://github.com/seperman/deepdiff/pull/413) by [cohml](https://github.com/cohml)
+- New flag: `zip_ordered_iterables` for forcing iterable items to be compared one by one. 
 
-- Bugfix deephash for paths by [maggelus](https://github.com/maggelus)
-- Bugfix deephash compiled regex [maggelus](https://github.com/maggelus)
-- Fix tests dependent on toml by [martin-kokos](https://github.com/martin-kokos)
-- Bugfix for `include_paths` for nested dictionaries by [kor4ik](https://github.com/kor4ik)
-- Use tomli and tomli-w for dealing with tomli files by [martin-kokos](https://github.com/martin-kokos)
-- Bugfix for `datetime.date` by [Alex Sauer-Budge](https://github.com/amsb)
-
-
-DeepDiff 6-3-0
-
-- [`PrefixOrSuffixOperator`](https://zepworks.com/deepdiff/current/custom.html#prefix-or-suffix-operator-label): This operator will skip strings that are suffix or prefix of each other.
-- [`include_obj_callback`](https://zepworks.com/deepdiff/current/ignore_types_or_values.html#include-obj-callback-label) and `include_obj_callback_strict` are added by [Håvard Thom](https://github.com/havardthom).
-- Fixed a corner case where numpy's `np.float32` nans are not ignored when using `ignore_nan_equality` by [Noam Gottlieb](https://github.com/noamgot)
-- `orjson` becomes optional again.
-- Fix for `ignore_type_in_groups` with numeric values so it does not report number changes when the number types are different.
 
 ## Installation
 
@@ -93,11 +82,11 @@ Thank you!
 
 How to cite this library (APA style):
 
-    Dehpour, S. (2023). DeepDiff (Version 6.3.1) [Software]. Available from https://github.com/seperman/deepdiff.
+    Dehpour, S. (2023). DeepDiff (Version 6.4.0) [Software]. Available from https://github.com/seperman/deepdiff.
 
 How to cite this library (Chicago style):
 
-    Dehpour, Sep. 2023. DeepDiff (version 6.3.1).
+    Dehpour, Sep. 2023. DeepDiff (version 6.4.0).
 
 # Authors
 

deepdiff/__init__.py

@@ -1,6 +1,6 @@
 """This module offers the DeepDiff, DeepSearch, grep, Delta and DeepHash classes."""
 # flake8: noqa
-__version__ = '6.3.1'
+__version__ = '6.4.0'
 import logging
 
 if __name__ == '__main__':

deepdiff/deephash.py

@@ -144,6 +144,7 @@ def __init__(self,
                  parent="root",
                  encodings=None,
                  ignore_encoding_errors=False,
+                 ignore_iterable_order=True,
                  **kwargs):
         if kwargs:
             raise ValueError(
@@ -190,6 +191,7 @@ def __init__(self,
         self.ignore_private_variables = ignore_private_variables
         self.encodings = encodings
         self.ignore_encoding_errors = ignore_encoding_errors
+        self.ignore_iterable_order = ignore_iterable_order
 
         self._hash(obj, parent=parent, parents_ids=frozenset({get_id(obj)}))
 
@@ -424,7 +426,9 @@ def _prep_iterable(self, obj, parent, parents_ids=EMPTY_FROZENSET):
                 '{}|{}'.format(i, v) for i, v in result.items()
             ]
 
-        result = sorted(map(str, result))  # making sure the result items are string and sorted so join command works.
+        result = map(str, result) # making sure the result items are string so join command works.
+        if self.ignore_iterable_order:
+            result = sorted(result)  
         result = ','.join(result)
         result = KEY_TO_VAL_STR.format(type(obj).__name__, result)
 

deepdiff/diff.py

@@ -13,6 +13,7 @@
 from math import isclose as is_close
 from collections.abc import Mapping, Iterable, Sequence
 from collections import defaultdict
+from inspect import getmembers
 from itertools import zip_longest
 from ordered_set import OrderedSet
 from deepdiff.helper import (strings, bytes_type, numbers, uuids, datetimes, ListItemRemovedOrAdded, notpresent,
@@ -142,6 +143,7 @@ def __init__(self,
                  ignore_type_in_groups=None,
                  ignore_type_subclasses=False,
                  iterable_compare_func=None,
+                 zip_ordered_iterables=False,
                  log_frequency_in_sec=0,
                  math_epsilon=None,
                  max_diffs=None,
@@ -166,7 +168,7 @@ def __init__(self,
                 "number_format_notation, exclude_paths, include_paths, exclude_types, exclude_regex_paths, ignore_type_in_groups, "
                 "ignore_string_type_changes, ignore_numeric_type_changes, ignore_type_subclasses, truncate_datetime, "
                 "ignore_private_variables, ignore_nan_inequality, number_to_string_func, verbose_level, "
-                "view, hasher, hashes, max_passes, max_diffs, "
+                "view, hasher, hashes, max_passes, max_diffs, zip_ordered_iterables, "
                 "cutoff_distance_for_pairs, cutoff_intersection_for_pairs, log_frequency_in_sec, cache_size, "
                 "cache_tuning_sample_size, get_deep_distance, group_by, cache_purge_level, "
                 "math_epsilon, iterable_compare_func, _original_type, "
@@ -208,6 +210,7 @@ def __init__(self,
             self.include_obj_callback_strict = include_obj_callback_strict
             self.number_to_string = number_to_string_func or number_to_string
             self.iterable_compare_func = iterable_compare_func
+            self.zip_ordered_iterables = zip_ordered_iterables
             self.ignore_private_variables = ignore_private_variables
             self.ignore_nan_inequality = ignore_nan_inequality
             self.hasher = hasher
@@ -415,20 +418,25 @@ def _diff_enum(self, level, parents_ids=frozenset(), local_tree=None):
 
     def _diff_obj(self, level, parents_ids=frozenset(), is_namedtuple=False, local_tree=None):
         """Difference of 2 objects"""
+        processing_error = False
         try:
             if is_namedtuple:
                 t1 = level.t1._asdict()
                 t2 = level.t2._asdict()
-            else:
+            elif all('__dict__' in dir(t) for t in level):
                 t1 = detailed__dict__(level.t1, ignore_private_variables=self.ignore_private_variables)
                 t2 = detailed__dict__(level.t2, ignore_private_variables=self.ignore_private_variables)
-        except AttributeError:
-            try:
+            elif all('__slots__' in dir(t) for t in level):
                 t1 = self._dict_from_slots(level.t1)
                 t2 = self._dict_from_slots(level.t2)
-            except AttributeError:
-                self._report_result('unprocessed', level, local_tree=local_tree)
-                return
+            else:
+                t1 = {k: v for k, v in getmembers(level.t1) if not callable(v)}
+                t2 = {k: v for k, v in getmembers(level.t2) if not callable(v)}
+        except AttributeError:
+            processing_error = True
+        if processing_error is True:
+            self._report_result('unprocessed', level, local_tree=local_tree)
+            return
 
         self._diff_dict(
             level,
@@ -655,7 +663,6 @@ def _compare_in_order(
         Default compare if `iterable_compare_func` is not provided.
         This will compare in sequence order.
         """
-
         if t1_from_index is None:
             return [((i, i), (x, y)) for i, (x, y) in enumerate(
                 zip_longest(
@@ -743,7 +750,8 @@ def _diff_iterable_in_order(self, level, parents_ids=frozenset(), _original_type
             child_relationship_class = NonSubscriptableIterableRelationship
 
         if (
-            isinstance(level.t1, Sequence)
+            not self.zip_ordered_iterables
+            and isinstance(level.t1, Sequence)
             and isinstance(level.t2, Sequence)
             and self._all_values_basic_hashable(level.t1)
             and self._all_values_basic_hashable(level.t2)
@@ -874,7 +882,8 @@ def _diff_by_forming_pairs_and_comparing_one_by_one(
                     x,
                     y,
                     child_relationship_class=child_relationship_class,
-                    child_relationship_param=j)
+                    child_relationship_param=j
+                )
                 self._diff(next_level, parents_ids_added, local_tree=local_tree)
 
     def _diff_ordered_iterable_by_difflib(
@@ -1527,7 +1536,7 @@ def _diff(self, level, parents_ids=frozenset(), _original_type=None, local_tree=
         if isinstance(level.t1, booleans):
             self._diff_booleans(level, local_tree=local_tree)
 
-        if isinstance(level.t1, strings):
+        elif isinstance(level.t1, strings):
             self._diff_str(level, local_tree=local_tree)
 
         elif isinstance(level.t1, datetimes):

deepdiff/helper.py

@@ -108,6 +108,40 @@ class pydantic_base_model_type:
 
 NUMERICS = frozenset(string.digits)
 
+
+def _int_or_zero(value):
+    """
+    Tries to extract some number from a string.
+
+    12c becomes 12
+    """
+    try:
+        return int(value)
+    except Exception:
+        result = []
+        for char in value:
+            if char in NUMERICS:
+                result.append(char)
+        if result:
+            return int(''.join(result))
+        return 0
+
+
+def get_semvar_as_integer(version):
+    """
+    Converts:
+
+    '1.23.5' to 1023005
+    """
+    version = version.split('.')
+    if len(version) > 3:
+        version = version[:3]
+    elif len(version) < 3:
+        version.extend(['0'] * (3 - len(version)))
+
+    return sum([10**(i * 3) * _int_or_zero(v) for i, v in enumerate(reversed(version))])
+
+
 # we used to use OrderedDictPlus when dictionaries in Python were not ordered.
 dict_ = dict
 
@@ -120,6 +154,10 @@ class pydantic_base_model_type:
 
 pypy3 = py3 and hasattr(sys, "pypy_translation_info")
 
+
+if get_semvar_as_integer(np.__version__) < 1019000:
+    sys.exit('The minimum required Numpy version is 1.19.0. Please upgrade your Numpy package.')
+
 strings = (str, bytes)  # which are both basestring
 unicode_type = str
 bytes_type = bytes
@@ -321,8 +359,8 @@ def type_in_type_group(item, type_group):
 
 def type_is_subclass_of_type_group(item, type_group):
     return isinstance(item, type_group) \
-           or (isinstance(item, type) and issubclass(item, type_group)) \
-           or type_in_type_group(item, type_group)
+        or (isinstance(item, type) and issubclass(item, type_group)) \
+        or type_in_type_group(item, type_group)
 
 
 def get_doc(doc_filename):

deepdiff/model.py

@@ -577,6 +577,10 @@ def __setattr__(self, key, value):
         else:
             self.__dict__[key] = value
 
+    def __iter__(self):
+        yield self.t1
+        yield self.t2
+
     @property
     def repetition(self):
         return self.additional['repetition']

docs/changelog.rst

@@ -5,6 +5,20 @@ Changelog
 
 DeepDiff Changelog
 
+-  v6-4-0
+
+   -  `Add Ignore List Order Option to
+      DeepHash <https://github.com/seperman/deepdiff/pull/403>`__ by
+      `Bobby Morck <https://github.com/bmorck>`__
+   -  `pyyaml to 6.0.1 to fix cython build
+      problems <https://github.com/seperman/deepdiff/pull/406>`__ by
+      `Robert Bo Davis <https://github.com/robert-bo-davis>`__
+   -  `Precompiled regex simple
+      diff <https://github.com/seperman/deepdiff/pull/413>`__ by
+      `cohml <https://github.com/cohml>`__
+   -  New flag: ``zip_ordered_iterables`` for forcing iterable items to
+      be compared one by one.
+
 -  v6-3-1
 
    -  Bugfix deephash for paths by

docs/conf.py

@@ -61,9 +61,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = '6.3.1'
+version = '6.4.0'
 # The full version, including alpha/beta/rc tags.
-release = '6.3.1'
+release = '6.4.0'
 
 load_dotenv(override=True)
 DOC_VERSION = os.environ.get('DOC_VERSION', version)

docs/deephash_doc.rst

@@ -123,6 +123,8 @@ ignore_private_variables: Boolean, default = True
 ignore_encoding_errors: Boolean, default = False
     If you want to get away with UnicodeDecodeError without passing explicit character encodings, set this option to True. If you want to make sure the encoding is done properly, keep this as False and instead pass an explicit list of character encodings to be considered via the encodings parameter.
 
+ignore_iterable_order: Boolean, default = True
+    If order of items in an iterable should not cause the hash of the iterable to be different.
 
 number_format_notation : string, default="f"
     number_format_notation is what defines the meaning of significant digits. The default value of "f" means the digits AFTER the decimal point. "f" stands for fixed point. The other option is "e" which stands for exponent notation or scientific notation.

docs/diff_doc.rst

@@ -129,6 +129,10 @@ ignore_encoding_errors: Boolean, default = False
     :ref:`ignore_encoding_errors_label` If you want to get away with UnicodeDecodeError without passing explicit character encodings, set this option to True. If you want to make sure the encoding is done properly, keep this as False and instead pass an explicit list of character encodings to be considered via the :ref:`encodings_label` parameter.
 
 
+zip_ordered_iterables: Boolean, default = False
+    :ref:`zip_ordered_iterables_label`:
+    When comparing ordered iterables such as lists, DeepDiff tries to find the smallest difference between the two iterables to report. That means that items in the two lists are not paired individually in the order of appearance in the iterables. Sometimes, that is not the desired behavior. Set this flag to True to make DeepDiff pair and compare the items in the iterables in the order they appear.
+
 iterable_compare_func:
     :ref:`iterable_compare_func_label`:
     There are times that we want to guide DeepDiff as to what items to compare with other items. In such cases we can pass a iterable_compare_func that takes a function pointer to compare two items. The function takes three parameters (x, y, level) and should return True if it is a match, False if it is not a match or raise CannotCompare if it is unable to compare the two.

docs/index.rst

@@ -4,7 +4,7 @@
    contain the root `toctree` directive.
 
 
-DeepDiff 6.3.1 documentation!
+DeepDiff 6.4.0 documentation!
 =============================
 
 *******
@@ -32,31 +32,21 @@ What is New
 ***********
 
 
-DeepDiff 6-3-1
+DeepDiff 6-4-0
 --------------
 
-This release includes many bug fixes.
+-  `Add Ignore List Order Option to
+  DeepHash <https://github.com/seperman/deepdiff/pull/403>`__ by
+  `Bobby Morck <https://github.com/bmorck>`__
+-  `pyyaml to 6.0.1 to fix cython build
+  problems <https://github.com/seperman/deepdiff/pull/406>`__ by
+  `Robert Bo Davis <https://github.com/robert-bo-davis>`__
+-  `Precompiled regex simple
+  diff <https://github.com/seperman/deepdiff/pull/413>`__ by
+  `cohml <https://github.com/cohml>`__
+-  New flag: ``zip_ordered_iterables`` for forcing iterable items to
+  be compared one by one.
 
--  Bugfix deephash for paths by `maggelus <https://github.com/maggelus>`__
--  Bugfix deephash compiled regex `maggelus <https://github.com/maggelus>`__
--  Fix tests dependent on toml by `martin-kokos <https://github.com/martin-kokos>`__
--  Bugfix for ``include_paths`` for nested dictionaries by `kor4ik <https://github.com/kor4ik>`__
--  Use tomli and tomli-w for dealing with tomli files by `martin-kokos <https://github.com/martin-kokos>`__
--  Bugfix for ``datetime.date`` by `Alex Sauer-Budge <https://github.com/amsb>`__
-
-
-DeepDiff 6-3-0
---------------
-
--  :ref:`prefix_or_suffix_operator_label`: This operator will skip strings that are
-   suffix or prefix of each other.
--  :ref:`include_obj_callback_label` and :ref:`include_obj_callback_strict_label` are
-   added by `Håvard Thom <https://github.com/havardthom>`__.
--  Fixed a corner case where numpy’s ``np.float32`` nans are not ignored
-   when using ``ignore_nan_equality`` by `Noam
-   Gottlieb <https://github.com/noamgot>`__
--  ``orjson`` becomes optional again.
--  Fix for ``ignore_type_in_groups`` with numeric values so it does not report number changes when the number types are different.
 
 *********
 Tutorials