Merge pull request #245 from seperman/dev

5.5.0

Author: seperman

AUTHORS.md

@@ -35,4 +35,5 @@ Authors in order of the contributions:
 - Florian Klien [flowolf](https://github.com/flowolf) for adding math_epsilon
 - Tim Klein [timjklein36](https://github.com/timjklein36) for retaining the order of multiple dictionary items added via Delta.
 - Wilhelm Schürmann[wbsch](https://github.com/wbsch) for fixing the typo with yml files.
-- [lyz-code](https://github.com/lyz-code) for adding support for regular expressions in DeepSearch.
+- [lyz-code](https://github.com/lyz-code) for adding support for regular expressions in DeepSearch and strict_checking feature in DeepSearch.
+- [dtorres-sf](https://github.com/dtorres-sf)for adding the option for custom compare function

CHANGELOG.md

@@ -1,6 +1,8 @@
 # DeepDiff Change log
 
-- v5-3-0: add support for regular expressions in DeepSearch
+- v5-5-0: adding iterable_compare_func for DeepDiff, adding output_format of list for path() in tree view.
+- v5-4-0: adding strict_checking for numbers in DeepSearch.
+- v5-3-0: add support for regular expressions in DeepSearch.
 - v5-2-3: Retaining the order of multiple dictionary items added via Delta. Fixed the typo with yml files in deep cli. Fixing Grep RecursionError where using non UTF-8 character. Allowing kwargs to be passed to to_json method.
 - v5-2-2: Fixed Delta serialization when None type is present.
 - v5-2-0: Removed Murmur3 as the preferred hashing method. Using SHA256 by default now. Added commandline for deepdiff. Added group_by. Added math_epsilon. Improved ignoring of NoneType.

README.md

@@ -1,4 +1,4 @@
-# DeepDiff v 5.3.0
+# DeepDiff v 5.5.0
 
 ![Downloads](https://img.shields.io/pypi/dm/deepdiff.svg?style=flat)
 ![Python Versions](https://img.shields.io/pypi/pyversions/deepdiff.svg?style=flat)
@@ -18,11 +18,11 @@ Tested on Python 3.6+ and PyPy3.
 
 **NOTE: The last version of DeepDiff to work on Python 3.5 was DeepDiff 5-0-2**
 
-- [Documentation](https://zepworks.com/deepdiff/5.3.0/)
+- [Documentation](https://zepworks.com/deepdiff/5.5.0/)
 
 ## What is new?
 
-Deepdiff 5.3.0 comes with regular expressions in the DeepSearch and grep modules:
+Deepdiff 5.5.0 comes with regular expressions in the DeepSearch and grep modules:
 
 ```python
 >>> from deepdiff import grep
@@ -66,13 +66,13 @@ Note: if you want to use DeepDiff via commandline, make sure to run `pip install
 
 DeepDiff gets the difference of 2 objects.
 
-> - Please take a look at the [DeepDiff docs](https://zepworks.com/deepdiff/5.3.0/diff.html)
-> - The full documentation of all modules can be found on <https://zepworks.com/deepdiff/5.3.0/>
+> - Please take a look at the [DeepDiff docs](https://zepworks.com/deepdiff/5.5.0/diff.html)
+> - The full documentation of all modules can be found on <https://zepworks.com/deepdiff/5.5.0/>
 > - Tutorials and posts about DeepDiff can be found on <https://zepworks.com/tags/deepdiff/>
 
 ## A few Examples
 
-> Note: This is just a brief overview of what DeepDiff can do. Please visit <https://zepworks.com/deepdiff/5.3.0/> for full documentation.
+> Note: This is just a brief overview of what DeepDiff can do. Please visit <https://zepworks.com/deepdiff/5.5.0/> for full documentation.
 
 ### List difference ignoring order or duplicates
 
@@ -276,8 +276,8 @@ Example:
 ```
 
 
-> - Please take a look at the [DeepDiff docs](https://zepworks.com/deepdiff/5.3.0/diff.html)
-> - The full documentation can be found on <https://zepworks.com/deepdiff/5.3.0/>
+> - Please take a look at the [DeepDiff docs](https://zepworks.com/deepdiff/5.5.0/diff.html)
+> - The full documentation can be found on <https://zepworks.com/deepdiff/5.5.0/>
 
 
 # Deep Search
@@ -309,17 +309,17 @@ And you can pass all the same kwargs as DeepSearch to grep too:
 {'matched_paths': {"root['somewhere']": 'around'}, 'matched_values': {"root['long']": 'somewhere'}}
 ```
 
-> - Please take a look at the [DeepSearch docs](https://zepworks.com/deepdiff/5.3.0/dsearch.html)
-> - The full documentation can be found on <https://zepworks.com/deepdiff/5.3.0/>
+> - Please take a look at the [DeepSearch docs](https://zepworks.com/deepdiff/5.5.0/dsearch.html)
+> - The full documentation can be found on <https://zepworks.com/deepdiff/5.5.0/>
 
 # Deep Hash
 (New in v4-0-0)
 
 DeepHash is designed to give you hash of ANY python object based on its contents even if the object is not considered hashable!
 DeepHash is supposed to be deterministic in order to make sure 2 objects that contain the same data, produce the same hash.
 
-> - Please take a look at the [DeepHash docs](https://zepworks.com/deepdiff/5.3.0/deephash.html)
-> - The full documentation can be found on <https://zepworks.com/deepdiff/5.3.0/>
+> - Please take a look at the [DeepHash docs](https://zepworks.com/deepdiff/5.5.0/deephash.html)
+> - The full documentation can be found on <https://zepworks.com/deepdiff/5.5.0/>
 
 Let's say you have a dictionary object.
 
@@ -367,8 +367,8 @@ Which you can write as:
 At first it might seem weird why DeepHash(obj)[obj] but remember that DeepHash(obj) is a dictionary of hashes of all other objects that obj contains too.
 
 
-> - Please take a look at the [DeepHash docs](https://zepworks.com/deepdiff/5.3.0/deephash.html)
-> - The full documentation can be found on <https://zepworks.com/deepdiff/5.3.0/>
+> - Please take a look at the [DeepHash docs](https://zepworks.com/deepdiff/5.5.0/deephash.html)
+> - The full documentation can be found on <https://zepworks.com/deepdiff/5.5.0/>
 
 
 # Using DeepDiff in unit tests
@@ -421,7 +421,7 @@ And here is more info: <http://zepworks.com/blog/diff-it-to-digg-it/>
 
 # ChangeLog
 
-Please take a look at the [changelog](changelog.md) file.
+Please take a look at the [CHANGELOG](CHANGELOG.md) file.
 
 # Releases
 

conftest.py

@@ -62,3 +62,21 @@ def nested_b_t2():
 def nested_b_result():
     with open(os.path.join(FIXTURES_DIR, 'nested_b_result.json')) as the_file:
         return json.load(the_file)
+
+
+@pytest.fixture(scope='class')
+def compare_func_t1():
+    with open(os.path.join(FIXTURES_DIR, 'compare_func_t1.json')) as the_file:
+        return json.load(the_file)
+
+
+@pytest.fixture(scope='class')
+def compare_func_t2():
+    with open(os.path.join(FIXTURES_DIR, 'compare_func_t2.json')) as the_file:
+        return json.load(the_file)
+
+
+@pytest.fixture(scope='class')
+def compare_func_result1():
+    with open(os.path.join(FIXTURES_DIR, 'compare_func_result1.json')) as the_file:
+        return json.load(the_file)

deepdiff/__init__.py

@@ -1,6 +1,6 @@
 """This module offers the DeepDiff, DeepSearch, grep, Delta and DeepHash classes."""
 # flake8: noqa
-__version__ = '5.3.0'
+__version__ = '5.5.0'
 import logging
 
 if __name__ == '__main__':

deepdiff/delta.py

@@ -260,9 +260,14 @@ def _del_elem(self, parent, parent_to_obj_elem, parent_to_obj_action,
                                         value=obj, action=parent_to_obj_action)
 
     def _do_iterable_item_added(self):
-        iterable_item_added = self.diff.get('iterable_item_added')
+        iterable_item_added = self.diff.get('iterable_item_added', {})
+        iterable_item_moved = self.diff.get('iterable_item_moved')
+        if iterable_item_moved:
+            added_dict = {v["new_path"]: v["value"] for k, v in iterable_item_moved.items()}
+            iterable_item_added.update(added_dict)
+
         if iterable_item_added:
-            self._do_item_added(iterable_item_added)
+            self._do_item_added(iterable_item_added, insert=True)
 
     def _do_dictionary_item_added(self):
         dictionary_item_added = self.diff.get('dictionary_item_added')
@@ -274,7 +279,7 @@ def _do_attribute_added(self):
         if attribute_added:
             self._do_item_added(attribute_added)
 
-    def _do_item_added(self, items, sort=True):
+    def _do_item_added(self, items, sort=True, insert=False):
         if sort:
             # sorting items by their path so that the items with smaller index
             # are applied first (unless `sort` is `False` so that order of
@@ -289,6 +294,11 @@ def _do_item_added(self, items, sort=True):
                 elements, parent, parent_to_obj_elem, parent_to_obj_action, obj, elem, action = elem_and_details
             else:
                 continue  # pragma: no cover. Due to cPython peephole optimizer, this line doesn't get covered. https://github.com/nedbat/coveragepy/issues/198
+
+            # Insert is only true for iterables, make sure it is a valid index.
+            if(insert and elem < len(obj)):
+                obj.insert(elem, None)
+
             self._set_new_value(parent, parent_to_obj_elem, parent_to_obj_action,
                                 obj, elements, path, elem, action, new_value)
 
@@ -397,7 +407,14 @@ def _do_item_removed(self, items):
             self._do_verify_changes(path, expected_old_value, current_old_value)
 
     def _do_iterable_item_removed(self):
-        iterable_item_removed = self.diff.get('iterable_item_removed')
+        iterable_item_removed = self.diff.get('iterable_item_removed', {})
+
+        iterable_item_moved = self.diff.get('iterable_item_moved')
+        if iterable_item_moved:
+            # These will get added back during items_added
+            removed_dict = {k: v["value"] for k, v in iterable_item_moved.items()}
+            iterable_item_removed.update(removed_dict)
+
         if iterable_item_removed:
             self._do_item_removed(iterable_item_removed)
 

deepdiff/diff.py

@@ -21,7 +21,7 @@
                              number_to_string, datetime_normalize, KEY_TO_VAL_STR, booleans,
                              np_ndarray, get_numpy_ndarray_rows, OrderedSetPlus, RepeatedTimer,
                              TEXT_VIEW, TREE_VIEW, DELTA_VIEW,
-                             np, get_truncate_datetime, dict_)
+                             np, get_truncate_datetime, dict_, CannotCompare)
 from deepdiff.serialization import SerializationMixin
 from deepdiff.distance import DistanceMixin
 from deepdiff.model import (
@@ -139,6 +139,7 @@ def __init__(self,
                  truncate_datetime=None,
                  verbose_level=1,
                  view=TEXT_VIEW,
+                 iterable_compare_func=None,
                  _original_type=None,
                  _parameters=None,
                  _shared_parameters=None,
@@ -154,7 +155,8 @@ def __init__(self,
                 "view, hasher, hashes, max_passes, max_diffs, "
                 "cutoff_distance_for_pairs, cutoff_intersection_for_pairs, log_frequency_in_sec, cache_size, "
                 "cache_tuning_sample_size, get_deep_distance, group_by, cache_purge_level, "
-                "math_epsilon, _original_type, _parameters and _shared_parameters.") % ', '.join(kwargs.keys()))
+                "math_epsilon, iterable_compare_func, _original_type, "
+                "_parameters and _shared_parameters.") % ', '.join(kwargs.keys()))
 
         if _parameters:
             self.__dict__.update(_parameters)
@@ -182,6 +184,7 @@ def __init__(self,
             self.ignore_string_case = ignore_string_case
             self.exclude_obj_callback = exclude_obj_callback
             self.number_to_string = number_to_string_func or number_to_string
+            self.iterable_compare_func = iterable_compare_func
             self.ignore_private_variables = ignore_private_variables
             self.ignore_nan_inequality = ignore_nan_inequality
             self.hasher = hasher
@@ -558,6 +561,71 @@ def _diff_iterable(self, level, parents_ids=frozenset(), _original_type=None):
         else:
             self._diff_iterable_in_order(level, parents_ids, _original_type=_original_type)
 
+    def _compare_in_order(self, level):
+        """
+        Default compare if `iterable_compare_func` is not provided.
+        This will compare in sequence order.
+        """
+
+        return [((i, i), (x, y)) for i, (x, y) in enumerate(
+            zip_longest(
+                level.t1, level.t2, fillvalue=ListItemRemovedOrAdded))]
+
+    def _get_matching_pairs(self, level):
+        """
+        Given a level get matching pairs. This returns list of two tuples in the form:
+        [
+          (t1 index, t2 index), (t1 item, t2 item)
+        ]
+
+        This will compare using the passed in `iterable_compare_func` if available.
+        Default it to compare in order
+        """
+
+        if(self.iterable_compare_func is None):
+            # Match in order if there is no compare function provided
+            return self._compare_in_order(level)
+        try:
+            matches = []
+            y_matched = set()
+            y_index_matched = set()
+            for i, x in enumerate(level.t1):
+                x_found = False
+                for j, y in enumerate(level.t2):
+
+                    if(j in y_index_matched):
+                        # This ensures a one-to-one relationship of matches from t1 to t2.
+                        # If y this index in t2 has already been matched to another x
+                        # it cannot have another match, so just continue.
+                        continue
+
+                    if(self.iterable_compare_func(x, y, level)):
+                        deep_hash = DeepHash(y,
+                                             hashes=self.hashes,
+                                             apply_hash=True,
+                                             **self.deephash_parameters,
+                                             )
+                        y_index_matched.add(j)
+                        y_matched.add(deep_hash[y])
+                        matches.append(((i, j), (x, y)))
+                        x_found = True
+                        break
+
+                if(not x_found):
+                    matches.append(((i, -1), (x, ListItemRemovedOrAdded)))
+            for j, y in enumerate(level.t2):
+
+                deep_hash = DeepHash(y,
+                                     hashes=self.hashes,
+                                     apply_hash=True,
+                                     **self.deephash_parameters,
+                                     )
+                if(deep_hash[y] not in y_matched):
+                    matches.append(((-1, j), (ListItemRemovedOrAdded, y)))
+            return matches
+        except CannotCompare:
+            return self._compare_in_order(level)
+
     def _diff_iterable_in_order(self, level, parents_ids=frozenset(), _original_type=None):
         # We're handling both subscriptable and non-subscriptable iterables. Which one is it?
         subscriptable = self._iterables_subscriptable(level.t1, level.t2)
@@ -566,10 +634,7 @@ def _diff_iterable_in_order(self, level, parents_ids=frozenset(), _original_type
         else:
             child_relationship_class = NonSubscriptableIterableRelationship
 
-        for i, (x, y) in enumerate(
-                zip_longest(
-                    level.t1, level.t2, fillvalue=ListItemRemovedOrAdded)):
-
+        for (i, j), (x, y) in self._get_matching_pairs(level):
             if self._count_diff() is StopIteration:
                 return  # pragma: no cover. This is already covered for addition.
 
@@ -586,10 +651,22 @@ def _diff_iterable_in_order(self, level, parents_ids=frozenset(), _original_type
                     notpresent,
                     y,
                     child_relationship_class=child_relationship_class,
-                    child_relationship_param=i)
+                    child_relationship_param=j)
                 self._report_result('iterable_item_added', change_level)
 
             else:  # check if item value has changed
+
+                if (i != j):
+                    # Item moved
+                    change_level = level.branch_deeper(
+                        x,
+                        y,
+                        child_relationship_class=child_relationship_class,
+                        child_relationship_param=i,
+                        child_relationship_param2=j
+                    )
+                    self._report_result('iterable_item_moved', change_level)
+
                 item_id = id(x)
                 if parents_ids and item_id in parents_ids:
                     continue
@@ -738,6 +815,7 @@ def _get_rough_distance_of_hashed_objs(
                 _shared_parameters=self._shared_parameters,
                 view=DELTA_VIEW,
                 _original_type=_original_type,
+                iterable_compare_func=self.iterable_compare_func,
             )
             _distance = diff._get_rough_distance()
             if cache_key and self._stats[DISTANCE_CACHE_ENABLED]:
@@ -788,6 +866,10 @@ def _get_most_in_common_pairs_in_iterables(
             pre_calced_distances = self._precalculate_numpy_arrays_distance(
                 hashes_added, hashes_removed, t1_hashtable, t2_hashtable, _original_type)
 
+        if hashes_added and hashes_removed and self.iterable_compare_func and len(hashes_added) > 1 and len(hashes_removed) > 1:
+            pre_calced_distances = self._precalculate_distance_by_custom_compare_func(
+                hashes_added, hashes_removed, t1_hashtable, t2_hashtable, _original_type)
+
         for added_hash in hashes_added:
             for removed_hash in hashes_removed:
                 added_hash_obj = t2_hashtable[added_hash]
@@ -797,9 +879,10 @@ def _get_most_in_common_pairs_in_iterables(
                 if id(removed_hash_obj.item) in parents_ids:
                     continue
 
+                _distance = None
                 if pre_calced_distances:
-                    _distance = pre_calced_distances["{}--{}".format(added_hash, removed_hash)]
-                else:
+                    _distance = pre_calced_distances.get("{}--{}".format(added_hash, removed_hash))
+                if _distance is None:
                     _distance = self._get_rough_distance_of_hashed_objs(
                         added_hash, removed_hash, added_hash_obj, removed_hash_obj, _original_type)
                 # Left for future debugging