You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardexpand all lines: CHANGELOG.md
+60
Original file line number
Diff line number
Diff line change
@@ -3,6 +3,66 @@
3
3
Used to document all changes from previous releases and collect changes
4
4
until the next release.
5
5
6
+
# Version 1.5.0
7
+
8
+
# Python 2 deprecation warning
9
+
A Python 2 deprecation warning for August 2020 has been added. See issue #387 for details.
10
+
11
+
# Validation feature update
12
+
See issues #377, #378 and #379 as well as Pull Request #389 for details.
13
+
14
+
An `IssueID` enum class as been added to provide identifiers to individual ValidationErrors. The `Validation` class itself has been refactored to provide the option to create standalone Validation instances with a different set of registered validations than the default library validation.
15
+
The `Validation` class now features the new `register_custom_handler`, `run_validation`and `report` methods to add custom validation handlers to an instance, re-run the validations of an existing Instance and provide a brief report of encountered errors and warnings. The general `ValidationError.__repr__` string has been shortened to make the individual ValidationErrors more convenient to print and read. The default Validation is always run when a Document is saved or loaded via the `ODMLParser` and the `Validation.report` method is used to provide a `warnings.warn` message of the following format:
16
+
```
17
+
UserWarning: The saved Document contains formal issues. Run 'odml.validation.Validation(doc)' to resolve them.
18
+
Validation found 0 errors and 3 warnings in 1 Sections and 1 Properties.
19
+
```
20
+
21
+
Further changes to the Validation class and behavior include:
22
+
- an odml `Document` now provides a `validate` method that will run a default Validation and return the Validation instance to provide users with access to encountered issues.
23
+
- a `validation_id` field has been added to the `ValidationError` class.
24
+
- standalone Sections and Properties can now be validated.
25
+
- Sections and Properties are validated on init.
26
+
- the `section_repository_present` validation has been removed from the default validation list. Since Sections rarely have repositories set, this validation can lead to spam when validating a Document.
27
+
28
+
# Cardinality feature
29
+
Property and Section now provide a cardinality feature. Users can now define a range how many Values a Property and how many Properties or Sections a Section should have. A cardinality can be set and read via its accessor method and can be set via an additional convenience method. Whenever a cardinality or an affected Value, Section or Property is set, a corresponding validation is triggered. If this a set cardinality for a Property or Section is violated, a message is printed to the command line directly and a warning is issued when a Document is saved or loaded. Every cardinality is saved to and loaded from all available file formats.
30
+
The full functionality of all cardinality features are documented in the tutorial and is available via readthedocs. For additional details see pull requests #374, #382, #383, #384 and issue #361.
31
+
32
+
# Update in Section type default behavior
33
+
With recent updates the library now respects and enforces `Section.type` as a required attribute and allows save only with documents where this requirement is satisfied.
34
+
To allow backwards file compatibility and ease usage, `Section.type` is by default set to the string `n.s.` (not specified), which means files where no `Section.type` had been specified can be loaded and saved, but will contain `n.s.` as value for every `Sections.type` that was previously not specified.
35
+
Further the validation run before a document can be saved will issue a warning, if a `Section.type` with value `n.s.` is encountered and will still refuse to save with an error, if an empty `Section.type` is encountered. See PR #376 for details.
36
+
37
+
# DictParser and ODMLParser fully support ignore errors
38
+
- the `DictParser.DictReader` is now able to ignore errors, collect warnings and print corresponding notifications and works now analogous to the `xmlparser.XMLReader` behaviour. See issue #367 for details.
39
+
- the `ODMLParser.ODMLReader` for JSON and YAML now uses `ignore_errors` by default e.g. when using the `odml.load` function for JSON and YAML odml files.
40
+
41
+
# Fixes
42
+
- fixes an exception when trying to append or extend a `Property` with dtype `tuple`. See issue #364 for details.
43
+
- when trying to set the `name` attribute to `None`, it now silently sets the name to `id` instead, since `name` must not be empty. It would be set to `id` on load and can cause `AttributeError` exceptions with some methods if its not set.
44
+
- a bug was fixed in `format.revmap` where the reverse mapping of an odml attribute would always return the case that the attribute is part of the format, even if it was not.
45
+
46
+
# Minor changes and updates
47
+
- all deprecation warnings now use the warnings module.
48
+
- the `Property.value` attribute deprecation warnings have been unified. See issue #360 for details.
49
+
- the `base.Sectionable.create_section` method has been updated to conform with `Section.__init__`. See issue #368 for details.
50
+
- all saved XML odML files now use the same XML header. See issue #339 for details.
51
+
- a function to manually refresh the terminology cache has been added. See issue #202 for details.
52
+
- a Validation to note non-human readable `Property` and `Section` names has been added. See issue #365 for details.
53
+
- getter and setter methods are added to the `odml.Document.origin_file_name` attribute. See issue #358 for details.
54
+
- the Exception type in `odml.tools.converters.VersionConverter` is changed to `odml.tools.parser_utils.ParserException`. See issue #359 for details.
55
+
- the `odml.Property.export_leaf` method now also includes sibling Properties on export.
56
+
- the `rdf_converter` has been cleaned up, see issues #211 and #345 for details.
57
+
- the test for the `Section`/`Property` order in documents obtained via the `RDFReader` has been expanded. See issue #265 for details.
58
+
- tests for Validation errors on `Section` or `Property` init have been added. See issue #369 for details.
59
+
- tests writing temporary files now properly clean up after themselves. See issue #381 for details.
60
+
- tests now use a common temporary directory to write files and use a constant for accessing the test/resources directory.
61
+
- the link to the odML tutorial in the README file now points to python-odml.readthedocs.org; the README file now also includes links to Travis and Coveralls.
62
+
- the tutorial now includes descriptions of the `pprint` method and a link to the odML templates hosting site. Further the tutorial has been updated to include descriptions of the cardinality feature and Validation usage.
63
+
- introduces major PEP8 fixes to basically all files of the library. See Pull Request #385 for details.
64
+
- the class reference now includes the Template, Terminology and Validation classes.
Python 2 has reached end of life. We will not keep any future versions of odml Python 2 compatible and also recommend using a Python version >= 3.6. If a Python version < 3.6 is a requirement, the following dependency needs to be installed as well:
96
+
Python 2 has reached end of life. We will not keep any future versions of odml Python 2 compatible and will completely drop support for Python 2 with August 2020. We also recommend using a Python version >= 3.6. If a Python version < 3.6 is a requirement, the following dependency needs to be installed as well:
Copy file name to clipboardexpand all lines: doc/tutorial.rst
+161-6
Original file line number
Diff line number
Diff line change
@@ -983,9 +983,168 @@ A cardinality is set via its convenience method:
983
983
>>> # or
984
984
>>> prop.val_cardinality =None
985
985
986
+
Please note that a set cardinality is not enforced. Users can set less or more entities than are specified allowed via a cardinality. Instead whenever a cardinality is not met, a warning message is displayed and any unment cardinality will show up as a Validation warning message whenever a document is saved or loaded.
986
987
987
-
Advanced knowledge on Values
988
-
----------------------------
988
+
Working with Validations
989
+
------------------------
990
+
991
+
odML Validations are a set of pre-defined checks that are run against an odML document automatically when it is saved or loaded. A document cannot be saved, if a Validation fails a check that is classified as an Error. Most validation checks are Warnings that are supposed to raise the overall data quality of the odml Document.
992
+
993
+
When an odML document is saved or loaded, tha automatic validation will print a short report of encountered Validation Warnings and it is up to the user whether they want to resolve the Warnings. The odML document provides the ``validate`` method to gain easy access to the default validations. A Validation in turn provides not only a specific description of all encountered warnings or errors within an odML document, but it also provides direct access to each and every odML entity i.e. an ``odml.Section`` or an ``odml.Property`` where an issue has been found. This enables the user to quickly access and fix an encountered issue.
994
+
995
+
A minimal example shows how a workflow using default validations might look like:
996
+
997
+
>>> # Create a minimal document with Section issues: name and type are not assigned
998
+
>>> doc = odml.Document()
999
+
>>> sec = odml.Section(parent=doc)
1000
+
>>> odml.save(doc, "validation_example.odml.xml")
1001
+
1002
+
This minimal example document will be saved, but will also print the following Validation report:
1003
+
1004
+
>>> UserWarning: The saved Document contains unresolved issues. Run the Documents 'validate' method to access them.
To fix the encountered warnings, users can access the validation via the documents' ``validate`` method:
1008
+
1009
+
>>> validation = doc.validate()
1010
+
>>> for issue in validation.errors:
1011
+
>>> print(issue)
1012
+
1013
+
This will show that the validation has encountered two Warnings and also displays the offending odml entity.
1014
+
1015
+
>>> ValidationWarning: Section[73f29acd-16ae-47af-afc7-371d57898e28] 'Section type not specified'
1016
+
>>> ValidationWarning: Section[73f29acd-16ae-47af-afc7-371d57898e28] 'Name not assigned'
1017
+
1018
+
To fix the "Name not assigned" warning the Section can be accessed via the validation entry and used to directly assign a human readable name to the Section in the original document. Re-running the validation will show, that the warning has been removed.
>>> # Check that the section name has been changed in the document
1022
+
>>> print(doc.sections)
1023
+
>>> # Re-running validation
1024
+
>>> validation = doc.validate()
1025
+
>>> for issue in validation.errors:
1026
+
>>> print(issue)
1027
+
1028
+
Similarly the second validation warning can be resolved before saving the document again.
1029
+
1030
+
Please note that the automatic validation is run whenever a document is saved or loaded using the ``odml.save`` and ``odml.load`` functions as well as the ``ODMLWriter`` or the ``ODMLReader`` class. The validation is not run when using any of the lower level ``xmlparser``, ``dict_parser`` or ``rdf_converter`` classes.
1031
+
1032
+
List of available default validations
1033
+
*************************************
1034
+
1035
+
The following contains a list of the default odml validations, their message and the suggested course of action to resolve the issue.
1036
+
1037
+
|Validation: ``object_required_attributes``
1038
+
|Message: "Missing required attribute 'xyz'"
1039
+
|Applies to: ``Document``, ``Section``, ``Property``
1040
+
|Course of action: Add an appropriate value to attribute 'xyz' for the reported odml entity.
1041
+
1042
+
|Validation: ``section_type_must_be_defined``
1043
+
|Message: "Section type not specified"
1044
+
|Applies to: ``Section``
1045
+
|Course of action: Fill in the ``type`` attribute of the reported Section.
1046
+
1047
+
|Validation: ``section_unique_ids``
1048
+
|Message: "Duplicate id in Section 'secA' and 'secB'"
1049
+
|Applies to: ``Section``
1050
+
|Course of action: IDs have to be unique and a duplicate id was found. Assign a new id for the reported Section.
1051
+
1052
+
|Validation: ``property_unique_ids``
1053
+
|Message: "Duplicate id in Property 'propA' and 'propB'"
1054
+
|Applies to: ``Property``
1055
+
|Course of action: IDs have to be unique and a duplicate id was found. Assign a new id for the reported Property
1056
+
1057
+
|Validation: ``section_unique_name_type``
1058
+
|Message: "name/type combination must be unique"
1059
+
|Applies to: ``Section``
1060
+
|Course of action: The combination of Section.name and Section.type has to be unique on the same level. Change either name or type of the reported Section.
1061
+
1062
+
|Validation: ``object_unique_name``
1063
+
|Message: "Object names must be unique"
1064
+
|Applies to: ``Document``, ``Section``, ``Property``
1065
+
|Course of action: Property name has to be unique on the same level. Change the name of the reported Property.
1066
+
1067
+
|Validation: ``object_name_readable``
1068
+
|Message: "Name not assigned"
1069
+
|Applies to: ``Section``, ``Property``
1070
+
|Course of action: When Section or Property names are left empty on creation or set to None, they are automatically assigned the entities uuid. Assign a human readable name to the reported entity.
1071
+
1072
+
|Validation: ``property_terminology_check``
1073
+
|Message: "Property 'prop' not found in terminology"
1074
+
|Applies to: ``Property``
1075
+
|Course of action: The reported entity is linked to a repository but the repository is not available. Check if the linked content has moved.
1076
+
1077
+
|Validation: ``property_dependency_check``
1078
+
|Message: "Property refers to a non-existent dependency object" or "Dependency-value is not equal to value of the property's dependency"
1079
+
|Applies to: ``Property``
1080
+
|Course of action: The reported entity depends on another Property, but this dependency has not been satisfied. Check the referenced Property and its value to resolve the issue.
1081
+
1082
+
|Validation: ``property_values_check``
1083
+
|Message: "Tuple of length 'x' not consistent with dtype 'dtype'!" or "Property values not of consistent dtype!".
1084
+
|Applies to: ``Property``
1085
+
|Course of action: Adjust the values or the dtype of the referenced Propery.
1086
+
1087
+
|Validation: ``property_values_string_check``
1088
+
|Message: "Dtype of property "prop" currently is "string", but might fit dtype "dtype"!"
1089
+
|Applies to: ``Property``
1090
+
|Course of action: Check if the datatype of the referenced Property.values has been loaded correctly and change the Property.dtype if required.
1091
+
1092
+
|Validation: ``section_properties_cardinality``
1093
+
|Message: "cardinality violated x values, y found)"
1094
+
|Applies to: ``Section``
1095
+
|Course of action: A cardinality defined for the number of Properties of a Section does not match. Add or remove Properties until the cardinality has been satisfied or adjust the cardinality.
1096
+
1097
+
|Validation: ``section_sections_cardinality``
1098
+
|Message: "cardinality violated x values, y found)"
1099
+
|Applies to: ``Section``
1100
+
|Course of action: A cardinality defined for the number of Sections of a Section does not match. Add or remove Sections until the cardinality has been satisfied or adjust the cardinality.
1101
+
1102
+
|Validation: ``property_values_cardinality``
1103
+
|Message: "cardinality violated x values, y found)"
1104
+
|Applies to: ``Property``
1105
+
|Course of action: A cardinality defined for the number of Values of a Property does not match. Add or remove Values until the cardinality has been satisfied or adjust the cardinality.
1106
+
1107
+
|Validation: ``section_repository_present``
1108
+
|Message: "A section should have an associated repository" or "Could not load terminology" or "Section type not found in terminology"
1109
+
|Applies to: ``Section``
1110
+
|Course of action: Optional validation. Will report any section that does not specify a repository. Add a repository to the reported Section to resolve.
1111
+
1112
+
Custom validations
1113
+
******************
1114
+
1115
+
Users can write their own validation and register them either with the default validation or add it to their own validation class instance.
1116
+
1117
+
A custom validation handler needs to ``yield`` a ``ValidationError``. See the ``validation.ValidationError`` class for details.
1118
+
1119
+
Custom validation handlers can be registered to be applied on "odML" (the odml Document), "section" or "property".
0 commit comments