[Parser Unit Test] Create DataType parser unit tests (project-chip#40477)

* DM XMLs: Add parsing for data types [WIP]

Note that anything with a within-data-type conformance will
have a conformance of optional currently. This is OK since we're
not actually using these for anything expect understanding P
markings currently.

TODO: Still not parsing multi-bit bitfields correctly.
TODO: Needs unit tests
TODO: Probably not passing CI on the test support tests
      because of the new errors with the bitfields.

* Updating changes for XML spec parsing and adding unit tests:
- Enhanced XmlDataTypeComponent with fields for docs, types, and constraints in spec parsing module
- Added support for detecting optional and nullable fields in XML in spec parsing module
- Improved constraint parsing for min/max values and attribute references in spec parsing module
- Added comprehensive test coverage using 1.4.1 cluster spec XML files in TestSpecParsingDataType unit test module
- Improved test output with detailed statistics on XML field usage in TestSpecParsingDataType unit test module and spec parsing module

* Updating TestSpecParsingDataType module:
- Updating some verbiage on comments as needed

* Updating test_metadata.yaml:
- Added TestSpecParsingDataType python3 module as it does not run against an app.

* Restyled by autopep8

* Restyled by isort

* Updating TestSpecParsingDataType unit testing module:
- Resolving linting errors

* Updated TestSpecParsingDataType unit test module:
- Resolving linting errors

* Restyled by autopep8

* Add comprehensive XML validation test for all data types in 1.4.2

Add test_exhaustive_all_fields_in_1_4_2() method to validate every struct,
enum, and bitmap field in all cluster XMLs for version 1.4.2. This test:

- Parses all clusters from /data_model/1.4.2/clusters/
- Validates every struct field has required attributes (name, id, type_info)
- Validates every enum item has required attributes (name, id)
- Validates every bitmap bitfield has required attributes (name, id)
- Skips type_info validation for deprecated fields (e.g., Key field in MonitoringRegistrationStruct)
- Accumulates all issues and reports them at the end instead of failing on first error
- Provides detailed error messages for each validation failure

This ensures full coverage of the data model XMLs and catches any missing
or malformed fields that could affect parsing or downstream analysis.

The test has already identified several clusters with missing type attributes
in ModeOptionStruct fields, which should be addressed in the XML source.

* Restyled by autopep8

* Added exception for possible issue 1.4.2 version Mode cluster XML's ModeOptionStruct struct appears to be missing the type_info attribute

* Restyled by autopep8

* Apply suggestions from code review from Gemini AI

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Fix TestSpecParsingDataType.py test logic for field ID validation:

- Fixed field_id validation to use explicit None check instead of falsy check (field_id=0 is valid but was being treated as empty)
- Added exception handling for fields without type_info that are intentionally missing types (references, ModeOptionStruct fields, etc.)
- Updated test to handle legitimate cases where fields don't have type attributes due to being references or having inferred types

* Restyled by autopep8

* changed chip -> matter for imports

* Restyled by isort

* Updating spec_parsing support module and TestSpecParsingDataType unit test module:
- fixed mypy errors:
-- Line 246: Added raw string prefix to regex pattern
-- Lines 458 & 548: Added type conversion and null checks for ClusterPathLocation calls
-- Line 510: Added null check for max_count.text before assignment
-- Lines 678-680: XmlCluster constructor now includes the required structs, enums, and bitmaps arguments
- Updated for validation of 1.5 XML's instead of 1.4.2

* Updated TestSpecParsingDatatype unit test module:
- Updated copyright date to 2025 from 2024
- Changed verbiage for skipped XML due to missing type_info field for mode
- Updated doc strings to include that we are testing version 1.5 now

* Adding parsing and verification of global data type XML's in version 1.5

* Removed some unneeded debug log outputs

* Restyled by autopep8

* Restyled by isort

* Resolving mypy errors

* Resolving linting errors

* Adding comments for why base=0 in init() of literal class in confomance.py

* Add comment explaining StrEnum usage in DataTypeEnum for XML parsing in spec_parsing.py

* Add comment explaining data_type field in XmlDataType in spec_parsing.py

* Apply suggestions from code review from Gemini-AI-bot

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Broke down _parse_components into 5 different focused helper methods for easier maintenance and provided clear documentation for each one

* Restyled by autopep8

* Resolving mypy error

* Updating spec_parsing.py to resolve Andrei's comments and suggestions:
- Renamed _determine_nullable_status → _isNullableField,, changed it to a static method as it doesn't use self, and changed its call to using the new name
- Renamed _determine_optional_status → _isOptionalField, changed it to a static method as it doesn't use self, and changed its call to using the new name
- Added a clarifying comment to explain why base=0 is used in the _parse_basic_field_attributes method.
- Simplified boolean field checks with direct returns and case-insensitive parsing
- Added comment explaining None vs {} return choice in _parse_field_constraints

* Updating spec_parsing.py to resolve some additional comments/suggestions from Andrei:
- Reduce nesting in _parse_field_conformance using early return pattern
- Add duplicate ID detection in component parsing to catch invalid XML data
- Add duplicate name detection in data type parsing to catch XML conflicts

* Updating unit test TestSpecParsingDataType module:
- Refactoring from suggestion by Gemini AI bot for repetitive skip condition
- Resolved issue where spec_parsing and TestSpecParsingDataType had gone out of sync due to recent updates

* Resolving linting errors

* Restyled by autopep8

* Restyled by autopep8

* Resolving linting error

* Moving TestSpecParsingDataType.py into test_testing directory and adding it back into tests.yaml

* Remove outdated skip conditions from TestSpecParsingDataType for fields that now have proper type_info

---------

Co-authored-by: cecille <[email protected]>
Co-authored-by: Restyled.io <[email protected]>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Author: 4 people

.github/workflows/tests.yaml

@@ -872,6 +872,7 @@ jobs:
                   scripts/run_in_python_env.sh out/venv 'python3 src/python_testing/test_testing/TestIdChecks.py'
                   scripts/run_in_python_env.sh out/venv 'python3 src/python_testing/test_testing/TestMatterTestingSupport.py'
                   scripts/run_in_python_env.sh out/venv 'python3 src/python_testing/test_testing/TestPics.py'
+                  scripts/run_in_python_env.sh out/venv 'python3 src/python_testing/test_testing/TestSpecParsingDataType.py'
                   scripts/run_in_python_env.sh out/venv 'python3 src/python_testing/test_testing/TestSpecParsingDeviceType.py'
                   scripts/run_in_python_env.sh out/venv 'python3 src/python_testing/test_testing/TestSpecParsingNamespace.py'
                   scripts/run_in_python_env.sh out/venv 'python3 src/python_testing/test_testing/TestSpecParsingSelection.py'

src/python_testing/matter_testing_infrastructure/matter/testing/conformance.py

@@ -134,6 +134,10 @@ def is_disallowed(conformance: Callable):
     return conformance(0, [], []).decision == ConformanceDecision.DISALLOWED
 
 
+def is_provisional(conformance: Callable):
+    return conformance(0, [], []).decision == ConformanceDecision.PROVISIONAL
+
+
 @dataclass
 class Conformance:
     def __call__(self, feature_map: uint, attribute_list: list[uint], all_command_list: list[uint]) -> ConformanceDecisionWithChoice:
@@ -203,7 +207,10 @@ def __str__(self):
 
 class literal(Conformance):
     def __init__(self, value: str):
-        self.value = int(value)
+        # base=0 allows automatic detection of number format from string prefix:
+        # "10" -> 10 (decimal), "0x10" -> 16 (hex), "0o10" -> 8 (octal), "0b10" -> 2 (binary)
+        # This is needed because XML literal values can be in different formats
+        self.value = int(value, 0)
 
     def __call__(self):
         # This should never be called