git commit -m "fix(models): convert CP1252 ontology files to UTF-8"

[pedropaulofb]

Summary

This pull request converts ontology source files that were encoded as Windows-1252/CP1252 to UTF-8.

The affected files are limited to:

• models/**/ontology.json
• models/**/ontology.ttl

Files that were already valid UTF-8, including ASCII-only files that are fully UTF-8-compatible, were left unchanged.

No conversion scripts, test utilities, configuration files, generated reports, metadata files, or unrelated repository changes are included in this pull request.

Motivation

The catalog currently contains ontology files using different source encodings. Although ASCII-only files are already valid UTF-8 and require no modification, some files contain CP1252-encoded non-ASCII characters and therefore cannot be decoded strictly as UTF-8.

Standardizing the affected ontology files as UTF-8 improves:

• cross-platform consistency;
• compatibility with Python, RDF, JSON, and text-processing tools;
• reproducibility of repository automation;
• reliability of downstream catalog processing;
• avoidance of platform-dependent decoding behavior.

This change does not attempt to infer or rewrite the historical encoding of ASCII-only files. ASCII bytes are already valid UTF-8 bytes, so rewriting those files would provide no benefit.

Encoding classification

Each target file was inspected using its raw bytes.

The classification procedure was:

1. detect and reject unsupported UTF-16 or UTF-32 byte-order marks;
2. recognize valid UTF-8 files with a UTF-8 BOM without modifying them;
3. attempt strict UTF-8 decoding;
4. classify ASCII-only files separately as UTF-8-compatible;
5. only when strict UTF-8 decoding failed, attempt strict CP1252 decoding;
6. require the decoded text to reproduce the original bytes exactly when re-encoded as CP1252;
7. convert only files that passed that deterministic CP1252 round-trip verification.

No probabilistic encoding detector was used to decide which files should be converted.

Conversion method

For every confirmed CP1252 file:

1. the original bytes were decoded strictly as CP1252;
2. the resulting Unicode text was encoded strictly as UTF-8;
3. the proposed UTF-8 bytes were decoded again;
4. the resulting Unicode text was verified to be identical, code point for code point, to the original CP1252-decoded text;
5. the original bytes were independently verified by re-encoding the source text as CP1252.

This ensures that the conversion changes only the byte representation of non-ASCII characters and does not change the textual content.

Content preservation

The files were converted directly at the byte/text-encoding level.

JSON and Turtle files were not parsed and reserialized as part of the conversion. In particular, the conversion did not use json.dump(), RDFLib serialization, or any formatter.

The following properties were preserved:

• all characters and Unicode code points;
• whitespace and indentation;
• blank lines;
• LF and CRLF line endings;
• final-newline presence or absence;
• JSON key and value ordering;
• Turtle prefix and triple ordering;
• comments;
• escape sequences as originally written;
• IRIs;
• literals;
• identifiers;
• JSON formatting;
• Turtle formatting.

The intended change is limited to replacing the CP1252 byte representation of non-ASCII characters with the corresponding UTF-8 byte representation.

Scope

This pull request contains only ontology files that were classified as requiring CP1252-to-UTF-8 conversion.

It does not include:

• the local conversion script;
• the local automated tests for the conversion script;
• .pre-commit-config.yaml;
• encoding audit reports;
• JSON or Turtle formatting changes;
• line-ending normalization;
• whitespace cleanup;
• Unicode normalization;
• metadata regeneration;
• semantic corrections;
• syntax corrections;
• changes to files already valid as UTF-8.

ASCII-only files reported as ASCII_UTF8_COMPATIBLE were intentionally not modified because their existing bytes already constitute valid UTF-8.

Safety measures

Before any file was modified, the complete target set was scanned and classified.

The conversion process was designed to abort before writing when encountering conditions such as:

• unsupported or uncertain encodings;
• unreadable files;
• missing expected ontology files;
• unexpected nested target files;
• symbolic links;
• multiple target paths referring to the same physical file;
• failed CP1252 round-trip validation;
• files changing between preflight and replacement.

Converted files were written through temporary files in the same directory and then atomically replaced. The original permission bits were preserved.

Validation performed

The following checks were performed after conversion:

• the conversion utility’s automated tests passed;
• every target file was rescanned;
• no CP1252_CONVERSION_REQUIRED classifications remained;
• no unsupported or uncertain encoding classifications remained;
• every target file decoded strictly as UTF-8;
• JSON and Turtle syntax validation completed without modifying the files;
• every converted file preserved the same Unicode text as its CP1252 source;
• only files previously classified as CP1252 were modified;
• the Git diff was reviewed for unrelated formatting or content changes;
• a second conversion execution produced no additional modifications, confirming idempotence.

Verification commands

python -m pytest -q scripts\tests\test_convert_ontology_encodings.py

python scripts\convert_ontology_encodings.py --check

python scripts\convert_ontology_encodings.py --check --validate-syntax

git diff --check

git diff --stat

git diff -- models

python scripts\convert_ontology_encodings.py --apply

The final --apply execution reported no additional converted files.

Expected result

After this change:

• all models/**/ontology.json files are valid UTF-8;
• all models/**/ontology.ttl files are valid UTF-8;
• ASCII-only ontology files remain byte-for-byte unchanged;
• files that were already valid UTF-8 remain byte-for-byte unchanged;
• only confirmed CP1252 files have different bytes;
• the recovered Unicode text and ontology semantics are unchanged.