Skip to content

fix: decode external data file content explicitly as UTF-8#1973

Merged
sisp merged 1 commit intocopier-org:masterfrom
sisp:fix/umlaut-in-external-data-files
Feb 18, 2025
Merged

fix: decode external data file content explicitly as UTF-8#1973
sisp merged 1 commit intocopier-org:masterfrom
sisp:fix/umlaut-in-external-data-files

Conversation

@sisp
Copy link
Member

@sisp sisp commented Feb 18, 2025

I've fixed a string encoding/decoding bug on Windows that broke loading external data with non-ASCII characters. See #1970 for more details, as the root cause for this problem is the same.

I've verified that the new test is failing on Windows 11 without the fix:

Expand to view the pytest log
$  poetry run pytest tests/test_answersfile.py::test_external_data_with_umlaut -n1
=============================================================================== test session starts ================================================================================
platform win32 -- Python 3.13.2, pytest-8.3.4, pluggy-1.5.0
rootdir: C:\Users\<REDACTED>\Documents\copier
configfile: pyproject.toml
plugins: cov-6.0.0, gitconfig-0.7.0, xdist-3.6.1
1 worker [1 item]
F                                                                                                                                                                             [100%]
===================================================================================== FAILURES =====================================================================================
__________________________________________________________________________ test_external_data_with_umlaut __________________________________________________________________________
[gw0] win32 -- Python 3.13.2 C:\Users\<REDACTED>\AppData\Local\pypoetry\Cache\virtualenvs\copier-mBQJSSRS-py3.13\Scripts\python.exe

tmp_path_factory = TempPathFactory(_given_basetemp=WindowsPath('C:/Users/local_<REDACTED>/Temp/pytest-of-<REDACTED>/pytest-5/popen-gw0'), _tr...ath('C:/Users/local_<REDACTED>/Temp/pytest-of-<REDACTED>/pytest-5/popen-gw0'), _retention_count=3, _retention_policy='all')

    def test_external_data_with_umlaut(
        tmp_path_factory: pytest.TempPathFactory,
    ) -> None:
        src, dst = map(tmp_path_factory.mktemp, ("src", "dst"))

        build_file_tree(
            {
                (src / "copier.yml"): (
                    """\
                    _external_data:
                        data: data.yml
                    ext_umlaut: "{{ _external_data.data.umlaut }}"
                    """
                ),
                (src / "{{ _copier_conf.answers_file }}.jinja"): (
                    "{{ _copier_answers|to_nice_yaml }}"
                ),
            }
        )
        git_save(src, tag="v1")

        (dst / "data.yml").write_text("umlaut: äöü", encoding="utf-8")

        copier.run_copy(str(src), dst, defaults=True, overwrite=True)
        answers = load_answersfile_data(dst, ".copier-answers.yml")
>       assert answers["ext_umlaut"] == "äöü"
E       AssertionError: assert 'äöü' == 'äöü'
E
E         - äöü
E         + äöü

tests\test_answersfile.py:252: AssertionError
------------------------------------------------------------------------------- Captured stdout call -------------------------------------------------------------------------------


------------------------------------------------------------------------------- Captured stderr call -------------------------------------------------------------------------------

Copying from template version 1
    create  .copier-answers.yml
============================================================================= short test summary info ==============================================================================
FAILED tests/test_answersfile.py::test_external_data_with_umlaut - AssertionError: assert 'äöü' == 'äöü'
================================================================================ 1 failed in 5.55s =================================================================================

@codecov
Copy link

codecov bot commented Feb 18, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.90%. Comparing base (21b486c) to head (0ff1e82).
Report is 125 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #1973   +/-   ##
=======================================
  Coverage   97.89%   97.90%           
=======================================
  Files          53       53           
  Lines        5567     5575    +8     
=======================================
+ Hits         5450     5458    +8     
  Misses        117      117           
Flag Coverage Δ
unittests 97.90% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@sisp sisp force-pushed the fix/umlaut-in-external-data-files branch from f116c40 to 0ff1e82 Compare February 18, 2025 13:32
@sisp sisp merged commit 3953603 into copier-org:master Feb 18, 2025
22 checks passed
@sisp sisp deleted the fix/umlaut-in-external-data-files branch February 18, 2025 17:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants