Summary

SecretsSettingsSource reads each secret with Path.read_text() without specifying an encoding. On Windows systems whose locale encoding is not UTF-8, a UTF-8 secret file can therefore be decoded using the active Windows code page.

For some byte sequences this does not raise UnicodeDecodeError; it silently produces a different string. The same implicit read_text() behavior also appears in NestedSecretsSettingsSource.

Relevant code:

• SecretsSettingsSource in v2.14.2
• NestedSecretsSettingsSource in v2.14.2

Environment

• pydantic-settings==2.14.2 from PyPI
• pydantic==2.13.4
• Python 3.12.10
• Windows 11 (10.0.22621)
• Windows locale encoding: cp936
• Python UTF-8 mode: disabled

Minimal reproduction

import locale
import sys
import tempfile
from pathlib import Path

from pydantic_settings import BaseSettings


class Settings(BaseSettings):
    token: str


with tempfile.TemporaryDirectory() as directory:
    secrets_dir = Path(directory)
    expected = "p\u00e4ssw\u00f6rd-\U0001f510"
    (secrets_dir / "token").write_bytes(expected.encode("utf-8"))

    actual = Settings(_secrets_dir=secrets_dir).token
    print("locale:", locale.getpreferredencoding(False))
    print("utf8_mode:", sys.flags.utf8_mode)
    print("expected:", ascii(expected))
    print("actual:  ", ascii(actual))
    assert actual == expected

Run normally on a Windows installation using a non-UTF-8 locale:

py -3.12 repro.py

Observed output:

locale: cp936
utf8_mode: 0
expected: 'p\xe4ssw\xf6rd-\U0001f510'
actual:   'p\u76f2ssw\u679ard-\u9983\u653c'
AssertionError

Control run:

py -3.12 -X utf8 repro.py

The control succeeds and returns the original value.

I reproduced the mismatch twice with the PyPI v2.14.2 package and twice with the current main checkout (1d0ea26). An expanded local check confirms the same result for both SecretsSettingsSource and NestedSecretsSettingsSource; both sources return the original value in the -X utf8 control run.

The relevant existing test suites still pass:

• tests/test_settings.py -k secrets: 14 passed, 6 skipped
• tests/test_source_nested_secrets.py: 36 passed

The current tests do not appear to cover a file written explicitly as UTF-8 and then read under a non-UTF-8 Windows locale.

Running the expanded reproduction with -X warn_default_encoding also emits Python's EncodingWarning at both read_text() calls. This matches the Python documentation: Path.read_text() delegates its optional encoding behavior to open(), whose default text encoding is locale-specific. See pathlib and Text Encoding.

Expected behavior

Loading a text secret should not silently change its Unicode value based on the host's locale encoding. Possible approaches might be:

1. define UTF-8 as the default encoding for secret files; or
2. add an explicit secret-file encoding setting, similar to env_file_encoding, and use it in both standard and nested secret sources.

I realize changing the default may have backward-compatibility implications for existing locale-encoded files, so an explicit configuration option may be preferable. If locale-dependent decoding is intentional, documenting that behavior would also remove the current ambiguity.

I searched open and closed issues and pull requests using encoding, UTF-8, secrets encoding Windows, secrets UTF-8, secret file encoding, read_text secrets, and cp936 secrets, and did not find a matching report.