Skip to content

generate_test_code.py does not support Unicode in .function files #64

Open
@gilles-peskine-arm

Description

@gilles-peskine-arm

generate_test_code.py supports non-ASCII characters in .data files, but not portably in .function files. Sample backtrace (from Mbed-TLS/mbedtls#9739 at Mbed-TLS/mbedtls@db226b4):

Traceback (most recent call last):
  File "../framework/scripts/generate_test_code.py", line 1294, in <module>
    main()
  File "../framework/scripts/generate_test_code.py", line 1289, in main
    c_file=out_c_file, out_data_file=out_data_file)
  File "../framework/scripts/generate_test_code.py", line 1217, in generate_code
    write_test_source_file(template_file, c_file, snippets)
  File "../framework/scripts/generate_test_code.py", line 1134, in write_test_source_file
    c_f.write(code)
UnicodeEncodeError: 'ascii' codec can't encode character '\xa7' in position 2012: ordinal not in range(128)

UTF-8 encoded characters work if the locale (LC_CTYPE) is based on UTF-8, or when using Python ≥3.7 in a non-Unicode locale. But on the CI, we try to use Python 3.6 and the locale is C, leading to the failure quoted above when I tried to use § in a .function file.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingsize-xsEstimated task size: extra small (a few hours at most)

    Type

    No type

    Projects

    Status

    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions