Skip to content

gh-83461: Don't allow datetime parsing to accept non-ASCII digits #131008

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 20 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion Doc/library/datetime.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2611,6 +2611,9 @@ differences between platforms in handling of unsupported format specifiers.
.. versionadded:: 3.12
``%:z`` was added.

.. versionchanged:: next
Non-ASCII digits are now rejected by ``strptime`` for numerical directives.

Technical Detail
^^^^^^^^^^^^^^^^

Expand Down Expand Up @@ -2650,7 +2653,8 @@ Notes:
Because the format depends on the current locale, care should be taken when
making assumptions about the output value. Field orderings will vary (for
example, "month/day/year" versus "day/month/year"), and the output may
contain non-ASCII characters.
contain non-ASCII characters. :meth:`~.datetime.strptime` rejects non-ASCII
digits for non-locale-specific numeric format codes (e.g. ``%Y``, ``%H``, etc).

(2)
The :meth:`~.datetime.strptime` method can parse years in the full [1, 9999] range, but
Expand Down
11 changes: 8 additions & 3 deletions Doc/library/time.rst
Original file line number Diff line number Diff line change
Expand Up @@ -567,6 +567,9 @@ Functions
When used with the :func:`strptime` function, ``%U`` and ``%W`` are only used in
calculations when the day of the week and the year are specified.

(5)
The :func:`strptime` function does not accept non-ASCII digits for numeric values.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we mention the "non-locale-specific numeric format codes" here or, since it's not officially supported, we can be a bit lazy?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pganssle Do you want to do this too? Or should I?


Here is an example, a format for dates compatible with that specified in the
:rfc:`2822` Internet email standard. [1]_ ::

Expand All @@ -590,9 +593,8 @@ Functions

.. function:: strptime(string[, format])

Parse a string representing a time according to a format. The return value
is a :class:`struct_time` as returned by :func:`gmtime` or
:func:`localtime`.
Parse a string representing a time according to a format string. The return
value is a :class:`struct_time` as returned by :func:`gmtime` or :func:`localtime`.

The *format* parameter uses the same directives as those used by
:func:`strftime`; it defaults to ``"%a %b %d %H:%M:%S %Y"`` which matches the
Expand Down Expand Up @@ -620,6 +622,9 @@ Functions
and thus does not necessarily support all directives available that are not
documented as supported.

.. versionchanged:: next
Non-ASCII digits are now rejected by ``strptime``.


.. class:: struct_time

Expand Down
16 changes: 16 additions & 0 deletions Doc/whatsnew/3.14.rst
Original file line number Diff line number Diff line change
Expand Up @@ -700,6 +700,13 @@ datetime
* Add :meth:`datetime.time.strptime` and :meth:`datetime.date.strptime`.
(Contributed by Wannes Boeykens in :gh:`41431`.)

* When using digits in :meth:`datetime.date.strptime`,
:meth:`datetime.datetime.strptime`, or :meth:`datetime.time.strptime`, a
:exc:`ValueError` is raised if non-ASCII digits are specified for
non-locale-specific numeric format codes.
(Contributed by Stan Ulbrych in :gh:`131008`.)


decimal
-------

Expand Down Expand Up @@ -1211,13 +1218,22 @@ sys.monitoring
:monitoring-event:`BRANCH_RIGHT`. The ``BRANCH`` event is deprecated.


time
----

* When using digits in :func:`time.strptime`, a :exc:`ValueError` is raised if
non-ASCII digits are specified for non-locale-specific numeric format codes.
(Contributed by Stan Ulbrych in :gh:`131008`.)


sysconfig
---------

* Add ``ABIFLAGS`` key to :func:`sysconfig.get_config_vars` on Windows.
(Contributed by Xuehai Pan in :gh:`131799`.)



threading
---------

Expand Down
22 changes: 11 additions & 11 deletions Lib/_strptime.py
Original file line number Diff line number Diff line change
Expand Up @@ -286,23 +286,23 @@ def __init__(self, locale_time=None):
base = super()
mapping = {
# The " [1-9]" part of the regex is to make %c from ANSI C work
'd': r"(?P<d>3[0-1]|[1-2]\d|0[1-9]|[1-9]| [1-9])",
'd': r"(?P<d>3[0-1]|[1-2][0-9]|0[1-9]|[1-9]| [1-9])",
'f': r"(?P<f>[0-9]{1,6})",
'H': r"(?P<H>2[0-3]|[0-1]\d|\d)",
'H': r"(?P<H>2[0-3]|[0-1][0-9]|[0-9])",
'I': r"(?P<I>1[0-2]|0[1-9]|[1-9]| [1-9])",
'G': r"(?P<G>\d\d\d\d)",
'j': r"(?P<j>36[0-6]|3[0-5]\d|[1-2]\d\d|0[1-9]\d|00[1-9]|[1-9]\d|0[1-9]|[1-9])",
'G': r"(?P<G>[0-9][0-9][0-9][0-9])",
'j': r"(?P<j>36[0-6]|3[0-5][0-9]|[1-2][0-9][0-9]|0[1-9][0-9]|00[1-9]|[1-9][0-9]|0[1-9]|[1-9])",
'm': r"(?P<m>1[0-2]|0[1-9]|[1-9])",
'M': r"(?P<M>[0-5]\d|\d)",
'S': r"(?P<S>6[0-1]|[0-5]\d|\d)",
'U': r"(?P<U>5[0-3]|[0-4]\d|\d)",
'M': r"(?P<M>[0-5][0-9]|[0-9])",
'S': r"(?P<S>6[0-1]|[0-5][0-9]|[0-9])",
'U': r"(?P<U>5[0-3]|[0-4][0-9]|[0-9])",
'w': r"(?P<w>[0-6])",
'u': r"(?P<u>[1-7])",
'V': r"(?P<V>5[0-3]|0[1-9]|[1-4]\d|\d)",
'V': r"(?P<V>5[0-3]|0[1-9]|[1-4][0-9]|[0-9])",
# W is set below by using 'U'
'y': r"(?P<y>\d\d)",
'Y': r"(?P<Y>\d\d\d\d)",
'z': r"(?P<z>[+-]\d\d:?[0-5]\d(:?[0-5]\d(\.\d{1,6})?)?|(?-i:Z))",
'y': r"(?P<y>[0-9][0-9])",
'Y': r"(?P<Y>[0-9]{4})",
'z': r"(?P<z>[+-][0-9][0-9]:?[0-5][0-9](:?[0-5][0-9](\.[0-9]{1,6})?)?|(?-i:Z))",
'A': self.__seqToRE(self.locale_time.f_weekday, 'A'),
'a': self.__seqToRE(self.locale_time.a_weekday, 'a'),
'B': self.__seqToRE(self.locale_time.f_month[1:], 'B'),
Expand Down
10 changes: 9 additions & 1 deletion Lib/test/datetimetester.py
Original file line number Diff line number Diff line change
Expand Up @@ -2921,6 +2921,14 @@ def test_strptime(self):
with self.assertRaises(ValueError): strptime("-000", "%z")
with self.assertRaises(ValueError): strptime("z", "%z")

# test only ascii is allowed
with self.assertRaises(ValueError): strptime('٢025-0٢-٢9', '%Y-%m-%d')
with self.assertRaises(ValueError): strptime('1٢:02:٢7', '%H:%M:%S')
with self.assertRaises(ValueError): strptime('٢5', '%y')
with self.assertRaises(ValueError): strptime('٢555', '%G')
with self.assertRaises(ValueError): strptime('٢/0٢ 0٢a٢', '%j/%y %I%p:%M:%S')
with self.assertRaises(ValueError): strptime('0٢/٢/200٢', '%U/%V')

def test_strptime_single_digit(self):
# bpo-34903: Check that single digit dates and times are allowed.

Expand Down Expand Up @@ -4052,7 +4060,7 @@ def test_strptime_tz(self):
self.assertEqual(strptime("UTC", "%Z").tzinfo, None)

def test_strptime_errors(self):
for tzstr in ("-2400", "-000", "z"):
for tzstr in ("-2400", "-000", "z", "٢"):
with self.assertRaises(ValueError):
self.theclass.strptime(tzstr, "%z")

Expand Down
4 changes: 4 additions & 0 deletions Lib/test/test_time.py
Original file line number Diff line number Diff line change
Expand Up @@ -357,6 +357,10 @@ def test_strptime_leap_year(self):
r'.*day of month without a year.*'):
time.strptime('02-07 18:28', '%m-%d %H:%M')

def test_strptime_non_ascii(self):
with self.assertRaises(ValueError):
time.strptime('٢025', '%Y')

def test_asctime(self):
time.asctime(time.gmtime(self.t))

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Digits passed to :meth:`datetime.date.strptime`, :meth:`datetime.datetime.strptime`,
:meth:`datetime.time.strptime` and :meth:`time.strptime` must now be ASCII in
non-locale-specific numeric format codes.
Loading