whitespace in strip()/lstrip()/rstrip()

[DimitriPapadopoulosOrfanos]

BPO	25433
Nosy	@jwilk, @ezio-melotti, @nitishch, @joelsmithjohnson, @oulenz, @eedmunds, @rompe
PRs	bpo-25433: Streamline whitespace definitions #14753 bpo-25433: Align bytearray strip methods to those found in byteobject.c #14771 bpo-25433: Streamline whitespace documentation #14775
Files	whitespace_regex.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2015-10-18.12:15:36.240>
labels = ['easy', '3.8', 'type-feature', '3.7', 'docs']
title = 'whitespace in strip()/lstrip()/rstrip()'
updated_at = <Date 2019-07-14.11:58:14.923>
user = 'https://bugs.python.org/DimitriPapadopoulosOrfanos'

bugs.python.org fields:

activity = <Date 2019-07-14.11:58:14.923>
actor = 'rompe'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation']
creation = <Date 2015-10-18.12:15:36.240>
creator = 'Dimitri Papadopoulos Orfanos'
dependencies = []
files = ['47769']
hgrepos = []
issue_num = 25433
keywords = ['patch', 'easy']
message_count = 8.0
messages = ['253152', '257449', '257450', '314668', '314681', '315193', '324272', '347853']
nosy_count = 9.0
nosy_names = ['jwilk', 'ezio.melotti', 'docs@python', 'Dimitri Papadopoulos Orfanos', 'nitishch', 'joel.johnson', 'oulenz', 'edmundselliot@gmail.com', 'rompe']
pr_nums = ['14753', '14771', '14775']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue25433'
versions = ['Python 2.7', 'Python 3.7', 'Python 3.8']

Linked PRs

• gh-69619: Add whitespace term to glossary and reference in stdtypes.rst #132568

[DimitriPapadopoulosOrfanos]

The documentation of strip() / lstrip() / rstrip() should define "whitespace" more precisely.

The Python 3 documentation refers to "ASCII whitespace" for bytes.strip() / bytes.lstrip() / bytes.rstrip() and "whitespace" for str.strip() / str.lstrip() / str.rstrip(). I suggest the following improvements:

• add a link from "ASCII whitespace" to string.whitespace or bytes.isspace(),
• define plain "whitespace" more precisely (possibly with a link to str.isspace()).

The Python 2 documentation refers to plain "whitespace". As far as I know strip() removes ASCII whitespaces only. If so, please:

• add a link to string.whitespace or str.isspace(),
• improve the string.whitespace documentation and explain that it is locale-dependent (see documentation of str.isspace()).

[DimitriPapadopoulosOrfanos]

In Python 2, as far as I can understand, string.whitespace and str.isspace() are different:

• str.isspace() is built upon the C isspace() function and is therefore locale-dependant. Python heavily relies on isspace() to detect "whitespace" characters.
• string.whitespace is a list of "ASCII whitespace characters" carved in stone. As far as I can see string.whitespace is defined but not used anywhere in Python source code.

See source code:
* Modules/stringobject.c around line 3319:
  [...]
  string_isspace(PyStringObject *self)
  {
  [...]
      e = p + PyString_GET_SIZE(self);
      for (; p < e; p++) {
          if (!isspace(*p))
              return PyBool_FromLong(0);
      }
      return PyBool_FromLong(1);
  [...]
* Lib/string.py near line 23:
  whitespace = ' \t\n\r\v\f'

Functions strip()/lstrip()/rstrip() use str.isspace() and have nothing to do with string.whitespace:

* Modules/stringobject.c around line 1861:
[...]
do_strip(PyStringObject *self, int striptype)
{
[...]
    i = 0;
    if (striptype != RIGHTSTRIP) {
        while (i < len && isspace(Py_CHARMASK(s[i]))) {
            i++;
        }
    }
[...]

Therefore I suggest the documentation of Python 2.7 points to str.isspace() wherever the term "whitespace" is used in the documentation - including this specific case of strip()/lstrip()/rstrip().

[DimitriPapadopoulosOrfanos]

In Python 3 the situation is similar:

• The Py_UNICODE_ISSPACE macro is used internally to define str.isspace() and wherever Python needs to detect "whitespace" characters in strings.
• There is an equivalent function Py_ISSPACE for bytes/bytearray.
• The bytearray.strip() implementation for bytearray relies on hardcoded ASCII whitespaces instead of Py_ISSPACE.
• string.whitespace is a list of "ASCII whitespace characters" carved in stone. As far as I can see string.whitespace is defined but not used anywhere in Python source code.

Therefore I suggest the documentation of Python 3 points to str.isspace() wherever the term "whitespace" is used in any documentation related to strings - including this specific case of strip()/lstrip()/rstrip().

[joelsmithjohnson]

I have started working on this and will have a pull request submitted by the end of the week.

The term "whitespace" appears in several contextual situations throughout the documentation. While all situations would benefit from the definition of "whitespace" contained in the str.isspace() documentation, not all of the situations would benefit from a link to str.isspace() whose primary goal is to document the str.isspace() function and not to provide a global definition of what a whitespace character is.

Therefore I suggest the documentation of Python 3 create a new glossary definition of "whitespace" (which contains the definition currently in the str.isspace() documentation) and is pointed to wherever the term "whitespace" is used in any documentation related to strings - including this specific case of strip()/lstrip()/rstrip().

[DimitriPapadopoulosOrfanos]

I agree on avoiding a link to str.isspace() and defining "whitespace" instead.

However please note there are many de facto definitions of "whitespace". All of them must be documented - or at least the conceptual classes of "whitespace" and clarify which class each of the following belongs to:

• Unicode whitespaces are by very far the most common: str.isspace(), strip()/lstrip()/rstrip(), Py_UNICODE_ISSPACE.

• Py_ISSPACE targets byte/bytearray but is never used!

• bytearray.strip() does not use Py_ISSPACE but a hardcoded list of ASCII whitespaces instead.

• finally string.whitespace is probably equivalent to the list used by bytearray.strip().

Beyond the docs, I think Python 3 should rationalize bytearray.strip() / Py_ISSPACE / string.whitespace, probably having bytearray.strip() rely on Py_ISSPACE, and Py_ISSPACE rely on string.whitespace unless string.whitespace is obsoleted.

[oulenz]

Slightly tangential, but it would be great if the documentation of lstrip() and rstrip() could include an equivalent definition in terms of re.sub(), e.g.:

lstrip(foo) == re.sub(r'(?u)\A\s*', '', foo)
rstrip(foo) == re.sub(r'(?u)\s*\Z', '', foo)

(Or whatever else is correct.)

[eedmunds]

Not sure how helpful it would be to have the re.sub expressions for lstrip and rstrip, but I think it would look like:

l_stripped = re.sub(r'^\s*', '', foo)
r_stripped = re.sub(r'\s*$', '', foo)

[rompe]

Using a re.sub() call as documentation:

1. wouldn't be helpful for many developers. If they need to look up the documentation of a simple method they shouldn't be forced to learn about a more complex one as well to understand it.
2. would be wild guessing since the re module defines its own whitespace as well.

I have created a pull request that aligns the strip methods of bytearray to those of bytes objects. The implementation found there is cleaner and even a little bit faster.

Current master:

./python -m timeit -s 'bla = bytearray(b" foo ")' 'bla.lstrip()'
1000000 loops, best of 5: 245 nsec per loop
./python -m timeit -s 'bla = bytearray(b" foo ")' 'bla.rstrip()'
1000000 loops, best of 5: 245 nsec per loop
./python -m timeit -s 'bla = bytearray(b" foo ")' 'bla.strip()'
1000000 loops, best of 5: 260 nsec per loop

Using my patch:

./python -m timeit -s 'bla = bytearray(b" foo ")' 'bla.lstrip()'
1000000 loops, best of 5: 235 nsec per loop
./python -m timeit -s 'bla = bytearray(b" foo ")' 'bla.rstrip()'
1000000 loops, best of 5: 235 nsec per loop
./python -m timeit -s 'bla = bytearray(b" foo ")' 'bla.strip()'
1000000 loops, best of 5: 239 nsec per loop

I have also updated the documentation, adding "whitespace" to the glossary and linking to it from many places in the documentation of standard types.