text/x-rst README analysis broken with the latest version (6.1.0)

[ralonsoh]

Is there an existing issue for this?

• I have searched the existing issues (open and closed), and could not find an existing issue

What keywords did you use to search existing issues?

Since the release of twine 6.1.0, the parsing of the README files (format text/x-rst) fails in the OpenStack CI [1].

Error snippet: https://paste.opendev.org/show/bpIn5YiVeRrpNDMM06Ez/

Logs (of the snippet): https://04824dc10f811bf71cc7-f60cbd2bdbb8b5648c0b0982a5f4272f.ssl.cf1.rackcdn.com/939769/3/check/test-release-openstack/cd3a31f/job-output.txt

Source code from these logs: https://github.com/openstack/python-openstackclient

README file: https://raw.githubusercontent.com/openstack/python-openstackclient/refs/heads/master/README.rst

[1]https://zuul.opendev.org/t/openstack/builds?job_name=test-release-openstack&skip=0

What operating system are you using?

Linux

If you selected 'Other', describe your Operating System here

Ubuntu 24.04

What version of Python are you running?

Python 3.12

How did you install twine? Did you use your operating system's package manager or pip or something else?

sudo python -m pip install twine!=1.12.0 requests-toolbelt!=0.9.0

(from https://opendev.org/zuul/zuul-jobs/src/commit/7932e7ccc9d689e8612322292386740d5726c18d/roles/ensure-twine/tasks/main.yaml)

What version of twine do you have installed (include the complete output)

$ twine --version
twine version 6.1.0 (keyring: 25.6.0, packaging: 24.2, requests: 2.32.3, requests-toolbelt: 1.0.0, urllib3: 1.26.20, id: 1.5.0)

Which package repository are you using?

pypi

Please describe the issue that you are experiencing

(already described)

Please list the steps required to reproduce this behaviour

Using the OpenStack CI. The steps are:

• Build the sdist of the project
• Execute twine

Anything else you'd like to mention?

No response

[dnicolodi]

I assume the the metadata or the build process for python-openstackclient has not changed, and thus the metadata looks something like this https://inspector.pypi.io/project/python-openstackclient/7.2.1/packages/5e/df/ffc770c88d551f1845411efa572a9657f65300e24a82c5a558c5139b86c3/python-openstackclient-7.2.1.tar.gz/python-openstackclient-7.2.1/PKG-INFO

If this is the case, I see where the problem is: in twine 6.1 packaging is used to parse the metadata and it handles the continuation lines in the Description field differently than what pkginfo did in previous versions of twine. The metadata specification is very vague on how to treat continuation lines, thus both behaviors are correct according to the specification. However, the one chosen by packaging breaks this use case. I'll fix this ASAP.

It should also be noted that serializing the Description field like it is done here is the deprecated version of doing it (it is deprecated because of the precise problem of handling the continuation lines). Which version of setuptools is used to build the python-openstackclient distribution?

[ralonsoh]

Hi @dnicolodi. I was wrong when I provided the description of this error. I thought it was using Ubuntu 24.04, but is still using some old Ubuntu 20.04 VMs, thus the Python version is 3.8 and setuptools is 45.2.0-1.

How should we use "description"? If I'm not wrong and according to [1], the "description" field should be one line only. The README file should be embedded in "long_description", right?

[1]https://docs.python.org/3.11/distutils/setupscript.html

[ralonsoh]

For example, this is also failing in OpenStack Neutron. Here we are using:

• description_file = README.rst
• long_description_content_type = text/x-rst

E.g.: https://review.opendev.org/c/openstack/neutron/+/939806

Result: https://9441e1808f41b6da8d93-dcb12bf222f1c96a489b3ed464d5a157.ssl.cf1.rackcdn.com/939806/12/check/test-release-openstack/dfe25b5/job-output.txt

[dnicolodi]

Python version is 3.8 and setuptools is 45.2.0-1

That's a fairly old setuptools, and it explains why the metadata is generated the way it is.

How should we use "description"? If I'm not wrong and according to [1], the "description" field should be one line only. The README file should be embedded in "long_description", right?

Your project uses description_file in setup.cfg. I don't find this documented anywhere. However, at a cursory look at the generated metadata file, without double checking the metadata specification, it seems that the metadata is correct, despite using a deprecated way to store the description.

However, what matters is that twine was happy with your package metadata before, and it is not happy now. This needs to be fixed.

[dnicolodi]

Looking at the specification, it seems that setuptools is at fault here.

The specification for the Description field is here https://packaging.python.org/en/latest/specifications/core-metadata/#description and it appears that setuptools is encoding it wrong: the continuation lines are prefixed by 7 spaces, but the pipe character is missing, cfg https://inspector.pypi.io/project/python-openstackclient/7.2.1/packages/5e/df/ffc770c88d551f1845411efa572a9657f65300e24a82c5a558c5139b86c3/python-openstackclient-7.2.1.tar.gz/python-openstackclient-7.2.1/PKG-INFO

I guess a compatibility kludge for this setuptools non-standard behavior needs to be implemented.

[dnicolodi]

This is how pkginfo handles the description field:

def _collapse_leading_ws(header, txt):
    """
    ``Description`` header must preserve newlines; all others need not
    """
    if header.lower() == 'description':  # preserve newlines
        return '\n'.join([x[8:] if x.startswith(' ' * 8) else x
                          for x in txt.strip().splitlines()])
    else:
        return ' '.join([x.strip() for x in txt.splitlines()])

Therefore it seems that pkginfo supports the invalid format used by setuptools, but not the standard prescribed format.

[dnicolodi]

@ralonsoh Is this still an issue for OpenStack? There is a fix for the issue in #1220 but it has not received much attention

[pekkaklarck]

I believe I have encountered this when creating Robot Framework 7.3rc1. I first created the release packages by running:

python setup.py sdist bdist_wheel

After that twine check with twine 6.1.0 failed:

$ twine check dist/*
Checking dist/robotframework-7.3rc1-py3-none-any.whl: PASSED
Checking dist/robotframework-7.3rc1.tar.gz: FAILED
ERROR    `long_description` has syntax errors in markup and would not be rendered on PyPI.                                                                                                                                                    
         line 2: Severe: Unexpected section title or transition.                                                                                                                                                                              
                                                                                                                                                                                                                                              
         ===============                                                                 

I downgraded to twine 5.1.1 and there were no problems:

$ twine check dist/*
Checking dist/robotframework-7.3rc1-py3-none-any.whl: PASSED
Checking dist/robotframework-7.3rc1.tar.gz: PASSED

I then tried uploading with twine 6.1.0, but that failed:

$ twine upload dist/*
Uploading distributions to https://upload.pypi.org/legacy/
Uploading robotframework-7.3rc1-py3-none-any.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 802.5/802.5 kB • 00:01 • 537.1 kB/s
Uploading robotframework-7.3rc1.tar.gz
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 651.0/651.0 kB • 00:01 • 640.6 kB/s
WARNING  Error during upload. Retry with the --verbose option for more details.                                                                                                                                                               
ERROR    HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/                                                                                                                                                                      
         The description failed to render for 'text/x-rst'. See https://pypi.org/help/#description-content-type for more information.          

After downgrading again, upload succeeded:

$ twine upload dist/*
Uploading distributions to https://upload.pypi.org/legacy/
Uploading robotframework-7.3rc1-py3-none-any.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 802.8/802.8 kB • 00:01 • 975.6 kB/s
Uploading robotframework-7.3rc1.tar.gz
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 650.1/650.1 kB • 00:00 • 114.3 MB/s

View at:
https://pypi.org/project/robotframework/7.3rc1/

The distribution can be installed without problems and the description at https://pypi.org/project/robotframework/7.3rc1/ looks fine as well.

Our plan is to move to more modern packaging tech later this year, but it would naturally be nice if twine still worked with setuptools. We can keep using twine 5.1.1 (or 6.0.1 that also seems to work) as well, though.