4.9.1 has been retagged

[dvzrv]

Describe the bug
Hi! 👋
I package scons as a package for Arch Linux.
It appears that 4.9.1 has been retagged (see https://gitlab.archlinux.org/archlinux/packaging/packages/scons/-/issues/1).
This is a problem for downstreams, as it means that a) reproducibility is broken and b) anyone trying to build this package from source (e.g. other distributions) will not be able to do so.

Please create a new tag instead of re-tagging as this creates a lot of unnecessary work downstream.

Required information

• Link to SCons Users thread discussing your issue.: n/a
• Version of SCons: 4.9.1
• Version of Python: 3.13.2
• Which python distribution if applicable (python.org, cygwin, anaconda, macports, brew,etc) Arch Linux
• How you installed SCons: from source
• What Platform are you on? (Linux/Windows and which version): Arch Linux
• How to reproduce your issue? Please include a small self contained reproducer. Likely a SConstruct should do for most issues.: n/a
• How you invoke scons (The command line you're using "scons --flags some_arguments"): n/a

[bdbaddog]

I guess you didn't read the header which asks you to bring your issue to the mailing list or discord server before posting.

Where are you pulling your .tag.gz's from?
sourceforge or github?
(or pypi?)

[bdbaddog]

I looked at my notes, we found a couple errors where the version # hadn't been changed in CHANGES.txt

BTW we have a dedicated channel on our discord server for project maintainers and packagers.
We may push a new patch release shortly, or we can retag or update the tarball depending where you're pulling from.

[dvzrv]

I guess you didn't read the header which asks you to bring your issue to the mailing list or discord server before posting.

Sorry, but I maintain several hundreds of packages. I simply do not have the time to follow up on a mailing list or a proprietary chat platform (imagine being asked this by n upstreams that each have their own specific platforms on top of a simple issue tracker).
Development of this project appears to happen here, so that is where I am reporting issues with the projects.

Where are you pulling your .tag.gz's from?
sourceforge or github?
(or pypi?)

Currently we use both the locked, signed tag of this repository and the prebuilt sources from sourceforge (see https://gitlab.archlinux.org/archlinux/packaging/packages/scons/-/blob/feec8c772898ebd3b575ae85ad585a20ea407066/PKGBUILD#L33-34).
The latter we only use because we are unable to build the man pages ourselves from the git sources.

[bdbaddog]

@dvzrv - re tons of packages. Fair enough. (BTW thanks for your work on such.. we know it's mostly thankless)

Ok so if we update sourceforge's packages to match the tag this issue would be resolved for you?

[bdbaddog]

Do you need assistance in being able to build the manpages for SCons?

[mwichmann]

I know Debian has problems doing the full doc build as part of their validation... we can't really make the pdf generation quieter, as it's two pieces out of our control: docbook stylesheets (an old docbook, for that matter) vs. Apache fop. But you can skip that - this is a much quieter and quicker build of just the manpages, and without generating pdfs of them:

scons doc SKIP_DOC=pdf,api

[dvzrv]

Ok so if we update sourceforge's packages to match the tag this issue would be resolved for you?

The problem is not a tag/ custom source tarball mismatch in this case, but that which commit the tag 4.9.1 points at has been changed after pushing the tag.

As background info about downstream distributions and their builds: When we initially encounter a tag we lock the checksum of the content that it points at.
If the tag is changed, the checksum changes and successive package builds for the "same" sources fail (this breaks our reproducible-builds effort, as well as any distribution relying on our package sources).
FWIW: Changing the tag also changes the auto-generated tarball for the tag, so those kinds of sources are affected as well.

The downside with such a situation is, that we have to follow up with upstream to figure out what went wrong and afterwards adjust the locked checksum our build script.
Creating a new tag (e.g. 4.9.2) circumvents the situation from happening.
As such, this ticket is probably more of an FYI, so thank you for following up, it is really much appreciated!

But you can skip that - this is a much quieter and quicker build of just the manpages, and without generating pdfs of them:

scons doc SKIP_DOC=pdf,api

Thank you for the pointer, I will try that and report back!

[bdbaddog]

So sounds like you'd have to lock the hash between the initial tag and any updates to the tag?
With 4.9.1 a typo in CHANGES.txt was found the next day so that's why it was edited and the tag updated.
The errors in the CHANGES.txt would have made the existing 4.9.1 incorrect (at least the history would have been).

Are you automatically detecting on every tag push? Or is this a daily automated task, or other?
Trying to understand if there's any timeframe during which we can just update the tag and rebuild without breaking your process.

[mwichmann]

I think the information for us is to not move tags, once they're assigned. I don't recall that's happened in "recent memory". We had the odd case where there were some very minor niggles in files that don't contribute to operation at all - version strings that hadn't been updated in the changelog - which is what led to a tweak this time. I'd think we could just agree to "not do that".

[bdbaddog]

I think the information for us is to not move tags, once they're assigned. I don't recall that's happened in "recent memory". We had the odd case where there were some very minor niggles in files that don't contribute to operation at all - version strings that hadn't been updated in the changelog - which is what led to a tweak this time. I'd think we could just agree to "not do that".

Yup. Though in the past I'm pretty sure we've updated the tags, but fairly quickly. and rebuilt. Some of our release machinery could use some improvement. I'll work on that shortly...

[dvzrv]

Are you automatically detecting on every tag push? Or is this a daily automated task, or other?
Trying to understand if there's any timeframe during which we can just update the tag and rebuild without breaking your process.

You may be lucky if you move it really fast. However, as soon as the tag is pushed, semi-automated systems will pick it up and lock the checksum accordingly (for Arch this is currently still a mix of automated tooling and manual intervention, but we are currently working on a system that will drive the automation all the way to the "acknowledge that you want to publish this updated package" bit, which will make this process even faster and the "window of opportunity" much much smaller).

I think the information for us is to not move tags, once they're assigned.

That would be much appreciated and help all downstreams! 🙏

[dvzrv]

To quickly respond to the suggestion wrt manpage generation:

It appears the command as such does not work. After building the wheel from git sources and installing scons temporarily to a custom location, I'm running into rather obscure error messages that I don't understand (probably warrants a separate ticket):

scons: Building targets ...
/usr/bin/xmllint --xinclude main.xml > scons_xi.xml
main.xml:34: parser warning : Entity 'buildversion' not defined
    <title>SCons &buildversion;</title>
                               ^
main.xml:46: parser warning : Entity 'buildversion' not defined
    <releaseinfo>Version &buildversion;</releaseinfo>
                                       ^
main.xml:68: parser warning : Entity 'MergeFlags' not defined
    This chapter describes the &MergeFlags;, &ParseFlags;, and &ParseConfig;
                                           ^
main.xml:68: parser warning : Entity 'ParseFlags' not defined
    This chapter describes the &MergeFlags;, &ParseFlags;, and &ParseConfig;
                                                         ^
main.xml:68: parser warning : Entity 'ParseConfig' not defined
    This chapter describes the &MergeFlags;, &ParseFlags;, and &ParseConfig;
                                                                            ^
main.xml:69: parser warning : Entity 'consenv' not defined
    methods of a &consenv;, as well as the <parameter>parse_flags</parameter>
                          ^
o scons_ex.xml /build/scons/src/scons/build/doc/xslt/xinclude_examples.xslt scons_xi.xml
sh: line 1: o: command not found
/usr/bin/xmllint --xinclude scons_ex.xml > scons_exi.xml
warning: failed to load external entity "scons_ex.xml"
scons: *** [scons_exi.xml] Error 4
scons: building terminated because of errors.
scons: *** [build/doc/user/index.html] Error 2
scons: building terminated because of errors.

Full build log: scons-4.9.1-2-x86_64-build.log

[mwichmann]

That means it's not picking up the bits that define the SCons-specific entities that extend DocBook. There's an earlier complaint that I don't recognize:

/usr/lib/python3.13/site-packages/setuptools/command/build_py.py:212: _Warning: Package 'SCons.Tool.docbook.docs' is absent from the `packages` configuration.

Which may be related.

Also, we're going to run into a problem we can't resolve, so there's little use going further down this path for now: lxml got stricter as of 5.0 and will not validate the SCons doc tree. Last I knew that was true for the generic docbook xsl files of that version, and not specific to the SCons extension. We can work around that for the package builds we do by populating a virtualenv with a version of lxml from PyPI that's pinned to < 5.0, but that's not available for Python 3.13 and greater, as the project has simply not released old wheels to enable that, unsurprisingly. I don't expect you'd want to build against an old lxml version anyway. We had a volunteer working on a fix, but that went silent, and we're otherwise lacking the necessary skills.

[mwichmann]

I poked a bit, and the path your build went down was trying to use the fallback if lxml is not found. It may be that code path has bitrotted in some way. At the point yours went south, mine looks like this (taking the lxml path):

cd /home/mats/github/scons/build/doc/user && "/home/mats/.pyenv/versions/venv-system312/bin/python3.12" /home/mats/github/scons/scripts/scons.py SKIP_PDF=1
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
__xinclude_lxml(["scons_xi.xml"], ["main.xml"])
__build_lxml(["scons_ex.xml"], ["scons_xi.xml"])
__xinclude_lxml(["scons_exi.xml"], ["scons_ex.xml"])
__build_lxml(["scons_db.xml"], ["scons_exi.xml"])
__build_lxml(["index.html"], ["scons_db.xml"])
__build_lxml_noresult(["scons-user/index.html"], ["scons_db.xml"])
scons: done building targets.

while yours had none of the xinclude's, thus the error messages are indeed not surprising.

[bdbaddog]

@dvzrv - Are you running pip install -U -r requirements-pkg.txt when building SCons?