create an official Python port

[aspiers]

As discussed in #104 (comment), I'm increasingly convinced that it makes sense to port Stow to Python. Perl is a mostly dead language at this point (with no offence to the Perl community - I have a great fondness for and long history with Perl for the first chunk of my career). The reality is that despite looking for co-maintainers in #104, none have been forthcoming and I believe this is largely due to most people not knowing Perl these days.

Python is the obvious candidate for a port:

1. It's a hugely popular, well-known, and approachable language.
2. For the Stow use case, compiled languages are overkill. Its performance is entirely I/O-bound, not CPU-bound.
3. I know Python very well.
4. There are already at least two unofficial ports:
   • https://github.com/isarandi/stow-python which is a fantastic start (EDIT: see more details in comments below)
   • https://github.com/r-burns/stow.py

Also, AI tooling makes this increasingly easier, e.g. @isarandi managed his in about a day.

So at this point, the main thing I'm still debating is whether the port should be done:

1. in a subdirectory of the main branch of the current repo
2. in a python branch of the current repo
3. in a separate repo

Currently I'm leaning towards option 1, because:

1. this would allow a single commit to fix the same bug or add the same feature in both versions, and
2. it would allow the same test suite to test both versions.

These would be helpful because for a period of time both versions would need to be maintained in parallel to allow software distributions to migrate over.

Other considerations (added after lots of discussion below):

• Not planning any name change. It will still be GNU Stow, for many reasons.
• However, I think 3.x onwards should be the Python version only, which should avoid a lot of confusion. So 2.x would remain the Perl series which gets EOL'd at some point in time with a reasonable amount of notice.

[aspiers]

@isarandi So my current thinking for how to lock down both Perl and Python implementations to the exact same behaviour is to have a combination of integration and unit tests all inside the same branch:

• An integration test suite in Python that runs the stow/chkstow commands of both implementations, treating them as a black boxes rather than importing code and running it in a way that bypasses all of the command line parsing and so on. So to be clear, Python tests invoking both Perl and Python implementations.
• Keep the existing Perl unit tests which only test a single function at a time
• a similar Python unit test suite

Integration tests

These integration tests are the "meat" of the existing Perl tests. These are the ones which I would switch to the black box approach of testing their effects from the outside:

• chkstow.t
• cleanup_invalid_links.t
• dotfiles.t
• foldable.t
• stow.t
• unstow.t
• link_dest_within_stow_dir.t
• Some but not all of ignore.t?

Unit tests

These ones could be kept and reimplemented in Python (in fact you've already done that):

• cli_options.t
• cli.t
• defer.t
• examples.t
• find_stowed_path.t
• join_paths.t
• parent.t
• rc_options.t
• Some but not all of ignore.t?

Thoughts welcome!

[isarandi]

I've done pretty much this, comparing Perl and Python in their effects, generating test cases with the Hypothesis library (and some generated with Claude assistance). I'll push it on another branch to my repo in a few hours / half day.

[aspiers]

Currently I see:

https://github.com/isarandi/stow-python/blob/ee94368ad119a8f462e749b233e891dcc9505c95/tests/testutil.py#L98

where the tests instantiate a Stow object. If this was switched to just invoking the executable directly, then it could also test the Perl implementation. Maybe you're saying you've already done that switch but just not pushed it yet? If so, great!

[aspiers]

Another question is which version of the Stow codebase you based your port on? I have made some changes over recent months, although not too many.

[isarandi]

It's the 2.4.1 release. Changes should be fairly easy to port. You can check the pythonic refactor here: https://github.com/isarandi/stow-python/tree/pythonic

It supports library-like use as well, a calling stow() function from Python, the same CLI as the Perl one and also there is a script to bundle the multi-file codebase to a single py file for easier drop-in to $PATH.

And the new tests indeed compare directly calling the two CLI execuables.

I hope it helps!

[thesamesam]

Any rewrite is likely to introduce regressions (which is just how things go). I'd really suggest having a separate repo so it's very clear which versions of stow are Python and which aren't, and which had major changes and which didn't. It also means if people want to make minimal bug fixes to the old version, they can do so with more ease.

Dealing with packages as a distro maintainer where they got completely rewritten is usually very bumpy and more issues arise than expected - which is not to say you shouldn't do it, that's up to you, but I'd really recommend keeping it separate.

(Also, stow works just fine for me, and I'd prefer to have anything with a lot of churn (and possibly AI involvement in refactoring) completely elsewhere.)

There are already at least two unofficial ports:

Why not just continue development in one of those?

this would allow a single commit to fix the same bug or add the same feature in both versions, and

Backporting such commits won't be easy as a result though. I don't think this is an actual advantage even if it sounds nice in theory.

Also, having them in the same repo and the same branch (required by "single commit") would mean you need either two build systems, or one for each port.

[RaphGL]

Long time no see. I see that the rewrite is finally gonna happen.
I'm for any rewrite away from perl. But let me state my case against Python in hopes of that stirring the direction of the project even if a little bit.

Pros of using python:

• Same as perl, it comes pre-installed on linux and even on some windows machines
• Easy to use and beginner friendly
• "Batteries included"

Cons of using python:

• Distribution sucks, uv exists but it doesn't complete solve the problem, it's just a very good patch on top of a hacky solution. It's bad enough that even the port that prompted this issue resorts to doing a single file to avoid the tooling headache
• Not that stable, there are deprecations and subsequent removals in almost every release, python's "minor version" is a lie, the language will keep the python 3 label for as long as it can to avoid losing community good faith due to the disaster that was python 2
• You can't just ship the thing and be done with it, you need to install the appropriate python version, if the interpreter version changes behind your back, the code might or might not run

I know that you've discarded compiled languages for "not needing the performance" but in my opinion the reason to use one to begin with is that you can just keep the same binary forever and if you choose a language that is dogmatically into keeping stability (language stability not stability of the program), then you can essentially make the program and then let it exist forever until there's a real bit rot you have to handle.

Out of the languages out there, I think Go would likely be the better fit due to having an extensive stdlib, having simple tooling, being very stable and already having an ample community focused on building dev tools. So it achieves the goal of being attractive to new contributors.

[aspiers]

Any rewrite is likely to introduce regressions (which is just how things go).

Yes true, although I'm not really worried about that since we already have a comprehensive test suite which should get us to 99.9% compatibility straight away.

BTW @isarandi has already done an excellent job of documenting some differences:

https://github.com/isarandi/stow-python/blob/pythonic/docs/perl-differences.md

However I wouldn't really call these regressions; they're mostly corner cases which I expect hardly ever get hit, and even if they did, the fall-out would be minimal.

I'd really suggest having a separate repo so it's very clear which versions of stow are Python and which aren't

I'm not yet convinced - I think clarity comes more from good communication and documentation than from where code is located. But still open to different takes on this.

and which had major changes and which didn't. It also means if people want to make minimal bug fixes to the old version, they can do so with more ease.

Who are "people" in this context? 😜 Despite clear calls for help with maintenance, no one has stepped forward. (BTW not complaining about that! It's entirely understandable). So presumably this would only be me. And I wouldn't maintain the Perl version separately to a Python version; I'd maintain them in parallel for a while before EOL'ing the Perl version. The only question is how quickly I'd EOL the Perl version after releasing the Python one.

Dealing with packages as a distro maintainer where they got completely rewritten is usually very bumpy and more issues arise than expected - which is not to say you shouldn't do it, that's up to you, but I'd really recommend keeping it separate.

Thanks, this is very valuable feedback and I'd love to learn a bit more about the bumps you anticipate if you have time to explain. BTW I've also spent many years as a distro maintainer (openSUSE / SUSE) and can certainly confirm that switching an rpm .spec file from packaging Perl to packaging Python is not a small piece of work. I imagine a Debian package switch would be very similar.

That said, AI can make it quicker these days, and anyway I don't think most of that migration work would depend on whether the code lives in the same repo or a different one.

(Also, stow works just fine for me, and I'd prefer to have anything with a lot of churn (and possibly AI involvement in refactoring) completely elsewhere.)

I'm not aiming for any significant refactoring. Stow is already fairly reasonably structured.

There are already at least two unofficial ports:

Why not just continue development in one of those?

Well, firstly if I'm going to maintain something then I'll need full control over the repo. Maybe @isarandi would be open to transferring his. But anyway, creating a new repo or forking an existing one are both trivially easy. The reasons for keeping the two implementations in the same repo are mentioned above:

this would allow a single commit to fix the same bug or add the same feature in both versions, and

Backporting such commits won't be easy as a result though.

I wouldn't think of it as backporting, because the code changes would be made at basically the same time. And I don't see why they wouldn't be easy. If it's a bug in the underlying logic / design then most likely it won't be language-specific. And if it's language-specific then it only needs to be fixed on one side.

Also, having them in the same repo and the same branch (required by "single commit") would mean you need either two build systems, or one for each port.

We'll need two build systems regardless. The only difference is where they live.

Another reason to keep them unified is the documentation. I certainly don't want two parallel sets of documentation - that goes not just for the manual, but all the other docs.

[aspiers]

Long time no see. I see that the rewrite is finally gonna happen. I'm for any rewrite away from perl. But let me state my case against Python in hopes of that stirring the direction of the project even if a little bit.

Greatly appreciate the alternative view-point! It's very helpful for sanity-checking my assumptions.

Cons of using python:

• Distribution sucks, uv exists but it doesn't complete solve the problem, it's just a very good patch on top of a hacky solution. It's bad enough that even the port that prompted this issue resorts to doing a single file to avoid the tooling headache

Please can you elaborate? Funnily enough I've found the exact opposite with uv - a breath of "just works" fresh air after decades of miserably bad UX and DX for Python packaging. Why wouldn't installation would be as easy as uvx tool install stow.

• Not that stable, there are deprecations and subsequent removals in almost every release, python's "minor version" is a lie, the language will keep the python 3 label for as long as it can to avoid losing community good faith due to the disaster that was python 2

Can you give some examples of breaking changes which would be likely to affect Stow? Bearing in mind that in terms of language usage and interaction with the OS / libraries, it's a stupidly simple program which doesn't need to achieve anything other than reading/writing to the filesystem and outputting to STDOUT/STDERR. So I struggle to imagine how even irresponsible decisions in Python releases would disrupt it.

• You can't just ship the thing and be done with it, you need to install the appropriate python version, if the interpreter version changes behind your back, the code might or might not run

If Stow needed to make heavy usage of third party Python modules with non-Python (e.g. binary) dependencies, then I'd maybe worry about that. But it really doesn't, as explained above. I've used every Python major version since maybe 2003 or so, and I don't really recall any of the breakage you allude to. It's entirely possible I've missed things, in which case hearing about concrete examples of breakage would be very helpful.

I know that you've discarded compiled languages for "not needing the performance" but in my opinion the reason to use one to begin with is that you can just keep the same binary forever

That's only really true with statically-linked binaries, which is not a good way to ship user-facing software IMHO.

and if you choose a language that is dogmatically into keeping stability (language stability not stability of the program), then you can essentially make the program and then let it exist forever until there's a real bit rot you have to handle.

Again, with more complex programs with lots of dependencies, bit-rot is a valid concern, but not with Stow. It hasn't changed significantly for years. People seem to appreciate that it's highly stable, and just does one simple job well.

Out of the languages out there, I think Go would likely be the better fit due to having an extensive stdlib, having simple tooling, being very stable and already having an ample community focused on building dev tools. So it achieves the goal of being attractive to new contributors.

I have nothing against Go, but:

• All those things can also be said about Python IMHO.
• Stow doesn't need an extensive stdlib.
• It has less marketshare than Python.
• I have hardly any experience in Go, and no time to learn it properly. Sure AI can help a ton, but I would still be at a disadvantage.

[aspiers]

It's the 2.4.1 release. Changes should be fairly easy to port. You can check the pythonic refactor here: isarandi/stow-python@pythonic

It supports library-like use as well, a calling stow() function from Python, the same CLI as the Perl one and also there is a script to bundle the multi-file codebase to a single py file for easier drop-in to $PATH.

And the new tests indeed compare directly calling the two CLI execuables.

This is excellent work, thanks!

It looks to me that the original test suite still doesn't use the black-box method, but instead there are some new tests in https://github.com/isarandi/stow-python/blob/main/tests/test_oracle.py which do - do I understand correctly?

The new tests look great, and I love the approach of comparing STDOUT / STDERR and the filesystem state between the two implementations. However I really think we need to run the Python version of the existing (original) tests against both implementations, because they exercise a ton of edge cases and this would prove both (a) that the Python port of those tests is correct, and (b) that the new Python implementation passes them correctly.

[aspiers]

BTW I just checked and it seems the only changes I've made since 2.4.1 are external to the implementation, i.e. tests, docs, and other misc changes.

[RaphGL]

Please can you elaborate? Funnily enough I've found the exact opposite with uv - a breath of "just works" fresh air after decades of miserably bad UX and DX for Python packaging. Why woutldn't installation would be as easy as uvx tool install stow.

I've been away from doing anything in python that involves more than a single file for a while now (got tired of dealing with tooling)
So I haven't had much of a chance of using UV, but my friend has and has had very good impressions of it.

When I said it was a hack was that UV only builds on top of what is already there. It solves python's breaking between versions by just downloading a specified version. It solves the global dependencies by using "virtual environments" which was a hack created in 2012 by one guy and just spread around. (it works so can't complain ig?)

So I don't think UV is bad. but it's just the tool people are slowly converging to, it's not really a standard. So that's my only real criticism of it now.

If Stow needed to make heavy usage of third party Python modules with non-Python (e.g. binary) dependencies, then I'd maybe worry about that. But it really doesn't, as explained above. I've used every Python major version since maybe 2003 or so, and I don't really recall any of the breakage you allude to. It's entirely possible I've missed things, in which case hearing about concrete examples of breakage would be very helpful.

Breakages do happen if you go and read the changelogs you can see constant deprecations and removals. Obviously certain modules have more churn than others. And if you interact with C code you're a lot more likely to hit constant breakages.

I went looking for recent breaking changes in os, shutil and pathlib, I couldn't find anything major there, so these modules are stable (for now at least). But there's some deprecations with scheduled removals for the PathPure object in pathlib and re.split for regex.

That's only really true with statically-linked binaries, which is not a good way to ship user-facing software IMHO.

Go is statically linked yes. The most stable programs out there are old win32 programs and most of these programs ship their own DLLs instead of relying on a globally installed library. I'm not really sure why you think statically linking is bad for user facing software. Since they tend to reduce the amount of surface for breaking changes. The major advantage of dynamic linking is less effort for security patches and lower memory consumption (but modern software already takes a lot more memory than just statically linking a dependency for no reason)

---

It has less marketshare than Python.
I have hardly any experience in Go, and no time to learn it properly. Sure AI can help a ton, but I would still be at a disadvantage

You having experience with Python for a long time and unwilling/not having the time to learn a new language is a valid reason to use it to be honest. I'm just here playing devil's advocate.

[aspiers]

Please can you elaborate? Funnily enough I've found the exact opposite with uv - a breath of "just works" fresh air after decades of miserably bad UX and DX for Python packaging. Why woutldn't installation would be as easy as uvx tool install stow.

I've been away from doing anything in python that involves more than a single file for a while now (got tired of dealing with tooling) So I haven't had much of a chance of using UV, but my friend has and has had very good impressions of it.

When I said it was a hack was that UV only builds on top of what is already there. It solves python's breaking between versions by just downloading a specified version. It solves the global dependencies by using "virtual environments" which was a hack created in 2012 by one guy and just spread around. (it works so can't complain ig?)

So I don't think UV is bad. but it's just the tool people are slowly converging to, it's not really a standard. So that's my only real criticism of it now.

Ah, I see. I do agree that virtualenvs in their original form were pretty horrible, and remained that way for years afterwards with a series of incomplete solutions which all had problems and annoying differences. It's one of my biggest complaints with Python. However I think that era is mostly over now that we have uv.

Also I don't think there's anything fundamentally wrong with the idea of virtual environments or creating an environment which provides a specific language version. Indeed the same approach is used by:

• almost every single Docker container
• Ruby (via rvm and similar, plus bundler)
• NodeJS (via nvm and npm/yarn/pnpm)

and so on. Environment isolation does make for much more reliable results. It's just unfortunate that the trend emerged of doing it at the language level, which is a royal pain for distros built around package managers.

Breakages do happen if you go and read the changelogs you can see constant deprecations and removals. Obviously certain modules have more churn than others. And if you interact with C code you're a lot more likely to hit constant breakages.

I went looking for recent breaking changes in os, shutil and pathlib, I couldn't find anything major there, so these modules are stable (for now at least). But there's some deprecations with scheduled removals for the PathPure object in pathlib and re.split for regex.

Right. As already proven in https://github.com/isarandi/stow-python/, Stow doesn't need much beyond os, sys, re, and a few other core modules. Certainly not anything interacting with C code.

Deprecations and removals are fine as long as they're communicated and versioned clearly. BTW Python doesn't follow SemVer, so it's not reasonable to expect that a minor version bump won't break anything. But that's fine because I think they do a good job of communicating changes.

That's only really true with statically-linked binaries, which is not a good way to ship user-facing software IMHO.

Go is statically linked yes.

I believe it supports both static and dynamic linking.

I'm not really sure why you think statically linking is bad for user facing software. Since they tend to reduce the amount of surface for breaking changes. The major advantage of dynamic linking is less effort for security patches and lower memory consumption

Yes exactly.

(but modern software already takes a lot more memory than just statically linking a dependency for no reason)

Not sure I follow. We shouldn't give up on efficiency just because things are already wasteful, right?

You having experience with Python for a long time and unwilling/not having the time to learn a new language is a valid reason to use it to be honest.

Don't get me wrong, I'd love to be able to justify spending time learning Go properly. But I barely have time for Stow already, let alone extra burden.

I'm just here playing devil's advocate.

And I'm very grateful for that! It forces me to challenge my own assumptions, after which I'll either change my mind or have more conviction in my decision 😁