There appears to be a canonical regex for package names, unless I'm getting that wrong. I'm not quite sure why we're not using it, though.

It does not seem to be compatible with unicode packages, though, so we should probably try and fix this at the spec level.

Unfortunately it does seem to be the only way to go at this time.
The official spec for identifiers is:

The identifier syntax is <XID_Start> <XID_Continue>.
ID_Start is defined as all characters having one of the general categories uppercase letters (Lu), lowercase letters (Ll), titlecase letters (Lt), modifier letters (Lm), other letters (Lo), letter numbers (Nl), the underscore, and characters carrying the Other_ID_Start property. XID_Start then closes this set under normalization, by removing all characters whose NFKC normalization is not of the form ID_Start ID_Continue anymore.

(From PEP 3131.)

Unfortunately to use these unicode classes, we have to use the regex, and not the builtin re module. Am I right that adding regex as a dependency to setuptools would be a bad idea?

If so, we are I believe effectively stuck with ugly unicode ranges.

Unfortunately it does seem to be the only way the go at this time.
The official spec for identifiers is:

That is for identifiers, this is for package names and versions. It's not clear to me that they are the same (though there's some case to be made that they should be harmonized in the official spec).

Unfortunately to use these unicode classes, we have to use the regex, and not the builtin re module. Am I right that adding regex as a dependency to setuptools would be a bad idea?

If so, we are I believe effectively stuck with ugly unicode ranges.

Well, the other option is to not use a regular expression, for example, this will cover most of what your regex does. Presumably unicodedata can also get the Other_ID_start property as well.

def fc(x):
    out = ""
    allowed_categories = {"Lt", "Lm", "Lu", "Ll", "Lt", "Lm", "Lo", "NI"}
    for c in x:
        if unicodedata.category(c) in allowed_categories:
            out += c
        else:
            out += "-"
    return re.sub("-+", "-", out).strip("-")

But you'll note that in fact "safe names" are not valid identifiers, since they actually replace all _s with -.

Am I right that adding regex as a dependency to setuptools would be a bad idea.

I'd like to think not, but it's not as simple as adding a dependency in another project. Any dependency (and its dependencies) must be vendored. This shouldn't be done with abandon, as it does impact downstream system packagers that work to unvendor the vendored packages.

Thank you for the feedback! I think I will try out @pganssle 's approach first. By the way, pardon my ignorance, but what is the practical difference between the package "safe name" and its identifier in terms of how Python (or setuptools) uses them?

@julienmalard The two things are nearly entirely different. For something to be a valid identifier means that it can be used as a variable name in Python. For something to be a valid package name means that it can be used for the name of a package to be installed. For example my-package is a valid package name, but not a valid identifier, and in fact my-package, my_package and My-Package all refer to the same package (my-package is the canonical name).

Thank you @pganssle for your clear explanation!
I have implemented your alternative to the regex as well and it seems to be passing all tests now.

Please let me know what you think!

I started down the path of removing the stylistic changes, and re-organizing the commits into first a test failure. However, I noticed that safe_name isn't implicated in the traceback of the original report, so I'm not even confident the additional test here exhibits the failure reported in the issue.

It's not obvious to me that the reported issue is even related to the fix herein. Let's go back to the issue and work on creating a minimal reproducer that exhibits the failure.