Behaviour using version-pioneer for conda-forge builds

[will-s-hart]

When using versioneer to build conda-forge packages from GitHub archives, strings that were substituted into _version.py upon git archive are used to determine the version number during the conda-forge recipe build. Since git archive support is not included in version-pioneer, version inference fails during conda-forge recipe builds: instead, inferred versions are based on (incorrect) tags taken from a git repo on the build machine that contains the downloaded source code several directories down.

I'm going to try to get git archive working at least in my use case by editing the _version.py file. Not sure if there's an appetite to restore git archive support more widely - if not, hopefully my experience is nonetheless useful if anyone else wants to use version-pioneer with conda-forge (will link my version script in case useful for anyone if I get things working)

(Edited since I realised an earlier alternative attempt at fixing my conda-forge build wouldn't work)

[will-s-hart]

Update: copying the relevant code from versioneer for version inference based on keywords substituted upon git archive into my (non-vendored) version file - see here https://github.com/idm-oxford/climate-epidemics/blob/main/climepi/_version.py - as well as adding climepi/_version.py export-subst to my .gitattributes, seemed to do the job for me. May be useful to others but not sure how robust to different use cases it is.

[kiyoon]

I appreciate you sharing this!

Quick question: does conda require using git archive, or how is this feature related to conda builds?

The reason I didn't include git archive support is because

1. It depends on the git repo settings, which a lot of users can miss when simply copying the versionscript. My python project became huge and I often use a project template, and I often forgot to change git attributes after forking so it broke often. Thus I have the "single versionscript without other file dependency" philosophy, so it's more maintainable and possible to do whatever you want. Your approach is the correct way to use version-pioneer!
2. Archiving directly modifying the script may cause issues in production, IMO. If I used git archive, I personally prefer saving the info in an external dotenv or config file and let versionscript read the file. Everyone thinks differently, so I let them choose the workflow freely.

That said, I think your work is truly meaningful to others. I'm pasting the solution below just in case the link breaks in the future.

Solution adding git archive support

import re
from pathlib import Path
from typing import Any

from version_pioneer.api import get_version_dict_wo_exec


class NotThisMethod(Exception):
    """Exception raised if a method is not valid for the current scenario."""


def get_keywords() -> dict[str, str]:
    """Get the keywords needed to look up the version information."""
    # these strings will be replaced by git during git-archive.
    # setup.py/versioneer.py will grep for the variable names, so they must
    # each be defined on a line of their own. _version.py will just call
    # get_keywords().
    git_refnames = "$Format:%d$"
    git_full = "$Format:%H$"
    git_date = "$Format:%ci$"
    keywords = {"refnames": git_refnames, "full": git_full, "date": git_date}
    return keywords


def git_versions_from_keywords(
    keywords: dict[str, str],
    tag_prefix: str,
    verbose: bool,
) -> dict[str, Any]:
    """Get version information from git keywords."""
    if "refnames" not in keywords:
        raise NotThisMethod("Short version file found")
    date = keywords.get("date")
    if date is not None:
        # Use only the last line.  Previous lines may contain GPG signature
        # information.
        date = date.splitlines()[-1]

        # git-2.2.0 added "%cI", which expands to an ISO-8601 -compliant
        # datestamp. However we prefer "%ci" (which expands to an "ISO-8601
        # -like" string, which we must then edit to make compliant), because
        # it's been around since git-1.5.3, and it's too difficult to
        # discover which version we're using, or to work around using an
        # older one.
        date = date.strip().replace(" ", "T", 1).replace(" ", "", 1)
    refnames = keywords["refnames"].strip()
    if refnames.startswith("$Format"):
        if verbose:
            print("keywords are unexpanded, not using")
        raise NotThisMethod("unexpanded keywords, not a git-archive tarball")
    refs = {r.strip() for r in refnames.strip("()").split(",")}
    # starting in git-1.8.3, tags are listed as "tag: foo-1.0" instead of
    # just "foo-1.0". If we see a "tag: " prefix, prefer those.
    TAG = "tag: "
    tags = {r[len(TAG) :] for r in refs if r.startswith(TAG)}
    if not tags:
        # Either we're using git < 1.8.3, or there really are no tags. We use
        # a heuristic: assume all version tags have a digit. The old git %d
        # expansion behaves like git log --decorate=short and strips out the
        # refs/heads/ and refs/tags/ prefixes that would let us distinguish
        # between branches and tags. By ignoring refnames without digits, we
        # filter out many common branch names like "release" and
        # "stabilization", as well as "HEAD" and "master".
        tags = {r for r in refs if re.search(r"\d", r)}
        if verbose:
            print("discarding '%s', no digits" % ",".join(refs - tags))
    if verbose:
        print("likely tags: %s" % ",".join(sorted(tags)))
    for ref in sorted(tags):
        # sorting will prefer e.g. "2.0" over "2.0rc1"
        if ref.startswith(tag_prefix):
            r = ref[len(tag_prefix) :]
            # Filter out refs that exactly match prefix or that don't start
            # with a number once the prefix is stripped (mostly a concern
            # when prefix is '')
            if not re.match(r"\d", r):
                continue
            if verbose:
                print("picking %s" % r)
            return {
                "version": r,
                "full-revisionid": keywords["full"].strip(),
                "dirty": False,
                "error": None,
                "date": date,
            }
    # no suitable tags, so version is "0+unknown", but full hex is still there
    if verbose:
        print("no suitable tags, using unknown + full revision id")
    return {
        "version": "0+unknown",
        "full-revisionid": keywords["full"].strip(),
        "dirty": False,
        "error": "no suitable tags",
        "date": None,
    }


def get_version_dict():
    # NOTE: during installation, __file__ is not defined
    # When installed in editable mode, __file__ is defined
    # When installed in standard mode (when built), this file is replaced to a compiled versionfile.
    if "__file__" in globals():
        cwd = Path(__file__).parent
    else:
        cwd = Path.cwd()

    try:
        return git_versions_from_keywords(get_keywords(), tag_prefix="v", verbose=False)
    except NotThisMethod:
        pass

    return get_version_dict_wo_exec(
        cwd=cwd,
        style="pep440",
        tag_prefix="v",
    )

[will-s-hart]

Makes sense!

conda-forge builds packages from source tarballs, usually pulled from PyPI or from GitHub release artifacts (not full git checkouts). For packages built from GitHub release tarballs (as for my package - unfortunately I have a key conda-only dependency, so putting on PyPI doesn't make much sense), the git archive behaviour (as executed in generating those tarballs) is then what provides versioneer the info needed to infer the version number during conda-forge recipe builds.

[kiyoon]

okay, thanks for sharing. I would need to think of a solution for adding support for git release tarballs.

But if your directory name contains the version info, it reads from that as well. So if the name of the directory isn't changed from the source tarball, it will get the version, although I think it is a very hacky solution.

[will-s-hart]

Unfortunately the conda-forge build process renames the directory to something like 'work', so the directory name-based version inference can't be used (my unsuccessful first attempt to get things working involved swapping the order to try inferring version from directory name before git tags, because otherwise version-pioneer will try to take the version from a git repo a few levels up the build machine file system)