[build system] look for system-wide installed binaries

[paride]

A minor, wishlist issue relevent for downstream packaging.

It would be nicer if the build scripts did look if the sfnt2woff-zopfli-build and woff2_compress binaries are installed system-wide (i.e., in the $PATH), before looking in the user's home directory.

As an alternative, it would be nice to be able to override the SFNTWOFF_BIN and WOFF2_BIN variables with environment variables.

[chrissimpkins]

We're currently doing this with our Python build dependencies and I was actually thinking through whether to begin to pin those to specific versions. The problem on our end is that it is very difficult to keep up with all of the build dependency releases and confirm that there were not changes that affect our compiles in a way that requires manual visual review before we release. To my knowledge, there is not an approach available to automate testing of rendering diffs so that we can have confidence that the shapes that users see do not change as build dependencies are modified. It is possible to perform a text dump of the binary data in the font files then diff this, but they are enormous files and the last attempt at this with my visual diff/merge tool ran my system out of memory... When you can view the diff, there are always changes with timestamps and other parts of the compiled file (version number strings, etc) that change so there is not a simple approach to say that the file did not change vs. it did change.

I would definitely be willing to implement the env variable override of these settings with the above caveat. If you do override, I would suggest that you be prepared to visually proof the files or have some form of diff approach on hand before you release them to Debian/Ubuntu users. Is this something that you would like to PR in to the shell scripts that we use for the build?

[paride]

I have prepared a simple patch so the scripts fallback to system-wide installed binaries, but only if these are not found in the "unprivileged" ($HOME/...) installation directories used by --install-dependencies. The "unprivileged" installation always takes precedence, so the behavior of your script is largely untouched. I'll wait for the sfnt2woff-zopfli package to be accepted in Debian before submitting the PRs (I think @fabiangreffrath will review it shortly), to be sure that the patches do work well for what will finally land in Debian.

About the visual check, no problem in doing it. I may prepare a LaTeX document to compile and check before releasing new versions. I want the Debian packaged Hack font to be excellent.

[chrissimpkins]

Prefer to use an environment variable approach instead of a hard coded path so that this can be modified by others on different paths as necessary? I liked this idea that you had. It seems to be the most flexible for all. Thoughts?

[chrissimpkins]

About the visual check, no problem in doing it. I may prepare a LaTeX document to compile and check before releasing new versions. I want the Debian packaged Hack font to be excellent.

Paride, we modify the version strings as compiled with fontmake from UFO using the font-v tool. This takes place outside of the current scripted build workflow. I haven't added another dependency because I know that you and the Fedora team are working through the licensing issues with the current set of dependencies that we have here. It isn't mandatory but it might be helpful to you. We currently add the git commit SHA1 hash at build time to the version string so that we have a link back to the source state at build. This git commit SHA1 stamp is automated (detection of the git commit and write to the OpenType tables) with the tool. It also provides the option to add any Debian and/or Ubuntu specific metadata to the version string that might be helpful to you in these packaged builds. This can be any free text that you'd like.

It's available here if any of this is of use:

https://github.com/source-foundry/font-v

cc: @spstarr : see above as it also applies to Fedora builds if helpful

[paride]

Thanks Chris. I think we know quite precisely what is built and distributed in Debian (it's a precise git tag). At the same time we can't assume the font is built from a git repository (it is built from a tarball that is generated from git, but it's not a git repository itself). Still adding some information could be useful, even the simple fact that it is a Debian build. If I package font-v I'll let you know...

[chrissimpkins]

Sounds good. Figured it would be worth mentioning as there may be Debian specific metadata now or in the future that is worth capturing in your builds.

[fabiangreffrath]

We currently add the git commit SHA1 hash at build time to the version string so that we have a link back to the source state at build.

We should make sure that this doesn't get in the way of the reproducible builds effort:
https://wiki.debian.org/ReproducibleBuilds

[chrissimpkins]

@fabiangreffrath @paride see the new issue reports that I just posted today about pinning Python dependencies and using an explicit git commit time to define data in the OpenType tables:

#400

#401

Please weigh in and let us know how this affects you on Debian.

[chrissimpkins]

We are trying to make these builds completely determined by source at a specific commit. The compiler currently timestamps OpenType tables in the font binaries so the time of build will make your text diff of an OpenType table dump and hash comparison of font binaries differ from those that we push as releases based upon this difference (but nothing else in the font binary changes). Interested in your opinions about these issues from the Linux packaging standpoint.

[chrissimpkins]

If helpful, we will be adding this shell script to the repository. It does not add any additional dependencies (save for git but I am assuming that is not an issue...) and can be used to directly diff the OpenType tables across any two font builds.

#402

[chrissimpkins]

We should make sure that this doesn't get in the way of the reproducible builds effort:
https://wiki.debian.org/ReproducibleBuilds

@fabiangreffrath this is not currently implemented as part of our make based build workflow. I suggested this to Paride as it is an option if you need to maintain any state metadata in the compiled binaries for any reason. As is, your make builds will compile a version string with the definitions that fontmake uses based upon the source files along with ttfautohint applied metadata. This will include version strings that look like this:

Version X.XXX; [ttfautohint metadata]

Our changes do not lead to non-reproducible builds assuming that you build at the same git commit. Our release file version strings look like this:

Version X.XXX;abcd123-release;[ttfautohint metadata]

where abcd123 represents the git commit sha1 short hash for the git commit used to build and -release indicates that it is a release build (contrasted with -dev builds that represent WIP towards the next release).