Skip to content

Refactor build scripting to support Python 3 only builds, pinned build dependency versions, optional system installed build dependencies #416

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 39 commits into from
Apr 9, 2018
Merged

Refactor build scripting to support Python 3 only builds, pinned build dependency versions, optional system installed build dependencies #416

Show file tree
Hide file tree
Changes from 32 commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
8490024
add support for pipenv Python dependency version pinning
chrissimpkins Mar 7, 2018
384f3fa
copies build-ttf.sh as build-ttf-system.sh to use as basis for script…
chrissimpkins Mar 7, 2018
fe6e6ba
[build-pipenv.sh] added new shell script that supports builds of venv…
chrissimpkins Mar 8, 2018
e3c0e97
updated script documentation
chrissimpkins Mar 8, 2018
1700e78
[build-pipenv.sh] modified if blocks in shell script
chrissimpkins Mar 8, 2018
74d359e
[build-pipenv.sh] add Python interpreter version to environment report
chrissimpkins Mar 8, 2018
0eb8a96
[build-ttf.sh] transitioned Python build tool calls to use of pipenv …
chrissimpkins Mar 8, 2018
176a14a
[build-subsets.sh] eliminated build dependency checks, these are now …
chrissimpkins Mar 8, 2018
bda185c
[build-subsets.sh] transitioned to use of pipenv virtualenv installs …
chrissimpkins Mar 8, 2018
da20882
[Makefile] added pipenv venv setup script to the build process
chrissimpkins Mar 8, 2018
7de5d65
added Python interpreter version definition to virtualenv scripting
chrissimpkins Mar 15, 2018
10c5444
modified build variable name
chrissimpkins Mar 15, 2018
aff9f86
removed unnecessary build shell script
chrissimpkins Mar 15, 2018
4d8a9a1
added new sfnt2woff-zopfli local install script
chrissimpkins Mar 15, 2018
953c35b
[build-ttf.sh] updated script to support pinned and system installed …
chrissimpkins Mar 15, 2018
14b1912
[Makefile] updated make targets
chrissimpkins Mar 15, 2018
dd71f61
[build-woff.sh] added support for pinned version and system installed…
chrissimpkins Mar 15, 2018
353455d
[build-ttf.sh] fixes incorrect executable path checks with system ins…
chrissimpkins Mar 15, 2018
0da551b
[build-woff.sh] refactored default build approach path test for depen…
chrissimpkins Mar 15, 2018
c0f9b0c
added new woff2-compress-build shell script for local builds of woff2…
chrissimpkins Mar 15, 2018
f1c3c53
[Makefile] updated make target shell script path for local woff2_comp…
chrissimpkins Mar 15, 2018
8b05246
[build-woff2.sh] added support for pinned dependency builds and rolli…
chrissimpkins Mar 15, 2018
63c6f05
[build-woff2.sh] removed unnecessary variable
chrissimpkins Mar 15, 2018
83eac63
[Makefile] updated shellcheck target deps
chrissimpkins Mar 15, 2018
2b1d273
[build-subsets.sh] updated shell script to support pinned build depen…
chrissimpkins Mar 15, 2018
843e689
new build-archives.sh shell script
chrissimpkins Mar 15, 2018
1028b7e
[Makefile] updated archives make target with new shell script path
chrissimpkins Mar 15, 2018
842a21f
[Makefile] added subsets-system make target
chrissimpkins Mar 16, 2018
4c48555
[Makefile] added subsets-system target to .PHONY
chrissimpkins Mar 16, 2018
e20d786
[fix-dsig.py] removed from __future__ imports, code formatting = pass…
chrissimpkins Mar 16, 2018
0a01f04
[fix-fstype.py] source code formatting changes = passes flake8 tests
chrissimpkins Mar 16, 2018
96a35cb
[woff2-compress-build.sh] convert to upstream woff2 at v1.0.2 (includ…
chrissimpkins Mar 16, 2018
ee180fb
[build-pipenv.sh] modified command to create the pipenv virtual envir…
chrissimpkins Mar 16, 2018
8b7feda
[Makefile] added documentation and process specific target arrangement
chrissimpkins Mar 17, 2018
d6df277
[BUILD.md] updated build documentation with v4.x build tooling changes
chrissimpkins Mar 18, 2018
1dd3664
[BUILD.md] added TOC
chrissimpkins Mar 18, 2018
b1ebe12
[BUILD.md] minor text revisions
chrissimpkins Mar 18, 2018
bc6a683
[BUILD.md] added documentation for system PATH installed build depend…
chrissimpkins Mar 18, 2018
2a46cad
[woff2-compress-build.sh] modified woff2_compress to use brotli v1.0.3
chrissimpkins Mar 21, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 40 additions & 15 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,40 +1,65 @@
all: build

archives:
postbuild_processing/archive_prep/archiver.sh
./build-archives.sh

build: ttf webfonts

build-with-dependencies: source/*.ufo
./build-ttf.sh --install-dependencies
./build-woff.sh --install-dependencies
./build-woff2.sh --install-dependencies
./build-subsets.sh
build-with-dependencies: compile-local-dep ttf webfonts

build-system: ttf-system webfonts-system

build-local-ttfa:
tools/scripts/install/ttfautohint-build.sh

build-local-sfnt2woffzopfli:
tools/scripts/install/sfnt2woff-zopfli-build.sh

build-local-woff2:
tools/scripts/install/woff2-compress-build.sh

compile-local-dep: build-local-ttfa build-local-sfnt2woffzopfli build-local-woff2

css:
tools/scripts/css/css-build.sh

lint: shellcheck ufolint

shellcheck: build-ttf.sh build-woff.sh build-woff2.sh build-subsets.sh tools/scripts/install/ttfautohint-build.sh postbuild_processing/archive_prep/archiver.sh tools/scripts/css/css-build.sh
pipenv:
./build-pipenv.sh

shellcheck: *.sh tools/scripts/css/*.sh tools/scripts/install/*.sh
$@ $^

subsets: source/*.ufo
subsets: pipenv
./build-subsets.sh

ttf:
subsets-system:
./build-subsets.sh --system

ttf: pipenv
./build-ttf.sh

ttf-system:
./build-ttf.sh --system

ufolint: source/*.ufo
$@ $^

webfonts:
./build-woff.sh
./build-woff2.sh
./build-subsets.sh
webfonts: woff woff2 subsets

woff:
webfonts-system: woff-system woff2-system subsets-system

woff: pipenv
./build-woff.sh

woff2:
woff-system:
./build-woff.sh --system

woff2: pipenv
./build-woff2.sh

woff2-system:
./build-woff2.sh --system

.PHONY: all archives build build-with-dependencies build-local-ttfa build-local-sfnt2woffzopfli build-local-woff2 build-system compile-local-dep css lint pipenv shellcheck subsets subsets-system ttf ttf-system ufolint webfonts webfonts-system woff woff-system woff2 woff2-system
20 changes: 20 additions & 0 deletions Pipfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
[[source]]

url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"


[packages]

fonttools = "*"
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you don't need to specify fonttools in addition to fontmake, because the former is a dependency of the latter.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Safe to depend on the fact that this will always be the case? We have local Python scripts that depend on fontTools and will not pull it in with a pip install. pipenv simply ignores it since it was pulled in with fontmake.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, if you've scripts that directly depend on fonttools then it makes sense to have it there.

fontmake = "*"


[dev-packages]



[requires]

python_version = "3.6"
144 changes: 144 additions & 0 deletions Pipfile.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 5 additions & 5 deletions ...build_processing/archive_prep/archiver.sh → build-archives.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,18 @@

# ////////////////////////////////////////////////////////////////////
#
# archiver.sh
# build-archives.sh
# A shell script that packages .zip, tar.gz, and tar.xz font archives
# Copyright 2018 Christopher Simpkins
# MIT License
#
# Usage: ./archiver.sh
# Usage: ./build-archives.sh
#
# ////////////////////////////////////////////////////////////////////

HACK_VERSION="v3.003"
HACK_ARCHIVES_DIR="../../../Hack-archives"
HACK_BUILD_DIR="../../build"
HACK_VERSION="v4.000"
HACK_ARCHIVES_DIR="../../Hack-archives"
HACK_BUILD_DIR="build"


# Make build directory the current working directory
Expand Down
54 changes: 54 additions & 0 deletions build-pipenv.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
#!/bin/sh

# /////////////////////////////////////////////////////////////////
#
# build-pipenv.sh
# A shell script that creates a virtualenv for Hack font builds
# Copyright 2018 Christopher Simpkins
# MIT License
#
# Usage: ./build-pipenv.sh
#
# /////////////////////////////////////////////////////////////////

BUILD_PYTHON_VERSION="3.6"

if ! which pipenv
then
echo "Unable to detect a pipenv install. Please install with 'pip install pipenv' then repeat your build attempt." 1>&2
exit 1
fi

# install fontTools and fontmake build dependencies with pipenv
pipenv install --python $BUILD_PYTHON_VERSION --ignore-pipfile fontmake fontTools
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if you checked in the Pipfile (which already states the top-level dependencies), why do you need to duplicate them here on the command line? Shouldn't pipenv install just work?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Believe that you are correct. We should be able to simply do this now that I have defined the venv in my development repo with the above command:

pipenv install --ignore-pipfile

My understanding is that without the --ignore-pipfile flag, it defaults to the specification in the Pipfile rather than Pipfile.lock (i.e. `ignore pipfile.lock is default). Kenneth's docs state that Pipfile should keep versions unpinned. Unclear why this is the approach. It seems to me that if a Pipfile.lock is present, the behavior should be to default to pinned versions defined there, else use Pipfile spec which is defined by the versions contained in the current versions of the dependencies at any given time.

I will test in a cloned repo so that I can replicate what a user's development repo will see when they go through the build.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hm.. what's the point of you checking in the Pipfile.lock if you're going to ignore it?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, ok --ignore-pipfile option

Ignore Pipfile when installing, using the Pipfile.lock.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kenneth's docs state that Pipfile should keep versions unpinned.

correct. Because the abstract Pipfile is what generates the concrete Pipfile.lock (a bit like setup.py vs requirements.txt)

If you did pyenv install without the --ignore-pipfile the Pipfile.lock would probably be updated with the current versions? I'm not a pipenv users so I'm not familiar with it.

Copy link
Member Author

@chrissimpkins chrissimpkins Mar 17, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you did pyenv pipenv install without the --ignore-pipfile the Pipfile.lock would probably be updated with the current versions? I'm not a pipenv users so I'm not familiar with it.

Yes, believe that this is correct Cosimo. Docs state that it is a goal of the project to use current versions of packages when possible for safety (security patch releases) reasons. You can default to Pipfile.lock specified pinned installs with the above flag as I understand it.

Updated the shell script with your suggestion here and it is working as intended. It eliminates forced rebuilds of the entire venv with each font compile. Thanks for catching this!

Copy link
Member Author

@chrissimpkins chrissimpkins Mar 17, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will be interested to see how this works once system Py3 bumps to Py3.7 on my development machine. The intended behavior is for pyenv to automatically pull the Py3.6 interpreter and install into the venv when it is not available locally. pipenv is supposed to automate this if pyenv is installed.


# test for fontmake install in venv
if ! pipenv run fontmake --version
then
echo "Unable to detect fontmake install with pipenv. Please repeat your build attempt." 1>&2
exit 1
fi

# test for fontTools install in venv
if ! pipenv run python -c "import fontTools"
then
echo "Unable to detect fontTools install with pipenv. Please repeat your build attempt." 1>&2
exit 1
fi

# print environment used for build to std output stream

echo "================================="
echo " PYTHON BUILD ENVIRONMENT"
echo "================================="
echo " "
echo "Python interpreter version:"
pipenv run python --version
echo " "
pipenv graph

echo " "
echo "================================="
echo " "
echo "================================="
echo " "
Loading