Skip to content

Commit 0b1284a

Browse files
brianeggeclaude
brianegge
and
claude
committed
Pin the generators the checked-in parser and lexer are made with (Covered by dco/Brian_Egge.md)
hexpr.parse.C, hexpr.parse.H and hexpr.lex.C are checked in, generated from hexpr.y and hexpr.l by pgen.sh -- with whatever bison and flex are on the path. Nothing recorded which, and it matters: regenerating the unmodified hexpr.l produced 997 lines of churn with Apple's flex and 809 with one GNU flex 2.6.4 build against the checked-in file from another, which is how morganstanley#548 came to carry a thousand-line diff for a two-line change to the lexer. A diff like that cannot be reviewed, and a generated file nobody can regenerate identically is one nobody can safely change. So pin them. pgen.sh now runs the generators in a container -- Ubuntu 24.04, pinned by digest, which packages GNU Bison 3.8.2 and flex 2.6.4, the versions the checked-in files name -- using podman or docker, whichever is installed. PGEN_NATIVE=1 runs them from the path instead, for CI and for a machine that has exactly those versions, and refuses anything else by whole version line, since Apple's flex calls itself 2.6.4 too. One thing the script now also does: re-apply the single hand edit the checked-in parser carries. bison's generated cleanup frees the parser stack only if it was moved off the initial array, GCC cannot see that and warns (-Wfree-nonheap-object, gcc bugzilla 98753), and -Werror turns the warning into a failed build, so someone once spelled the condition out by hand in hexpr.parse.C. Regenerating dropped it. pgen.sh puts it back, so the edit survives regeneration and is documented where it is made. With the pin, regenerating all three files from the unmodified grammar and lexer -- on top of morganstanley#548, whose lexer was regenerated with this same flex -- reproduces the checked-in files byte for byte. A new workflow does exactly that on every change under lib/hobbes/read/pgen or include/hobbes/read/pgen and fails on any difference: a change to the grammar has to come with its regenerated output, and that output has to be the pinned generators'. Built on morganstanley#548, which it depends on: the pinned flex produces morganstanley#548's lexer, not the one before it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016rGT4C394qeh2DBQhqy3Tb
1 parent 16fef2a commit 0b1284a

2 files changed

Lines changed: 127 additions & 8 deletions

File tree

‎.github/workflows/pgen.yml‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
name: "Generated parser"
2+
3+
# hexpr.parse.C, hexpr.parse.H and hexpr.lex.C are checked in, generated from
4+
# hexpr.y and hexpr.l by the bison and flex versions lib/hobbes/read/pgen/pgen.sh
5+
# pins. This regenerates them with those versions and fails if the result
6+
# differs from what is checked in: a change to a grammar file has to come with
7+
# its regenerated output, and the output has to come from the pinned
8+
# generators, so that the diff reviewers see is the change and not the
9+
# generator.
10+
on:
11+
push:
12+
branches: [main]
13+
paths:
14+
- 'lib/hobbes/read/pgen/**'
15+
- 'include/hobbes/read/pgen/**'
16+
- '.github/workflows/pgen.yml'
17+
pull_request:
18+
paths:
19+
- 'lib/hobbes/read/pgen/**'
20+
- 'include/hobbes/read/pgen/**'
21+
- '.github/workflows/pgen.yml'
22+
23+
permissions:
24+
contents: read
25+
26+
jobs:
27+
regenerate:
28+
# ubuntu-24.04 packages the pinned versions (flex 2.6.4, bison 3.8.2), so
29+
# the generators run natively here; pgen.sh checks the versions before it
30+
# runs them, and locally it runs the same thing in a container of the same
31+
# release
32+
runs-on: ubuntu-24.04
33+
steps:
34+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
35+
- name: Install the pinned generators
36+
run: |
37+
sudo apt-get update
38+
sudo apt-get install -y --no-install-recommends bison flex
39+
bison --version | head -1
40+
flex --version
41+
- name: Regenerate
42+
run: PGEN_NATIVE=1 lib/hobbes/read/pgen/pgen.sh
43+
- name: Compare with the checked-in files
44+
run: |
45+
git status --short -- lib/hobbes/read/pgen include/hobbes/read/pgen
46+
git diff --exit-code -- lib/hobbes/read/pgen include/hobbes/read/pgen

‎lib/hobbes/read/pgen/pgen.sh‎

Lines changed: 81 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,86 @@
11
#!/usr/bin/env bash
2+
# Regenerate the parser and lexer from hexpr.y and hexpr.l.
3+
#
4+
# The generated files are checked in, and the generators they were made with
5+
# are pinned: GNU Bison 3.8.2 and flex 2.6.4, as packaged by Ubuntu 24.04.
6+
# Regenerating with anything else produces a large diff of unrelated churn --
7+
# a different flex build of the same version number changes several hundred
8+
# lines -- which buries the real change and cannot be reviewed. So by default
9+
# this script runs the generators in a container pinned to that image, with
10+
# podman or docker, whichever is installed. CI regenerates the same way and
11+
# fails if the checked-in files differ from what the pinned generators
12+
# produce, so a change to hexpr.y or hexpr.l has to come with its regenerated
13+
# output, made by this script.
14+
#
15+
# lib/hobbes/read/pgen/pgen.sh # regenerate in the pinned container
16+
# PGEN_NATIVE=1 lib/hobbes/read/pgen/pgen.sh # use bison/flex from $PATH (CI, or
17+
# # a machine with exactly these versions)
18+
set -euo pipefail
219

3-
cd `dirname ${BASH_SOURCE[0]}`
20+
cd "$(dirname "${BASH_SOURCE[0]}")"
421

5-
# generate the LALR(1) parser and token definitions
6-
${BISON:-bison} -d -ohexpr.parse.C hexpr.y
7-
sed -i 's,#include "hexpr.parse.H",//&,' hexpr.parse.C
22+
# Ubuntu 24.04 (noble) as of the digest below ships flex 2.6.4-8.2build1 and
23+
# bison 2:3.8.2+dfsg-1build2. The digest pins the image; the versions are
24+
# asserted after install, so a moved image or repository cannot silently
25+
# change what generates these files.
26+
PGEN_IMAGE="docker.io/library/ubuntu:24.04@sha256:7607b6f97024ef850f1bd6e91a89273beb5973d04432c5b87f15f813d64b9c05"
27+
PGEN_FLEX_VERSION="2.6.4"
28+
PGEN_BISON_VERSION="3.8.2"
829

9-
# obey internal convention for division of source and header files
10-
mv hexpr.parse.H ../../../../include/hobbes/read/pgen/
30+
generate() {
31+
# the whole version line, not just the number: Apple's flex reports itself
32+
# as "flex 2.6.4 Apple(flex-35)" and generates several hundred lines
33+
# differently from GNU flex 2.6.4
34+
local have_flex have_bison
35+
have_flex="$(flex --version 2>/dev/null | head -1 || true)"
36+
have_bison="$(bison --version 2>/dev/null | head -1 || true)"
37+
if [ "$have_flex" != "flex $PGEN_FLEX_VERSION" ] || [ "$have_bison" != "bison (GNU Bison) $PGEN_BISON_VERSION" ]; then
38+
echo "pgen.sh: need GNU flex $PGEN_FLEX_VERSION and GNU Bison $PGEN_BISON_VERSION on \$PATH" >&2
39+
echo "pgen.sh: have '${have_flex:-no flex}' and '${have_bison:-no bison}'" >&2
40+
echo "pgen.sh: run without PGEN_NATIVE to use the pinned container instead" >&2
41+
exit 1
42+
fi
1143

12-
# generate the lexer to tokenize string input
13-
${LEX:-flex} -ohexpr.lex.C hexpr.l
44+
# generate the LALR(1) parser and token definitions
45+
bison -d -ohexpr.parse.C hexpr.y
46+
sed -i 's,#include "hexpr.parse.H",//&,' hexpr.parse.C
47+
48+
# bison's generated cleanup frees the parser stack only if it was moved off
49+
# the initial array, but GCC cannot see that and warns (-Wfree-nonheap-object,
50+
# https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98753) -- which -Werror turns
51+
# into a failed build. Spell the condition out where GCC can see it. This is
52+
# the one hand edit the checked-in parser carries.
53+
sed -i 's|^ YYSTACK_FREE (yyss);$| YYSTACK_FREE (yyss == yyssa ? nullptr : yyss);|' hexpr.parse.C
54+
sed -i 's|^#ifndef yyoverflow$|#ifndef yyoverflow\n // false positive https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98753|' hexpr.parse.C
55+
56+
# obey internal convention for division of source and header files
57+
mv hexpr.parse.H ../../../../include/hobbes/read/pgen/
58+
59+
# generate the lexer to tokenize string input
60+
flex -ohexpr.lex.C hexpr.l
61+
}
62+
63+
if [ -n "${PGEN_NATIVE:-}" ]; then
64+
generate
65+
exit 0
66+
fi
67+
68+
if command -v podman >/dev/null 2>&1; then
69+
RUNTIME=podman
70+
elif command -v docker >/dev/null 2>&1; then
71+
RUNTIME=docker
72+
else
73+
echo "pgen.sh: neither podman nor docker found; install one, or set PGEN_NATIVE=1 with flex $PGEN_FLEX_VERSION and bison $PGEN_BISON_VERSION on \$PATH" >&2
74+
exit 1
75+
fi
76+
77+
# the repository root is mounted so the header can land in include/
78+
ROOT="$(cd ../../../.. && pwd)"
79+
exec "$RUNTIME" run --rm \
80+
-v "$ROOT:/hobbes:Z" -w /hobbes/lib/hobbes/read/pgen \
81+
-e PGEN_NATIVE=1 \
82+
"$PGEN_IMAGE" \
83+
bash -c 'export DEBIAN_FRONTEND=noninteractive
84+
apt-get update -qq >/dev/null
85+
apt-get install -y -qq --no-install-recommends bison flex >/dev/null
86+
exec ./pgen.sh'

0 commit comments

Comments
 (0)