Skip to content

Add opt-in -reuse-build to go-publish: skip the redundant -publish re-render#1328

Open
KyleOps wants to merge 1 commit into
HL7:masterfrom
KyleOps:feat/reuse-build
Open

Add opt-in -reuse-build to go-publish: skip the redundant -publish re-render#1328
KyleOps wants to merge 1 commit into
HL7:masterfrom
KyleOps:feat/reuse-build

Conversation

@KyleOps

@KyleOps KyleOps commented Jun 18, 2026

Copy link
Copy Markdown

Problem

When a publication pipeline runs an -ig build and then -go-publish, the IG is rendered twice
the pipeline's -ig step, then go-publish's own internal -publish render (and a third -milestone
render for milestone releases). Each render is many minutes, so the redundant one dominates publish time.

The re-render isn't gratuitous: a plain -ig build renders in MANUAL mode, so its output bakes
file:// self-paths and skips the previous-version comparison pages — it isn't publishable. go-publish
re-renders in PUBLICATION mode purely to bake the final canonical path (pathVer). But if the
pipeline's -ig step is itself run in publication mode (-ig … -publish <path>), its output is already
exactly what go-publish would produce — so the re-render is pure duplication.

Change (opt-in, off by default)

A new -go-publish -reuse-build flag. When set, go-publish adopts the pre-built source/output/
instead of re-rendering — only after verifying it is a PUBLICATION build for the exact pathVer.

  • Publisher writes a "publication-url" marker into qa.json only in PUBLICATION mode
    (additive; absent and inert for every other build).
  • PublicationProcess.doPublish reuses the output iff qa.publication-url == pathVer. Any mismatch
    (no marker, MANUAL build, stale path, unreadable qa.json) falls back to the existing full render
    it can never publish stale/non-publication output.
  • For IGs that declare no related IGs, the -milestone render is reused too — isMilestoneBuild()
    only changes related-IG link targets, so with none declared the milestone render is byte-identical to
    the -publish render. IGs with related IGs keep the dedicated -milestone render.

Default behaviour is unchanged — without -reuse-build, the code path is identical to today.
And it's forward/backward compatible: -publish <path> is existing functionality, so a pipeline can
adopt publication-mode -ig against an older jar that ignores -reuse-build and simply re-renders as
before; the speedup switches on once a jar with the flag is deployed. No flag-day.

Why it's a good change

  • Cuts working publishes from 2 → 1 render and (related-IG-free) milestones from 3 → 1.
  • Opt-in + verified + fail-closed: zero risk to existing go-publish users.
  • General — benefits any multi-version IG with a render-then-publish pipeline, not a one-off.

Verification

End-to-end in the hl7fhir/ig-publisher-base container against a real IG (HL7 AU Base, ~5,600 output
files):

  1. Marker — a PUBLICATION render writes "publication-url": "<path>" to qa.json; the render also
    produced the comparison pages and 665,871 links, 0 broken.
  2. Reuse fires, render skipped — go-publish logged [-reuse-build] Adopting pre-built publication output … (skipping the internal -publish render …) and went straight to the post-render steps;
    publish completed normally (version folder + regenerated history/feeds/registry/publish-box).
  3. Fallback — against a MANUAL -ig output (no marker), -reuse-build logs the mismatch and
    performs the full render.
  4. Gold-standard A/B byte-diff — published the same source with vs without -reuse-build and diffed
    the two fhir/<version>/ folders:
    • same 5,616-file set; 0 substantive content differences; 0 file:// leaks; identical
      canonical paths.
    • The files that differ do so only by per-render nondeterminism present between any two renders:
      build timestamps, randomly-generated element/anchor IDs (incl. comparison-table filterTree ids),
      and HashMap/terminology-expansion ordering. (Expected — reuse copies bytes; it cannot introduce
      content.)
    • Wall-clock (go-publish phase): 725 s → 100 s (~86% faster) — the delta is exactly the render
      reuse skips.

When to enable / trade-off

Low-risk by construction — it's opt-in and fail-closed: any marker mismatch falls back to the full
render, so it can never publish stale or non-publication output. The only requirement is that the
pipeline render its -ig step in publication mode (-ig … -publish <path>); if it doesn't, -reuse-build
simply re-renders as today (no speedup, no harm).

There's no real correctness trade-off — reuse copies the exact bytes the internal -publish render
would produce (verified: 0 substantive content diffs). The only reason to leave it off is if you
deliberately want a second, independent render from source as a belt-and-suspenders check, or your
pipeline can't render -ig in publication mode. IGs with related IGs keep their dedicated
-milestone render automatically (reuse applies to the -publish step) — not a user choice.

Scope / notes

  • Independent of any other open PR; touches only Publisher.recordOutcome (the additive marker) and
    PublicationProcess.doPublish (+ small private helpers).
  • Pipelines opt in by (a) rendering their -ig step as -ig … -publish <path> and (b) passing
    -reuse-build to -go-publish.

…blish render)

go-publish always re-renders the IG in PUBLICATION mode even though the
pipeline already ran an -ig render, because the -ig (MANUAL) output bakes
file:// self-paths and skips comparison pages. The -reuse-build flag (off by
default) lets go-publish adopt a pre-built output that was itself rendered in
PUBLICATION mode for the exact pathVer, verified via a new qa.json
"publication-url" marker. Falls back to a full render on any mismatch, so it
can never publish stale/non-publication output. For IGs with no related IGs the
milestone render is reused too (render-identical to the -publish render).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

1 participant