Add opt-in -reuse-build to go-publish: skip the redundant -publish re-render#1328
Open
KyleOps wants to merge 1 commit into
Open
Add opt-in -reuse-build to go-publish: skip the redundant -publish re-render#1328KyleOps wants to merge 1 commit into
KyleOps wants to merge 1 commit into
Conversation
…blish render) go-publish always re-renders the IG in PUBLICATION mode even though the pipeline already ran an -ig render, because the -ig (MANUAL) output bakes file:// self-paths and skips comparison pages. The -reuse-build flag (off by default) lets go-publish adopt a pre-built output that was itself rendered in PUBLICATION mode for the exact pathVer, verified via a new qa.json "publication-url" marker. Falls back to a full render on any mismatch, so it can never publish stale/non-publication output. For IGs with no related IGs the milestone render is reused too (render-identical to the -publish render).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When a publication pipeline runs an
-igbuild and then-go-publish, the IG is rendered twice —the pipeline's
-igstep, then go-publish's own internal-publishrender (and a third-milestonerender for milestone releases). Each render is many minutes, so the redundant one dominates publish time.
The re-render isn't gratuitous: a plain
-igbuild renders in MANUAL mode, so its output bakesfile://self-paths and skips the previous-version comparison pages — it isn't publishable. go-publishre-renders in PUBLICATION mode purely to bake the final canonical path (
pathVer). But if thepipeline's
-igstep is itself run in publication mode (-ig … -publish <path>), its output is alreadyexactly what go-publish would produce — so the re-render is pure duplication.
Change (opt-in, off by default)
A new
-go-publish -reuse-buildflag. When set, go-publish adopts the pre-builtsource/output/instead of re-rendering — only after verifying it is a PUBLICATION build for the exact
pathVer.Publisherwrites a"publication-url"marker intoqa.jsononly in PUBLICATION mode(additive; absent and inert for every other build).
PublicationProcess.doPublishreuses the output iffqa.publication-url == pathVer. Any mismatch(no marker, MANUAL build, stale path, unreadable qa.json) falls back to the existing full render —
it can never publish stale/non-publication output.
-milestonerender is reused too —isMilestoneBuild()only changes related-IG link targets, so with none declared the milestone render is byte-identical to
the
-publishrender. IGs with related IGs keep the dedicated-milestonerender.Default behaviour is unchanged — without
-reuse-build, the code path is identical to today.And it's forward/backward compatible:
-publish <path>is existing functionality, so a pipeline canadopt publication-mode
-igagainst an older jar that ignores-reuse-buildand simply re-renders asbefore; the speedup switches on once a jar with the flag is deployed. No flag-day.
Why it's a good change
Verification
End-to-end in the
hl7fhir/ig-publisher-basecontainer against a real IG (HL7 AU Base, ~5,600 outputfiles):
"publication-url": "<path>"toqa.json; the render alsoproduced the comparison pages and
665,871 links, 0 broken.[-reuse-build] Adopting pre-built publication output … (skipping the internal -publish render …)and went straight to the post-render steps;publish completed normally (version folder + regenerated history/feeds/registry/publish-box).
-igoutput (no marker),-reuse-buildlogs the mismatch andperforms the full render.
-reuse-buildand diffedthe two
fhir/<version>/folders:file://leaks; identicalcanonical paths.
build timestamps, randomly-generated element/anchor IDs (incl. comparison-table
filterTreeids),and HashMap/terminology-expansion ordering. (Expected — reuse copies bytes; it cannot introduce
content.)
reuse skips.
When to enable / trade-off
Low-risk by construction — it's opt-in and fail-closed: any marker mismatch falls back to the full
render, so it can never publish stale or non-publication output. The only requirement is that the
pipeline render its
-igstep in publication mode (-ig … -publish <path>); if it doesn't,-reuse-buildsimply re-renders as today (no speedup, no harm).
There's no real correctness trade-off — reuse copies the exact bytes the internal
-publishrenderwould produce (verified: 0 substantive content diffs). The only reason to leave it off is if you
deliberately want a second, independent render from source as a belt-and-suspenders check, or your
pipeline can't render
-igin publication mode. IGs with related IGs keep their dedicated-milestonerender automatically (reuse applies to the-publishstep) — not a user choice.Scope / notes
Publisher.recordOutcome(the additive marker) andPublicationProcess.doPublish(+ small private helpers).-igstep as-ig … -publish <path>and (b) passing-reuse-buildto-go-publish.