Skip to content

docs-infra: move 1.3 legacy PDF upload from Cloudflare REST to R2 S3 API - #2

Draft
andamasov wants to merge 2 commits into
productionfrom
claude/r2-s3-pdf-upload
Draft

docs-infra: move 1.3 legacy PDF upload from Cloudflare REST to R2 S3 API#2
andamasov wants to merge 2 commits into
productionfrom
claude/r2-s3-pdf-upload

Conversation

@andamasov

@andamasov andamasov commented Aug 17, 2026

Copy link
Copy Markdown
Member

What

Moves the 1.3 legacy PDF seeding step in .github/workflows/deploy.yml off the Cloudflare REST object endpoint (wrangler r2 object get/put) and onto R2's S3-compatible API (aws s3api head-object / put-object).

The connection shape follows the precedent set by docs-preview-deploy.yml and docs-preview-cleanup.yml in vyos/vyos-documentation (introduced in vyos-documentation#2140) — same endpoint form, same AWS_DEFAULT_REGION: auto, same secret names. Precedent for the shape only: those workflows' S3 write path has never actually executed (the preview deploy has always exited at its authorization gate, and the nightly cleanup has only ever listed), so it is not evidence that the AWS CLI's defaults transact successfully against R2. See the checksum section below.

Why

The PDF step runs before the Worker deploy and gates it: if the PDF cannot be secured in R2, no Worker goes live. That preserves the spec §5 site+PDF consistency invariant, but it also means the step blocks unrelated changes when it fails.

Two dispatches on 2026-08-15 — runs 31885425647 and 31886303447 — both failed with Cloudflare 10429 Too Many Requests on PUT /accounts/*/r2/buckets/vyos-docs-artifacts/objects/legacy/1.3/vyos-documentation.pdf.

The cause of that 429 is not established. Size is not it (29.2 MB against a 300 MB REST ceiling), and the identical command succeeded on 2026-07-10. This PR does not claim to fix the 429. The rationale is that the REST object endpoint is the wrong tool for bulk object transfer — Cloudflare's own documentation for that endpoint says:

For most production workloads, it is recommended to use the S3-compatible API or a Cloudflare Worker with an R2 binding instead of this direct API method.

Changes

Existence probe. Was wrangler r2 object get --pipe, which downloads the entire 29.2 MB object purely to test existence — the CLI exposes no head-only verb. Now aws s3api head-object: metadata only. This alone removes the bulk of the traffic against the rate-limited key.

Probe failures are classified, not swallowed. vyos-documentation#2209 fixed a near-identical bug where a probe that caught every failure could not tell "object absent" from an auth or network error, so a transient failure silently took a safe-looking path. The failure signalling here was established empirically against the AWS CLI rather than assumed:

Condition exit status stderr
absent 254 An error occurred (404) when calling the HeadObject operation: Not Found
permission denied 254 An error occurred (403) when calling the HeadObject operation: Forbidden
unreachable endpoint 255 Could not connect to the endpoint URL: "..."

404 and 403 share an exit status, so the exit code alone cannot be trusted. Only an explicit (404) / (NoSuchKey) stderr shape falls through to fetch+upload; everything else fails the step loudly with the captured stderr.

Upload. aws s3api put-object, not aws s3 cpcp switches to a multipart upload above its 8 MiB threshold, which would put this object on R2's composite-checksum path; put-object is one deterministic single-part PUT, far below the 5 GiB single-PUT ceiling. --content-type application/pdf is explicit and load-bearing: wrangler r2 object put infers the type from the file extension, the S3 API does not, and the apex Worker serves the object's stored type. Without it the PDF would be served as application/octet-stream.

Cached-object validation (from Phase 0 review). The skip path previously accepted any object that merely existed. The probe now reads ContentLength + ContentType in the same HEAD — no extra request — and refuses to short-circuit on an object outside the 25–35 MiB bound or not typed application/pdf, rather than letting a truncated or mistyped seed become the deploy's §5 fallback source.

Post-upload verification. A head-object size comparison after the PUT, replacing the on-the-wire integrity check disabled below.

Removed: the npm ci (existed only for npx wrangler; the Deploy step keeps its own) and CLOUDFLARE_API_TOKEN (unused by this step now) from the PDF step.

⚠️ Repo secrets required before this workflow can run

R2_ACCESS_KEY_ID and R2_SECRET_ACCESS_KEY do not exist in this repository yet. The same-named pair exists in vyos/vyos-documentation from IS-579. This workflow will fail until an operator adds them here. The account ID keeps coming from the existing CLOUDFLARE_ACCOUNT_ID secret, via R2_ENDPOINT.

The aws-chunked / checksum risk

R2 implements only part of S3's request-integrity surface: for a single-part (FULL_OBJECT) PUT it supports CRC64NVME and not CRC32/CRC32C/SHA-1/SHA-256. Cloudflare's guidance for its own Java SDK is to disable chunked encoding, which otherwise produces a signature mismatch (HTTP 403). boto3/botocore ≥ 1.36 defaults request_checksum_calculation to when_supported, and the AWS CLI v2 is built on botocore, so it is exposed to the same behaviour.

What I did: pinned both AWS_REQUEST_CHECKSUM_CALCULATION and AWS_RESPONSE_CHECKSUM_VALIDATION to when_required, so the request is a plain, unadorned PutObject. Env-var spelling and semantics verified against the AWS SDK/CLI reference; the runner (ubuntu-latest → Ubuntu 24.04) ships AWS CLI 2.36.20, well past the 2.23 release that introduced these settings.

Honest caveat: AWS CLI v2's default checksum algorithm is CRC64NVME, which R2 does support for FULL_OBJECT — so the default might well have worked. What cannot be determined without live R2 is whether the CLI delivers it as a plain header or as an aws-chunked trailing checksum. when_required is the conservative setting, not a claim that the default would have failed.

This behaviour is unverifiable outside a live run against real R2 — no local test can exercise it. It is the highest-risk part of this change. The post-upload size read-back is the compensating control for the integrity check given up.

Verification

No live R2 exercise was possible (no credentials in this repo, and dispatching a deploy to test is out of scope).

  • actionlint — clean on the changed workflow, and clean on origin/production baseline. No new findings.
  • shellcheck on the extracted step body — clean (only SC2050 on the force_pdf_refresh literal, an artifact of substituting the GitHub expression for the local run).
  • Stubbed-aws fixtures — the run: body is extracted from the YAML with yq (not hand-copied) and run against stubbed aws/curl. 8/8 pass:
scenario result put-object attempted
object present + force_pdf_refresh=false skips, rc=0 no
object present + force_pdf_refresh=true re-fetches + re-uploads, rc=0 yes
object absent fetch + upload, rc=0 yes
auth failure (403) loud fail, rc=1 no
network failure loud fail, rc=1 no
post-upload size mismatch loud fail, rc=1 yes
cached object truncated loud fail, rc=1 no
cached object mistyped loud fail, rc=1 no

The auth- and network-failure cases assert that no upload was attempted — the vyos-documentation#2209 lesson in executable form.

Preserved unchanged

force_pdf_refresh semantics, the strictly-between 25–35 MiB size check, the %PDF- header check, the fail-fast ordering ahead of the Worker deploy, and the idempotent short-circuit.

Phase 0 review disposition

Two local CodeRabbit rounds. Round 1 (cached object accepted without validation) is fixed — see "Cached-object validation" above. Round 2 suggested pinning a trusted SHA-256 digest of the PDF and verifying stored objects against it via R2 checksum storage; declined: there is no independently trusted digest to pin (the artifact is fetched from RTD, so a hardcoded digest would only record an unverified fetch), and it would make the step depend on exactly the R2 checksum path this PR cannot verify without a live run.

Why merging this matters beyond the PDF

workers/branch/wrangler.legacy.jsonc on vyos/vyos-documentation@rolling already carries "html_handling": "none", but the legacy Worker was last deployed 2026-07-10 and still runs auto-trailing-slash. Docs 1.3 and 1.2 currently return 307 on every explicit .html URL. Because the PDF step gates the deploy, that fix cannot land until this step can complete.

Advances: IS-572

🤖 Generated by robots

The PDF step gates the Worker deploy (fail-fast, spec §5 site+PDF
consistency). Two dispatches on 2026-08-15 failed with Cloudflare 10429
on the REST object endpoint, blocking unrelated Worker changes. The
cause of the 429 is not established; the reason for this change is that
Cloudflare documents the REST object endpoint as the wrong tool for bulk
object transfer ("For most production workloads, it is recommended to
use the S3-compatible API or a Cloudflare Worker with an R2 binding
instead of this direct API method").

- Existence probe: `wrangler r2 object get --pipe` downloaded the whole
  29.2 MiB object just to test existence (the CLI has no head-only
  verb). Replaced with `aws s3api head-object` — metadata only.
- Probe failures are classified, not swallowed: 403 shares HeadObject's
  exit status 254 with 404 and connection errors exit 255, so only an
  explicit 404/NoSuchKey stderr shape may mean "absent"; auth, network,
  and permission failures fail the step loudly.
- Upload: `aws s3api put-object` (single-part PUT; `aws s3 cp` would
  multipart above 8 MiB) with an explicit `--content-type
  application/pdf`, which the S3 API does not infer from the extension.
- Request/response checksum calculation pinned to `when_required`: R2
  accepts only CRC64NVME for a single-part FULL_OBJECT PUT, and the AWS
  CLI opts into a computed checksum by default. A post-upload
  head-object size comparison replaces the integrity check given up.
- Dropped the now-unused `npm ci` and `CLOUDFLARE_API_TOKEN` from this
  step; the Deploy step keeps its own.

force_pdf_refresh semantics, the 25-35 MiB size check, the %PDF- header
check, the fail-fast ordering, and the idempotent short-circuit are
unchanged.
Phase 0 CodeRabbit: the skip path accepted any object that merely
existed. Now that the fresh-upload path verifies what it stored, hold
the cached object to the same bar — the existence probe reads
ContentLength + ContentType in the same HEAD (no extra request) and
fails loudly when the stored seed is outside the 25-35 MiB range or is
not application/pdf, rather than letting a truncated or mistyped object
become the deploy's §5 fallback source. min/max hoisted so the bound has
one definition shared by both paths.
@coderabbitai

coderabbitai Bot commented Aug 17, 2026

Copy link
Copy Markdown

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro

Run ID: 788fb909-27f3-4e57-bdf5-421ffcb15352

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant