(PTFE-3070) Add multi part upload for large file

[matthiasL-scality]

This pull request updates the GitHub Actions workflows to use the latest actions/checkout@v6 and introduces improved multipart upload support for large files in the artifact upload action. It also updates the test workflows to cover the multipart upload path, and updates the container image reference for artifacts. These changes improve compatibility, performance, and test coverage.

Workflow and Dependency Updates:

• All workflows now use actions/checkout@v6 instead of older versions, ensuring compatibility and access to the latest features and security updates. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21]

• The artifacts container image in test-promote.yaml is updated from registry.scality.com/artifacts/artifacts:4.2.6 to ghcr.io/scality/artifacts:4.3.3, ensuring the use of the latest image from GitHub Container Registry.

Multipart Upload Support:

• Added multipart upload support for files larger than 100 MB in src/artifacts.ts, with each part set to 64 MB and up to 4 parts uploaded concurrently. This improves reliability and performance for large file uploads.

• A new test job, test-upload-multipart, is added to test-upload.yaml to verify multipart upload functionality and ensure file integrity for files above the 100 MB threshold.

• The artifact upload workflow dependencies are updated to include the new multipart upload test, ensuring it runs as part of the CI process.

These changes collectively modernize the CI workflows and strengthen support for large artifact uploads.

[claude]

• No timeout on raw https.request calls in fileUploadPresigned (line 144) and fileUploadMultipart.uploadPart (line 286): a stalled S3 connection will hang the action indefinitely. Add a timeout option and a req.on('timeout', ...) handler to destroy hung requests.
  • Add timeout: 300000 to the request options object
  • Add req.on('timeout', () => req.destroy(new Error('S3 upload timed out'))) after creating the request
• Multipart initiate (line 235) and complete (line 371) calls are not wrapped in retryWithBackoff, unlike the individual part uploads. A transient failure on either call fails or wastes the entire upload.
  • Wrap both in retryWithBackoff — both operations are idempotent

Review by Claude Code

[claude]

• retryWithBackoff retries all errors including non-transient 4xx (401, 403), causing ~100s of wasted delay on auth failures. Consider skipping retries for 4xx like the existing retryArtifacts does for 5xx.
  • Add a guard to throw immediately on AxiosError with status 400-499
• fileStream / partStream not destroyed on retry failure in fileUploadPresigned (line 137) and uploadPart (line 292), leaking file descriptors (up to 10 per file).
  • Wrap the S3 PUT promise in try/finally with stream.destroy()
• node20 to node24 in action.yaml is a breaking change for callers — should be called out prominently.
  • Consider node22 as a stepping stone, or document the requirement

Review by Claude Code

[claude]

• File descriptor leak on upload retry: fileStream (line 181) and partStream (line 350) in src/artifacts.ts are not destroyed when https.request errors or times out. On retry, new streams are created but old ones leak FDs. Fix: call stream.destroy() in the error handler.
  - Orphaned S3 parts when complete fails: the abort logic (lines 374-386) only covers part-upload errors. If the complete POST (line 402) fails after exhausting retries, parts remain in S3 consuming storage. Fix: move the complete call inside the existing try block.
  - Breaking change — node24 runtime: action.yaml now uses node24, which requires runner >= 2.323.0. Callers on older self-hosted runners will fail. Document this in release notes.
  
  Review by Claude Code

[claude]

• Breaking change: action.yaml jumps from node20 to node24, skipping node22. Self-hosted runners on versions older than v2.322.0 will fail. Document the minimum runner version in the release notes.

The multipart upload implementation is well-structured: proper abort on failure to avoid orphaned S3 parts, presigned URLs re-fetched on each retry attempt (handling expiry), resource cleanup via finally blocks, and a clear concurrency model (8 files × 4 parts = 32 peak connections matching maxSockets). The capability probing fallback to legacy fileUpload is a clean migration path.

Review by Claude Code

[claude]

• Breaking change not documented: action.yaml changes using: 'node20' to using: 'node24'. This affects every repo that consumes this action and is not mentioned in the PR description.
  • Call this out in the PR description and verify runner compatibility.
• S3 4xx errors retried unnecessarily: retryWithBackoff only short-circuits retries for AxiosError with status < 500, but presigned and multipart uploads use raw https.request and throw plain Error. Permanent S3 failures (403, 400) are retried 10 times (~3.5 min wasted).
  • Introduce a typed error with a status field so the retry logic can distinguish permanent from transient failures across both Axios and raw HTTP calls.
• Capability probe false positives on 5xx: probeServerCapabilities treats any non-404 response as capability available. A transient 500 would falsely enable multipart/presigned paths.
  • Add && resp.status < 500 to the probe check.

Review by Claude Code

[claude]

Breaking change: node20 → node24 changes the runtime for every consumer of this action. Workflows running on self-hosted runners without Node.js 24 will fail. This is independent of the multipart upload feature and should be called out in the PR description and release notes so callers know to expect a runtime change.

— Claude Code

[claude]

When one worker rejects, Promise.all rejects immediately but the remaining workers continue running. Their eventual rejections become unhandled promise rejections, which in Node.js 24 defaults to crashing the process (--unhandled-rejections=throw). Consider using Promise.allSettled and rethrowing the first failure so all worker promises settle cleanly before the catch block aborts the upload.

— Claude Code

[claude]

• action.yaml: node20 → node24 is a breaking runtime change for all consumers of this action, independent of the multipart feature. Should be documented in the PR description and release notes. - Call this out explicitly so callers on self-hosted runners without Node 24 aren't caught off guard.
  - src/artifacts.ts:393: Promise.all over the worker pool can leave orphaned worker promises whose rejections are unhandled under Node.js 24's default --unhandled-rejections=throw. - Use Promise.allSettled and rethrow the first failure, or drain the shared queue on error to prevent workers from picking up new parts.
  
  Review by Claude Code

[claude]

Breaking change: switching the action runtime from node20 to node24 affects every caller. Node.js 24 is still in Current (not LTS). Self-hosted runners without node24 installed will fail. This should be called out as a breaking change in the PR description.

— Claude Code

[claude]

• Breaking change not documented: action.yaml switches runtime from node20 to node24. Node.js 24 is Current, not LTS. Callers on self-hosted runners without node24 will break. This should be called out as a breaking change.
  - Document the node24 requirement in the PR description and consider whether this warrants a major version bump.
  - No unit tests for new multipart/presigned upload code: retryWithBackoff, fileUploadPresigned, fileUploadMultipart, and probeServerCapabilities are complex functions with retry logic and concurrency, but only covered by a workflow integration test. Unit tests would catch regressions faster and more reliably.
  - Add unit tests covering at least: retry behavior on transient vs permanent errors, multipart part counting, abort-on-failure path, and capability probe logic.
  
  Review by Claude Code