tools: add script to upgrade operator-sdk by ezekiel-alexrod · Pull Request #4816 · scality/metalk8s

ezekiel-alexrod · 2026-03-12T13:29:56Z

Add tool to upgrade operator-sdk projects

Summary

A generic, config-driven tool that automates operator-sdk upgrades by scaffolding
fresh projects and applying customizations via GNU patch files. Designed to be
reusable across repositories.

Closes: MK8S-190

Execution flow

flowchart TD
    A["Load config.yaml"] --> B["Confirm upgrade (--yes to skip)"]
    B --> C["Install operator-sdk (cached)"]
    C --> D["Phase 1: Backup operator/ → operator.bak/"]
    D --> E["Phase 2: Scaffold fresh project"]
    E --> E1["operator-sdk init + create api"]
    E1 --> E2["Delete unwanted files (delete config)"]
    E2 --> F["Phase 2b: Detect latest versions"]
    F --> F1["operator-sdk: GitHub releases/latest"]
    F --> F2["Go toolchain: go.dev latest patch"]
    F --> F3["k8s.io libs: module proxy latest patch"]
    F1 & F2 & F3 --> G["Reconcile: compare detected vs pinned"]
    G --> H["Phase 3: Restore custom code (raw_copy)"]
    H --> I["Phase 4: Apply patch files"]
    I --> J["Substitute __GOTOOLCHAIN__ placeholder"]
    J --> K["Phase 5: go get k8s.io + go mod tidy"]
    K --> L["make manifests generate"]
    L --> M["make fmt vet"]
    M --> N["Extra commands (make metalk8s)"]
    N --> O["Upgrade complete"]

Usage

python3 tools/upgrade-operator-sdk/upgrade.py \
    --operator-dir operator \
    tools/upgrade-operator-sdk/operator

python3 tools/upgrade-operator-sdk/upgrade.py \
    --operator-dir storage-operator \
    tools/upgrade-operator-sdk/storage-operator

Option	Description
`--operator-dir`	Path to the operator project directory (required)
`--skip-backup`	Reuse an existing `.bak` directory
`--clean-tools`	Remove tool cache after upgrade
`--yes, -y`	Skip the confirmation prompt

Env variable	Description
`GITHUB_TOKEN`	Optional. Raises GitHub API rate limit (60 → 5000 req/hour)

Prerequisites: go, curl, patch, pyyaml

Configuration

Each operator has a config directory with config.yaml and patches/:

tools/upgrade-operator-sdk/
  upgrade.py
  operator/
    config.yaml
    patches/
      Dockerfile.patch          # extra COPY dirs, ldflags, LABEL block
      Makefile.patch            # GOTOOLCHAIN, metalk8s make target
      clusterconfig_types.patch # CRD type definition
      clusterconfig_controller.patch
      virtualippool_types.patch
      virtualippool_controller.patch
  storage-operator/
    config.yaml
    patches/
      Dockerfile.patch
      Makefile.patch
      volume_types.patch
      volume_controller.patch

config.yaml fields

Field	Required	Description
`operator_sdk_version`	yes	Target operator-sdk release (e.g. `v1.42.1`)
`repo`	yes	Go module path for `operator-sdk init --repo`
`domain`	yes	CRD domain for `operator-sdk init --domain`
`apis`	yes	List of CRDs with `group`, `version`, `kind`, `namespaced`
`go_toolchain`	no	Pin Go toolchain (e.g. `go1.24.13`). If absent, detected from go.dev
`k8s_libs`	no	Pin k8s.io libs (e.g. `v0.33.10`). If absent, detected from module proxy
`raw_copy`	no	Dirs/files to copy from backup (purely custom, no scaffold equivalent)
`delete`	no	Dirs/files to remove from scaffold after generation
`extra_commands`	no	Commands to run after `make fmt vet` (e.g. `["make", "metalk8s"]`)

Version reconciliation (CI-friendly)

After scaffolding, the tool queries go.dev, the Go module proxy, and GitHub to detect the latest available versions. It compares them with the YAML pins:

Situation	Behavior
No pin in YAML	Use detected version for this run (YAML not modified)
Pin matches detected	All good
Pin is older than detected	Warning with newer version available, keeps pinned value
Pin is newer than detected	Warning, uses detected value

Zero interactive input during reconciliation -- safe for CI.

Design decisions

Patch files over programmatic modifications: all scaffold customizations are plain diff -u files, readable and editable without understanding the Python code
raw_copy for purely custom code: only dirs/files with no scaffold equivalent are copied from backup; everything else uses patches
delete config field: configurable removal of scaffold files (.devcontainer, .github, incompatible test stubs)
--operator-dir as CLI arg: not in the YAML, so the tool works from any directory
No auto-pin: the tool warns about newer versions but never modifies the config file
__GOTOOLCHAIN__ placeholder: only runtime placeholder, detected dynamically (latest Go patch from go.dev) and substituted in the Makefile
Jinja expressions hardcoded in patches: each operator's Makefile.patch contains its own build_image_name() call directly, no placeholder needed
--forward flag on patch: prevents false "reversed patch" detection on patches that remove most of a file's content

Add a script to upgrade the operator-sdk It detects the latest compatible versions of operator-sdk, Go toolchain and golangci-lint from their respective public APIs, scaffolds fresh projects, and merges the existing custom code back. It applies to both `operator/` and `storage-operator/`. Closes: MK8S-190

bert-e · 2026-03-12T13:30:04Z

Hello ezekiel-alexrod,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

bert-e · 2026-03-12T13:30:12Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

Peer approvals must include at least 1 approval from the following list:

scripts/pyproject.toml

scripts/upgrade-operator-sdk.py

bert-e · 2026-03-13T14:28:33Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

Peer approvals must include at least 1 approval from the following list:

bert-e · 2026-03-13T17:01:55Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

Peer approvals must include at least 1 approval from the following list:

Keep scaffold-generated versions for Dockerfile and .golangci.yml, applying MetalK8s customizations programmatically on top (extra COPY layers, ldflags, LABEL block) via a new DockerfilePatch dataclass. This ensures upstream scaffold improvements are picked up automatically on future upgrades. Also cleans up redundant entries (.devcontainer, cover.out) from the merge policy and removes the golangci-lint install/migrate logic.

bert-e · 2026-03-16T13:11:43Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

Peer approvals must include at least 1 approval from the following list:

Using the tool for upgrading operator-sdk. See PR #4816 python scripts/upgrade-operator-sdk.py Closes: MK8S-110,MK8S-111

bert-e · 2026-03-16T16:56:37Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

Peer approvals must include at least 1 approval from the following list:

scripts/upgrade-operator-sdk.py

bert-e · 2026-03-18T15:17:06Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

Peer approvals must include at least 1 approval from the following list:

Replace the monolithic script with a generic, config-driven tool that takes an operator directory as argument. Each operator has its own config.yaml and patches/ directory under scripts/upgrade-operator-sdk/. All versions (operator-sdk, Go toolchain, k8s.io libs) are now pinned in the YAML config — the script makes no API calls for version detection. Key changes per review feedback: - YAML config per operator instead of hardcoded OPERATORS dict - GNU patch files for scaffold customizations (Dockerfile, Makefile) - Inclusion-list merge (backup_paths) instead of exclusion-list - No global mutable state (VersionInfo removed) - No SourceFix mechanism (manual fixes by the developer) - API scope (--namespaced) configurable per CRD - Empty group hack removed (operator-sdk v1.42.1 supports it) - Backup/scaffold file conflicts shown as errors with diff - Extra commands (make metalk8s) configurable in YAML

bert-e · 2026-03-19T08:10:23Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

Peer approvals must include at least 1 approval from the following list:

After scaffolding, read the generated go.mod to detect the latest stable Go patch (go.dev) and k8s.io patch (module proxy). Also check the latest operator-sdk release on GitHub. Version reconciliation is CI-friendly (zero interactive input): - No pin in YAML: auto-pin detected version and update the file - Pin matches detected: all good - Pin is older: warn with the newer version, keep pinned value - Pin is newer than detected: warn, use detected Re-adds GOTOOLCHAIN export to Makefile patches (single hunk at end).

TeddyAndrieux

I would suggest also to move this to tools it has nothing to do with scripts (this directory is for scripts that will be used by the end user not by the developers)

scripts/upgrade-operator-sdk/operator/config.yaml

scripts/upgrade-operator-sdk/upgrade.py

tools/upgrade-operator-sdk/upgrade.py

scripts/upgrade-operator-sdk.py

… new patches - Move operator_dir from YAML config to --operator-dir CLI argument - Remove REPO_ROOT and __file__-based path deduction (KISS) - Rename backup_paths to raw_copy, reduce to purely custom dirs (pkg/, version/, config/metalk8s/, salt/) - Generate patch files for CRD types and controllers (previously raw-copied from backup, now scaffold + patch) - Remove Dockerfile modification (scaffold FROM golang is kept as-is) - Fail on missing raw_copy paths instead of warning - Use Path.is_dir() instead of endswith("/") convention - Remove zz_generated filtering (no longer needed) - Neutralize scaffold-generated controller tests via patches (incompatible with our delegation pattern, never used) - Add --forward flag to patch command to avoid false "reversed patch"

tools/upgrade-operator-sdk/operator/patches/clusterconfig_controller.patch

tools/upgrade-operator-sdk/operator/patches/clusterconfig_controller_test.patch

tools/upgrade-operator-sdk/upgrade.py

- Hardcode Jinja build_image_name() directly in Makefile patches, remove __IMAGE__ placeholder and image_placeholder config field - Add 'delete' config field to remove scaffold files (replaces hardcoded .devcontainer removal and controller_test patches) - Simplify controller patches: keep scaffold kubebuilder marker format, remove Copyright changes and commented-out code - Improve raw_copy directory handling: diff and skip if identical, error with diff if different (idempotent re-runs) - Remove auto-pin of YAML config (never modify the config file) - Remove _update_yaml() function (no longer needed) - Fix f-string formatting per review suggestions - Sync BUMPING.md: remove stale image_placeholder references

ezekiel-alexrod · 2026-03-20T16:55:28Z

/approve

bert-e · 2026-03-20T16:55:37Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

Peer approvals must include at least 1 approval from the following list:

The following options are set: approve

ezekiel-alexrod changed the title ~~scripts: add upgrade-operator-sdk.py~~ scripts: add script to upgrade operator-sdk Mar 12, 2026

TeddyAndrieux reviewed Mar 12, 2026

View reviewed changes

ezekiel-alexrod added 4 commits March 16, 2026 13:08

scripts: fetch last patch for k8s go libraries

7121a7d

scripts: constants for regexp

669c94c

scripts: update target python version to 3.10

7e6dd52

ezekiel-alexrod force-pushed the improvement/tool-upgrade-operator-sdk branch from 2657ffa to 9569a9d Compare March 16, 2026 13:11

ezekiel-alexrod marked this pull request as ready for review March 16, 2026 13:11

ezekiel-alexrod requested a review from a team as a code owner March 16, 2026 13:11

scripts: corrected regex for Dockerfile

521bae0

ezekiel-alexrod added a commit that referenced this pull request Mar 16, 2026

operator,storage-operator: upgrade to v1.42.1

99510fa

Using the tool for upgrading operator-sdk. See PR #4816 python scripts/upgrade-operator-sdk.py Closes: MK8S-110,MK8S-111

ezekiel-alexrod mentioned this pull request Mar 17, 2026

Bump operator-sdk to v1.42.1 in operator and storage-operator #4818

Merged

scripts: change strategy for GNU patch files

800dc94

ezekiel-alexrod force-pushed the improvement/tool-upgrade-operator-sdk branch from 3c7cbaf to 800dc94 Compare March 17, 2026 13:03

TeddyAndrieux reviewed Mar 17, 2026

View reviewed changes

TeddyAndrieux reviewed Mar 19, 2026

View reviewed changes

ezekiel-alexrod force-pushed the improvement/tool-upgrade-operator-sdk branch from e8f96e9 to 05260c0 Compare March 19, 2026 15:54

scripts,tools: move upgrade-operator-sdk to tools/ directory

432380b

TeddyAndrieux reviewed Mar 20, 2026

View reviewed changes

ezekiel-alexrod changed the title ~~scripts: add script to upgrade operator-sdk~~ tools: add script to upgrade operator-sdk Mar 20, 2026

ezekiel-alexrod force-pushed the improvement/tool-upgrade-operator-sdk branch from 9638577 to a85a05f Compare March 20, 2026 11:46

ezekiel-alexrod force-pushed the improvement/tool-upgrade-operator-sdk branch from a85a05f to ecda27e Compare March 20, 2026 14:03

TeddyAndrieux approved these changes Mar 20, 2026

View reviewed changes

Conversation

ezekiel-alexrod commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Add tool to upgrade operator-sdk projects

Summary

Execution flow

Usage

Configuration

config.yaml fields

Version reconciliation (CI-friendly)

Design decisions

Uh oh!

bert-e commented Mar 12, 2026

Hello ezekiel-alexrod,

Uh oh!

bert-e commented Mar 12, 2026

Waiting for approval

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bert-e commented Mar 13, 2026

Waiting for approval

Uh oh!

bert-e commented Mar 13, 2026

Waiting for approval

Uh oh!

bert-e commented Mar 16, 2026

Waiting for approval

Uh oh!

bert-e commented Mar 16, 2026

Waiting for approval

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bert-e commented Mar 18, 2026

Waiting for approval

Uh oh!

bert-e commented Mar 19, 2026

Waiting for approval

Uh oh!

TeddyAndrieux left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ezekiel-alexrod commented Mar 20, 2026

Uh oh!

bert-e commented Mar 20, 2026

Waiting for approval

Uh oh!

Reviewers

Assignees

ezekiel-alexrod commented Mar 12, 2026 •

edited

Loading