Add target distro option and initial conversion enablement #1438

matejmatuska · 2025-10-14T15:29:09Z

This series of patches is the initial work required for upgrading and converting in one leapp run. This is just an initial enablement, there are still several issues.

A --target-os CLI option is added for specifying the distro to upgrade (and convert) to. Currently, all the distros that can be upgraded (RHEL, CS, AL) can be converted to and from.
A LEAPP_DEVEL_TARGET_OS environment variable is added for development.

Included changes overview:

Add the envar and CLI option.
Store target distro in IPUConfig message and handle target version determination.
- See the particular commit for info on how that is determined.
Replace usage of distro.get_distro_id() with distro.get_{source,target}_distro().
Update the repomapping handlers to enable mapping across the source and target distro.
~~Extend removeobsoletegpgkeys actor to remove RPM GPG keys from the source distro.~~
~~Add some missing obsolete RPM GPG keys to remove for RHEL and CS - this could be a separate PR.~~
Add --target-version CLI option as an to --target - this makes is more ambiguous next to --target-os.

Jira: RHEL-111672, ~~RHEL-111163~~

TODO

I will squash a lot of the commits. I can also split some commits into separate PR, but before the --target-os option is added all of the other commits should probably be there . That's just so we dont merge something that just tracebacks.
Docs page/update.
Add jira ticket to commits.

github-actions · 2025-10-14T15:29:18Z

Thank you for contributing to the Leapp project!

Please note that every PR needs to comply with the leapp-repository contribution and development guidelines and must pass all tests in order to be mergeable.
If you want to request a review or rebuild a package in copr, you can use following commands as a comment:

review please @oamg/developers to notify leapp developers of the review request
/packit copr-build to submit a public copr build using packit

Packit will automatically schedule regression tests for this PR's build and latest upstream leapp build.
However, here are additional useful commands for packit:

/packit test to re-run manually the default tests
/packit retest-failed to re-run failed tests manually
/packit test oamg/leapp#42 to run tests with leapp builds for the leapp PR#42 (default is latest upstream - main - build)

Note that first time contributors cannot run tests automatically - they need to be started by a reviewer.

It is possible to schedule specific on-demand tests as well. Currently 2 test sets are supported, beaker-minimal and kernel-rt, both can be used to be run on all upgrade paths or just a couple of specific ones.
To launch on-demand tests with packit:

/packit test --labels kernel-rt to schedule kernel-rt tests set for all upgrade paths
/packit test --labels beaker-minimal-8.10to9.4,kernel-rt-8.10to9.4 to schedule kernel-rt and beaker-minimal test sets for 8.10->9.4 upgrade path

See other labels for particular jobs defined in the .packit.yaml file.

Please open ticket in case you experience technical problem with the CI. (RH internal only)

Note: In case there are problems with tests not being triggered automatically on new PR/commit or pending for a long time, please contact leapp-infra.

During a containerized tests run, the commands/ directory is not used when running the commands tests. Instead, the commands are imported from tut/lib/python3.X/site-packages/leapp/cli/commands/ where python3.X is the python used in the tut/ virtualenv. When there are changes breaking the commands tests, the test are still passing because the files are not being updated in the directory mentioned above. This can be currently fixed by rebuilding the container which takes a lot of time. This patch adds copying of the commands/ dir to the tut/lib/python3.X/site-packages/leapp/cli/commands/. This helped while working on oamg#1438 because I caught the failing tests only after opening a the PR. The CI containerized tests are always created from scratch so it works there.

During a containerized tests run, the commands/ directory is not used when running the commands tests. Instead, the commands are imported from tut/lib/python3.X/site-packages/leapp/cli/commands/ where python3.X is the python used in the tut/ virtualenv. When there are changes breaking the commands tests, the test are still passing because the files are not being updated in the directory mentioned above. This can be currently fixed by rebuilding the container which takes a lot of time. This patch adds copying of the commands/ dir to the tut/lib/python3.X/site-packages/leapp/cli/commands/. This helped while working on #1438 because I caught the failing tests only after opening a the PR. The CI containerized tests are always created from scratch so it works there.

matejmatuska · 2025-10-22T21:40:53Z

Okay I've split out the removeobsoletegpgkeys conversions refactor to a separate PR (not yet open).

matejmatuska · 2025-10-23T10:58:09Z

I think this is now ready for some initial review.

MichalHe

A substantive piece of work that imprints admiration, good job.

MichalHe · 2025-10-27T06:40:01Z

commands/command_utils.py

+            'Unexpected format of {} version: {}. The required format is \'{}\'.'
        )
-        raise CommandError(error_str.format(version_str, desired_format.human_readable))
+        raise CommandError(error_str.format(version_kind, version_str, desired_format.human_readable))


The error message will look something like this:

'Unexpected format of _VersionKind.SOURCE version: ...

I believe that the intention was different 'source'/'target' instead of '_VersionKind.SOURCE. If this is the case, version_kind.value needs to be used.

You are right.

MichalHe · 2025-10-27T06:47:00Z

commands/command_utils.py


-    target_ver = env_version_override or args.target
+    target_ver = env_version_override or args.target_version
+    distro_id = os.getenv('LEAPP_TARGET_OS')


I feel like distro_id is ambiguous (we can have different source/target distro IDs).

Suggested change

distro_id = os.getenv('LEAPP_TARGET_OS')

target_distro_id = os.getenv('LEAPP_TARGET_OS')

MichalHe · 2025-10-27T06:50:13Z

commands/command_utils.py

    db_config.store()
+
+
+def get_target_os_options():


I was a bit confused whether, based on the function name, a list of leapp's CLI options related to target distro will be returned, or something similar.

Suggested change

def get_target_os_options():

def get_available_target_distros():

I went with get_available_target_distro_ids.

MichalHe · 2025-10-27T07:00:22Z

repos/system_upgrade/common/actors/ipuworkflowconfig/libraries/ipuworkflowconfig.py

+            ' provide any virtual version for version {}'
+        ).format(CENTOS_VIRTUAL_VERSIONS_KEY, version)
+        raise StopActorExecutionError('Failed to identify virtual minor version number for the system.',
+                                      details=details)


Suggested change

details=details)

details={'details': details|)

MichalHe · 2025-10-27T07:00:40Z

repos/system_upgrade/common/actors/ipuworkflowconfig/libraries/ipuworkflowconfig.py

+    virtual_versions = centos_upgrade_paths.get(CENTOS_VIRTUAL_VERSIONS_KEY, {})
+    if not virtual_versions:  # Unlikely, only if using old upgrade_paths.json, but the user should not touch the file
+        details = {'details': 'The file does not contain any information about virtual versions of CentOS'}
+        raise StopActorExecutionError('The internal upgrade_paths.json file is invalid.')


Suggested change

raise StopActorExecutionError('The internal upgrade_paths.json file is invalid.')

raise StopActorExecutionError('The internal upgrade_paths.json file is invalid.', details=details)

Good catch.

MichalHe · 2025-10-27T07:08:02Z

repos/system_upgrade/common/actors/ipuworkflowconfig/libraries/ipuworkflowconfig.py

+    assert (
+        len(exposed_supported_paths) == 1
+    ), "Expected only 1 IPUSourceToPossibleTargets on CS->RHEL upgrade"


Use StopActorExecutionError. When I started in upgrades, I tried to use assert as well, and I have received a list of reasons why I should not use them in production code. The primary reason I remember is that you can disable all asserts when executing the python interpreter.

Okay, thanks for the explanation.

MichalHe · 2025-10-27T07:16:39Z

repos/system_upgrade/common/actors/peseventsscanner/libraries/peseventsscanner_repomap.py

    """

-    def __init__(self, repo_map, distro='', cloud_provider='', default_channels=None):
+    def __init__(self, repo_map, src_distro='', dst_distro='', cloud_provider='', default_channels=None):


Note that elsewhere in the PR we are using target_distro whereas here it is dst_distro. I would prefer having a single, consistent terminology.

MichalHe · 2025-10-27T07:17:49Z

repos/system_upgrade/common/actors/peseventsscanner/libraries/peseventsscanner_repomap.py

-        :type distro: str
+        :param src_distro: The distribution to map repos from, default to current
+        :type src_distro: str
+        :param dst_distro: The distribution to map repos to, default to current


Suggested change

:param dst_distro: The distribution to map repos to, default to current

:param dst_distro: The distribution to map repos to, default to current target distro

MichalHe · 2025-10-27T07:21:41Z

repos/system_upgrade/common/actors/peseventsscanner/libraries/peseventsscanner_repomap.py

+            # It seems obvious we should check it, but it's not clear why we
+            # don't and investigation might be required.


AFAIK, it is not strictly necessary here since later when we are actually picking out repoids (final stages of the mapping process), we filter the candidates by architecture.

A bit of context: the comments that were there before my changes were both added in the same PR due to misunderstanding. I just modified the comments to make a bit more sense.

MichalHe · 2025-10-27T07:29:58Z

repos/system_upgrade/common/libraries/config/__init__.py

    return CONSUMED_DATA_STREAM_ID


-def get_distro_id():


Since this is a public library, shouldn't we first deprecate it?

matejmatuska · 2025-10-27T09:48:07Z

Rebased, resolved a conflict and got the new linter rule changes.

matejmatuska · 2025-10-27T11:31:24Z

Everything should be addressed by the fixup commits.

Allow setting the target OS for development purposes. Internally (cannot be set by the user) the LEAPP_TARGET_OS envar is set, similarly to how this works other arguments. Note that this patch only adds the variable and stores it into LEAPP_TARGET_OS, otherwise it is unhandled.

The distros are now stored similarly to how versions are stored. CurrentActorMocked is modified to take src_distro and dst_distro as arguments. Although there is already release_id parameter which is kept for compatibility.

This is required for conversions. The existing error message was improved to contain the key that is actually missing in the virtual versions map.

Notable changes: - removeobsoletegpgkeys - actor needs to handle RPM GPG key removal differently during converting, for now it just returns early.

…lers This should be squashed into the src/dst distro split, breaks on it's own. RepoMapDataHandler is modified to accept source and target distro independently and therefore perform mapping across different distros.

Currently the version "autocorrection" is performed only when the source system is centos, however we might get a major only version even when the source doesn't use them and the target does. For example when matching whether a version is between source and target version as is done in peseventsscanner.

The option specifies the target OS (distribution) to upgrade to. This sets the value of LEAPP_TARGET_OS, however LEAPP_DEVEL_TARGET_OS has precedence. If none of the envars are defined, default to the source distro i.e. only upgrade, no conversion, will be performed. The available options are 'rhel', 'centos' and 'almalinux'. Note that the "ID" value from /etc/os-release is used and therefore 'centos' really refers to Centos Stream.

With the introduction of --target-os option, the --target option is a bit ambiguous, therefore --target-version is added as an alias.

On CS to RHEL upgrades, particularly on 9->10, there is only one upgrade path defined, CS 9 -> latest RHEL 10 (10.X). However a situation may occur, in which the latest RHEL version has not yet been publicly released, e.g. in pre-release builds. This is problematic because the upgrade fails if the content is not yet available. If this happens the user is informed via a hint in the error message to specify the latest available RHEL version (the previous minor version) manually using the --target-version CLI option. To make leapp consider the previous minor version as supported without adding it to upgrade_paths.json (which would affect RHEL->RHEL upgrades), it is added to the IPUConfig message after processing the json. --- Also, in the error message, move the existing proxy hints from 'details' to 'hint', otherwise the original exeception message, which is in 'details' is overwritten.

…ap handlers

matejmatuska · 2025-11-06T17:19:43Z

Rebased to resolve a conflict after the merge of #1426.

matejmatuska added this to the 8.10/9.8 milestone Oct 14, 2025

matejmatuska added the enhancement New feature or request label Oct 14, 2025

matejmatuska changed the title ~~WIP Target distro option~~ Add target distro option and initial conversion enablement Oct 16, 2025

matejmatuska mentioned this pull request Oct 17, 2025

Makefile: Properly copy commands dir to test containers #1440

Merged

matejmatuska force-pushed the target-distro-option branch 2 times, most recently from 122101f to 7d78e54 Compare October 22, 2025 21:34

matejmatuska force-pushed the target-distro-option branch 2 times, most recently from 01cc2e6 to cc6183a Compare October 23, 2025 10:56

matejmatuska marked this pull request as ready for review October 23, 2025 10:58

matejmatuska force-pushed the target-distro-option branch from cc6183a to e24e3f0 Compare October 23, 2025 14:38

MichalHe requested changes Oct 27, 2025

View reviewed changes

matejmatuska force-pushed the target-distro-option branch from e24e3f0 to 6a36b15 Compare October 27, 2025 09:46

matejmatuska added 10 commits November 6, 2025 18:12

commands: Get target version based on target distro

7dc5114

ipuconfig: Add source and target distros to config

73df9dd

The distros are now stored similarly to how versions are stored. CurrentActorMocked is modified to take src_distro and dst_distro as arguments. Although there is already release_id parameter which is kept for compatibility.

ipuconfig: Allow for one of the virtual version to be missing

0827f3f

This is required for conversions. The existing error message was improved to contain the key that is actually missing in the virtual versions map.

ipuconfig: Map source to target distro paths when converting

e2c4b6b

Split get_distro_id into variants for source and target distros

bc90c1f

Notable changes: - removeobsoletegpgkeys - actor needs to handle RPM GPG key removal differently during converting, for now it just returns early.

Add --target-version as an alias for the --target option

cfdc167

With the introduction of --target-os option, the --target option is a bit ambiguous, therefore --target-version is added as an alias.

matejmatuska added 6 commits November 6, 2025 18:15

fixup! commands: Get target version based on target distro

0dbd0cf

fixup! Add --target-os CLI option

d76d62b

fixup! ipuconfig: Allow for one of the virtual version to be missing

9212eaf

fixup! cs->rhel: Handle potentially unreleased target version

4ba7199

fixup! SQUASH Handle source and target distros independently in repom…

8f13093

…ap handlers

matejmatuska force-pushed the target-distro-option branch from 197aef7 to 8f13093 Compare November 6, 2025 17:17

matejmatuska mentioned this pull request Nov 7, 2025

Fix unsupported source version crash #1443

Open

	distro_id = os.getenv('LEAPP_TARGET_OS')
	target_distro_id = os.getenv('LEAPP_TARGET_OS')

	def get_target_os_options():
	def get_available_target_distros():

	raise StopActorExecutionError('The internal upgrade_paths.json file is invalid.')
	raise StopActorExecutionError('The internal upgrade_paths.json file is invalid.', details=details)

	:param dst_distro: The distribution to map repos to, default to current
	:param dst_distro: The distribution to map repos to, default to current target distro

		# It seems obvious we should check it, but it's not clear why we
		# don't and investigation might be required.

Add target distro option and initial conversion enablement #1438

Are you sure you want to change the base?

Add target distro option and initial conversion enablement #1438

Uh oh!

Conversation

matejmatuska commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Included changes overview:

TODO

Uh oh!

github-actions bot commented Oct 14, 2025

Thank you for contributing to the Leapp project!

Uh oh!

matejmatuska commented Oct 22, 2025

Uh oh!

matejmatuska commented Oct 23, 2025

Uh oh!

MichalHe left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

matejmatuska commented Oct 27, 2025

Uh oh!

matejmatuska commented Oct 27, 2025

Uh oh!

matejmatuska commented Nov 6, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

matejmatuska commented Oct 14, 2025 •

edited

Loading