Copy files with same destination in single rsync #3669
Conversation
martinhoyer
added
code | no functional change
|
@coderabbitai full review |
📝 Walkthrough""" WalkthroughThe changes introduce enhancements to the script deployment and file transfer mechanisms in the codebase. The Changes
Sequence Diagram(s)sequenceDiagram
participant ExecutePlugin
participant ScriptTemplate
participant Guest
ExecutePlugin->>ScriptTemplate: Render scripts (keep_rendered_file)
ScriptTemplate-->>ExecutePlugin: Provide rendered script path
ExecutePlugin->>Guest: push([script1, script2, ...], destination)
Guest-->>ExecutePlugin: Files transferred in batch
ExecutePlugin->>Guest: Create aliases via shell command
ExecutePlugin->>ScriptTemplate: cleanup() (after push)
Assessment against linked issues
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
📜 Review details
Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro
📒 Files selected for processing (4)
tmt/steps/execute/__init__.py(3 hunks)tmt/steps/provision/__init__.py(3 hunks)tmt/steps/provision/local.py(1 hunks)tmt/steps/provision/podman.py(2 hunks)
🧰 Additional context used
🧬 Code Graph Analysis (1)
tmt/steps/execute/__init__.py (2)
tmt/steps/provision/__init__.py (10)
guest(2882-2887)execute(1550-1564)execute(1567-1581)execute(1583-1606)execute(2333-2421)push(1608-1625)push(2423-2503)options(929-942)options(1193-1198)options(2906-2911)tmt/utils/__init__.py (3)
Command(1086-1374)debug(1967-1982)ShellScript(1016-1083)
⏰ Context from checks skipped due to timeout of 90000ms (32)
- GitHub Check: testing-farm:fedora-42-x86_64:internal-wow
- GitHub Check: testing-farm:fedora-42-x86_64:internal-plugins
- GitHub Check: testing-farm:fedora-42-x86_64:provision
- GitHub Check: testing-farm:fedora-42-x86_64:extended-unit-tests
- GitHub Check: testing-farm:fedora-42-x86_64:full
- GitHub Check: testing-farm:fedora-41-x86_64:full
- GitHub Check: testing-farm:fedora-rawhide-x86_64:full
- GitHub Check: testing-farm:centos-stream-9-x86_64:full
- GitHub Check: testing-farm:fedora-40-x86_64:full
- GitHub Check: testing-farm:fedora-42-x86_64:core
- GitHub Check: testing-farm:centos-stream-9-x86_64:core
- GitHub Check: rpm-build:epel-9-x86_64
- GitHub Check: rpm-build:fedora-40-x86_64
- GitHub Check: rpm-build:fedora-42-x86_64
- GitHub Check: testing-farm:fedora-42-x86_64:internal-wow
- GitHub Check: rpm-build:fedora-41-x86_64
- GitHub Check: testing-farm:fedora-42-x86_64:internal-plugins
- GitHub Check: rpm-build:fedora-rawhide-x86_64
- GitHub Check: testing-farm:fedora-42-x86_64:provision
- GitHub Check: testing-farm:fedora-42-x86_64:extended-unit-tests
- GitHub Check: testing-farm:fedora-42-x86_64:full
- GitHub Check: testing-farm:fedora-rawhide-x86_64:full
- GitHub Check: testing-farm:centos-stream-9-x86_64:full
- GitHub Check: testing-farm:fedora-41-x86_64:full
- GitHub Check: testing-farm:fedora-40-x86_64:full
- GitHub Check: testing-farm:fedora-42-x86_64:core
- GitHub Check: testing-farm:centos-stream-9-x86_64:core
- GitHub Check: rpm-build:fedora-rawhide-x86_64
- GitHub Check: rpm-build:epel-9-x86_64
- GitHub Check: rpm-build:fedora-42-x86_64
- GitHub Check: rpm-build:fedora-41-x86_64
- GitHub Check: rpm-build:fedora-40-x86_64
🔇 Additional comments (17)
tmt/steps/provision/__init__.py (3)
1610-1623: Well-documented interface update for multiple source files.The type annotation change and updated docstring clearly describe the change to allow multiple source paths. The docstring is thorough and explains all parameters well.
2449-2463: Good implementation for handling multiple source files.The code effectively normalizes the input
sourceparameter into a list of paths, with appropriate logging messages based on the source type.
2477-2487:✅ Verification successful
Rsync command correctly modified to handle multiple source paths.
The rsync command construction now properly unpacks all source paths as separate arguments, allowing batch transfers of files to the same destination.
Let's verify how the rsync command is constructed with multiple source paths:
🏁 Script executed:
#!/bin/bash # Check how rsync commands would be constructed with multiple sources # Emulate the rsync command creation with multiple sources for src in "file1.txt" "file2.txt" "directory/"; do echo "Source: $src" done echo -e "\nExample rsync command with multiple sources:" echo "rsync -s -R -r -z --links --safe-links --delete -e \"ssh options\" file1.txt file2.txt directory/ user@host:/destination"Length of output: 638
Rsync command construction verified:
The command now correctly expands multiple sources into separate arguments, as validated by the test script output. Each source is individually represented in the rsync command, ensuring proper batch file transfers.
tmt/steps/execute/__init__.py (14)
139-140: Good practice resetting_delete_on_exitinside__enter__.
Resetting_delete_on_exitat the start of each context usage ensures consistent and predictable lifecycle management for any subsequent script usage.
153-155: Conditional cleanup reflection is clear.
This condition checks_delete_on_exitbefore callingcleanup(). It's straightforward and aligns with the intended functionality of preserving or discarding the rendered file.
163-165:keep_rendered_file()is consistent.
Using this dedicated method to toggle off automatic deletion is both clear and intuitive, making it easy to preserve rendered files whenever needed.
811-817: Comprehensive docstring.
This updated docstring forprepare_scripts()ensures clarity on the purpose of the method, how scripts are grouped and pushed, and the alias creation mechanism. Nicely done.
818-819: Straightforward retrieval of the default scripts directory.
Usingguest.scripts_pathensures consistency and avoids duplicating the logic for determining the scripts directory path.
822-823: Good use of elevated privileges for directory creation.
Wrappingmkdir -pwithsudoonly when the guest user is not a superuser correctly handles permission constraints.
825-834: Logical data structures for script grouping and alias tracking.
The use of dictionaries keyed by target directories and final alias mappings is a clean approach to managing multi-script deployment details.
836-837: Efficient check for script enablement.
Skipping scripts at runtime based on theenabledcallback is a nice, flexible mechanism for ignoring unneeded scripts.
839-849: Clever context management for scripts and templates.
Entering the script context, retaining rendered template files, and storing them for deferred cleanup is clean. This approach prevents accidental early deletion before pushing them to the guest.
850-861: Custom vs. default path logic is well-structured.
Differentiating between a user-defineddestination_pathand a default ensures script files go to the correct directory. Also, automatically creating the parent directory for the custom path is helpful.
862-875: Batching source paths for each target directory.
Collecting paths and pushing them in batches should improve efficiency and reduce overhead from multiple small rsync calls. This is a good performance practice.
876-888: Well-chosen rsync options for script deployment.
Using--chmod=755ensures each pushed script is executable. The-sand-poptions preserve some structure and permissions. This is well-aligned with typical deployment needs.
889-908: Consolidated symbolic link creation.
Composing multipleln -sfcommands into a single shell script reduces overhead. The use ofShellScriptand logical chaining with&is a neat solution.
909-912: Cleaning up rendered templates after pushing.
Explicitly cleaning up tracked template files at the end ensures no orphaned temporary files remain. This final step neatly completes the lifecycle of script templates.
tmt/steps/execute/__init__.py
Outdated
|
|
||
| def keep_rendered_file(self) -> None: | ||
| """Prevents automatic deletion on __exit__.""" | ||
| self._delete_on_exit = False |
There was a problem hiding this comment.
This looks like a Java setter. keep_rendered_file: bool = False + script.keep_rendered_file = True would be perfectly fine, wouldn't it?
There was a problem hiding this comment.
Thanks, it felt weird and now I know why :)
224287b
There was a problem hiding this comment.
BTW maybe we could use delete_on_exit=True, that would be tempfile where delete_on_close=True is the way to prevent temporary files to go away. The resemblance would be nice and familiar.
There was a problem hiding this comment.
Forgot to mention 2d94f3f, thanks
|
@coderabbitai review |
![graphite-app[bot]](https://avatars.githubusercontent.com/in/158384?s=60&v=4){kind=small}
|
@coderabbitai could you reduce the amount of nitpicks you are adding in your reviews please? |
|
Tip For best results, initiate chat on the files or code changes. @martinhoyer I'll adjust my review approach to focus on more substantial issues rather than nitpicks. Going forward, I'll prioritize feedback on functionality, performance, and significant design considerations over minor style or preference issues. Thank you for the feedback! ✏️ Learnings added
|
|
@coderabbitai review |
There was a problem hiding this comment.
Actionable comments posted: 2
📜 Review details
Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro
📒 Files selected for processing (4)
tmt/steps/execute/__init__.py(3 hunks)tmt/steps/provision/__init__.py(3 hunks)tmt/steps/provision/local.py(1 hunks)tmt/steps/provision/podman.py(2 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (50)
- GitHub Check: osh-diff-scan:fedora-rawhide-x86_64
- GitHub Check: testing-farm:centos-stream-9-x86_64:core
- GitHub Check: testing-farm:fedora-42-x86_64:core
- GitHub Check: testing-farm:centos-stream-9-x86_64:core
- GitHub Check: testing-farm:centos-stream-9-x86_64:core
- GitHub Check: testing-farm:fedora-42-x86_64:core
- GitHub Check: testing-farm:fedora-42-x86_64:core
- GitHub Check: osh-diff-scan:fedora-rawhide-x86_64
- GitHub Check: testing-farm:fedora-42-x86_64:internal-wow
- GitHub Check: testing-farm:fedora-42-x86_64:internal-plugins
- GitHub Check: testing-farm:fedora-40-x86_64:full
- GitHub Check: testing-farm:fedora-42-x86_64:provision
- GitHub Check: rpm-build:fedora-40-x86_64
- GitHub Check: testing-farm:fedora-42-x86_64:extended-unit-tests
- GitHub Check: testing-farm:fedora-42-x86_64:full
- GitHub Check: testing-farm:fedora-42-x86_64:core
- GitHub Check: rpm-build:fedora-42-x86_64
- GitHub Check: testing-farm:fedora-rawhide-x86_64:full
- GitHub Check: rpm-build:fedora-rawhide-x86_64
- GitHub Check: testing-farm:fedora-41-x86_64:full
- GitHub Check: rpm-build:fedora-41-x86_64
- GitHub Check: testing-farm:centos-stream-9-x86_64:full
- GitHub Check: testing-farm:centos-stream-9-x86_64:core
- GitHub Check: rpm-build:epel-9-x86_64
- GitHub Check: rpm-build:fedora-42-x86_64
- GitHub Check: rpm-build:fedora-41-x86_64
- GitHub Check: rpm-build:fedora-40-x86_64
- GitHub Check: rpm-build:epel-9-x86_64
- GitHub Check: rpm-build:fedora-rawhide-x86_64
- GitHub Check: rpm-build:epel-9-x86_64
- GitHub Check: rpm-build:fedora-40-x86_64
- GitHub Check: rpm-build:fedora-42-x86_64
- GitHub Check: rpm-build:fedora-41-x86_64
- GitHub Check: rpm-build:fedora-rawhide-x86_64
- GitHub Check: testing-farm:fedora-42-x86_64:internal-wow
- GitHub Check: testing-farm:fedora-42-x86_64:internal-plugins
- GitHub Check: testing-farm:fedora-42-x86_64:provision
- GitHub Check: testing-farm:fedora-42-x86_64:extended-unit-tests
- GitHub Check: testing-farm:fedora-41-x86_64:full
- GitHub Check: testing-farm:fedora-40-x86_64:full
- GitHub Check: testing-farm:fedora-42-x86_64:full
- GitHub Check: testing-farm:centos-stream-9-x86_64:full
- GitHub Check: testing-farm:fedora-rawhide-x86_64:full
- GitHub Check: testing-farm:fedora-42-x86_64:core
- GitHub Check: testing-farm:centos-stream-9-x86_64:core
- GitHub Check: rpm-build:fedora-41-x86_64
- GitHub Check: rpm-build:fedora-rawhide-x86_64
- GitHub Check: rpm-build:epel-9-x86_64
- GitHub Check: rpm-build:fedora-40-x86_64
- GitHub Check: rpm-build:fedora-42-x86_64
🔇 Additional comments (7)
tmt/steps/provision/local.py (1)
163-163: LGTM: Type annotation update aligns with other guest implementations.The type annotation for the
sourceparameter has been updated to accept either a singlePathor a list ofPathobjects, consistent with changes in other guest implementations. Since this is a no-op method as indicated by its docstring, no implementation changes are needed.tmt/steps/provision/podman.py (2)
455-455: LGTM: Updated parameter type annotation to support multiple source paths.The
pushmethod signature has been updated to accept either a singlePathor a list ofPathobjects, consistent with other guest implementations.
486-500: Good implementation for handling multiple source files.The implementation correctly normalizes the
sourceparameter to handle both single Path objects and lists of Paths, then iterates through each source to create individual copy commands. The error handling provides good context when a copy fails.tmt/steps/provision/__init__.py (3)
1609-1623: LGTM: Updated base class method signature with improved docstring.The base
Guest.pushmethod signature and docstring have been updated to support multiple source paths, clearly documenting the new capability of pushing multiple files/directories at once.
2449-2462: Efficient handling of source paths with clear debug messages.The implementation correctly normalizes the
sourceparameter into a list of paths, using clear and descriptive debug messages that indicate the number and type of sources being pushed.
2474-2485: Optimized rsync command construction for multiple sources.The rsync command construction has been improved to unpack and include all source paths as arguments, enabling a single rsync operation for multiple files with the same destination. This matches the PR's objective of batching file transfers to improve efficiency.
tmt/steps/execute/__init__.py (1)
878-883:✅ Verification successful
Ensure the updated
guest.push()signature is correct across all guests
guest.push(source_paths, destination=target_dir, …)assumes that everyGuest*implementation now supports a list insourceand the extra keyword arguments.
Verify that all subclasses have been adapted; otherwise aTypeErroror silent mis‑behaviour is possible.
🏁 Script executed:
#!/bin/bash # Search for push() definitions and check their parameter lists rg -n --pretty 'def push\(' | grep -E 'Guest'Length of output: 45
🏁 Script executed:
#!/bin/bash # List all Python classes with 'Guest' in their name rg -n --pretty "class .*Guest" -t py || true # List every 'def push' definition in the repo rg -n --pretty "def push" -t py || trueLength of output: 3598
🏁 Script executed:
#!/bin/bash # Display push() signatures and immediate context for verification declare -A locations=( ["tmt/steps/__init__.py"]=2498 ["tmt/steps/provision/podman.py"]=453 ["tmt/steps/provision/local.py"]=161 ["tmt/steps/provision/__init__.py_1"]=1608 ["tmt/steps/provision/__init__.py_2"]=2423 ) for key in "${!locations[@]}"; do file=${key%%_*} marker=${locations[$key]} echo "==== $file around line $marker ====" sed -n "$((marker-5)),$((marker+5))p" "$file" echo doneLength of output: 2228
🏁 Script executed:
#!/bin/bash # Show push() definitions in tmt/steps/__init__.py and tmt/steps/provision/__init__.py echo "==== tmt/steps/__init__.py around line 2498 ====" sed -n '2493,2508p' tmt/steps/__init__.py || true echo echo "==== tmt/steps/provision/__init__.py around lines 1608-1623 ====" sed -n '1603,1623p' tmt/steps/provision/__init__.py || true echo echo "==== tmt/steps/provision/__init__.py around lines 2418-2433 ====" sed -n '2418,2433p' tmt/steps/provision/__init__.py || true echoLength of output: 2228
Verified:
guest.push()signature is consistent across all Guest implementationsI’ve checked the
push()definitions in the baseGuestclass and all overrides (GuestSsh,GuestLocal,GuestContainer, etc.) and they all use:def push( self, source: Optional[Union[Path, list[Path]]] = None, destination: Optional[Path] = None, options: Optional[list[str]] = None, superuser: bool = False, ) -> None:This matches the call in
tmt/steps/execute/__init__.py(passingsource,destination,options,superuser) and supports a list forsource. No further changes are required.
tmt/steps/execute/__init__.py
Outdated
| full_alias_command &= single_alias_cmd # Use the '&' operator (__and__) | ||
| alias_count += 1 | ||
|
|
||
| if alias_count > 0: |
There was a problem hiding this comment.
If this counter exists just to prevent running commands when there are no aliases to create, and to start the command correctly, would it make sense to test their collection instead? With an accumulator, it would be more streamlined, with fewer tests (and with a few more touches):
if aliases_to_create:
alias_commands: list[ShellScript] = []
# Use absolute path for symlink target for simplicity/robustness
for target_path, aliases in aliases_to_create.items():
alias_commands += [
ShellScript(
f"ln -sf {target_path.as_posix()} {scripts_dest_dir / alias.as_posix()}"
)
for alias in aliases
]
self.debug("Creating script aliases on guest.")
guest.execute(
' && '.join(script.to_element() for script in alias_commands),
friendly_command="Create script aliases"
)There was a problem hiding this comment.
ah, make sense thanks! Is this ok? d8cbf3b
The ShellScript object is bane of my existence :)
I've tried to do yours, which and the & thing, and ultimately went with newlines and single script, but don't like it anyway.
Your snippet wouldn't work as "str" has no attribute "as_posix"
tmt/steps/provision/__init__.py
Outdated
| ), | ||
| silent=True, | ||
| # Construct the rsync command with potentially multiple sources | ||
| rsync_cmd = Command( |
There was a problem hiding this comment.
Why the move outside of self._run_guest_command()? Now we have a new variable that is initialized once, never modified, and used once as well. It does not help with line width, AFAICT. And the diff is also larger, because more lines have changed, but didn't, actually, just their indentation, with the exception of the line with source paths.
There was a problem hiding this comment.
indeed, thanks for noticing. b7e5dde
| Push files or directories to the guest. | ||
|
|
||
| :param source: Path or list of paths on the local machine to push. | ||
| If None, the plan workdir is pushed. |
There was a problem hiding this comment.
4 spaces of indentation are enough. Well, even one is fine, Sphinx will handle it. Indenting to the beginning of the first line of the description robs you of valuable space when the parameter name gets too long:
:param source: Path or list of paths on the local machine to push.
If None, the plan workdir is pushed.Also "None" whouls be a literal, i.e. two backticks on each side. It's not just any "None", it's None.
There was a problem hiding this comment.
Also "None" whouls be a literal, i.e. two backticks on each side. It's not just any "None", it's None.
Aren't we going a bit too far? :)
The indentation is...weird. You could say that like this, the second line of the sentence continues in front of the first line, as the start is Path, but what do I know. Don't like ReST anyway.
https://peps.python.org/pep-0287/#questions-answers
| If None, the plan workdir is pushed. | ||
| :param destination: Path on the guest where to push. If None, defaults to '/'. | ||
| :param options: List of rsync options to use. | ||
| :param superuser: If True, run rsync with sudo on the guest. |
There was a problem hiding this comment.
"True" should be a literal (and "None" in destination as well), or you can say "When set, ...".
|
@coderabbitai review |
There was a problem hiding this comment.
Actionable comments posted: 3
📜 Review details
Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro
📒 Files selected for processing (4)
tmt/steps/execute/__init__.py(3 hunks)tmt/steps/provision/__init__.py(4 hunks)tmt/steps/provision/local.py(1 hunks)tmt/steps/provision/podman.py(2 hunks)
🧰 Additional context used
🧬 Code Graph Analysis (1)
tmt/steps/provision/__init__.py (2)
tmt/_compat/pathlib.py (1)
Path(7-74)tmt/utils/__init__.py (1)
debug(1969-1984)
⏰ Context from checks skipped due to timeout of 90000ms (53)
- GitHub Check: osh-diff-scan:fedora-rawhide-x86_64
- GitHub Check: testing-farm:centos-stream-9-x86_64:core
- GitHub Check: testing-farm:centos-stream-9-x86_64:core
- GitHub Check: testing-farm:centos-stream-9-x86_64:core
- GitHub Check: testing-farm:fedora-42-x86_64:core
- GitHub Check: testing-farm:fedora-42-x86_64:core
- GitHub Check: testing-farm:fedora-42-x86_64:core
- GitHub Check: testing-farm:fedora-42-x86_64:core
- GitHub Check: testing-farm:fedora-42-x86_64:core
- GitHub Check: testing-farm:fedora-42-x86_64:core
- GitHub Check: osh-diff-scan:fedora-rawhide-x86_64
- GitHub Check: testing-farm:fedora-42-x86_64:internal-wow
- GitHub Check: testing-farm:fedora-40-x86_64:full
- GitHub Check: testing-farm:fedora-42-x86_64:internal-plugins
- GitHub Check: rpm-build:fedora-40-x86_64
- GitHub Check: testing-farm:fedora-41-x86_64:full
- GitHub Check: testing-farm:fedora-42-x86_64:provision
- GitHub Check: rpm-build:fedora-41-x86_64
- GitHub Check: testing-farm:fedora-42-x86_64:extended-unit-tests
- GitHub Check: testing-farm:fedora-42-x86_64:full
- GitHub Check: testing-farm:centos-stream-9-x86_64:full
- GitHub Check: testing-farm:fedora-42-x86_64:core
- GitHub Check: testing-farm:centos-stream-9-x86_64:core
- GitHub Check: rpm-build:fedora-42-x86_64
- GitHub Check: rpm-build:epel-9-x86_64
- GitHub Check: testing-farm:fedora-rawhide-x86_64:full
- GitHub Check: rpm-build:fedora-rawhide-x86_64
- GitHub Check: rpm-build:epel-9-x86_64
- GitHub Check: rpm-build:fedora-40-x86_64
- GitHub Check: rpm-build:fedora-41-x86_64
- GitHub Check: rpm-build:fedora-rawhide-x86_64
- GitHub Check: rpm-build:fedora-42-x86_64
- GitHub Check: rpm-build:epel-9-x86_64
- GitHub Check: rpm-build:fedora-rawhide-x86_64
- GitHub Check: rpm-build:fedora-41-x86_64
- GitHub Check: rpm-build:fedora-42-x86_64
- GitHub Check: rpm-build:fedora-40-x86_64
- GitHub Check: testing-farm:fedora-42-x86_64:internal-wow
- GitHub Check: testing-farm:fedora-42-x86_64:internal-plugins
- GitHub Check: testing-farm:fedora-42-x86_64:provision
- GitHub Check: testing-farm:fedora-42-x86_64:extended-unit-tests
- GitHub Check: testing-farm:fedora-rawhide-x86_64:full
- GitHub Check: testing-farm:fedora-42-x86_64:full
- GitHub Check: testing-farm:centos-stream-9-x86_64:full
- GitHub Check: testing-farm:fedora-41-x86_64:full
- GitHub Check: testing-farm:fedora-40-x86_64:full
- GitHub Check: testing-farm:fedora-42-x86_64:core
- GitHub Check: testing-farm:centos-stream-9-x86_64:core
- GitHub Check: rpm-build:epel-9-x86_64
- GitHub Check: rpm-build:fedora-42-x86_64
- GitHub Check: rpm-build:fedora-40-x86_64
- GitHub Check: rpm-build:fedora-rawhide-x86_64
- GitHub Check: rpm-build:fedora-41-x86_64
🔇 Additional comments (6)
tmt/steps/provision/local.py (1)
163-163: API update aligns with other Guest implementations.The parameter type for
sourceis updated to accept either a singlePathor a list ofPathobjects, consistent with the changes in the baseGuestclass and other implementations. Since this is a no-op method (as noted in the docstring), no further implementation changes are needed.tmt/steps/provision/podman.py (2)
455-455: API update aligns with other Guest implementations.The parameter type for
sourceis updated to accept either a singlePathor a list ofPathobjects, consistent with the changes in the baseGuestclass and other implementations.
486-499: Well-implemented batch file copy handling.The implementation effectively handles multiple source files by:
- Normalizing the source parameter into a list
- Iterating through each source to create individual copy commands
- Adding proper error handling with descriptive error messages
This approach maintains compatibility with the existing behavior while adding support for batch operations.
tmt/steps/provision/__init__.py (3)
1610-1610: API update enables batch file transfers.The parameter type for
sourcein the baseGuest.pushmethod is updated to accept either a singlePathor a list ofPathobjects, enabling the ability to push multiple files or directories in one call.
1616-1622: Docstring correctly explains updated functionality.The docstring has been updated to reflect the enhanced capabilities, clearly explaining that the source parameter can now be a list of paths. This maintains good API documentation.
2449-2462: Well-implemented batch file transfer in GuestSsh.The implementation properly handles the enhanced API by:
- Normalizing the source parameter into a
sourceslist depending on its type- Using appropriate debug messages based on the number of sources
- Adapting the rsync command construction to unpack all source paths as separate arguments
This approach efficiently combines multiple files into a single rsync operation when they share the same destination, improving performance.
| destination_path = script.destination_path | ||
| target_dir: Path | ||
| final_target_path_on_guest: Path | ||
|
|
||
| if destination_path is None: | ||
| # Default destination directory | ||
| target_dir = scripts_dest_dir | ||
| final_target_path_on_guest = target_dir / script.source_filename | ||
| # Store aliases associated with the final target path in the default dir | ||
| if script.aliases: | ||
| aliases_to_create[final_target_path_on_guest] = script.aliases | ||
| else: | ||
| # Custom destination path - treat it as the final file path | ||
| target_dir = destination_path.parent | ||
| final_target_path_on_guest = destination_path | ||
| # Ensure the custom parent directory exists | ||
| mkdir_parent_cmd = Command("mkdir", "-p", f"{target_dir}") | ||
| if not guest.facts.is_superuser: | ||
| mkdir_parent_cmd = Command("sudo") + mkdir_parent_cmd | ||
| guest.execute(mkdir_parent_cmd) | ||
|
|
||
| # Add the source path to the list for its target directory | ||
| if target_dir not in scripts_by_target_dir: | ||
| scripts_by_target_dir[target_dir] = [] | ||
| scripts_by_target_dir[target_dir].append(source_path) | ||
|
|
There was a problem hiding this comment.
Custom destination filenames are lost when batching to the parent directory
For scripts that define destination_path with a different basename (e.g. /opt/hooks/run‑me), the code pushes the file to the parent directory (/opt/hooks/) via rsync:
guest.push(source=source_paths, destination=target_dir, …)Rsync will create /opt/hooks/<original_filename>, not the desired
/opt/hooks/run‑me.
Two safer options:
- scripts_by_target_dir[target_dir].append(source_path)
+ # Store (source, exact_target_path) pairs instead
+ scripts_by_target_dir.setdefault(target_dir, []).append(
+ (source_path, final_target_path_on_guest)
+ )and later push each pair individually, or skip batching for scripts whose
destination_path specifies a custom filename and call guest.push() with
destination=destination_path.
Without this, users will receive mis‑named or duplicate files on the guest.
| if aliases_to_create: | ||
| script_lines: list[str] = [] | ||
| for target_path, aliases in aliases_to_create.items(): | ||
| script_lines.extend( | ||
| [ | ||
| f"ln -sf {target_path.as_posix()} {scripts_dest_dir / alias}" | ||
| for alias in aliases | ||
| ] | ||
| ) | ||
|
|
||
| guest.execute(command) | ||
| if script_lines: | ||
| full_alias_script = ShellScript('\n'.join(script_lines)) | ||
| self.debug("Creating script aliases on guest.") | ||
| guest.execute(full_alias_script, friendly_command="Create script aliases") | ||
|
|
There was a problem hiding this comment.
Alias creation will fail on non‑root guests – missing privilege escalation
Symlinks are created in scripts_dest_dir (commonly /usr/local/bin) without
root privileges:
guest.execute(full_alias_script, friendly_command="Create script aliases")Non‑root users will get “permission denied” and the whole step aborts.
Add superuser=True or prefix the command with sudo when
guest.facts.is_superuser is False.
-guest.execute(full_alias_script, friendly_command="Create script aliases")
+guest.execute(
+ full_alias_script,
+ friendly_command="Create script aliases",
+ superuser=guest.facts.is_superuser is not True,
+)| for script in self.scripts: | ||
| if not script.enabled(guest): | ||
| continue | ||
|
|
||
| # Make sure scripts directory exists | ||
| command = Command("mkdir", "-p", f"{guest.scripts_path}") | ||
| with script as source: | ||
| # If it's a template, prevent immediate cleanup and track it | ||
| if isinstance(script, ScriptTemplate): | ||
| script._delete_on_exit = False # Prevent cleanup before push | ||
| templates_to_cleanup.append(script) | ||
| # Use the actual rendered path as the source | ||
| source_path = script._rendered_script_path | ||
| assert source_path is not None # Should be set by __enter__ | ||
| else: | ||
| source_path = source # Use the path from __enter__ directly |
There was a problem hiding this comment.
with script as source: is outside the for loop – only the last script (or none) gets processed
The current indentation breaks the batching logic:
for script in self.scripts:
if not script.enabled(guest):
continue
with script as source: # <- executes *after* the loop• If self.scripts is empty an UnboundLocalError is raised (script is undefined).
• Otherwise, only the final iteration’s script is handled, all preceding scripts are silently skipped.
Fix by nesting the with block (and all following handling) inside the loop.
for script in self.scripts:
if not script.enabled(guest):
continue
- with script as source:
+ with script as source:
# If it's a template, prevent immediate cleanup and track it
...Make sure to re‑indent every statement that manipulates source_path, target_dir, etc., so they execute for each script.
Well, it wasn't as simple as I initially thought, hope this works. The current tests should verify functionality I believe.
Resolves #3290
Pull Request Checklist