Skip to content

core: support chunked transfer for image files #10820

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 3 commits into
base: 4.20
Choose a base branch
from
Draft

core: support chunked transfer for image files #10820

wants to merge 3 commits into from
+71 −28

Conversation

shwstppr
Copy link
Contributor

@shwstppr shwstppr commented May 6, 2025 •
edited
Loading

Description

Fixes #10658
Fixes download of with chunked transfer encoding for non-directdownload templates and ISOs

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • build/CI
  • test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

How Has This Been Tested?

Tested download of:

SSVM logs - tail -f /var/log/cloud.log | grep -E "HttpTemplateDownloader|DownloadManagerImpl" 
2025-05-11T12:53:29,498 INFO  [storage.template.HttpTemplateDownloader] (AgentRequest-Handler-7:[]) No credentials configured for host=10.1.3.130:80
2025-05-11T12:53:29,503 DEBUG [storage.template.HttpTemplateDownloader] (pool-2-thread-2:[]) Remote size for http://10.1.3.130/cks/setup-1.26.0.iso found to be (660.03 MB) 692094976
2025-05-11T12:53:29,505 INFO  [storage.template.HttpTemplateDownloader] (pool-2-thread-2:[]) Starting download from http://10.1.3.130/cks/setup-1.26.0.iso to /mnt/SecStorage/037ced3e-bc30-388e-a0c2-d307b6f8ffb1/template/tmpl/2/210/dnld16851435041626252415tmp_ remoteSize=(660.03 MB) 692094976 , max size=(50.00 GB) 53687091200
2025-05-11T12:53:29,528 DEBUG [storage.template.HttpTemplateDownloader] (pool-2-thread-2:[]) Verified format of downloading file /mnt/SecStorage/037ced3e-bc30-388e-a0c2-d307b6f8ffb1/template/tmpl/2/210/dnld16851435041626252415tmp_ is supported
2025-05-11T12:53:31,223 DEBUG [storage.template.HttpTemplateDownloader] (pool-2-thread-2:[]) Reached expected remote size limit: 692094976 bytes
2025-05-11T12:53:31,241 INFO  [storage.template.DownloadManagerImpl] (pool-2-thread-2:[]) Download Completion for jobId: 94db7f97-1dd0-4df0-923c-4d86c4c2c208, status=DOWNLOAD_FINISHED
2025-05-11T12:53:31,242 INFO  [storage.template.DownloadManagerImpl] (pool-2-thread-2:[]) local: /mnt/SecStorage/037ced3e-bc30-388e-a0c2-d307b6f8ffb1/template/tmpl/2/210/dnld16851435041626252415tmp_, bytes=(660.03 MB) 692094976, error=Downloaded (660.03 MB) 692094976 bytes (download complete remote=unknown bytes), pct=100
2025-05-11T12:53:31,242 INFO  [storage.template.DownloadManagerImpl] (pool-2-thread-2:[]) No checksum available for 'dnld16851435041626252415tmp_'
2025-05-11T12:53:35,477 DEBUG [storage.template.DownloadManagerImpl] (pool-2-thread-2:[]) computed checksum: {SHA-512}ad026d55e0fda7dcb65764660258084f9985d8800b684228a367cf7f90a44d05fb3ff83d9d4e6bc0da2b7049c8bdbfa8c432b342faf96386f92666bcbec8a216
2025-05-11T12:53:35,482 DEBUG [storage.template.DownloadManagerImpl] (pool-2-thread-2:[]) Executing command [/usr/local/cloud/systemvm/scripts/storage/secondary/createtmplt.sh -s 2 -S 53687091200 -d non-chunked -h -n 210-2-66ee1468-58ee-388e-bf9f-9c8aca93b27d.iso -t /mnt/SecStorage/037ced3e-bc30-388e-a0c2-d307b6f8ffb1/template/tmpl/2/210 -f /mnt/SecStorage/037ced3e-bc30-388e-a0c2-d307b6f8ffb1/template/tmpl/2/210/dnld16851435041626252415tmp_ -u ].
2025-05-11T12:53:35,596 DEBUG [storage.template.DownloadManagerImpl] (pool-2-thread-2:[]) Successfully executed process [10257] for command [/usr/local/cloud/systemvm/scripts/storage/secondary/createtmplt.sh -s 2 -S 53687091200 -d non-chunked -h -n 210-2-66ee1468-58ee-388e-bf9f-9c8aca93b27d.iso -t /mnt/SecStorage/037ced3e-bc30-388e-a0c2-d307b6f8ffb1/template/tmpl/2/210 -f /mnt/SecStorage/037ced3e-bc30-388e-a0c2-d307b6f8ffb1/template/tmpl/2/210/dnld16851435041626252415tmp_ -u ].
2025-05-11T12:53:35,681 DEBUG [storage.template.DownloadManagerImpl] (pool-2-thread-2:[]) Cleaning-up temporary download file /mnt/SecStorage/037ced3e-bc30-388e-a0c2-d307b6f8ffb1/template/tmpl/2/210/dnld16851435041626252415tmp_
2025-05-11T12:53:35,684 DEBUG [storage.template.DownloadManagerImpl] (pool-2-thread-2:[]) Deleting directory /mnt/SecStorage/037ced3e-bc30-388e-a0c2-d307b6f8ffb1/template/tmpl/2/210, if empty, as part of cleanup
2025-05-11T12:54:16,591 INFO  [storage.template.HttpTemplateDownloader] (AgentRequest-Handler-10:[]) No credentials configured for host=releases.rancher.com:443
2025-05-11T12:54:17,039 DEBUG [storage.template.HttpTemplateDownloader] (pool-2-thread-3:[]) https://releases.rancher.com/harvester/v1.3.2/harvester-v1.3.2-amd64-net-install.iso is using chunked transfer encoding, possible remote size: 883715325
2025-05-11T12:54:17,043 DEBUG [storage.template.HttpTemplateDownloader] (pool-2-thread-3:[]) Remote size for https://releases.rancher.com/harvester/v1.3.2/harvester-v1.3.2-amd64-net-install.iso found to be (842.78 MB) 883715325
2025-05-11T12:54:17,051 INFO  [storage.template.HttpTemplateDownloader] (pool-2-thread-3:[]) Starting download from https://releases.rancher.com/harvester/v1.3.2/harvester-v1.3.2-amd64-net-install.iso to /mnt/SecStorage/037ced3e-bc30-388e-a0c2-d307b6f8ffb1/template/tmpl/2/211/dnld6098655088962065044tmp_ remoteSize=(842.78 MB) 883715325 , max size=(50.00 GB) 53687091200
2025-05-11T12:54:17,533 DEBUG [storage.template.HttpTemplateDownloader] (pool-2-thread-3:[]) Verified format of downloading file /mnt/SecStorage/037ced3e-bc30-388e-a0c2-d307b6f8ffb1/template/tmpl/2/211/dnld6098655088962065044tmp_ is supported
2025-05-11T12:54:45,608 DEBUG [storage.template.HttpTemplateDownloader] (pool-2-thread-3:[]) Reached expected remote size limit: 883715325 bytes
2025-05-11T12:54:46,113 INFO  [storage.template.DownloadManagerImpl] (pool-2-thread-3:[]) Download Completion for jobId: 472028ff-d07e-4b67-abb3-d5065a87d401, status=DOWNLOAD_FINISHED
2025-05-11T12:54:46,114 INFO  [storage.template.DownloadManagerImpl] (pool-2-thread-3:[]) local: /mnt/SecStorage/037ced3e-bc30-388e-a0c2-d307b6f8ffb1/template/tmpl/2/211/dnld6098655088962065044tmp_, bytes=(842.78 MB) 883722310, error=Downloaded (842.78 MB) 883722310 bytes (download complete remote=unknown bytes), pct=100
2025-05-11T12:54:46,115 INFO  [storage.template.DownloadManagerImpl] (pool-2-thread-3:[]) No checksum available for 'dnld6098655088962065044tmp_'
2025-05-11T12:54:52,088 DEBUG [storage.template.DownloadManagerImpl] (pool-2-thread-3:[]) computed checksum: {SHA-512}44fb1821bbaef863c68da65d84c0363a86efe24592f6b2eff64957b163f1d700041ccde856af418c0a9cc369a1188cb79d44dcaf665fafdebe17e7906742db2d
2025-05-11T12:54:52,092 DEBUG [storage.template.DownloadManagerImpl] (pool-2-thread-3:[]) Executing command [/usr/local/cloud/systemvm/scripts/storage/secondary/createtmplt.sh -s 2 -S 53687091200 -d chunked -h -n 211-2-21936ca5-db4d-3030-8f63-ed051b48865a.iso -t /mnt/SecStorage/037ced3e-bc30-388e-a0c2-d307b6f8ffb1/template/tmpl/2/211 -f /mnt/SecStorage/037ced3e-bc30-388e-a0c2-d307b6f8ffb1/template/tmpl/2/211/dnld6098655088962065044tmp_ -u ].
2025-05-11T12:54:52,212 DEBUG [storage.template.DownloadManagerImpl] (pool-2-thread-3:[]) Successfully executed process [10285] for command [/usr/local/cloud/systemvm/scripts/storage/secondary/createtmplt.sh -s 2 -S 53687091200 -d chunked -h -n 211-2-21936ca5-db4d-3030-8f63-ed051b48865a.iso -t /mnt/SecStorage/037ced3e-bc30-388e-a0c2-d307b6f8ffb1/template/tmpl/2/211 -f /mnt/SecStorage/037ced3e-bc30-388e-a0c2-d307b6f8ffb1/template/tmpl/2/211/dnld6098655088962065044tmp_ -u ].
2025-05-11T12:54:52,263 DEBUG [storage.template.DownloadManagerImpl] (pool-2-thread-3:[]) Cleaning-up temporary download file /mnt/SecStorage/037ced3e-bc30-388e-a0c2-d307b6f8ffb1/template/tmpl/2/211/dnld6098655088962065044tmp_
2025-05-11T12:54:52,265 DEBUG [storage.template.DownloadManagerImpl] (pool-2-thread-3:[]) Deleting directory /mnt/SecStorage/037ced3e-bc30-388e-a0c2-d307b6f8ffb1/template/tmpl/2/211, if empty, as part of cleanup

How did you try to break this feature and the system with this change?

@codecov Codecov
Copy link

codecov bot commented May 6, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 0% with 49 lines in your changes missing coverage. Please review.

Project coverage is 16.13%. Comparing base (f6d0590) to head (0b6185a).
Report is 7 commits behind head on 4.20.

Files with missing lines Patch % Lines
...cloud/storage/template/HttpTemplateDownloader.java 0.00% 49 Missing ⚠️
Additional details and impacted files 
@@             Coverage Diff              @@
##               4.20   #10820      +/-   ##
============================================
- Coverage     16.13%   16.13%   -0.01%     
  Complexity    13220    13220              
============================================
  Files          5651     5651              
  Lines        496740   496785      +45     
  Branches      60183    60187       +4     
============================================
+ Hits          80148    80150       +2     
- Misses       407674   407710      +36     
- Partials       8918     8925       +7
Flag Coverage Δ
uitests 4.00% <ø> (+<0.01%) ⬆️
unittests 16.98% <0.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@DaanHoogland DaanHoogland mentioned this pull request May 6, 2025
Open
@DaanHoogland
Copy link
Contributor

@blueorangutan package

@blueorangutan
Copy link

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 13273

@shwstppr
fix
7989d58 
Signed-off-by: Abhishek Kumar <[email protected]>
@DaanHoogland
Copy link
Contributor

@blueorangutan package

@blueorangutan
Copy link

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13275

@DaanHoogland
Copy link
Contributor

@blueorangutan test

@blueorangutan
Copy link

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

@blueorangutan
Copy link

[SF] Trillian test result (tid-13224)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 53033 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10820-t13224-kvm-ol8.zip
Smoke tests completed. 141 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File

@shwstppr
fix
0b6185a 
Signed-off-by: Abhishek Kumar <[email protected]>
@shwstppr
Copy link
Contributor Author

@blueorangutan package

@blueorangutan
Copy link

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13329

@shwstppr
Copy link
Contributor Author

@blueorangutan test

@blueorangutan
Copy link

@shwstppr a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@shwstppr @DaanHoogland @blueorangutan