Skip to content

Extract index generation into a separate module from upload_package_repo.py#3566

Merged
erman-gurses merged 4 commits intomainfrom
users/erman-gurses/split-index-html-generation
Feb 25, 2026
Merged

Extract index generation into a separate module from upload_package_repo.py#3566
erman-gurses merged 4 commits intomainfrom
users/erman-gurses/split-index-html-generation

Conversation

@erman-gurses
Copy link
Copy Markdown
Contributor

@erman-gurses erman-gurses commented Feb 23, 2026 •
edited
Loading

Motivation

Progress on #3328
This PR extracts the HTML index generation logic from upload_package_repo.py into a separate module, generate_package_indexes.py.

Technical Details

  • Moved S3-based index generation functions into a new script:
    • generate_index_from_s3
    • generate_top_index_from_s3
  • Updated upload_package_repo.py to import and call these functions from the new module.
  • The new script can also be executed standalone via CLI to regenerate indexes for an existing repository prefix.

Test Plan

Run on CI

Test Result

Test: https://github.com/ROCm/TheRock/actions/runs/22338748161/job/64637174767

Submission Checklist

@erman-gurses
Copy link
Copy Markdown
Contributor Author

erman-gurses commented Feb 24, 2026 •
edited
Loading

Test: https://github.com/ROCm/TheRock/actions/runs/22338748161/job/64637174767

@erman-gurses erman-gurses marked this pull request as ready for review February 24, 2026 06:31
@marbre
Copy link
Copy Markdown
Member

marbre commented Feb 24, 2026

Test: https://github.com/ROCm/TheRock/actions/runs/22338748161/job/64637174767

Can you please post the link to the entry point allowing to browse the index files?

@marbre marbre requested a review from araravik-psd February 24, 2026 11:28
@marbre
Copy link
Copy Markdown
Member

marbre commented Feb 24, 2026

@araravik-psd, please review. This addresses #3328, more refactoring might will be needed to fully address #3329.

@erman-gurses
Copy link
Copy Markdown
Contributor Author

erman-gurses commented Feb 24, 2026 •
edited
Loading

Test: https://github.com/ROCm/TheRock/actions/runs/22338748161/job/64637174767

Can you please post the link to the entry point allowing to browse the index files?

Sure, based on the log in here:
https://github.com/ROCm/TheRock/actions/runs/22338748161/job/64637174767#step:10:41

Uploading to s3://therock-dev-packages/rpm/20260224-22322854136/

This should be the link below. However, I got Access Denied
https://therock-dev-packages.s3.us-east-2.amazonaws.com/rpm/20260224-22322854136/

@marbre
Copy link
Copy Markdown
Member

marbre commented Feb 24, 2026

This should be the link below. However, I got Access Denied https://therock-dev-packages.s3.us-east-2.amazonaws.com/rpm/20260224-22322854136/

In that case something went wrong here and you'll need to check via the AWS console. CF should be set-up correctly as I see redirects to index.html files for other runs working at https://rocm.devreleases.amd.com/rpm/.

@erman-gurses
Copy link
Copy Markdown
Contributor Author

erman-gurses commented Feb 24, 2026 •
edited
Loading

This should be the link below. However, I got Access Denied https://therock-dev-packages.s3.us-east-2.amazonaws.com/rpm/20260224-22322854136/

In that case something went wrong here and you'll need to check via the AWS console. CF should be set-up correctly as I see redirects to index.html files for other runs working at https://rocm.devreleases.amd.com/rpm/.

I think I was looking at the wrong entry point above. These two links are working fine:
https://rocm.devreleases.amd.com/rpm/20260224-22322854136/index.html
https://rocm.devreleases.amd.com/rpm/20260224-22322854136/x86_64/index.html

CC: @araravik-psd

nunnikri
nunnikri approved these changes Feb 24, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@nunnikri nunnikri left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
The jobs are showing the index files uploaded properly.
Generating indexes from S3: s3://therock-dev-packages/rpm/20260224-22322854136/ (recursive)
Uploading index: rpm/20260224-22322854136/x86_64/repodata/index.html
Uploading index: rpm/20260224-22322854136/x86_64/index.html
Uploading index: rpm/20260224-22322854136/index.html
Generated and uploaded 3 index files from S3 state
Generating top index from S3: s3://therock-dev-packages/rpm/
Uploading top index: rpm/index.html

marbre
marbre reviewed Feb 24, 2026
View reviewed changes
Comment thread build_tools/packaging/linux/generate_package_indexes.py
@erman-gurses erman-gurses requested a review from marbre February 25, 2026 16:10
@erman-gurses erman-gurses merged commit 7a38803 into main Feb 25, 2026
4 checks passed
@erman-gurses erman-gurses deleted the users/erman-gurses/split-index-html-generation branch February 25, 2026 21:43
@github-project-automation github-project-automation Bot moved this from TODO to Done in TheRock Triage Feb 25, 2026
@erman-gurses erman-gurses mentioned this pull request Feb 25, 2026
Merged
@erman-gurses erman-gurses linked an issue Feb 25, 2026 that may be closed by this pull request
Split index.html generation from upload_package_repo.py #3328
Closed
erman-gurses added a commit that referenced this pull request Mar 2, 2026
@erman-gurses
Add unittests for generate_package_indexes_test.py script (#3623)
5a77600 
Closes #3328
Follow up PR for #3566
Add unittests for `generate_package_indexes_test.py` script
```
pytest build_tools/packaging/tests/generate_package_indexes_test.py -vv
=================================================================================== test session starts ===================================================================================
platform linux -- Python 3.12.12, pytest-8.4.2, pluggy-1.6.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /home/nod/ergurses/TheRock/build_tools
configfile: pyproject.toml
collected 4 items                                                                                                                                                                         

build_tools/packaging/tests/generate_package_indexes_test.py::GeneratePackageIndexesTest::test_generate_index_from_s3_creates_indexes_and_links PASSED                              [ 25%]
build_tools/packaging/tests/generate_package_indexes_test.py::GeneratePackageIndexesTest::test_generate_index_from_s3_respects_max_depth PASSED                                     [ 50%]
build_tools/packaging/tests/generate_package_indexes_test.py::GeneratePackageIndexesTest::test_generate_index_html_skips_dotfiles PASSED                                            [ 75%]
build_tools/packaging/tests/generate_package_indexes_test.py::GeneratePackageIndexesTest::test_generate_top_index_from_s3_lists_subfolders PASSED                                   [100%]

==================================================================================== 4 passed in 0.11s ====================================================================================
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@nunnikri nunnikri nunnikri approved these changes

@ammallya ammallya Awaiting requested review from ammallya

@araravik-psd araravik-psd Awaiting requested review from araravik-psd

@marbre marbre Awaiting requested review from marbre

Assignees

No one assigned

Labels

None yet

Projects

TheRock Triage
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Split index.html generation from upload_package_repo.py

3 participants

@erman-gurses @marbre @nunnikri