Skip to content

Conversation

@andrewdnolan
Copy link
Collaborator

@andrewdnolan andrewdnolan commented Oct 16, 2025

This PR speeds up the file permission updating by removing nested for loops and added the ability to parallelize the file permission departures over threads.

Checklist

  • User's Guide has been updated if needed
  • Developer's Guide has been updated if needed
  • API documentation lists any new or modified class, method, or function
  • Documentation builds cleanly and changes look as expected
  • Tests pass and new features are covered by tests
  • PR description includes a summary and any relevant issue references
  • Testing comment, if appropriate, in the PR documents testing used to verify the changes

Remove nested for loops and parallelize updating using threads.
Also switched to tqdm for the progress bar updating.
@andrewdnolan andrewdnolan added the enhancement New feature or request label Oct 16, 2025
Add docstring and make progress bar optional.
Copy link
Collaborator

@xylar xylar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@xylar
Copy link
Collaborator

xylar commented Oct 16, 2025

Feel free to merge when you're ready.

Also switched to pass from continue so that subdirs are
properly itterated.
@andrewdnolan
Copy link
Collaborator Author

Testing:

Setting up the test directory:

$ mkdir mache_testing
$ cp -r /lcrc/soft/climate/e3sm-unified/e3sm_unified_1_12_0rc1 mache_testing/
$ python -c 'from mache.permissions import update_permissions; update_permissions("mache_testing/spack", "cels", show_progress=True, group_writable=True, other_readable=False)

Updating file permissions for: mache_testing/e3sm_unified_1_12_0rc1
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 534374/534374 [04:17<00:00, 2075.54item/s]

I intentionally changed the group_writable and other_readable value from what e3sm-unified uses, to make sure actual changes were being applied. This branch finished in 4 min 17 sec, whereas it took over 2 hour using the old version to deploy on chrysalis yesterday.

@andrewdnolan andrewdnolan merged commit f7f4fba into E3SM-Project:main Oct 17, 2025
5 checks passed
@andrewdnolan andrewdnolan deleted the speed_up_permission_updates branch October 20, 2025 16:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants