Limit what we iterate over in providers

[jstack-godaddy]

Summary

The current implementation of applying/creating users, grants and databases was creating extremely long puppet runs in one of our environments. The environment is a shared hosting implementation that has 1000+ schema and 4 users per schema for RW/RO access. These schema and users are managed by another team, so we do not want to prune. The other team also does not want to limit users as it would create a split in workflow between on-prem and cloud provisioning. Puppet runs with --tag mysql were taking close to 14 minutes to loop over all users, all grants, and all databases. In this change we are determining what the users, grants and databases we care about and only iterating over them. There's little to gain by looping over everything if we only care about the users, grants, and databases we are managing via hiera and manifests. The result is astounding. The runtime dropped from 14 minutes to 11 seconds with no change in expected behavior.

[root@HOST ~]# puppet agent --test --tags mysql
....
Notice: Applied catalog in 820.98 seconds

after:

[root@HOST ~]# puppet agent --test --tags mysql
....
Notice: Applied catalog in 11.32 seconds

Additional Context

Add any additional context about the problem here.

• To see long puppet application times, create a large amount of users and schema. In our case 1000+ schema and 4000+ users.
• Speed up application times, resulting in faster delivery of changes in the manifests and less resource usage on database servers.

Related Issues (if any)

N/A

Checklist

• 🟢 Spec tests.
• 🟢 Acceptance tests.
• Manually verified. (For example puppet apply)

[CLAassistant]

All committers have signed the CLA.

[jst-cyr]

@jstack-godaddy : This looks like a good addition, thanks for submitting it! And congrats on the performance improvement for your environment! I believe that typically we would need to have some tests that are validating the new functionality. The existing tests will hopefully cover regression issues, but are you able to add a test that represents your scenario so we make sure that once this is introduced it continues to function?

[jstack-godaddy]

@jst-cyr Thank you for your response. I have added some new spec. I did use Claude to assist and a co-worker proficient in ruby to do a code-review as I'm not a ruby programmer by trade. I also made sure Rubocop passes locally for syntax. Tested in a test environment and manifests compile properly as well. Any other requests?

[jst-cyr]

Thank you @jstack-godaddy ! I did a rebase ran the spec tests again (it failed originally because of an stdlib version mismatch with metadata which was corrected in main). CI is running green again. It will probably be some time before engineering can do their review, but at least we've gotten it to a good place now!