[RFC0010] Test Scripts Migration Discussion

[jayhawk-commits]

Related PR: #3907

[jayhawk-commits]

This seems like substantial effort for relatively little benefit. Why create automation for syncing? Would a manual copy once be sufficient?

— @ScottTodd in #3907

Looking at the commit history of the test_executable_scripts directory, there are a handful of changes per week. Some automated synchronization would help not disrupt other development touching these scripts while the work for this transition was happening, which could take some time.

I agree that the effort for a synchronization workflow would be more substantial if the files get moved to the individual project folders (Option A) instead of a singular test scripts folder on the super repos (Option B). It might be better to just do manual copies at the beginning and end of the transition, where the latter considers the commit history.

[jayhawk-commits]

Some of the new imports of that module are misguided... that is not a general utility file, it is just for interacting with the github API. Need to fix that independent of this migration work.

— @ScottTodd in #3907

In general I'd like for these tests/scripts to be runnable by developers and not just CI systems, so I've never liked their location in build_tools/github_actions/test_executable_scripts. We shouldn't have a split there at all. They shouldn't be "build tools" or be tied to "github actions".

— @ScottTodd in #3907

What I envision is that TheRock repo will have a proper place to have common functions the testing CI would want for things like sharding setup, test log uploading/parsing, posting results to database, etc. These functions would be called outside these scripts being moved, so that developers can easily run these test scripts on their local setups. The interface for these test scripts can be limited to setting environment variables that work locally and can reproduce some of the CI behaviour (e.g., running only the tests shard 2 of 4 would run). This effort will happen separately. I think moving the scripts is more urgent.

[jayhawk-commits]

Filed #3968 for this other work mentioned that is not part of the migration, but would be affected by it.