Improve Efficiency of Variant Generation

[GordonsKitchen]

I understand this'll likely be a massive undertaking, but it would be good to review variant generation and try to reduce the amount of time it takes to run a global ("all") job. I've been adding lots of new combos recently and I've noticed the time balloon significantly just over a few months.

This job, for example, was ran on May 31, 2025 and took 7:43 to complete: https://backend.commanderspellbook.com/admin/spellbook/job/51263/change/?_changelist_filters=created__month%3D5%26created__year%3D2025%26name%3Dgenerate_variants

This job, ran August 20th (today), took 14:17 to complete: https://backend.commanderspellbook.com/admin/spellbook/job/60047/change/?_changelist_filters=created__month%3D8%26created__year%3D2025%26name%3Dgenerate_variants

Even looking at jobs from August, the job time has increased by roughly 3 minutes just this month alone.
This is almost a double time increase not quite 3 months. At this rate, job times will start to get very large quickly and become unwieldly for us to get as much done as possible. This isn't an urgent fix per se but will need to be addressed at some point relatively soon.

You may need to reach out to SCM for more resources, and/or possibly look into eliminating hardware constraints (like RAM limits). Keep in mind as well the database is only going to get larger, so try to make it as efficient as possible for future-proofing.

It's possible that this time leak could also be caused by some other issue that could be patched to save time. However, it certainly couldn't hurt to improve the process overall regardless.

[ldeluigi]

I've done many hourse of profiling, changing a lot of code in the process trying to improve it, but I haven't managed to beat the current performance (14m) by optimizing the code. The next thing to try for me would be to drop one of the core dependency and re-write its code myself to make it faster for our use case. I calculated that doing so would buy us about 1/7 of the time, bringing the current 14 minutes to 12. I feel that isn't enough of a difference to pursue the change, let me know if it's worh for you.

The real brakethrough could only be achieved by re-writing the code in a different language, probably C, but that's one of the worst nightmares for me right now. I'll reach out to SCM to ask for support.

[ldeluigi]

Now that we have Cython, many doors open:
https://cython.readthedocs.io/en/latest/src/tutorial/pure.html

[ldeluigi]

It takes 7 minutes in total now, we'll repone this issue when the time grows

[ldeluigi]

I misread the timer