Windows support for pip

[ahmet-umut]

Hello. Since this seems like a relatively informal discussion channel, I will narrativitize the potential support for using loopy in Windows here. Read anything weird, note that I am not quite native English speaker. Just want to learn about Windows support of this library without any fun? Read from end then, or just use a LLM to extract.

I am a new-grad computer scientist who is yet-unemployed. After 1,5 years of a pretty tough education and hot 6 months of preparing for interviews etc., finishing all pending interviews; during the time I wait for results from multiple companies; -this corresponds to the last week- I wanted to free myself to lean over my ambitions since I feel like this time is the last free time of my life until I am A - quite old or B - unexpectedly rich. I value this unemployed time very much.

2 months ago ,during my interview/career speedrun, I had been busy designing a funny totally experimental LLM model in my head. I then wanted to proceed with experimenting. That resulted in me overthinking about which language to implement this experimental yet scalable -in theory- model. The model is so experimental that -even without search- I guarantee you that it is harder -if not impossible- to find a meaningfully optimal previously hardware-accelerated library/way of implementing the model I have been fantasizing. So I -as I love to do so- decided to not lose my ambition/fun time searching suitable libraries forever -in the risk of finding a suboptimal unmaintained (sub-)library which I need to tweak nevertheless- and just write the experiment code myself.

Then, the implementations of neural networks I have done since I learned programming flashed before my eyes. Those times, I had no idea what parallelism is, or how GPUs work. So I was always writing custom neural network inference and back-propagation functions in a single thread -"by design". Now, since I am a graduated computer scientist, I know how GPUs work and how parallelism can be achieved -at least in theory. Then -I thought-, I had to do something about it... Why not accelerate my own code -for the first time. I decided to write the core functionality in HLSL compute shader code and using shader in c#. Those decisions were just for the sake of starting from somewhere, not a highly educated / carefully thought ones.

I have been grappling with Direct3D11 and 12, setting up lots of things just to be able to use single-precision shaders. Then I wanted to see if I could also write code that works efficiently on half-precision data. That was a horrible week for me. Using LLMs only to create the ugliest pipeline I have ever seen was so bad. I later had a heated discussion with a friend about my project. He convinced me that, compute shaders -as opposite of what I had known- were just for the cases when it is already going to be part of a graphical pipeline.

I then abandoned my unborn baby project of HLSL-c# and went back on looking for how could I do GPGPU programming. I already know I could start with OpenCL now, but I wanted to consider my options before using pure OpenCL. So I stumbled upon numba-dpex. (Disclaimer: My GPU is a intel iGPU. So I cannot use CUDA or similar.) I spent 2 weeks to just make it run. It did not happen, because, apparently, the package is just a unmaintained package and requires sycl6.dll, where the current oldest OneAPI you can download officially had sycl7 or 8.dll.

After 2 long weeks of disappointment, I gave up on numba-dpex with sadness (The idea of auto-converting python code to GPGPU code with decorators was very exciting to me!). It was then I found Loopy. The idea behind this library was also very exciting to me and I wanted to install and use it right away.

However, I had Windows and a bad experience with Linux enough to not reinstall it (my current installation is broken) before it is strictly necessary. And loopy seems to be compatible with Windows except that one package: islpy. And islpy seems not to have any Windows-compatible pip package. I did not want to ditch pip or use conda. So, I wanted to install islpy using pip. I later found old a old issue set that is centered around building isl (C library islpy wraps around) in/for Windows. I applied the commits and solved different bugs for some days to eventually be able to compile isl in Windows without any bugs -in hope. So, I installed islpy using that, then installed loopy and it worked. I tested it for 30 minutes of different programs that is timed and guaranteed to run in my GPU.

Since everything I have done so far was so new to me, I decided to load this binary I created that works on My Windows into PyPI so that anyone has a chance to use loopy or any library that depends on islpy on Windows. Feel free to give feedback to me thru this discussion, since this is my first ever Python package upload. Also, can my package be defined as a fall-back package if loopy is wanted to use in a Windows environment?