Freethreading - module markers + CI

[H-Plus-Time]

Yet another fork (this time in setup-python), what exactly was it about setup-uv in windows that broke builds?

Verified wheel building run over at https://github.com/H-Plus-Time/arro3/actions/runs/12349504140.

[kylebarron]

I was hitting this #263. If you find a way to use uv on Windows I'd be happy to use it. Maybe it's an issue with PATH?

[kylebarron]

It's also an issue that the buffer protocol is even less sound in Python 3.13t.

See discussion: https://discord.com/channels/1209263839632424990/1310572181138178129/1310572181138178129, https://alexgaynor.net/2022/oct/23/buffers-on-the-edge/, PyO3/pyo3#2824

I'm not sure what to do here. It's very beneficial to have the buffer protocol as an option. So perhaps just document to Python users that it's unsafe to mutate the Python buffer after passing it to rust?

[H-Plus-Time]

Yeah - I'm actually having a great deal of difficulty thinking of buffer protocol scenarios where freethreading would be unsound but not GIL-enabled; there's definitely a difference, but the complete lack of immutability guarantees is 99% of the problem (and serialized python execution (the GIL) addresses the remaining 1%).

I think the safety note on into_arrow_array, surfaced prominently should do the trick.

By the sounds of (amongst other parts of that document) https://py-free-threading.github.io/porting/#dealing-with-thread-unsafe-objects and the comments on how numpy is going ahead with this, pathological cases (like resizing an array while it's being read) can just be left as "you're holding it wrong, enjoy the segfault", right up until the first bug report.

[kylebarron]

Hmm it looks like I can't trigger the wheel builds on this repo from your PR.

I think the safety note on into_arrow_array, surfaced prominently should do the trick.

Ideally this would also be surfaced to Python to be seen by Python users, but I'm not sure the right level of obtrusiveness for that.

[kylebarron]

It would be nice if we didn't have to use a fork of numpy. Do you know if there's progress towards getting that merged?

[H-Plus-Time]

Hmm it looks like I can't trigger the wheel builds on this repo from your PR.

Yeah, I noticed that in my own fork, though that seems to be fine now 🤔. It's likely a matter of: jobs can be triggered manually via workflow_dispatch, but not on pull requests (the wheels would need on: pull_request, which might be ok with cancel-in-progress); there doesn't seem to be a non-convoluted way to do manually triggered PR checks.

[H-Plus-Time]

Mm, not yet, probably mid-January (given the whole Christmas/NYE period is upon us) - it looks like it's down to a single mutex, probably test reconfiguration too.

Obtrusiveness-wise, I can stick a (concise) spiel in the docstring of Array::frombuffer; I'd be hesitant to log anything at runtime higher than trace or debug level.

[kylebarron]

I'm inclined not to merge this until numpy gets a stable release including the free-threading feature

[H-Plus-Time]

Yep, I'll stick this one (and related) in draft.

[kylebarron]

Numpy merged its free-threading pr: PyO3/rust-numpy#471 (comment)

[H-Plus-Time]

Excellent, hopefully shouldn't be too much longer until that gets a release. I'll re-pin to the tip of main and see if anything strange crops up in the build in the meantime.

[H-Plus-Time]

Hrm, blocked by the pyO3 breaking change in 0.23.4 involving the IntoPyObject constraint and Chrono Tz. With the suggested workaround (map(|v| v.fixed_offset()) specifically), seems to compile fine. By the looks of apache/arrow-rs#7173 and associates, everything should pan out roughly around the same time (the numpy release + patch to obviate the fixed_offset workaround)

[kylebarron]

Hrm, blocked by the pyO3 breaking change in 0.23.4 involving the IntoPyObject constraint and Chrono Tz.

Can you merge in latest main? It should be fixed (with a temporary fixed offset workaround)