One may write pseudocode for foldxt and foldxd as something like:
function foldxt(step, xf, xs)
parts = split_into_parts(xs)
output = init_output()
@threads for part ∈ parts
insert!(output, foldxl(step, xf, part))
end
combine(xf, output)
end
This is fairly intuitive, however currently Transducers.jl doesn't really expose any of these intermediate steps, and the internals of it are such that it might not be all that obvious what's going on until close inspection (at least it wasn't to me).
I propose that we create a public API for each of these pieces such that a user could easily write its own foldxt. As I see it, some benefits of this would be:
- The semantics of
foldxt might be somewhat more obvious. This is important because the combine step can cause it to behave quite differently from foldxl.
- Users could create custom
combine. Currently users have to rely on Transducers to provide combine functionality.
- Users could more easily create custom splitting strategies.
On the other hand, exposing these details could make it significantly easier to write unsafe code, though note that a custom combine needn't rely on parallelism at all.
One may write pseudocode for
foldxtandfoldxdas something like:This is fairly intuitive, however currently Transducers.jl doesn't really expose any of these intermediate steps, and the internals of it are such that it might not be all that obvious what's going on until close inspection (at least it wasn't to me).
I propose that we create a public API for each of these pieces such that a user could easily write its own
foldxt. As I see it, some benefits of this would be:foldxtmight be somewhat more obvious. This is important because thecombinestep can cause it to behave quite differently fromfoldxl.combine. Currently users have to rely on Transducers to providecombinefunctionality.On the other hand, exposing these details could make it significantly easier to write unsafe code, though note that a custom
combineneedn't rely on parallelism at all.