Skip to content

Incorporating should_run in nx-parallel #77

Open
@Schefflera-Arboricola

Description

I'd like to propose integrating the should_run mechanism into nx-parallel for functions whose original networkx implementation is already really fast or the functions that are faster for small graphs. Some functions that could benefit include number_of_isolates and almost all shortest paths generator functions.

We can take inspiration from cugraph's should_run in PR rapidsai/cugraph#4348. Additionally, in nx-cugraphs using should_run significantly improves speedups because the graph conversion cost in nx-cugraph is high. But, in nx-parallel that's not the case, so it's hard to tell how much will this benefit in terms of speedup improvement.

Also, if we do choose to implement should_run in nx-parallel then what should be the criterion for initiating should_run? Like, how small is a small graphs? will it be same for all algorithms?

Another concern of mine about these algorithms that are already really fast is whether we should omit chunking by default for such algorithms and instead feed the node list or edge list directly in the joblib.Parallel. Because chunking seems like an extra step for such algorithms that doesn't actually improve the speedups that much, I think. Also, for really fast algorithms it's hard to tell if chunking is actually improving the speedups or not because they are so fast.

Any feedback or thoughts on this?

For reference - networkx/networkx#7257 - added should_run in the main networkx library.

Thank you :)

PS: Should we also consider introducing a can_run in nx-parallel?

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Labels

    questionFurther information is requestedtype: EnhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions