Loop Type Networks

[PaulSchweizer]

A potential user of flowpipe pointed out the fact that we do not support loop type networks as described in the original Flow-Based Programming definition: http://www.jpaulmorrison.com/fbp/loop.htm

The example given, was reading a file line by line and acting on each line. I imagine that graph to look something like this:

+------------------------------------------------+
|                                                |
|   +---------------+          +--------------+  |
+-->|   Read Lines  |          |  Print Line  |--+
    |---------------|          |--------------|
    o file<>        |     +--->o line<>       |
    |        line<> o-----+    +--------------+
    +---------------+       

I am not sure how to best implement this yet, it will involve quite a bit of reshuffeling and redesigning I think.

[globant-camilo-rubio]

Hi Paul! Actually I don't think that there must be a link between Print Line output and Read Lines input. The general idea here is to pass every line read from file through the entire network, one by one, and not finish execution until all of them have been processed. I think it can be achieved by the loop you describe, but it would force to process next event only when the last one already finished. That is not needed and will affect performance.

I'll show an example that I tried using this project: https://github.com/baffelli/pyperator.
We have this file:

what we want is to process every line in the file as an IP flowing through the network. For this example we will only print every line.

First I developed a component for reading the file:

Notice how every line is warpped as an InformationPacket on line 26, and then flowed to next component through the port in line 28
Notice how we send a special IP named EndOfStream, which tell's the execution to stop because no more events are expected on line 29

Then I developed a component to print IP's:

Then we only put all the pieces together:

The result is:

I hope this clears some doubts, and I would be glad to help anyway I can.

[PaulSchweizer]

Thanks for explaining and illustrating the issue so well!

I have no experience with pyperator unfortunately, but it seems that the big difference is that in flowpipe, you have to construct your entire graph before execution, while pyperator is more "dynamic" in that way.

It would be a really cool to be able to do that in flowpipe as well, but as it stands we'd have to make some pretty big changes to the architecture, so I can't promise any quick solution at all. Sorry for these news.

Does anyone else have different thoughts on this?

[neuneck]

I concur with @PaulSchweizer - this would require fundamental changes to flowpipe. Most notably, as it stands, flowpipe has individual batches of work as inputs as a hard assumption. Each node is considered ready to operate only when all it's dependencies have finished their tasks. Thus, to accomodate a continuous, streaming approach would require us to revamp the graph execution logic as well as a completely different way that nodes are connected.