Recurrent network interface updates/design

[mkschleg]

While we were discussing #1675 and #1671 several improvements/updates to the recurrent network API came up. Instead of taking over #1675, @ToucheSir and myself thought it would be best to separate out the needed improvements into a separate issue so they can be worked on and discussed here. That way we can finish #1675, and move on with the other changes in lock-step.

• Add Support for 3d-arrays (i.e. FoldedRNNs)
• Adding CuDNN support back in (after FoldedRNNs)
• Potentially add TimeDistributed from TFs API
• Adding parameters to change gate activation functions
• Lifting eltype restrictions on all recurrent cells (ref Recurrent cell eltype restriction breaks outputsize #1565)
• Allowing for non-dense operations in RNN cells (ref ConvRNN Cell and ConvLSTM Cell #91)
• Bidirectional RNNs, and fusing multiple layers (related to CuDNN, ref Adding support for folding RNNs over 3d arrays #1686).

Any others I'm missing?

[ToucheSir]

I added the ConvLSTM issue in the top post so we can track that here as well.

[mkschleg]

I've started working on the Folded interface. It actually should be pretty easy to add without disrupting the current api, but I haven't thought through how CuDNN fits yet. I'll make a pr after we get #1675 merged, so we can iterate.

[mkschleg]

I separated out CuDNNs from 3d array support. They both influence each other, but I think getting the FoldedRNN's api right needs a bit of iteration first.

[mkschleg]

I added Bidirectional RNNs as per the conversation in #1686.

[CarloLucibello]

Now that #1686 is merged I think we should disambiguate the interface (e.g. a 3d tensor input could be a single time step on batched grey images [width, height, batch_size] or multiple time steps on batched 1d inputs [num_features, batch_size, seq_length]).

I think that starting from next breaking release we should always assume that the input's last dimensions are batch and time, and start introducing a deprecation path.

[DhairyaLGandhi]

It's not something that needs deprecation yet.

[CarloLucibello]

If we decide to go with this

we should always assume that the input's last dimensions are batch and time, and start introducing a deprecation path.

at the very least we should immediately update the docs

[ToucheSir]

👍 for a docs update. I don't think we need to deprecate anything though, because it's not at all clear that Recur is the right struct to hang all this off of long term. What might be nice is if we took this opportunity to create the proposed experimental package/namespace and started prototyping there. My assumption is that most of the back and forth there will be about naming and API design, so being able to change both at a whim while still having something concrete for users to test would be nice.

[dcecchini]

I could not find the reference to Bidirectional on #1686 so I created a wrapper inspired by the way it is applied in Keras. The PR with more details can be found here.

The errors on the checks were due to the instability of Github this Sunday (could not connect to github.com).

[mkschleg]

@ToucheSir @DhairyaLGandhi @CarloLucibello and others. Sorry for the recent static, but this semester was absolutely chaotic/brutal for me.

I've started a RNN repo where we can play with the interface. I think the first thing we need to do is figure out the weirdness in the API that was introduced by #1686 . As discussed, it would not be possible to have convolutional RNNs w/ this interface. I'm thinking the recur struct needs to know some amount of information about its cell type for this to work. It is also possible that we could use traits to accomplish this.

This would also be a good place to test CuDNN paths.

I'm curious to see what people think so we can get the RNN api in a better place. I have some ideas and will hopefully get to working on instantiating them over the next few days.

The current repo just a copy of the current interface, and the tests are also the same (but just for the RNN layers).