Add support for many additional Keras 3 layers - #465
Merged
Conversation
Restored recurrent layers (LSTM, GRU, SimpleRNN, Bidirectional) that were removed in commit a60717c when Keras 3 broke the previous implementation, and added a broad set of layers that had been listed as unsupported. New layers: - LSTM, GRU, SimpleRNN, Bidirectional (all merge_modes) - ConvLSTM1D, ConvLSTM2D, ConvLSTM3D - RNN wrapping LSTMCell / GRUCell / SimpleRNNCell / StackedRNNCells - EinsumDense (generic einsum interpreter) - GroupedQueryAttention (with optional use_gate) - AdaptiveAvg/MaxPooling 1D/2D/3D - DepthwiseConv1D - RMSNormalization, GroupNormalization - Discretization, IntegerLookup - Masking (passthrough; mask propagation through recurrent layers is not modeled) - Many training-only image-augmentation layers as identity passthroughs (RandomBrightness, RandomCrop, RandomHue, AugMix, CutMix, MixUp, RandAugment, AutoContrast, etc.) Caveats: - Stateful recurrent layers and return_state-as-input remain unsupported. - ConvLSTM go_backwards / unroll / stateful remain unsupported. - GroupNormalization currently only supports axis=-1. A new test_model_recurrent test target exercises each new layer with multiple distinct configurations. README: pruned stale unsupported entries (ThresholdedReLU, LocallyConnected1D/2D, CuDNNGRU/LSTM) that were removed in Keras 3. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fixes clang-format-lint failures: alphabetical include order in fdeep.hpp and import_model.hpp, ternary indentation in IntegerLookup creator, plus a drive-by indentation fix in attention_layer.hpp that was already drifting. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CI uses clang-format-16 which prefers the previous indentation; my local clang-format-20 reformatted it differently. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
C++ correctness: - Fix UB: MappedRowMajorMatrixXf used Eigen::Aligned over std::vector data which is not SIMD-aligned. Switch the typedef to const + Unaligned and drop the const_cast at every call site. - IntegerLookup: replace abs(v) UB at INT64_MIN with a single-OOV-bucket assertion (Keras hashes OOV across multiple buckets via FarmHash, which fdeep does not currently replicate, so num_oov_indices > 1 is rejected). - EinsumDense: assert rhs has at least 2 chars before slicing off the leading batch dim. - GRU: replace silent zero-fallback for malformed bias with an explicit size assertion that respects reset_after. - GroupedQueryAttention: assert num_key_value_heads > 0 before the modulo. - AutoContrast removed from the passthrough dispatch — it transforms deterministically at inference and isn't actually a no-op. Performance: - EinsumDense: precompute summed_strides instead of recomputing the inner stride loop per output element. Style: - Make member fields const in newly-added layer classes for consistency with the rest of the codebase. - show_bidirectional_layer: validate merge_mode and reject forward_layer.go_backwards explicitly. Test coverage (added in get_test_model_recurrent): - LSTM with return_state=True (h, c outputs) and unit_forget_bias=False. - GroupedQueryAttention with num_query_heads == num_key_value_heads (= MultiHeadAttention) and with distinct K/V sequence lengths. - ConvLSTM1D with dilation_rate, two ConvLSTM2D layers chained. - GroupNormalization with groups=1 (= LayerNorm) and groups=n_features (= InstanceNorm). - Discretization, Masking, RandomBrightness, RandomFlip, RandomCrop now exercised end-to-end. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The catch-all get_test_model_recurrent had drifted to include layers that aren't recurrent at all (DepthwiseConv1D, EinsumDense, Adaptive*Pooling, GroupedQueryAttention, RMS/GroupNormalization, Discretization, Random* image augmentation). Extract those into a new get_test_model_extended driving a parallel test_model_extended_test target so each test name reflects what it covers. test_model_recurrent now only contains genuinely-recurrent layers: LSTM, GRU, SimpleRNN, Bidirectional, Masking, RNN with cells, ConvLSTM 1D/2D/3D. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per review feedback the previous split that introduced a separate test_model_extended duplicated what test_model_exhaustive is for. Drop test_model_extended and append DepthwiseConv1D, EinsumDense, RMSNormalization, GroupNormalization, AdaptiveAvg/MaxPooling, GroupedQueryAttention, Discretization, and the Random* augmentation passthroughs to test_model_exhaustive, reusing existing inputs (notably inputs[49]/[50]/[51] for the attention layers and inputs[22] for image shapes). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Split deeply-nested apply_impl bodies into named helpers: - adaptive_pooling_3d_layer: extracted pool_window() and adapt_range() (also folded the in_size==1 special case into adapt_range), plus output_shape_for() to keep the apply_impl loop body shallow. - einsum_dense_layer: extracted derive_char_sizes(), contract_one(), add_bias(), and decode_index/encode_offset helpers. - group_query_attention_layer: split into project(), attention_distribution(), compute_attention(), apply_sigmoid_gate(), and output_projection(). No functional change — all existing tests still pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Latent bugs fixed: - EinsumDense bias ordering: Keras stores the bias variable with dims in rhs char order, not bias_axes-string order. Reorder bias_axes_ chars to match rhs_ before computing strides so the right element is read. - EinsumDense output_shape index: Keras's output_shape excludes the batch dim, so indexing rhs[i] against full_output_shape_[i] was off by one for output-only chars. Prepend a -1 placeholder for the batch char in the C++ creator to keep positional indexing consistent. Defensive guards: - IntegerLookup now requires num_oov_indices == 1 (was <= 1, which silently accepted strict mode that Keras would have raised on). - Add n_units > 0 assertion to lstm_impl, gru_impl, simple_rnn_impl. - Add groups > 0 / group_volume > 0 assertions to GroupNormalization. - Add out_size > 0 assertion to AdaptivePooling adapt_range. Style: - Make IntegerLookup's vocab_index_ const and populate via a helper called from the member init list, matching the rest of the new layer files. Test coverage: - LSTM had return_state=True but GRU and SimpleRNN didn't — added. - GroupedQueryAttention with num_query_heads == num_key_value_heads now also exercises distinct K/V sequence lengths (cross-attention shape). - IntegerLookup is now reached by an end-to-end model (chained after Discretization to get an integer-valued tensor). - Several more Random* augmentation passthroughs (RandomContrast, RandomRotation, RandomTranslation, RandomZoom, RandomHue, RandomSaturation, RandomSharpness) now go through the test pipeline. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The four prior EinsumDense test cases all used bias_axes in canonical
(rhs-matching) order, so the bias-ordering bug fixed in the previous
commit was invisible to the suite. Add an explicit case where bias_axes
is permuted ('ed' on rhs='abde'). Verified by temporarily reverting the
fix: this case fails the 1e-5 tolerance, confirming the test now catches
the regression.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Repository owner
deleted a comment
May 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Restores the recurrent layers (
LSTM,GRU,SimpleRNN,Bidirectional) that were removed ina60717cwhen Keras 3 broke the previous implementation, and adds a broad set of layers that had been listed as unsupported in the README.Newly supported layers
Recurrent
LSTM,GRU,SimpleRNN,Bidirectional(concat/sum/mul/avemerge modes)ConvLSTM1D,ConvLSTM2D,ConvLSTM3DRNNwrappingLSTMCell/GRUCell/SimpleRNNCell/StackedRNNCellsAttention / dense
EinsumDense(generic einsum interpreter; covers transformer-style projections)GroupedQueryAttention(with optionaluse_gate=True)Pooling / convolution
AdaptiveAveragePooling1D/2D/3D,AdaptiveMaxPooling1D/2D/3DDepthwiseConv1DNormalization
RMSNormalizationGroupNormalization(onlyaxis=-1for now)Preprocessing
Discretization,IntegerLookupMasking(passthrough)RandomBrightness,RandomCrop,RandomHue,RandomSaturation,RandomShear,RandomGrayscale,RandomInvert,RandomPosterization,RandomGaussianBlur,RandomColorJitter,RandomColorDegeneration,RandomElasticTransform,RandomErasing,RandomPerspective,RandomSharpness,AugMix,CutMix,MixUp,RandAugment,Equalization,Solarization,Pipeline,MaxNumBoundingBoxes,AutoContrastOut of scope (still unsupported)
return_state-as-inputConvLSTMgo_backwards/unroll/statefulHashing,HashedCrossing(would need a FarmHash port to match Keras's bucket assignments)MelSpectrogram,STFTSpectrogram(would need an FFT)StringLookup,TextVectorization(fdeep's tensor is float-only)Maskingend-to-end mask propagation through recurrent layers (the layer is registered, but Keras 3 lowersMasking → LSTMto a graph withNotEqual/Anyops that this PR doesn't model)Tests
A new
test_model_recurrenttest target exercises each new layer with multiple distinct configurations (e.g.LSTMwithuse_bias=Falseand withrelu/hard_sigmoidactivations,GroupNormalizationwithscale=Falseand withcenter=False,ConvLSTM2Dwithpadding='valid'anduse_bias=False, etc.). Numerical outputs are verified against Keras at conversion time as usual.README's unsupported-layer list was pruned of stale entries that no longer exist in Keras 3 (
ThresholdedReLU,LocallyConnected1D/2D,CuDNNGRU,CuDNNLSTM).Test plan
ctest --output-on-failure— all 7 tests pass locally with the existingtest_model_*plus the newtest_model_recurrent_test🤖 Generated with Claude Code