onnx output nan

[jeffkrupa]

Hi,

I am converting the model in from saved-model to onnx using python -m tf2onnx.convert --saved-model saved-model --output model.onnx. More details are below.

The model inference seems to work in keras

>>> from keras.models import load_model
>>> import tensorflow as tf
>>> keras_model = load_model('saved-model')
>>> x = np.zeros(shape=(1,60,18))
>>> y = np.zeros(shape=(1,5,6))
>>> keras_model.predict([x,y])
array([[9.994024e-01, 5.975381e-04]], dtype=float32)

however the output is nan in onnxruntime on the onnx file produced with tf2onnx

>>> import onnxruntime as rt
>>> sess = rt.InferenceSession('model.onnx')
>>> x = x.astype(np.float32)
>>> y = y.astype(np.float32)
>>> sess.run([sess.get_outputs()[0].name],{sess.get_inputs()[0].name:x, sess.get_inputs()[1].name:y})
[array([[nan, nan]], dtype=float32)]

There doesn't seem to be anything in the conversion step that is failing. What else could be going wrong? I can share the model if needed. Thank you.

>>> keras_model.summary()
Model: "model"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
inputParticle (InputLayer)      [(None, 60, 18)]     0                                            
__________________________________________________________________________________________________
inputSV (InputLayer)            [(None, 5, 6)]       0                                            
__________________________________________________________________________________________________
inputNormParticle (BatchNormali (None, 60, 18)       72          inputParticle[0][0]              
__________________________________________________________________________________________________
inputNormSV (BatchNormalization (None, 5, 6)         24          inputSV[0][0]                    
__________________________________________________________________________________________________
XdotRR (Lambda)                 (None, 3540, 18)     0           inputNormParticle[0][0]          
__________________________________________________________________________________________________
XdotRS (Lambda)                 (None, 3540, 18)     0           inputNormParticle[0][0]          
__________________________________________________________________________________________________
XdotRK (Lambda)                 (None, 300, 18)      0           inputParticle[0][0]              
__________________________________________________________________________________________________
YdotRV (Lambda)                 (None, 300, 6)       0           inputNormSV[0][0]                
__________________________________________________________________________________________________
Bpp (Lambda)                    (None, 3540, 36)     0           XdotRR[0][0]                     
                                                                 XdotRS[0][0]                     
__________________________________________________________________________________________________
Bvp (Lambda)                    (None, 300, 24)      0           XdotRK[0][0]                     
                                                                 YdotRV[0][0]                     
__________________________________________________________________________________________________
convOneParticle (Conv1D)        (None, 3540, 60)     2220        Bpp[0][0]                        
__________________________________________________________________________________________________
convOneSV (Conv1D)              (None, 300, 60)      1500        Bvp[0][0]                        
__________________________________________________________________________________________________
convTwoParticle (Conv1D)        (None, 3540, 30)     1830        convOneParticle[0][0]            
__________________________________________________________________________________________________
convTwoSV (Conv1D)              (None, 300, 30)      1830        convOneSV[0][0]                  
__________________________________________________________________________________________________
convThreeParticle (Conv1D)      (None, 3540, 20)     620         convTwoParticle[0][0]            
__________________________________________________________________________________________________
convThreeSV (Conv1D)            (None, 300, 20)      620         convTwoSV[0][0]                  
__________________________________________________________________________________________________
Epp (BatchNormalization)        (None, 3540, 20)     80          convThreeParticle[0][0]          
__________________________________________________________________________________________________
Evp (BatchNormalization)        (None, 300, 20)      80          convThreeSV[0][0]                
__________________________________________________________________________________________________
EppBar (Lambda)                 (None, 60, 20)       0           Epp[0][0]                        
__________________________________________________________________________________________________
EvpBar (Lambda)                 (None, 60, 20)       0           Evp[0][0]                        
__________________________________________________________________________________________________
C (Lambda)                      (None, 60, 58)       0           inputParticle[0][0]              
                                                                 EppBar[0][0]                     
                                                                 EvpBar[0][0]                     
__________________________________________________________________________________________________
convPredictOne (Conv1D)         (None, 60, 60)       3540        C[0][0]                          
__________________________________________________________________________________________________
convPredictTwo (Conv1D)         (None, 60, 30)       1830        convPredictOne[0][0]             
__________________________________________________________________________________________________
O (Conv1D)                      (None, 60, 24)       744         convPredictTwo[0][0]             
__________________________________________________________________________________________________
OBar (Lambda)                   (None, 24)           0           O[0][0]                          
__________________________________________________________________________________________________
denseEndOne (Dense)             (None, 50)           1250        OBar[0][0]                       
__________________________________________________________________________________________________
normEndOne (BatchNormalization) (None, 50)           200         denseEndOne[0][0]                
__________________________________________________________________________________________________
denseEndTwo (Dense)             (None, 20)           1020        normEndOne[0][0]                 
__________________________________________________________________________________________________
denseEndThree (Dense)           (None, 10)           210         denseEndTwo[0][0]                
__________________________________________________________________________________________________
denseEndFour (Dense)            (None, 5)            55          denseEndThree[0][0]              
__________________________________________________________________________________________________
output (Dense)                  (None, 2)            12          denseEndFour[0][0]               
==================================================================================================
Total params: 17,737
Trainable params: 17,509
Non-trainable params: 228
__________________________________________________________________________________________________

python -m tf2onnx.convert --saved-model saved-model --output model.onnx
  constant_folding: Graph size after: 318 nodes (-125), 417 edges (-159), time = 17.982ms.
  function_optimizer: function_optimizer did nothing. time = 0.218ms.
  constant_folding: Graph size after: 318 nodes (0), 417 edges (0), time = 6.774ms.
  function_optimizer: function_optimizer did nothing. time = 0.169ms.

2021-05-25 13:26:13,213 - INFO - Using tensorflow=2.4.1, onnx=1.8.0, tf2onnx=1.8.4/cd55bf
2021-05-25 13:26:13,213 - INFO - Using opset <onnx, 9>
2021-05-25 13:26:13,235 - INFO - Computed 0 values for constant folding
2021-05-25 13:26:13,433 - INFO - Optimizing ONNX model
2021-05-25 13:26:14,032 - INFO - After optimization: Cast -3 (18->15), Concat -2 (15->13), Const -68 (117->49), Gather -2 (12->10), Identity -12 (12->0), ReduceProd -2 (12->10), Reshape -1 (12->11), Shape -1 (6->5), Transpose -1 (30->29), Unsqueeze -2 (21->19)
2021-05-25 13:26:14,043 - INFO - 
2021-05-25 13:26:14,043 - INFO - Successfully converted TensorFlow model saved-model to ONNX
2021-05-25 13:26:14,043 - INFO - Model inputs: ['inputparticle:0', 'inputsv:0']
2021-05-25 13:26:14,043 - INFO - Model outputs: ['output']
2021-05-25 13:26:14,043 - INFO - ONNX model is saved at model.onnx

Versions:

• tf2onnx: 1.8.4
• tf: 2.4.1
• onnxruntime: 1.7.0
• keras: 2.4.0
• python: 3.7.10

[TomWildenhain-Microsoft]

Yes please share a zip with the saved model and the onnx model. Normally the model produces correct results if it converts without error but we do have some some transformations that are technically incorrect for NaN values. Did you try any inputs other than np.zeros? You might want to try it on actual test data. I'm wondering if some normalization of the input is creating a division by 0 when all the inputs are 0.

[guschmue]

I assume this was resolved.

[juliakorovsky]

For @jeffkrupa and anybody who will encounter this: try to downgrade tf2onnx. I had the same issue with my model and downgrading to 1.7.2 worked for me.

[TomWildenhain-Microsoft]

Hey @juliakorovsky can you upload a saved model and onnx file so we can reproduce this? I'm thinking optimization of Select/Where ops may be coming up here. Also double-check using the latest tf2onnx. We did a fix for a somewhat similar issue recently.

pip uninstall tf2onnx
pip install git+https://github.com/onnx/tensorflow-onnx

[juliakorovsky]

@TomWildenhain-Microsoft No, unfortunately I can't share the model. But I can create the list of all the ops used in the model, do you want me to do that? I checked with latest version hour ago, it does not work. After version 1.7.2 it doesn't work (even if installing directly from Git).

[TomWildenhain-Microsoft]

Can you generate and share a similar model with the same error? (you can probably skip the training step)

[juliakorovsky]

@TomWildenhain-Microsoft Actually I have no idea what could cause this, so I don't know what I should simulate. I know it's not much help but at least all the ops from my Tensorflow Graph:
'LoopCond', 'Sigmoid', 'TensorArrayV3', 'StopGradient', 'TensorArrayReadV3', 'TensorArrayGatherV3', 'Softmax', 'Max', 'PlaceholderWithDefault', 'GatherV2', 'Sum', 'TensorArrayWriteV3', 'Select', 'Assert', 'MatMul', 'ExpandDims', 'All', 'LessEqual', 'Tanh', 'Const', 'GreaterEqual', 'Enter', 'Add', 'TensorArrayScatterV3', 'Maximum', 'LogicalAnd', 'Switch', 'Shape', 'Split', 'NextIteration', 'Identity', 'Squeeze', 'GatherNd', 'Tile', 'ReverseSequence', 'Range', 'Fill', 'Mul', 'Minimum', 'Exit', 'Cast', 'Relu', 'Conv2D', 'Reshape', 'Prod', 'Placeholder', 'Rsqrt', 'BiasAdd', 'StridedSlice', 'Less', 'LogicalOr', 'Pack', 'Transpose', 'Sub', 'Equal', 'LogicalNot', 'BatchMatMul', 'Greater', 'TensorArraySizeV3', 'ConcatV2', 'Merge'.

I hope someone who could share their model will help. Also I don't know if this is important, because I don't know if you use tensorflow fold_constants function in this repo in any way, but in latest versions of TF fold_constants break the graph of my model. Maybe these issues are connected, maybe not, but I thougth I should mention it just in case.

[TomWildenhain-Microsoft]

Interesting. That op list might help. Are you able to get an op list of the working and failing onnx models? I suspect you will find "Where" in the working model but not in the failing model. Also can you send a picture of the graph where the Select op occurs using Netron?

[juliakorovsky]

@TomWildenhain-Microsoft I know how to extract ops in Tensorflow, how should I do it with Onnx (I'm using Python and onnx and onnxruntime libraries)?

[TomWildenhain-Microsoft]

Try this:

from collections import Counter
import onnx
model = onnx.load("path")
ops = Counter(node.op_type for node in model.graph.node)

[juliakorovsky]

@TomWildenhain-Microsoft Working model:

'Transpose': 23, 'Add': 17, 'Cast': 16, 'Unsqueeze': 13, 'Squeeze': 10, 'Conv': 8, 'Mul': 8, 'Concat': 8, 'Shape': 5, 'Tanh': 5, 'Reshape': 5, 'Relu': 4, 'Slice': 4, 'Gather': 3, 'Scan': 2, 'Less': 2, 'ReverseSequence': 2, 'Gemm': 2, 'ReduceProd': 2, 'Sub': 1, 'ReduceMin': 1, 'Greater': 1, 'Not': 1, 'And': 1, 'GRU': 1, 'GatherND': 1, 'Tile': 1, 'Range': 1, 'MatMul': 1, 'Expand': 1, 'Loop': 1

Total 31 different ops

Non-working model:

'Add': 17, 'Cast': 13, 'Unsqueeze': 11, 'Squeeze': 10, 'Reshape': 10, 'Conv': 8, 'Mul': 8, 'Transpose': 6, 'Concat': 6, 'Tanh': 5, 'Relu': 4, 'Shape': 4, 'Slice': 4, 'Gather': 2, 'Less': 2, 'Scan': 2, 'ReverseSequence': 2, 'Gemm': 2, 'Sub': 1, 'ReduceMin': 1, 'Greater': 1, 'Not': 1, 'And': 1, 'GRU': 1, 'GatherND': 1, 'Tile': 1, 'Range': 1, 'MatMul': 1, 'Expand': 1, 'Loop': 1

Total 30 different ops.

[juliakorovsky]

@TomWildenhain-Microsoft Also forgot to mention I use opset 11 and graphdef format (I got successful conversion with 1.7.2 and opsets 11 and 12).