AttributeError: '_UserObject' object has no attribute 'add_slot'

[snowuyl]

Describe the bug
Can't convert TensorFlow MNIST saved_model.pb to onnx model.

Urgency
none.

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 18.04*): Ubuntu 22.04
• TensorFlow Version: 2.18.0
• Python version: 3.10.12
• ONNX version (if applicable, e.g. 1.11*): 1.17.0
• ONNXRuntime version (if applicable, e.g. 1.11*): 1.20.0

To Reproduce
pip3 install git+https://github.com/onnx/tensorflow-onnx
python3 -m tf2onnx.convert --saved-model ./ --output model.onnx

Additional context
python3 -m tf2onnx.convert --saved-model ./ --output model.onnx
2024-11-09 15:41:08.659796: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1731138068.674427 22280 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1731138068.678814 22280 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-11-09 15:41:08.694085: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/usr/lib/python3.10/runpy.py:126: RuntimeWarning: 'tf2onnx.convert' found in sys.modules after import of package 'tf2onnx', but prior to execution of 'tf2onnx.convert'; this may result in unpredictable behaviour
warn(RuntimeWarning(msg))
W0000 00:00:1731138071.688361 22280 gpu_device.cc:2344] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2024-11-09 15:41:11,690 - WARNING - '--tag' not specified for saved_model. Using --tag serve
Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/snowuyl/samba/workspace_AI/tensorflow-onnx/tf2onnx/convert.py", line 714, in
main()
File "/home/snowuyl/samba/workspace_AI/tensorflow-onnx/tf2onnx/convert.py", line 242, in main
graph_def, inputs, outputs, initialized_tables, tensors_to_rename = tf_loader.from_saved_model(
File "/home/snowuyl/samba/workspace_AI/tensorflow-onnx/tf2onnx/tf_loader.py", line 636, in from_saved_model
_from_saved_model_v2(model_path, input_names, output_names,
File "/home/snowuyl/samba/workspace_AI/tensorflow-onnx/tf2onnx/tf_loader.py", line 570, in _from_saved_model_v2
imported = tf.saved_model.load(model_path, tags=tag) # pylint: disable=no-value-for-parameter
File "/home/snowuyl/.local/lib/python3.10/site-packages/tensorflow/python/saved_model/load.py", line 912, in load
result = load_partial(export_dir, None, tags, options)["root"]
File "/home/snowuyl/.local/lib/python3.10/site-packages/tensorflow/python/saved_model/load.py", line 1042, in load_partial
loader = Loader(object_graph_proto, saved_model_proto, export_dir,
File "/home/snowuyl/.local/lib/python3.10/site-packages/tensorflow/python/saved_model/load.py", line 223, in init
self._load_all()
File "/home/snowuyl/.local/lib/python3.10/site-packages/tensorflow/python/saved_model/load.py", line 320, in _load_all
self._load_nodes()
File "/home/snowuyl/.local/lib/python3.10/site-packages/tensorflow/python/saved_model/load.py", line 529, in _load_nodes
slot_variable = optimizer_object.add_slot(
AttributeError: '_UserObject' object has no attribute 'add_slot'

[snowuyl]

saved_model.zip

[Leogendra]

Did you managed to resolve it?

[cameronbronstein]

I also ran into this issue under certain conditions. I was able to convert an old tensorflow model that was trained on TF 2.4.1 to onnx by installing tf2onnx and running the following:

$ python3 --version
Python 3.8.10
$ python3 -c "import tf2onnx; import tensorflow; print(tf2onnx.__version__); print(tensorflow.__version__)"
WARNING:root:Limited tf.compat.v2.summary API due to missing TensorBoard installation.
WARNING:root:Limited tf.compat.v2.summary API due to missing TensorBoard installation.
WARNING:root:Limited tf.compat.v2.summary API due to missing TensorBoard installation.
WARNING:root:Limited tf.summary API due to missing TensorBoard installation.
1.16.1
2.4.1
% python3 -m tf2onnx.convert --saved-model saved_model --output model.onnx
WARNING:root:Limited tf.compat.v2.summary API due to missing TensorBoard installation.
WARNING:root:Limited tf.compat.v2.summary API due to missing TensorBoard installation.
WARNING:root:Limited tf.compat.v2.summary API due to missing TensorBoard installation.
WARNING:root:Limited tf.summary API due to missing TensorBoard installation.
/usr/lib/python3.8/runpy.py:127: RuntimeWarning: 'tf2onnx.convert' found in sys.modules after import of package 'tf2onnx', but prior to execution of 'tf2onnx.convert'; this may result in unpredictable behaviour
  warn(RuntimeWarning(msg))
2025-03-22 00:43:59,928 - WARNING - '--tag' not specified for saved_model. Using --tag serve
2025-03-22 00:44:01,355 - INFO - Signatures found in model: [serving_default].
2025-03-22 00:44:01,355 - WARNING - '--signature_def' not specified, using first signature: serving_default
2025-03-22 00:44:01,356 - INFO - Output names: ['tf.identity_1']
2025-03-22 00:44:01,938 - INFO - Using tensorflow=2.4.1, onnx=1.15.0, tf2onnx=1.16.1/15c810
2025-03-22 00:44:01,938 - INFO - Using opset <onnx, 15>
2025-03-22 00:44:02,389 - INFO - Computed 0 values for constant folding
2025-03-22 00:44:02,755 - INFO - Optimizing ONNX model
2025-03-22 00:44:03,742 - INFO - After optimization: Cast -1 (5->4), Const -59 (135->76), Gather +2 (0->2), Identity -2 (2->0), Reshape +1 (1->2), Squeeze -3 (4->1), Transpose -48 (50->2), Unsqueeze -4 (4->0)
2025-03-22 00:44:03,788 - INFO - 
2025-03-22 00:44:03,788 - INFO - Successfully converted TensorFlow model saved_model to ONNX
2025-03-22 00:44:03,788 - INFO - Model inputs: ['input_1']
2025-03-22 00:44:03,788 - INFO - Model outputs: ['tf.identity_1']
2025-03-22 00:44:03,788 - INFO - ONNX model is saved at model.onnx

However, I am not able to do this if my environment contains a newer version of tensorflow.

% python3 --version
Python 3.12.0
% python3 -c "import tf2onnx; import tensorflow; print(tf2onnx.__version__); print(tensorflow.__version__)" 
1.16.1
2.16.1
% python3 -m tf2onnx.convert --saved-model saved_model --output model.onnx --opset 15
<frozen runpy>:128: RuntimeWarning: 'tf2onnx.convert' found in sys.modules after import of package 'tf2onnx', but prior to execution of 'tf2onnx.convert'; this may result in unpredictable behaviour
2025-03-24 15:50:24,599 - WARNING - ***IMPORTANT*** Installed protobuf is not cpp accelerated. Conversion will be extremely slow. See https://github.com/onnx/tensorflow-onnx/issues/1557
2025-03-24 15:50:24,605 - WARNING - '--tag' not specified for saved_model. Using --tag serve
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/cameron/Library/Caches/pypoetry/virtualenvs/udm2-scripts-ws4UZpvZ-py3.12/lib/python3.12/site-packages/tf2onnx/convert.py", line 714, in <module>
    main()
  File "/Users/cameron/Library/Caches/pypoetry/virtualenvs/udm2-scripts-ws4UZpvZ-py3.12/lib/python3.12/site-packages/tf2onnx/convert.py", line 242, in main
    graph_def, inputs, outputs, initialized_tables, tensors_to_rename = tf_loader.from_saved_model(
                                                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cameron/Library/Caches/pypoetry/virtualenvs/udm2-scripts-ws4UZpvZ-py3.12/lib/python3.12/site-packages/tf2onnx/tf_loader.py", line 636, in from_saved_model
    _from_saved_model_v2(model_path, input_names, output_names,
  File "/Users/cameron/Library/Caches/pypoetry/virtualenvs/udm2-scripts-ws4UZpvZ-py3.12/lib/python3.12/site-packages/tf2onnx/tf_loader.py", line 570, in _from_saved_model_v2
    imported = tf.saved_model.load(model_path, tags=tag)  # pylint: disable=no-value-for-parameter
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cameron/Library/Caches/pypoetry/virtualenvs/udm2-scripts-ws4UZpvZ-py3.12/lib/python3.12/site-packages/tensorflow/python/saved_model/load.py", line 912, in load
    result = load_partial(export_dir, None, tags, options)["root"]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cameron/Library/Caches/pypoetry/virtualenvs/udm2-scripts-ws4UZpvZ-py3.12/lib/python3.12/site-packages/tensorflow/python/saved_model/load.py", line 1042, in load_partial
    loader = Loader(object_graph_proto, saved_model_proto, export_dir,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cameron/Library/Caches/pypoetry/virtualenvs/udm2-scripts-ws4UZpvZ-py3.12/lib/python3.12/site-packages/tensorflow/python/saved_model/load.py", line 223, in __init__
    self._load_all()
  File "/Users/cameron/Library/Caches/pypoetry/virtualenvs/udm2-scripts-ws4UZpvZ-py3.12/lib/python3.12/site-packages/tensorflow/python/saved_model/load.py", line 320, in _load_all
    self._load_nodes()
  File "/Users/cameron/Library/Caches/pypoetry/virtualenvs/udm2-scripts-ws4UZpvZ-py3.12/lib/python3.12/site-packages/tensorflow/python/saved_model/load.py", line 529, in _load_nodes
    slot_variable = optimizer_object.add_slot(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: '_UserObject' object has no attribute 'add_slot'

My understanding is that my older model, or perhaps my custom objects in the model are just not compatible with newer versions of tensorflow. If I want to load this model with a modern tensorflow/keras (i.e. tf 2.16.1), I have to ensure use of legacy Keras 2.

import os 

os.environ["TF_USE_LEGACY_KERAS"] = "1"

import tensorflow as tf
from tensorflow import keras

...
custom_objects = {}
model = keras.models.load_model(saved_model_path, compile=False, custom_objects=custom_objects)

It'd be great to have support for these older models or to have a recommendation for what to do. My next attempt will be to first convert the model to h5 and see if I can convert it to onnx after...

[cameronbronstein]

It turns out installing tf_keras into my environment fixes the issue.

I was lucky to discover a solution by accident. As mentioned in my response above, using the os.environ["TF_USE_LEGACY_KERAS"] = "1" to have tensorflow load Keras 2 solved a similar issuew in another environment.

My understanding is that tf_keras works as a shim for compatibility between newer versions of tensorflow that by default load Keras 3.

So I installed tf-keras to refresh my memory with some of the behavior of model loading/saving with different formats. Then much to my surprise, the conversion was working again. It apperas that tf_keras changes the behavior of tf2onnx in these situations! Hopefully this can help solve other people's issues!