Structured data learning with TabTransformer Tutorial Working on Colab, broken on Local #1946
Description
Issue Type
Bug
Source
source
Keras Version
Keras 3.4.1
Custom Code
No
OS Platform and Distribution
macOS Ventura 13.6.3
Python version
3.10.12
GPU model and memory
Apple M2 Pro
Current Behavior?
After running the cell with run_experiment() I expect the model to train and continue with the tutorial. Instead it errors out (log below).
I would expect running this locally vs colab to have the same working result. I have tried to ensure that my local has the same Python version, Keras version, numpy version, etc as the Colab notebook to reduce the amount of possible culprits for the error.
Based on some debugging, I think that the model is not correctly mapping the named inputs to their named input layers. ie: it appears that an input like native_country is being passed to the relationship input ( 45 is not in [0, 6) ).
Standalone code to reproduce the issue or tutorial link
Tutorial link - https://keras.io/examples/structured_data/tabtransformer/
Colab link (working) - https://colab.research.google.com/github/keras-team/keras-io/blob/master/examples/structured_data/ipynb/tabtransformer.ipynb
Github Link (downloaded to local, broken) - https://github.com/keras-team/keras-io/blob/master/examples/structured_data/ipynb/tabtransformer.ipynbRelevant log output
{
"name": "InvalidArgumentError",
"message": "Graph execution error:
Detected at node functional_7_1/embedding_12_1/GatherV2 defined at (most recent call last):
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/runpy.py\", line 196, in _run_module_as_main
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/runpy.py\", line 86, in _run_code
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/ipykernel_launcher.py\", line 18, in <module>
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/traitlets/config/application.py\", line 1075, in launch_instance
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/ipykernel/kernelapp.py\", line 739, in start
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/tornado/platform/asyncio.py\", line 205, in start
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/asyncio/base_events.py\", line 603, in run_forever
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/asyncio/base_events.py\", line 1909, in _run_once
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/asyncio/events.py\", line 80, in _run
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/ipykernel/kernelbase.py\", line 545, in dispatch_queue
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/ipykernel/kernelbase.py\", line 534, in process_one
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/ipykernel/kernelbase.py\", line 437, in dispatch_shell
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/ipykernel/ipkernel.py\", line 362, in execute_request
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/ipykernel/kernelbase.py\", line 778, in execute_request
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/ipykernel/ipkernel.py\", line 449, in do_execute
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/ipykernel/zmqshell.py\", line 549, in run_cell
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3075, in run_cell
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3130, in _run_cell
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/IPython/core/async_helpers.py\", line 128, in _pseudo_sync_runner
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3334, in run_cell_async
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3517, in run_ast_nodes
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3577, in run_code
File \"/var/folders/pt/yxcmy21n099708ptn0358ntw0000gn/T/ipykernel_20599/232252535.py\", line 1, in <module>
File \"/var/folders/pt/yxcmy21n099708ptn0358ntw0000gn/T/ipykernel_20599/1288716379.py\", line 24, in run_experiment
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py\", line 117, in error_handler
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py\", line 318, in fit
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py\", line 121, in one_step_on_iterator
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py\", line 108, in one_step_on_data
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py\", line 51, in train_step
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py\", line 117, in error_handler
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/layers/layer.py\", line 882, in __call__
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py\", line 117, in error_handler
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/ops/operation.py\", line 46, in __call__
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py\", line 156, in error_handler
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/models/functional.py\", line 175, in call
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/ops/function.py\", line 171, in _run_through_graph
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/models/functional.py\", line 556, in call
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py\", line 117, in error_handler
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/layers/layer.py\", line 882, in __call__
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py\", line 117, in error_handler
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/ops/operation.py\", line 46, in __call__
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py\", line 156, in error_handler
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/layers/core/embedding.py\", line 140, in call
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/ops/numpy.py\", line 4875, in take
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/backend/tensorflow/numpy.py\", line 1951, in take
indices[0] = 45 is not in [0, 6)
\t [[{{node functional_7_1/embedding_12_1/GatherV2}}]] [Op:__inference_one_step_on_iterator_13726]",
"stack": "---------------------------------------------------------------------------
InvalidArgumentError Traceback (most recent call last)
Cell In[15], line 1
----> 1 history = run_experiment(
2 model=baseline_model,
3 train_data_file=train_data_file,
4 test_data_file=test_data_file,
5 num_epochs=NUM_EPOCHS,
6 learning_rate=LEARNING_RATE,
7 weight_decay=WEIGHT_DECAY,
8 batch_size=BATCH_SIZE,
9 )
Cell In[8], line 24, in run_experiment(model, train_data_file, test_data_file, num_epochs, learning_rate, weight_decay, batch_size)
21 validation_dataset = get_dataset_from_csv(test_data_file, batch_size)
23 print(\"Start training the model...\")
---> 24 history = model.fit(
25 train_dataset, epochs=num_epochs, validation_data=validation_dataset
26 )
27 print(\"Model training finished\")
29 _, accuracy = model.evaluate(validation_dataset, verbose=0)
File ~/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py:122, in filter_traceback.<locals>.error_handler(*args, **kwargs)
119 filtered_tb = _process_traceback_frames(e.__traceback__)
120 # To get the full stack trace, call:
121 # `keras.config.disable_traceback_filtering()`
--> 122 raise e.with_traceback(filtered_tb) from None
123 finally:
124 del filtered_tb
File ~/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/tensorflow/python/eager/execute.py:53, in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
51 try:
52 ctx.ensure_initialized()
---> 53 tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
54 inputs, attrs, num_outputs)
55 except core._NotOkStatusException as e:
56 if name is not None:
InvalidArgumentError: Graph execution error:
Detected at node functional_7_1/embedding_12_1/GatherV2 defined at (most recent call last):
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/runpy.py\", line 196, in _run_module_as_main
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/runpy.py\", line 86, in _run_code
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/ipykernel_launcher.py\", line 18, in <module>
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/traitlets/config/application.py\", line 1075, in launch_instance
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/ipykernel/kernelapp.py\", line 739, in start
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/tornado/platform/asyncio.py\", line 205, in start
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/asyncio/base_events.py\", line 603, in run_forever
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/asyncio/base_events.py\", line 1909, in _run_once
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/asyncio/events.py\", line 80, in _run
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/ipykernel/kernelbase.py\", line 545, in dispatch_queue
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/ipykernel/kernelbase.py\", line 534, in process_one
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/ipykernel/kernelbase.py\", line 437, in dispatch_shell
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/ipykernel/ipkernel.py\", line 362, in execute_request
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/ipykernel/kernelbase.py\", line 778, in execute_request
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/ipykernel/ipkernel.py\", line 449, in do_execute
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/ipykernel/zmqshell.py\", line 549, in run_cell
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3075, in run_cell
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3130, in _run_cell
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/IPython/core/async_helpers.py\", line 128, in _pseudo_sync_runner
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3334, in run_cell_async
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3517, in run_ast_nodes
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3577, in run_code
File \"/var/folders/pt/yxcmy21n099708ptn0358ntw0000gn/T/ipykernel_20599/232252535.py\", line 1, in <module>
File \"/var/folders/pt/yxcmy21n099708ptn0358ntw0000gn/T/ipykernel_20599/1288716379.py\", line 24, in run_experiment
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py\", line 117, in error_handler
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py\", line 318, in fit
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py\", line 121, in one_step_on_iterator
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py\", line 108, in one_step_on_data
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py\", line 51, in train_step
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py\", line 117, in error_handler
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/layers/layer.py\", line 882, in __call__
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py\", line 117, in error_handler
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/ops/operation.py\", line 46, in __call__
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py\", line 156, in error_handler
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/models/functional.py\", line 175, in call
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/ops/function.py\", line 171, in _run_through_graph
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/models/functional.py\", line 556, in call
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py\", line 117, in error_handler
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/layers/layer.py\", line 882, in __call__
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py\", line 117, in error_handler
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/ops/operation.py\", line 46, in __call__
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py\", line 156, in error_handler
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/layers/core/embedding.py\", line 140, in call
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/ops/numpy.py\", line 4875, in take
File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/backend/tensorflow/numpy.py\", line 1951, in take
indices[0] = 45 is not in [0, 6)
\t [[{{node functional_7_1/embedding_12_1/GatherV2}}]] [Op:__inference_one_step_on_iterator_13726]"
}