Unable to retrieve variable & freezed console

[O-Memis]

I am using Spyder (version 6) because of easy access to the variable explorer and observing my signal / image data.

When I import my EEG dataset about 5-6 GB into dataframes or ndarrays, I cannot see the numbers inside for a quick visual inspection.

More than that, the console & IDE start freezing after some point. Showing no outputs to interrupts, code execution, and does not apply variable deletion. It's a serious issue that makes me restart my pipeline again and again.

It's definitely not about my device; I have 64 GB RAM with an i7 CPU.
I hope I have the correct settings.

Error message down here:

Traceback (most recent call last):
File "C:\ProgramData\spyder-6\envs\spyder-runtime\Lib\site-packages\spyder\plugins\ipythonconsole\widgets\namespacebrowser.py", line 150, in get_value
).get_value(name, encoded=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ProgramData\spyder-6\envs\spyder-runtime\Lib\site-packages\spyder_kernels\comms\commbase.py", line 614, in call
return self._comms_wrapper._get_call_return_value(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ProgramData\spyder-6\envs\spyder-runtime\Lib\site-packages\spyder\plugins\ipythonconsole\comms\kernelcomm.py", line 171, in _get_call_return_value
return super()._get_call_return_value(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ProgramData\spyder-6\envs\spyder-runtime\Lib\site-packages\spyder_kernels\comms\commbase.py", line 467, in _get_call_return_value
self._wait_reply(comm_id, call_id, call_name, timeout)
File "C:\ProgramData\spyder-6\envs\spyder-runtime\Lib\site-packages\spyder\plugins\ipythonconsole\comms\kernelcomm.py", line 195, in _wait_reply
self._wait(got_reply, self._sig_got_reply, timeout_msg, timeout)
File "C:\ProgramData\spyder-6\envs\spyder-runtime\Lib\site-packages\spyder\plugins\ipythonconsole\comms\kernelcomm.py", line 232, in _wait
raise TimeoutError(timeout_msg)
TimeoutError: Timeout while waiting for {'ef06cab6ec0d46549bed6e5c9eb700db': (True, None), '132f4996b625464a8ac3800162d1dcdc': (True, None)}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\ProgramData\spyder-6\envs\spyder-runtime\Lib\site-packages\spyder\plugins\variableexplorer\widgets\collectionsdelegate.py", line 196, in createEditor
value = self.get_value(index)
^^^^^^^^^^^^^^^^^^^^^
File "C:\ProgramData\spyder-6\envs\spyder-runtime\Lib\site-packages\spyder\widgets\collectionseditor.py", line 2139, in get_value
return self.parent().get_value(name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ProgramData\spyder-6\envs\spyder-runtime\Lib\site-packages\spyder\widgets\collectionseditor.py", line 2232, in get_value
value = self.shellwidget.get_value(name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ProgramData\spyder-6\envs\spyder-runtime\Lib\site-packages\spyder\plugins\ipythonconsole\widgets\namespacebrowser.py", line 155, in get_value
raise ValueError(msg % reason_big)
ValueError:
The variable is too big to be retrieved

Note: If you consider this to be a valid error that needs to be fixed by the Spyder team, please report it on Github.

[O-Memis]

as a clear example; it's not responding / executing the code (I cannot see the new variable there)

[ccordoba12]

Hey @O-Memis, thanks for reporting. You said:

When I import my EEG dataset about 5-6 GB into dataframes or ndarrays, I cannot see the numbers inside for a quick visual inspection.

Unfortunately, those dataframes and arrays are too big for Spyder to handle them, sorry. The error you posted above mentions it at the end: The variable is too big to be retrieved.

More than that, the console & IDE start freezing after some point. Showing no outputs to interrupts, code execution, and does not apply variable deletion. It's a serious issue that makes me restart my pipeline again and again.

Sorry about that. It could happen after you tried to view your big datasets, so please don't do that and check again if the console freezes then too.

If that's the case, then please report back.

as a clear example; it's not responding / executing the code (I cannot see the new variable there)

That problem should be fixed in our latest version (6.1.4), so please update to it if you're using it and try again.

[O-Memis]

Thank you for the resonse. I am using the latest version. No freezing when I don't click to observe the variable, I have already noticed that. But I wonder why this happens. IDE also fails to observe for any other variable > 1 GB

[ccordoba12]

I am using the latest version.

Sorry but I just checked and the problem you reported here

is not happening in 6.1.4 (variables are correctly sorted alphabetically in the Variable Explorer, but they weren't before). So please make sure you really have that version installed (Anaconda is always one or more versions behind).

But I wonder why this happens. IDE also fails to observe for any other variable > 1 GB

I don't think that's the problem. Most probably Spyder is not cleaning the memory used to retrieve the variable, which makes entering new commands sluggish. However, that's not an easy fix.