How to solve this memory error when using to_dataframe?

[markusmat97]

i am having the following error when using the to_dataframe:
code : mdf_data = Data.to_dataframe(use_interpolation=False)
errror:
Traceback (most recent call last):
File "C:\Program Files (x86)\JetBrains\PyCharm Community Edition 2021.3.2\plugins\python-ce\helpers\pydev\pydevd.py", line 1483, in _exec
pydev_imports.execfile(file, globals, locals) # execute the script
File "C:\Program Files (x86)\JetBrains\PyCharm Community Edition 2021.3.2\plugins\python-ce\helpers\pydev_pydev_imps_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "C:/Users/xxxxx/PycharmProjects/pythonProject1/Seq_3 final.py", line 175, in
df_init = initialization(Data)
File "C:/Users/xxxxx/PycharmProjects/pythonProject1/Seq_3 final.py", line 17, in initialization
mdf_data = Data.to_dataframe(use_interpolation=False)
File "C:\Users\xxxxx\PycharmProjects\pythonProject1\venv\lib\site-packages\asammdf\mdf.py", line 4265, in to_dataframe
df = pd.DataFrame(nonstrings, index=master)
File "C:\Users\xxxxxx\PycharmProjects\pythonProject1\venv\lib\site-packages\pandas\core\frame.py", line 636, in init
mgr = dict_to_mgr(data, index, columns, dtype=dtype, copy=copy, typ=manager)
File "C:\Users\xxxx\PycharmProjects\pythonProject1\venv\lib\site-packages\pandas\core\internals\construction.py", line 502, in dict_to_mgr
return arrays_to_mgr(arrays, columns, index, dtype=dtype, typ=typ, consolidate=copy)
File "C:\Users\xxxxx\PycharmProjects\pythonProject1\venv\lib\site-packages\pandas\core\internals\construction.py", line 156, in arrays_to_mgr
return create_block_manager_from_column_arrays(
File "C:\Users\xxxxx\PycharmProjects\pythonProject1\venv\lib\site-packages\pandas\core\internals\managers.py", line 1959, in create_block_manager_from_column_arrays
mgr._consolidate_inplace()
File "C:\Users\xxxxx\PycharmProjects\pythonProject1\venv\lib\site-packages\pandas\core\internals\managers.py", line 1685, in _consolidate_inplace
self.blocks = tuple(_consolidate(self.blocks))
File "C:\Users\xxxxx\PycharmProjects\pythonProject1\venv\lib\site-packages\pandas\core\internals\managers.py", line 2084, in _consolidate
merged_blocks = _merge_blocks(
File "C:\Users\xxxxx\PycharmProjects\pythonProject1\venv\lib\site-packages\pandas\core\internals\managers.py", line 2111, in _merge_blocks
new_values = np.vstack([b.values for b in blocks]) # type: ignore[misc]
File "<array_function internals>", line 180, in vstack
File "C:\Users\xxxxx\PycharmProjects\pythonProject1\venv\lib\site-packages\numpy\core\shape_base.py", line 282, in vstack
return _nx.concatenate(arrs, 0)
File "<array_function internals>", line 180, in concatenate
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 11.6 GiB for an array with shape (567, 2747749) and data type float64
Breakpoint reached: Any exception
python-BaseException

[danielhrisca]

You need to use the raster argument (see https://asammdf.readthedocs.io/en/latest/api.html#asammdf.mdf.MDF.to_dataframe)

[markusmat97]

is it possible to give two channel name in raster?

[danielhrisca]

You can give an np array that you compute on your own

[markusmat97]

You can give an np array that you compute on your own

for example,if i have two channels : 'Cycle counter' and 'Log_flag'.How can i give this two channels as raster?Would you show me an example ?also i didn't understand the raster?What it does?

[danielhrisca]

import numpy as np

cntr, log_flag = mdf.select([ 'Cycle counter', 'Log_flag'])
my_raster = np.union1d(cntr.timestamps, log_flag.timestamps)

mdf.to_dataframe(raster=my_raster)

raster is probably not the best word, but it should be the timebase used for resampling