Skip to content

Unable to allocate 8.41 GiB for an array with shape (2258174000,) and data type int32 #86

@BudiZhu

Description

@BudiZhu

I tried to match string with 1M rows dataset against 10M rows dataset. N Block I set 10:2000.
Error occured with message:
File "", line 164, in match_strings\n File "", line 623, in fit\n File "", line 461, in _fit_blockwise_manual\n File "c:\program files\python38\lib\site-packages\topn\awesome_topn.py", line 88, in awesome_hstack_topn\n r = np.concatenate([b.indptr for b in blocks])\n File "<array_function internals>", line 180, in concatenate\nnumpy.core._exceptions._ArrayMemoryError: Unable to allocate 8.41 GiB for an array with shape (2258174000,) and data type int32\n<traceback object at 0x0000028A1BC113C0>.

Any solutions?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions