Skip to content

Pickling a histogram creates an extra copy of the array in memory #828

Open
@bendavid

Description

@bendavid

Pickling a large boost histogram results in a second copy being created in memory. Via the use of the PickleBuffer mechanism it should be possible to avoid this.

e.g.

import boost_histogram as bh

axis = bh.axis.Regular(1024*1024*1024, 0., 1)
htest = bh.Histogram(axis)

uses 8GB of memory as expected

import boost_histogram as bh
import pickle

axis = bh.axis.Regular(1024*1024*1024, 0., 1)
htest = bh.Histogram(axis)

with open("test.pkl", "wb") as f:
    pickle.dump(htest, f, protocol = pickle.HIGHEST_PROTOCOL)

uses 16GB

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions