High memory consumption when reading big files

[rsarili]

Used version: 1.8.6

I have a snappy parquet file which is ~60MB has ~1.000.000 rows and ~300 columns. I use following code snippet to read data.

const reader = await parquet.ParquetReader.openFile('.file.snappy.parquet)
const cursor = reader.getCursor()
while (await cursor.next()) {}

If memory limit is 10GB I got heap allocation error and if it is ~23GB(I put the limit ) it takes ~4 minutes to process.
I tried to set maxSpan and maxLength options too to fix the issue or have a batch processing effect but it didn't work.

I can read the same file in ~30 seconds and ~170MB memory with python pandas using following code snippet

pf = pq.ParquetFile("file.snappy.parquet")

for batch in pf.iter_batches(batch_size=1000):
    df = pa.Table.from_batches([batch]).to_pandas()
    for row in df.itertuples(index=False):
        pass

I am not sure whether this kind of a bug or I am incorrectly using this library.
Could you please share your comments?

[wilwade]

@rsarili Are you able to share the file or a file that produces the same issues.

That said, the processing of some larger compressed parquet files is a bit hard and this library is not well optimized for it.

Might also be worth trying https://github.com/kylebarron/parquet-wasm, although I know it doesn't have all the features, it might be able to process more quickly.

[rsarili]

@wilwade, I am not able share to the file, but it was an AWS Cost Explorer export for monthly costs. I will check the other library options too, thank for you feedback.

[wilwade]

I did some testing with this file: https://aws-public-blockchain.s3.us-east-2.amazonaws.com/index.html#v1.0/eth/logs/date=2025-07-25/

Which is ~800mb snappy compressed. One major difference is that it has fewer columns, although still a good number of rows.

Parquetjs handled it fine. Nothing amazing, but nothing as bad as what you were seeing.

@rsarili if you are in a position to check, can you check the Row Group Size of the file?

You can use parquet-tools meta to export a bunch of metadata including the group sizes. Here's the output for the blockchain file as an example and so others can compare.

As much of that data as you can provide would be helpful in tracking down the issue.

$ parquet meta large-blockchain-dataset.snappy.parquet

File path:  large-blockchain-dataset.snappy.parquet
Created by: parquet-mr version 1.10.1 (build 65f31597b18a0f2718a129fd2d69af0168952c55)
Properties:
                   org.apache.spark.version: 3.1.1
  org.apache.spark.sql.parquet.row.metadata: {"type":"struct","fields":[{"name":"address","type":"string","nullable":true,"metadata":{}},{"name":"topics","type":{"type":"array","elementType":"string","containsNull":true},"nullable":true,"metadata":{}},{"name":"data","type":"string","nullable":true,"metadata":{}},{"name":"transaction_index","type":"long","nullable":true,"metadata":{}},{"name":"log_index","type":"long","nullable":true,"metadata":{}},{"name":"transaction_hash","type":"string","nullable":true,"metadata":{}},{"name":"block_number","type":"long","nullable":true,"metadata":{}},{"name":"block_hash","type":"string","nullable":true,"metadata":{}},{"name":"block_timestamp","type":"timestamp","nullable":true,"metadata":{}},{"name":"date","type":"string","nullable":true,"metadata":{}},{"name":"last_modified","type":"timestamp","nullable":true,"metadata":{}}]}
Schema:
message spark_schema {
  optional binary address (STRING);
  optional group topics (LIST) {
    repeated group list {
      optional binary element (STRING);
    }
  }
  optional binary data (STRING);
  optional int64 transaction_index;
  optional int64 log_index;
  optional binary transaction_hash (STRING);
  optional int64 block_number;
  optional binary block_hash (STRING);
  optional int96 block_timestamp;
  optional binary date (STRING);
  optional int96 last_modified;
}


Row group 0:  count: 710779  188.71 B records  start: 4  total(compressed): 127.915 MB total(uncompressed):286.300 MB
--------------------------------------------------------------------------------
                     type      encodings count     avg size   nulls   min / max
address              BINARY    S _ R     710779    2.99 B     0       "0x000000000000003607fce1a..." / "0xffffffaeff0b96ea8e4f94b..."
topics.list.element  BINARY    S _ R_ F  1917000   26.31 B    3823    "0x00000000000000000000000..." / "0xfffffffffffffffffffffff..."
data                 BINARY    S _ R_ F  710779    37.08 B    0       "0x" / "0xfffffffffffffffffffffff..."
transaction_index    INT64     S _ R     710779    1.26 B     0       "0" / "1008"
log_index            INT64     S _ R     710779    1.59 B     0       "0" / "10249"
transaction_hash     BINARY    S _ R_ F  710779    62.10 B    0       "0x00002a9c7d3ec2f2ba87258..." / "0xfffff62ed4c66cdf03a9905..."
block_number         INT64     S _ R     710779    1.67 B     0       "22992451" / "22999594"
block_hash           BINARY    S _ R     710779    2.25 B     0       "0x000132755aa4d3c7a7f29a9..." / "0xfffd73719cd5869b3f5ebc5..."
block_timestamp      INT96     S _ R     710779    1.69 B     0
date                 BINARY    S _ R     710779    0.00 B     0       "2025-07-25" / "2025-07-25"
last_modified        INT96     S _ R_ F  710779    7.11 B     0

Row group 1:  count: 705620  188.57 B records  start: 134128955  total(compressed): 126.897 MB total(uncompressed):283.841 MB
--------------------------------------------------------------------------------
                     type      encodings count     avg size   nulls   min / max
address              BINARY    S _ R     705620    3.00 B     0       "0x000000000000003607fce1a..." / "0xffffffaeff0b96ea8e4f94b..."
topics.list.element  BINARY    S _ R_ F  1902499   26.30 B    3705    "0x00000000000000000000000..." / "0xfffffffffffffffffffffff..."
data                 BINARY    S _ R_ F  705620    36.95 B    0       "0x" / "0xfffffffffffffffffffffff..."
transaction_index    INT64     S _ R     705620    1.26 B     0       "0" / "1009"
log_index            INT64     S _ R     705620    1.54 B     0       "0" / "10232"
transaction_hash     BINARY    S _ R_ F  705620    62.17 B    0       "0x00000c7be3fdbec2045da0c..." / "0xffff7df9a41f494d417683b..."
block_number         INT64     S _ R     705620    1.67 B     0       "22992451" / "22999594"
block_hash           BINARY    S _ R     705620    2.27 B     0       "0x000132755aa4d3c7a7f29a9..." / "0xfffd73719cd5869b3f5ebc5..."
block_timestamp      INT96     S _ R     705620    1.69 B     0
date                 BINARY    S _ R     705620    0.00 B     0       "2025-07-25" / "2025-07-25"
last_modified        INT96     S _ R_ F  705620    7.11 B     0

Row group 2:  count: 710779  188.64 B records  start: 267190165  total(compressed): 127.873 MB total(uncompressed):286.357 MB
--------------------------------------------------------------------------------
                     type      encodings count     avg size   nulls   min / max
address              BINARY    S _ R     710779    2.99 B     0       "0x000000000000003607fce1a..." / "0xffffffaeff0b96ea8e4f94b..."
topics.list.element  BINARY    S _ R_ F  1917538   26.29 B    3733    "0x00000000000000000000000..." / "0xfffffffffffffffffffffff..."
data                 BINARY    S _ R_ F  710779    36.95 B    0       "0x" / "0xfffffffffffffffffffffff..."
transaction_index    INT64     S _ R     710779    1.26 B     0       "0" / "1072"
log_index            INT64     S _ R     710779    1.56 B     0       "0" / "10244"
transaction_hash     BINARY    S _ R_ F  710779    62.22 B    0       "0x00002a9c7d3ec2f2ba87258..." / "0xffffcca7393dab53fd2c5a6..."
block_number         INT64     S _ R     710779    1.67 B     0       "22992451" / "22999594"
block_hash           BINARY    S _ R     710779    2.26 B     0       "0x000132755aa4d3c7a7f29a9..." / "0xfffd73719cd5869b3f5ebc5..."
block_timestamp      INT96     S _ R     710779    1.69 B     0
date                 BINARY    S _ R     710779    0.00 B     0       "2025-07-25" / "2025-07-25"
last_modified        INT96     S _ R_ F  710779    7.11 B     0

Row group 3:  count: 710779  188.19 B records  start: 401274720  total(compressed): 127.566 MB total(uncompressed):285.563 MB
--------------------------------------------------------------------------------
                     type      encodings count     avg size   nulls   min / max
address              BINARY    S _ R     710779    2.99 B     0       "0x000000000000003607fce1a..." / "0xfffffffff15abf397da76f1..."
topics.list.element  BINARY    S _ R_ F  1916377   26.30 B    3852    "0x00000000000000000000000..." / "0xfffffffffffffffffffffff..."
data                 BINARY    S _ R_ F  710779    36.61 B    0       "0x" / "0xfffffffffffffffffffffff..."
transaction_index    INT64     S _ R     710779    1.26 B     0       "0" / "1008"
log_index            INT64     S _ R     710779    1.58 B     0       "0" / "10247"
transaction_hash     BINARY    S _ R_ F  710779    62.13 B    0       "0x00002a9c7d3ec2f2ba87258..." / "0xfffffcaca6942552a7be338..."
block_number         INT64     S _ R     710779    1.67 B     0       "22992451" / "22999594"
block_hash           BINARY    S _ R     710779    2.25 B     0       "0x000132755aa4d3c7a7f29a9..." / "0xfffd73719cd5869b3f5ebc5..."
block_timestamp      INT96     S _ R     710779    1.69 B     0
date                 BINARY    S _ R     710779    0.00 B     0       "2025-07-25" / "2025-07-25"
last_modified        INT96     S _ R_ F  710779    7.11 B     0

Row group 4:  count: 712376  188.10 B records  start: 535037554  total(compressed): 127.791 MB total(uncompressed):285.873 MB
--------------------------------------------------------------------------------
                     type      encodings count     avg size   nulls   min / max
address              BINARY    S _ R     712376    2.99 B     0       "0x000000000000003607fce1a..." / "0xfffae4a0f4ac251f4705717..."
topics.list.element  BINARY    S _ R_ F  1921846   26.14 B    3805    "0x00000000000000000000000..." / "0xfffffffffffffffffffffff..."
data                 BINARY    S _ R_ F  712376    36.90 B    0       "0x" / "0xfffffffffffffffffffffff..."
transaction_index    INT64     S _ R     712376    1.26 B     0       "0" / "1079"
log_index            INT64     S _ R     712376    1.59 B     0       "0" / "10248"
transaction_hash     BINARY    S _ R_ F  712376    62.12 B    0       "0x00002a9c7d3ec2f2ba87258..." / "0xfffff62ed4c66cdf03a9905..."
block_number         INT64     S _ R     712376    1.67 B     0       "22992451" / "22999594"
block_hash           BINARY    S _ R     712376    2.25 B     0       "0x000132755aa4d3c7a7f29a9..." / "0xfffd73719cd5869b3f5ebc5..."
block_timestamp      INT96     S _ R     712376    1.69 B     0
date                 BINARY    S _ R     712376    0.00 B     0       "2025-07-25" / "2025-07-25"
last_modified        INT96     S _ R_ F  712376    7.11 B     0

Row group 5:  count: 710779  188.57 B records  start: 669035798  total(compressed): 127.825 MB total(uncompressed):286.103 MB
--------------------------------------------------------------------------------
                     type      encodings count     avg size   nulls   min / max
address              BINARY    S _ R     710779    2.99 B     0       "0x000000000000003607fce1a..." / "0xffffffaeff0b96ea8e4f94b..."
topics.list.element  BINARY    S _ R_ F  1918448   26.28 B    3798    "0x00000000000000000000000..." / "0xfffffffffffffffffffffff..."
data                 BINARY    S _ R_ F  710779    36.99 B    0       "0x" / "0xfffffffffffffffffffffff..."
transaction_index    INT64     S _ R     710779    1.26 B     0       "0" / "1009"
log_index            INT64     S _ R     710779    1.59 B     0       "0" / "10251"
transaction_hash     BINARY    S _ R_ F  710779    62.10 B    0       "0x00002a9c7d3ec2f2ba87258..." / "0xfffff62ed4c66cdf03a9905..."
block_number         INT64     S _ R     710779    1.67 B     0       "22992451" / "22999594"
block_hash           BINARY    S _ R     710779    2.25 B     0       "0x000132755aa4d3c7a7f29a9..." / "0xfffd73719cd5869b3f5ebc5..."
block_timestamp      INT96     S _ R     710779    1.69 B     0
date                 BINARY    S _ R     710779    0.00 B     0       "2025-07-25" / "2025-07-25"
last_modified        INT96     S _ R_ F  710779    7.11 B     0

Row group 6:  count: 166191  187.84 B records  start: 803070006  total(compressed): 29.771 MB total(uncompressed):65.648 MB
--------------------------------------------------------------------------------
                     type      encodings count     avg size   nulls   min / max
address              BINARY    S _ R     166191    4.32 B     0       "0x00000000000000447e69651..." / "0xfffffffffffade86206f38e..."
topics.list.element  BINARY    S _ R_ F  447458    25.18 B    951     "0x00000000000000000000000..." / "0xfffffffffffffffffffffff..."
data                 BINARY    S _ R_ F  166191    36.55 B    0       "0x" / "0xfffffffffffffffffffffff..."
transaction_index    INT64     S _ R     166191    1.27 B     0       "0" / "1032"
log_index            INT64     S _ R     166191    1.46 B     0       "0" / "5785"
transaction_hash     BINARY    S _ R_ F  166191    61.51 B    0       "0x00002a9c7d3ec2f2ba87258..." / "0xffffe281cb16b8f03d48391..."
block_number         INT64     S _ R     166191    1.76 B     0       "22992451" / "22999594"
block_hash           BINARY    S _ R     166191    3.62 B     0       "0x000132755aa4d3c7a7f29a9..." / "0xfffb924915c3e15a766ff3a..."
block_timestamp      INT96     S _ R     166191    1.81 B     0
date                 BINARY    S _ R     166191    0.00 B     0       "2025-07-25" / "2025-07-25"
last_modified        INT96     S _ R_ F  166191    7.76 B     0

[rsarili]

@wilwade please see attached file for parquet-tools meta output

parquet_tools_meta.json

[wilwade]

@rsarili Hmm... Yeah. Lots of columns, but only one row group. Because Parquetjs loads in the entire row group at a time, that would be what is causing the high memory. It can handle lots of rows, but the number of columns is crushing it.

I'll have to see if I can generate a file like that to test. Thanks for the additional info!