SNOW-1747415 SNOW-1215393: Out of Memory Issue in memory-limited environment while streaming data

[kartikgupta2607]

Please answer these questions before submitting your issue.
In order to accurately debug the issue this information is required. Thanks!

1. What version of NodeJS driver are you using? ->snowflake-sdk -> Tried with 1.6.20, 1.9.3

2. What operating system and processor architecture are you using? -> macOS 14.3.1, arm64

3. What version of NodeJS are you using? -> v18.12.1

4. What are the component versions in the environment (npm list)? -> NA

5. Server version: -> 8.9.1

6. What did you do?
   Tried running this (snowflake_OOM.txt) script to export records from SNOWFLAKE_SAMPLE_DATA.TPCH_SF1000.CUSTOMER table, after limiting max-old-space-size=150 while running the script. A simple SELECT * FROM CUSTOMER; was used, but the node process exited after exporting some rows (200000) with FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory error but on modifying the same query with a LIMIT clause (with limit of 1,000,000), it was able to export 1M records with the same old-space limit. I tried workarounds mentioned in SNOW-750472 Out of memory issue #43 and the linked issues but none worked. Tried using the streamResult option in the connection config and while executing the query, tried degrading to 1.6.20.
   Following is the metadata of the source table

As it doesn't have any VARIANT type column, so #536 and the related issues because of it shouldn't matter in our case

7. What did you expect to see?

   It should be able to export the same number of records before the OOM issue occurs, also it seems a large buffer is fetched if the LIMIT isn't present in the query as it seems the GC triggers a bit late for query without LIMIT

8. Can you set logging to DEBUG and collect the logs? -> Can't upload logs due to company security policies.

var snowflake = require('snowflake-sdk');
snowflake.configure(
{
  logLevel: 'trace'
});

9. What is your Snowflake account identifier, if any? (Optional)

[sfc-gh-dszmolka]

hi - thanks for raising this issue with us, i'll take a look and see how we can proceed

[shelbylsmith]

I have the same issue with node 20.9.0 and snowflake-sdk 1.9.0.

[sfc-gh-dszmolka]

thank you again for the detailed reproduction, the issue could be reproduced even with the TPCH_SF100 version of the CUSTOMER table which 'only' has 15000000 rows.

Of course if one leaves the memory limit flag off, the query runs to completion and we can observe that memory usage goes up into the ~1G range with this table in this comment, then GC kicks in, and this goes until query completes.
Additionally if a smaller table is created e.g. with 1M rows from the original table, and the smaller table is queried instead with SELECT * FROM SMALLER_TABLE, then the query also runs to completion even with the memory limit.

I'm also not sure if this is the expected behaviour with the 'streaming rows' functionality and also comparing some heap snapshots in the 'bad' and 'good' scenario raises additional questions so now I involved the driver team to take a look at this.
Will keep this thread posted with the progress.

[MC7771]

You can try to fetch stream in specific range like this:
const stream = stmt.streamRows({ start, end })
This way you can limit number of records fetched from snowflake.
But you need to write your own loop and calculate start, end index.

[sfc-gh-dszmolka]

based on the same idea, perhaps the LIMIT ... OFFSET ... construct could be also a workaround as part of the big query, and alleviate the problem by 'partitioning' the query result and iterating over it. But it's still just a workaround, until we find a solution.