stitchBlobs memory uptake

[weathermanbarnes]

When stitching Blobs with stitch blob for a large dataset (say a 40 year ERA5 climatology), the memory usage is massive. Is this a normal problem to encounter and has there been anyone who has done this more efficiently?

[paullric]

Hard to say without knowing where the memory allocation is going. Do you have an example command line?

…

On Feb 16, 2025, at 6:49 PM, Michael Barnes ***@***.***> wrote: weathermanbarnes created an issue (ClimateGlobalChange/tempestextremes#94) When stitching Blobs with stitch blob for a large dataset (say a 40 year ERA5 climatology), the memory usage is massive. Is this a normal problem to encounter and has there been anyone who has done this more efficiently? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread. <#94> <https://github.com/notifications/unsubscribe-auth/ABIVPZHZ7LNAELFYNW6L5BD2QFE4TAVCNFSM6AAAAABXIJJHCOVHI2DSMVQWIX3LMV43ASLTON2WKOZSHA2TMNJYGQYTMNY> weathermanbarnes created an issue (ClimateGlobalChange/tempestextremes#94) <#94> When stitching Blobs with stitch blob for a large dataset (say a 40 year ERA5 climatology), the memory usage is massive. Is this a normal problem to encounter and has there been anyone who has done this more efficiently? — Reply to this email directly, view it on GitHub <#94>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABIVPZHZ7LNAELFYNW6L5BD2QFE4TAVCNFSM6AAAAABXIJJHCOVHI2DSMVQWIX3LMV43ASLTON2WKOZSHA2TMNJYGQYTMNY>. You are receiving this because you are subscribed to this thread.

[weathermanbarnes]

mpirun -np 1 tempestextremes/bin/StitchBlobs --in_list detect_files.txt --min_overlap_prev 25.0 --max_overlap_prev 100.0 --min_overlap_next 25.0 --max_overlap_next 100.0 --latname latitude --lonname longitude --out_list stitch_files.txt

The detect_files.txt contains a list of ncs from detectblobs of monthly files (6 hourly timesteps) for the globe. Changing -np X to a large value has no effect on memory usage.

[paullric]

Are there a large number of blobs at each time step? Can you send the DetectBlobs command as well?

…

On Feb 16, 2025, at 7:02 PM, Michael Barnes ***@***.***> wrote: weathermanbarnes left a comment (ClimateGlobalChange/tempestextremes#94) mpirun -np 1 tempestextremes/bin/StitchBlobs --in_list detect_files.txt --min_overlap_prev 25.0 --max_overlap_prev 100.0 --min_overlap_next 25.0 --max_overlap_next 100.0 --latname latitude --lonname longitude --out_list stitch_files.txt The detect_files.txt contains a list of ncs from detectblobs of monthly files (6 hourly timesteps) for the globe. Changing -np X to a large value has no effect on memory usage. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented. <#94 (comment)> <https://github.com/notifications/unsubscribe-auth/ABIVPZCTEZ2XM2GCNVVY7YD2QFGK5AVCNFSM6AAAAABXIJJHCOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNRRHA2TKMZTGM> weathermanbarnes left a comment (ClimateGlobalChange/tempestextremes#94) <#94 (comment)> mpirun -np 1 tempestextremes/bin/StitchBlobs --in_list detect_files.txt --min_overlap_prev 25.0 --max_overlap_prev 100.0 --min_overlap_next 25.0 --max_overlap_next 100.0 --latname latitude --lonname longitude --out_list stitch_files.txt The detect_files.txt contains a list of ncs from detectblobs of monthly files (6 hourly timesteps) for the globe. Changing -np X to a large value has no effect on memory usage. — Reply to this email directly, view it on GitHub <#94 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABIVPZCTEZ2XM2GCNVVY7YD2QFGK5AVCNFSM6AAAAABXIJJHCOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNRRHA2TKMZTGM>. You are receiving this because you commented.

[weathermanbarnes]

Not particularly. Here is the start of the output from stitchBlobs:
..Time 0 (1980-01-01 00:00:00)
....Finding blobs (13884 tagged points)
....Blobs detected: 3
....Done
..Time 1 (1980-01-01 06:00:00)
....Finding blobs (12437 tagged points)
....Blobs detected: 2
....Done
..Time 2 (1980-01-01 12:00:00)
....Finding blobs (14660 tagged points)
....Blobs detected: 3
....Done
..Time 3 (1980-01-01 18:00:00)
....Finding blobs (17043 tagged points)
....Blobs detected: 3
....Done
..Time 4 (1980-01-02 00:00:00)
....Finding blobs (18678 tagged points)
....Blobs detected: 3
....Done

Here is the detectBlob command:
mpirun -np 1 tempestextremes/bin/DetectBlobs --in_data_list input_files.txt --thresholdcmd z_anom,>=,1000.0,0.0 --geofiltercmd area,>=,500000.0km2 --timefilter 6hr --latname latitude --lonname longitude --out_list detect_files.txt

[hongyuchen1030]

Hi @weathermanbarnes

Thanks again for your information! We've been investigating the high memory usage and believe I’ve identified the root cause. We’ve opened a pull request (PR #97 ) that optimizes memory usage.

Would you be able to test the StitchBlobs MPI version from that PR branch on your 40-year ERA5 climatology dataset to confirm if the memory problem is resolved or significantly improved? And please let me know if you need any information from my end

[weathermanbarnes]

Wow! Thanks @hongyuchen1030! This is awesome! I am on leave for the next few weeks but will definitely give this a go the minute I get back and report what I find.

[weathermanbarnes]

Hi @hongyuchen1030, sorry for the delay in getting back to you. The update certainly does save some memory. However, my tests still seem to use a ton of memory. A 15-year stitchBlob fails with substantial resources allocation (n_cores=24 and mem=128GB allocation) and we would like to track for the full 40-year period. My blobs are generally around 3-4 per timestep with 30,000-40,000 tagged points at 6 hourly intervals. Is there a different strategy required for these long-run, global stitchBlobs? Or is it something wrong on my end?

..Time 37 (1980-01-10 06:00:00)
....Finding blobs (35564 tagged points)
....Blobs detected: 3
....Done
..Time 38 (1980-01-10 12:00:00)
....Finding blobs (39782 tagged points)
....Blobs detected: 2
....Done
..Time 39 (1980-01-10 18:00:00)
....Finding blobs (40530 tagged points)
....Blobs detected: 3
....Done
..Time 40 (1980-01-11 00:00:00)
....Finding blobs (36840 tagged points)
....Blobs detected: 4
....Done
..Time 41 (1980-01-11 06:00:00)
....Finding blobs (34989 tagged points)
....Blobs detected: 3
....Done
..Time 42 (1980-01-11 12:00:00)
....Finding blobs (37710 tagged points)
....Blobs detected: 4
....Done
..Time 43 (1980-01-11 18:00:00)
....Finding blobs (37132 tagged points)
....Blobs detected: 4
....Done

[paullric]

Hmm… So a quick back of the envelope calculation suggests that 40000 tagged points per time slice at 6 hour per day for 15 years should require around ~10GB of memory, which seems a fair bit less than the 128GB on the allocation. Is it possible the code is being run as serial on 24 cores? That would mean each core was trying to allocate around that much memory and doing a lot of redundant work in the process.

…

On May 5, 2025, at 12:19 AM, Michael Barnes ***@***.***> wrote: weathermanbarnes left a comment (ClimateGlobalChange/tempestextremes#94) <#94 (comment)> Hi @hongyuchen1030 <https://github.com/hongyuchen1030>, sorry for the delay in getting back to you. The update certainly does save some memory. However, my tests still seem to use a ton of memory. A 15-year stitchBlob fails with substantial resources allocation (n_cores=24 and mem=128GB allocation) and we would like to track for the full 40-year period. My blobs are generally around 3-4 per timestep with 30,000-40,000 tagged points at 6 hourly intervals. Is there a different strategy required for these long-run, global stitchBlobs? Or is it something wrong on my end? ..Time 37 (1980-01-10 06:00:00) ....Finding blobs (35564 tagged points) ....Blobs detected: 3 ....Done ..Time 38 (1980-01-10 12:00:00) ....Finding blobs (39782 tagged points) ....Blobs detected: 2 ....Done ..Time 39 (1980-01-10 18:00:00) ....Finding blobs (40530 tagged points) ....Blobs detected: 3 ....Done ..Time 40 (1980-01-11 00:00:00) ....Finding blobs (36840 tagged points) ....Blobs detected: 4 ....Done ..Time 41 (1980-01-11 06:00:00) ....Finding blobs (34989 tagged points) ....Blobs detected: 3 ....Done ..Time 42 (1980-01-11 12:00:00) ....Finding blobs (37710 tagged points) ....Blobs detected: 4 ....Done ..Time 43 (1980-01-11 18:00:00) ....Finding blobs (37132 tagged points) ....Blobs detected: 4 ....Done — Reply to this email directly, view it on GitHub <#94 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABIVPZCR7XWM5HFT7HZN3LT244GGTAVCNFSM6AAAAABXIJJHCOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQNJQGEYTQOJYHE>. You are receiving this because you commented.

[hongyuchen1030]

Hi @weathermanbarnes ,

Thanks for the information. I wonder if I can get your run command for this?

[weathermanbarnes]

Thanks both.
@paullric - I have run some tests in serial and I get similar memory uptake.
@hongyuchen1030 my stitchBlobs command is:
StitchBlobs --in_list detect_files.txt --minsize 1 --mintime 1 --min_overlap_prev 25.0 --max_overlap_prev 100.0 --min_overlap_next 25.0 --max_overlap_next 100.0 --latname latitude --lonname longitude --out_list stitch_files.txt

[hongyuchen1030]

Thanks both. @paullric - I have run some tests in serial and I get similar memory uptake. @hongyuchen1030 my stitchBlobs command is: StitchBlobs --in_list detect_files.txt --minsize 1 --mintime 1 --min_overlap_prev 25.0 --max_overlap_prev 100.0 --min_overlap_next 25.0 --max_overlap_next 100.0 --latname latitude --lonname longitude --out_list stitch_files.txt

Thanks for your information! We will take a close look into it after next week!

[hongyuchen1030]

Thanks both. @paullric - I have run some tests in serial and I get similar memory uptake. @hongyuchen1030 my stitchBlobs command is: StitchBlobs --in_list detect_files.txt --minsize 1 --mintime 1 --min_overlap_prev 25.0 --max_overlap_prev 100.0 --min_overlap_next 25.0 --max_overlap_next 100.0 --latname latitude --lonname longitude --out_list stitch_files.txt

Hi @weathermanbarnes,

Thanks again for sharing the details! We’ve been investigating the high memory usage, and we now have a solution. Could you please try the updated version in this PR:
#111

It still needs some code cleanup and is a bit of a scratch build, but it should run correctly. Please let us know if this helps on your end.

Also, you mentioned that your serial run encountered some issues—could you share more specifics? We haven’t seen memory uptake problems in serial runs before, so that would help us track it down.