Huge amount of memory used.

[ganju001]

Prerequisites

• I have written a descriptive issue title
• I have searched issues to ensure it has not already been reported

GitVersion package

GitVersion.Portable

GitVersion version

6.0.1

Operating system

Windows

What are you seeing?

Hello community,

We have mono-repo that is more that few gigabytes in size. We were using gitversion 5.x and it was calculating
the version in 15 minutes, after updating to gitversion 6.x, I got huge memory (RAM) usage: 40GB+ (yes 40, no mistake) and more than 2 hours...

Any suggestions?

What is expected?

I expect it will calculate version with reasonable RAM usage and time.

Steps to Reproduce

Internal repo..

RepositoryFixture Test

No response

Output log or link to your CI build (if appropriate).

[HHobeck]

I would suggest to wait until version v6.2 has been released. After that please analyze your execution and create a memory dump. You need to narrow down the locaction where the memory is allocated or the CPU time is used. I'm going to move this to the discussion forum.

Cheers

[ganju001]

@HHobeck I have a memory dump, I can try to give some insights, the problem is that LibGit2Sharp creates objects which takes gigabytes, gitversion wraps those objects (commits, collections) and those take gigabytes too, I Saw 400M+ of references and 300M+ of objects created.

Normal branches (non DevOps PR ones) takes ~1minute and 7GB RAM to calculate (that is kind of acceptable); but the PR ones are taking ages and more that expected RAM..

[HHobeck]

@HHobeck I have a memory dump, I can try to give some insights, the problem is that LibGit2Sharp creates objects which takes gigabytes, gitversion wraps those objects (commits, collections) and those take gigabytes too, I Saw 400M+ of references and 300M+ of objects created.

Normal branches (non DevOps PR ones) takes ~1minute and 7GB RAM to calculate (that is kind of acceptable); but the PR ones are taking ages and more that expected RAM..

We had some performance improvement in the upcoming version. Please wait with the analyzis. You say that LibGit2Sharp uses to much memory... Maybe we have a memory leak there. Normally the calls to LibGit2Sharp are wrapped and the results are cached. A code review might be an idea having this in mind.

[ganju001]

@HHobeck Few questions:

• Do we have some branch that I can try?
• I doubt it is memory leak, as the memory usage goes up and down many times. It seems like a caching. And having 300 000 000+ objects in memory for cache is an issue for sure. Too many strings are created too...

This is somewhere in the middle of calculation:

In example:
Commit person name/email strings takes 1.3GB. Some names are repeated 20k+ times - duplicates.

[HHobeck]

Do we have some branch that I can try?

The main branch includes the latest development version. If you want to do changes please fork the repository and create a feature branch. By creating a pull requst you can merge it back to the main branch of gitversion.

Commit person name/email strings takes 1.3GB. Some names are repeated 20k+ times - duplicates.

This properties are not really necessary for calculating the next version number. I'm not sure if we have it in control

[bentleyedby]

@HHobeck Using cache should be more quick and it actually saves more memory

[bentleyedby]

cache is the way to go, we just need to figure out the exact way

There are few places where the cache is needed, getting parent commit and commit collections should reuse same cache per repo.

[bentleyedby]

I will be waiting for you guys to fix it as I am not that familiar with a product.

[HHobeck]

For me it is still not clear what the solution is for the problem. Can you explain where exactly the caching is missing? Why does the caching is helping with memory consumption? Anyway it might be good to have a repository for testing. Please let me know what the outcome is.

[bentleyedby]

@HHobeck I did explain here: #4461 (reply in thread)

[HHobeck]

@bentleyedby: I have a few more questions and would be happy if you could answer them:

1. Are you and @ganju001 team-mates and are working together? The repository you are talking about is the same?
2. Is it possible to provide the repository which leads to the problem (or at least the anonymized repository without blobs)?
3. In your opinion, would the change in PR (Reuse created commits for all branches as commits are shared. #4553) solve the performance and memory issue for this repository?
4. You are not able to take the PR and do a refactoring by clean code aspects because you are not familiar with the project?
5. You have stated that with an empty tree and no blobs the memory consumption drops from 40 GB to 2 GB. Can you explain why this happens and why the PR is the solution? If I'm not wrong the PR just changes the way how the GitVersion Commit instances are created (the related Gitlib2Sharp Commit instance creation has not been touched). That means the blob and tree information are within Gitlib2Sharp and if this has not been touched why would it make a different?

Thank you very much!