Memory consumption grows linear #19622

nikdom · 2025-03-09T17:24:18Z

nikdom
Mar 9, 2025
Sponsor

Hi all,
I didnt create an issue yet, as I cannot test whether the issue still persists in the nightly build. Did anyone encounter a similar issue? Am I doing something wrong?

I run evcc in a proxmox lxc which is based on ubuntu 20.
Since the upgrade to v0.200.0 (at least since then I recognize) and above the memory consumption grows linear. No update since then changed the issue. Currently using v0.200.8. At some point the lxc will just crash as it is out of memory. A reboot is then necessary and fixes it for a day or two.

It does not matter whether a car is charging or if it is plugged to the load point at all.

My setup is:

sungrow sh5.0rt with sbr096
go-e homefix v3
some shelly pm1
tesla model y myteslamate integration
tibber

The log is not telling me anything suspicious:

[site  ] DEBUG 2025/03/09 16:54:14 ----
[lp-1  ] DEBUG 2025/03/09 16:54:14 charge power: 0W
[lp-1  ] DEBUG 2025/03/09 16:54:14 charge currents: [0 0 0]A
[site  ] DEBUG 2025/03/09 16:54:14 grid power: 15W
[site  ] DEBUG 2025/03/09 16:54:14 pv 1 power: 63W
[site  ] DEBUG 2025/03/09 16:54:14 pv 3 power: 3W
[site  ] DEBUG 2025/03/09 16:54:14 battery 1 power: 371W
[site  ] DEBUG 2025/03/09 16:54:14 pv 2 power: 48W
[site  ] DEBUG 2025/03/09 16:54:14 battery 1 soc: 98%
[site  ] DEBUG 2025/03/09 16:54:14 grid currents: [-0.514 0.369 0.309]A
[site  ] DEBUG 2025/03/09 16:54:14 pv power: 114W
[site  ] DEBUG 2025/03/09 16:54:14 site power: 636W
[lp-1  ] DEBUG 2025/03/09 16:54:14 charge total import: 2284.413kWh
[lp-1  ] DEBUG 2025/03/09 16:54:14 charger status: A
[site  ] DEBUG 2025/03/09 16:54:24 ----
[lp-1  ] DEBUG 2025/03/09 16:54:24 charge power: 0W
[lp-1  ] DEBUG 2025/03/09 16:54:24 charge currents: [0 0 0]A
[site  ] DEBUG 2025/03/09 16:54:24 grid power: -1W
[site  ] DEBUG 2025/03/09 16:54:24 pv 1 power: 62W
[site  ] DEBUG 2025/03/09 16:54:24 pv 2 power: 48W
[site  ] DEBUG 2025/03/09 16:54:24 pv 3 power: 3W
[site  ] DEBUG 2025/03/09 16:54:24 battery 1 power: 394W
[site  ] DEBUG 2025/03/09 16:54:24 battery 1 soc: 98%
[site  ] DEBUG 2025/03/09 16:54:24 grid currents: [-0.522 0.364 0.205]A
[site  ] DEBUG 2025/03/09 16:54:24 pv power: 113W
[site  ] DEBUG 2025/03/09 16:54:24 site power: 643W
[lp-1  ] DEBUG 2025/03/09 16:54:24 charge total import: 2284.413kWh
[lp-1  ] DEBUG 2025/03/09 16:54:24 charger status: A
[site  ] DEBUG 2025/03/09 16:54:34 ----
[lp-1  ] DEBUG 2025/03/09 16:54:34 charge power: 0W
[lp-1  ] DEBUG 2025/03/09 16:54:34 charge currents: [0 0 0]A
[site  ] DEBUG 2025/03/09 16:54:34 grid power: 18W
[site  ] DEBUG 2025/03/09 16:54:34 pv 1 power: 63W
[site  ] DEBUG 2025/03/09 16:54:34 pv 3 power: 3W
[site  ] DEBUG 2025/03/09 16:54:34 pv 2 power: 47W
[site  ] DEBUG 2025/03/09 16:54:34 battery 1 power: 369W
[site  ] DEBUG 2025/03/09 16:54:34 battery 1 soc: 98%
[site  ] DEBUG 2025/03/09 16:54:34 pv power: 113W
[site  ] DEBUG 2025/03/09 16:54:34 grid currents: [-0.516 0.366 0.279]A
[site  ] DEBUG 2025/03/09 16:54:34 site power: 637W
[lp-1  ] DEBUG 2025/03/09 16:54:34 charge total import: 2284.413kWh
[lp-1  ] DEBUG 2025/03/09 16:54:34 charger status: A
[site  ] DEBUG 2025/03/09 16:54:44 ----
[lp-1  ] DEBUG 2025/03/09 16:54:44 charge power: 0W
[lp-1  ] DEBUG 2025/03/09 16:54:44 charge currents: [0 0 0]A
[site  ] DEBUG 2025/03/09 16:54:44 grid power: 16W
[site  ] DEBUG 2025/03/09 16:54:44 pv 1 power: 63W
[site  ] DEBUG 2025/03/09 16:54:44 pv 3 power: 3W
[site  ] DEBUG 2025/03/09 16:54:44 battery 1 power: 372W
[site  ] DEBUG 2025/03/09 16:54:44 pv 2 power: 46W
[site  ] DEBUG 2025/03/09 16:54:44 battery 1 soc: 98%
[site  ] DEBUG 2025/03/09 16:54:44 grid currents: [-0.519 0.371 0.283]A
[site  ] DEBUG 2025/03/09 16:54:44 pv power: 112W
[site  ] DEBUG 2025/03/09 16:54:44 site power: 638W
[lp-1  ] DEBUG 2025/03/09 16:54:44 charge total import: 2284.413kWh
[lp-1  ] DEBUG 2025/03/09 16:54:44 charger status: A

I also found this #18098 but I assume the fix is already productive as it was resolved on Jan 6.

Thanks a lot and keep up the great work, I really love evcc!!

Answered by GrimmiMeloni

Mar 10, 2025

@GrimmiMeloni let me know whether you need the full pprof pb.gz dump.

Thanks, but no need - the data you provided is pretty clear already. And with the known leak in Yeagi I think there's not much else to analyze.

View full answer

GrimmiMeloni · 2025-03-09T17:37:38Z

GrimmiMeloni
Mar 9, 2025
Collaborator Sponsor

Can you run with --profile and create a pprof when memory is about to peak?

0 replies

nikdom · 2025-03-10T12:45:57Z

nikdom
Mar 10, 2025
Author Sponsor

Did that, here are three extracts. Seems it is also related to traefik/yaegi as github.com/traefik/yaegi/interp.(*Interpreter).ast.func1 is growing infinitely. Not sure if it is the same as mentioned by @andig already here traefik/yaegi#1678.

@GrimmiMeloni let me know whether you need the full pprof pb.gz dump.

1st after bootup

Time: Mar 10, 2025 at 10:42am (UTC)
Showing nodes accounting for 21022.18kB, 100% of 21022.18kB total
      flat  flat%   sum%        cum   cum%
 4609.26kB 21.93% 21.93%  4609.26kB 21.93%  github.com/traefik/yaegi/interp.(*Interpreter).ast.func1
 3585.55kB 17.06% 38.98%  3585.55kB 17.06%  github.com/aws/aws-sdk-go/aws/endpoints.init
 2057.14kB  9.79% 48.77%  2057.14kB  9.79%  github.com/evcc-io/evcc/util/templates.(*Template).ResolvePresets
    1025kB  4.88% 53.64%     1025kB  4.88%  encoding/pem.Decode
 1024.03kB  4.87% 58.51%  1024.03kB  4.87%  container/ring.New (inline)
  518.02kB  2.46% 60.98%   518.02kB  2.46%  github.com/andig/gosunspec/models/model214.init.0
  517.33kB  2.46% 63.44%   517.33kB  2.46%  github.com/evcc-io/evcc/tariff/entsoe.map.init.0
     514kB  2.45% 65.88%      514kB  2.45%  github.com/prometheus/client_golang/prometheus.newSummary
  513.50kB  2.44% 68.33%   513.50kB  2.44%  regexp/syntax.(*compiler).inst (inline)
  513.12kB  2.44% 70.77%   513.12kB  2.44%  regexp.onePassCopy
  512.44kB  2.44% 73.21%   512.44kB  2.44%  github.com/go-playground/validator/v10.map.init.4
  512.34kB  2.44% 75.64%   512.34kB  2.44%  net/http.(*Transport).tryPutIdleConn
  512.16kB  2.44% 78.08%   512.16kB  2.44%  math/big.nat.make (inline)
  512.05kB  2.44% 80.51%   512.05kB  2.44%  github.com/andig/gosunspec/models/model1.init.0
  512.05kB  2.44% 82.95%  1536.08kB  7.31%  github.com/evcc-io/evcc/util/logstash.(*logger).Write
  512.05kB  2.44% 85.39%   512.05kB  2.44%  github.com/nicksnyder/go-i18n/v2/i18n.setPluralTemplate (inline)
  512.05kB  2.44% 87.82%   512.05kB  2.44%  go/token.(*FileSet).AddFile
  512.03kB  2.44% 90.26%   512.03kB  2.44%  github.com/gorilla/mux.(*Route).addMatcher (inline)
  512.02kB  2.44% 92.69%   512.02kB  2.44%  crypto/x509.parseASN1String
  512.02kB  2.44% 95.13%  1024.07kB  4.87%  github.com/nicksnyder/go-i18n/v2/i18n.NewMessageTemplate
  512.01kB  2.44% 97.56%   512.01kB  2.44%  regexp.makeOnePass.func1
  512.01kB  2.44%   100%   512.01kB  2.44%  gopkg.in/yaml%2ev3.(*parser).scalar

2nd after an hour

Time: Mar 10, 2025 at 11:36am (UTC)
Showing nodes accounting for 26217.01kB, 100% of 26217.01kB total
    flat  flat%   sum%        cum   cum%
7682.05kB 29.30% 29.30%  7682.05kB 29.30%  github.com/traefik/yaegi/interp.(*Interpreter).ast.func1
3585.55kB 13.68% 42.98%  3585.55kB 13.68%  github.com/aws/aws-sdk-go/aws/endpoints.init
2057.14kB  7.85% 50.82%  2057.14kB  7.85%  github.com/evcc-io/evcc/util/templates.(*Template).ResolvePresets
1098.02kB  4.19% 55.01%  1098.02kB  4.19%  github.com/traefik/yaegi/interp.(*scope).add
  1025kB  3.91% 58.92%     1025kB  3.91%  encoding/pem.Decode
1024.03kB  3.91% 62.83%  1024.03kB  3.91%  container/ring.New (inline)
518.02kB  1.98% 64.80%   518.02kB  1.98%  github.com/andig/gosunspec/models/model214.init.0
517.33kB  1.97% 66.78%   517.33kB  1.97%  github.com/evcc-io/evcc/tariff/entsoe.map.init.0
   514kB  1.96% 68.74%      514kB  1.96%  github.com/prometheus/client_golang/prometheus.newSummary
513.50kB  1.96% 70.70%   513.50kB  1.96%  regexp/syntax.(*compiler).inst (inline)
513.12kB  1.96% 72.65%   513.12kB  1.96%  regexp.onePassCopy
512.44kB  1.95% 74.61%   512.44kB  1.95%  github.com/go-playground/validator/v10.map.init.4
512.34kB  1.95% 76.56%   512.34kB  1.95%  net/http.(*Transport).tryPutIdleConn
512.16kB  1.95% 78.52%   512.16kB  1.95%  math/big.nat.make (inline)
512.05kB  1.95% 80.47%   512.05kB  1.95%  github.com/andig/gosunspec/models/model1.init.0
512.05kB  1.95% 82.42%   512.05kB  1.95%  github.com/nicksnyder/go-i18n/v2/i18n.setPluralTemplate (inline)
512.05kB  1.95% 84.38%   512.05kB  1.95%  go/token.(*FileSet).AddFile
512.03kB  1.95% 86.33%   512.03kB  1.95%  github.com/gorilla/mux.(*Route).addMatcher (inline)
512.03kB  1.95% 88.28%   512.03kB  1.95%  github.com/traefik/yaegi/interp.convert
512.03kB  1.95% 90.24%   512.03kB  1.95%  go/token.(*File).AddLine
512.02kB  1.95% 92.19%   512.02kB  1.95%  crypto/x509.parseASN1String
512.02kB  1.95% 94.14%  1024.07kB  3.91%  github.com/nicksnyder/go-i18n/v2/i18n.NewMessageTemplate
512.01kB  1.95% 96.09%   512.01kB  1.95%  regexp.makeOnePass.func1
512.01kB  1.95% 98.05%   512.01kB  1.95%  go/scanner.(*Scanner).scanIdentifier
512.01kB  1.95%   100%   512.01kB  1.95%  gopkg.in/yaml%2ev3.(*parser).scalar

3rd close before memory was full (limited to 128mb for testing)

Time: Mar 10, 2025 at 12:37pm (UTC)
Showing nodes accounting for 34999.16kB, 100% of 34999.16kB total
      flat  flat%   sum%        cum   cum%
11779.30kB 33.66% 33.66% 11779.30kB 33.66%  github.com/traefik/yaegi/interp.(*Interpreter).ast.func1
 3585.55kB 10.24% 43.90%  3585.55kB 10.24%  github.com/aws/aws-sdk-go/aws/endpoints.init
 2057.14kB  5.88% 49.78%  2057.14kB  5.88%  github.com/evcc-io/evcc/util/templates.(*Template).ResolvePresets
 2051.85kB  5.86% 55.64%  3075.88kB  8.79%  github.com/evcc-io/evcc/util/logstash.(*logger).Write
 1681.03kB  4.80% 60.44%  1681.03kB  4.80%  github.com/traefik/yaegi/interp.(*scope).add
    1025kB  2.93% 63.37%     1025kB  2.93%  encoding/pem.Decode
 1024.03kB  2.93% 66.30%  1024.03kB  2.93%  container/ring.New (inline)
  518.02kB  1.48% 67.78%   518.02kB  1.48%  github.com/andig/gosunspec/models/model214.init.0
  517.33kB  1.48% 69.26%   517.33kB  1.48%  github.com/evcc-io/evcc/tariff/entsoe.map.init.0
     514kB  1.47% 70.73%      514kB  1.47%  bufio.NewWriterSize (inline)
     514kB  1.47% 72.19%      514kB  1.47%  github.com/prometheus/client_golang/prometheus.newSummary
  513.50kB  1.47% 73.66%   513.50kB  1.47%  regexp/syntax.(*compiler).inst (inline)
  513.12kB  1.47% 75.13%   513.12kB  1.47%  regexp.onePassCopy
  512.44kB  1.46% 76.59%   512.44kB  1.46%  github.com/go-playground/validator/v10.map.init.4
  512.34kB  1.46% 78.06%   512.34kB  1.46%  net/http.(*Transport).tryPutIdleConn
  512.16kB  1.46% 79.52%   512.16kB  1.46%  math/big.nat.make (inline)
  512.05kB  1.46% 80.98%   512.05kB  1.46%  github.com/andig/gosunspec/models/model1.init.0
  512.05kB  1.46% 82.44%   512.05kB  1.46%  github.com/nicksnyder/go-i18n/v2/i18n.setPluralTemplate (inline)
  512.05kB  1.46% 83.91%   512.05kB  1.46%  go/token.(*FileSet).AddFile
  512.03kB  1.46% 85.37%   512.03kB  1.46%  github.com/gorilla/mux.(*Route).addMatcher (inline)
  512.03kB  1.46% 86.83%   512.03kB  1.46%  github.com/traefik/yaegi/interp.convert
  512.03kB  1.46% 88.30%   512.03kB  1.46%  go/token.(*File).AddLine
  512.02kB  1.46% 89.76%   512.02kB  1.46%  crypto/x509.parseASN1String
  512.02kB  1.46% 91.22%  1024.07kB  2.93%  github.com/nicksnyder/go-i18n/v2/i18n.NewMessageTemplate
  512.02kB  1.46% 92.69%   512.02kB  1.46%  github.com/traefik/yaegi/interp.branch
  512.02kB  1.46% 94.15%   512.02kB  1.46%  github.com/traefik/yaegi/interp.genValue
  512.01kB  1.46% 95.61%   512.01kB  1.46%  regexp.makeOnePass.func1
  512.01kB  1.46% 97.07%   512.01kB  1.46%  encoding/json.typeFields
  512.01kB  1.46% 98.54%   512.01kB  1.46%  go/scanner.(*Scanner).scanIdentifier
  512.01kB  1.46%   100%   512.01kB  1.46%  gopkg.in/yaml%2ev3.(*parser).scalar

8 replies

andig Mar 10, 2025
Maintainer

Looks very much like it. Seems one of your templates is using the go plugin and that keeps growing. Unfortunately, we've never received an answer on the issue :(

GrimmiMeloni Mar 10, 2025
Collaborator Sponsor

@GrimmiMeloni let me know whether you need the full pprof pb.gz dump.

Thanks, but no need - the data you provided is pretty clear already. And with the known leak in Yeagi I think there's not much else to analyze.

Answer selected by nikdom

nikdom Mar 10, 2025
Author Sponsor

Yup, thanks both! I guess I have to wait and drink tea. :-) For now I will reboot evcc once a day and observe.

DiDu0815 Apr 30, 2025

There should be a smarter way to workaround it - at least as long as the leak still exists. I once started with a 256MB VM and now I'm at 1GB already because of that. Still not sufficient to keep evcc up 24x7 without restart. I do systemctl restart evcc by cron.daily, but still not sure if this is the recommended way to go.

andig May 1, 2025
Maintainer

If we can find which template you're using that causes this, maybe we can find a solution without the GO plugin. For time being it's unfixable: traefik/yaegi#1678

DiDu0815 May 1, 2025

I do not use custom GO: scripting (which I understood as the way to trigger the Yaegi leak). Just using the ordinary templates. If that were true, the leak must come from used core components. Is there a way to debug which evcc parts are consuming which amount of memory at runtime? How to export database configuration to an easy readable format (e.g. back to evcc.yaml) to show the config here? I remember screenshots are not very welcome.

tantive · 2025-05-01T20:09:57Z

tantive
May 1, 2025

templates/definition/meter/sungrow-hybrid.yaml
uses go to calculate and set power in two places.

I own a SH10RT and also observe the memory leak.

1 reply

DiDu0815 May 1, 2025

SH10RT here too. Yeah, the issue started somewhere around the time blocking charge/discharge home battery was invented.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Memory consumption grows linear #19622

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 9 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Memory consumption grows linear #19622

nikdom Mar 9, 2025 Sponsor

Replies: 3 comments · 9 replies

GrimmiMeloni Mar 9, 2025 Collaborator Sponsor

nikdom Mar 10, 2025 Author Sponsor

andig Mar 10, 2025 Maintainer

GrimmiMeloni Mar 10, 2025 Collaborator Sponsor

nikdom Mar 10, 2025 Author Sponsor

DiDu0815 Apr 30, 2025

andig May 1, 2025 Maintainer

DiDu0815 May 1, 2025

tantive May 1, 2025

DiDu0815 May 1, 2025

nikdom
Mar 9, 2025
Sponsor

Replies: 3 comments 9 replies

GrimmiMeloni
Mar 9, 2025
Collaborator Sponsor

nikdom
Mar 10, 2025
Author Sponsor

andig Mar 10, 2025
Maintainer

GrimmiMeloni Mar 10, 2025
Collaborator Sponsor

nikdom Mar 10, 2025
Author Sponsor

andig May 1, 2025
Maintainer

tantive
May 1, 2025