The move to generations and using them for routing requests during rolling restarts was a large help for us. One of the side effects was that during a rolling restart memory utilisation in our app goes up by some ~13-15GB for a minute or two. From today:
By contrast, this was Passenger 6.0.18 a year ago:
The red markers are deployments; we've significantly increased them over the past year but the scale of the memory growth around deployments is significantly more than it used to be. I think that this is because in our app we are terminating processes that are cold - they have low/no processed requests and therefore very little in the way of private memory utilisation after being forked from the pre-loader - first. When they are replaced we make their replacements serve requests because they're the newer generation, thus causing them to be warm immediately. We're killing a process with ~50-200MB of private memory and replacing it with one that grows to ~800MB within a few requests.
Eventually of course we reach the point of killing off older hot processes (~800MB) and the newer generation replacement processes stay cold (~50-200MB) hence the saw tooth. On occasion we've hurt ourselves with memory protection mechanisms/limits - situations where Passenger 6.0.18 would have been fine have resulted in triggering remediations due to this flow.
Would you be open to changing the replacement mechanism so we can replace old hot processes first, rather than working from the bottom of the pool first?
The move to generations and using them for routing requests during rolling restarts was a large help for us. One of the side effects was that during a rolling restart memory utilisation in our app goes up by some ~13-15GB for a minute or two. From today:
By contrast, this was Passenger 6.0.18 a year ago:
The red markers are deployments; we've significantly increased them over the past year but the scale of the memory growth around deployments is significantly more than it used to be. I think that this is because in our app we are terminating processes that are cold - they have low/no processed requests and therefore very little in the way of private memory utilisation after being forked from the pre-loader - first. When they are replaced we make their replacements serve requests because they're the newer generation, thus causing them to be warm immediately. We're killing a process with ~50-200MB of private memory and replacing it with one that grows to ~800MB within a few requests.
Eventually of course we reach the point of killing off older hot processes (~800MB) and the newer generation replacement processes stay cold (~50-200MB) hence the saw tooth. On occasion we've hurt ourselves with memory protection mechanisms/limits - situations where Passenger 6.0.18 would have been fine have resulted in triggering remediations due to this flow.
Would you be open to changing the replacement mechanism so we can replace old hot processes first, rather than working from the bottom of the pool first?