resource friendly projections required

[codeliner]

from the chat by @basz :
when I run several long runing php processes (projection) how much memory should I expect those to use? according to newrelic 11 processes consume ~500Mb. time three for dev, staging production that kind of adds up. I’m wondering how accurate that metric is… Also I thought php7 was memory friendly and several processes can ‘share’ memory… Just qurious here

Bradley Weston @bweston92 Mrz. 14 16:56
@basz PHP isn't very memory friendly it adds extra data around stuff (http://php.net/manual/en/internals2.variables.intro.php). Example a string isn't just a string. However most programs don't just decrease memory when things aren't used anymore you have to wait for garbage collection which has to be enabled with zend.enable_gc=1

Can we do something to improve the situation?

[codeliner]

Maybe an event loop can help here. At the moment we only run one projection per process in the standard blocking php way. We could also put different projections in an event loop and work with generator based coroutines and async database connections.

The amphp project offers some interesting packages to use async connections with MySql, Postgres and Redis

Thoughts?

[basz]

question: is the memory count reported by new relic accurate. I never know how to measure usage memory on linux

remark: I think my apps have some overhead as these scripts are in a monolithic setup and do include config/container.php. I did do some lazy loading setup for the service manager, but I think that's where a lot of mem usage goes. Unfortuanatly I stilldon't know how to do profiling with php

workaround: fewer projection scripts with a loop where projection->run(false /* $keepRunning);

[codeliner]

even if fewer memory is used we still have the problem, that we never know if the projection is really busy or just blocks resources because it is idle but does not free memory due to constantly querying the event store.

With a loop that runs let's say 10 projections you could share the service manager between them and reuse the already created objects to save memory. Furthermore an event loop would allow us to teardown projections at night or at the weekend and spin them up again when new events arrive in the write model (f.e. triggered by an http call made from a monitoring service or supervisor that watches traffic on the write side)