compiled interceptors module

[fsw]

Description (*)

At Creatuity we have developed a module that generates Interceptors code using app config instead of a boilerplate that reads config at runtime.

https://github.com/creatuity/magento2-interceptors

We believe it changes performance and debugging magento significantly and it is worth to consider making it a part of the core witch this pull request does.

Any comments welcome.

Manual testing scenarios (*)

to use in developer mode in app/etc/di.xml replace:

<item name="interceptor" xsi:type="string">\Magento\Framework\Interception\Code\Generator\Interceptor</item>

with:

<item name="interceptor" xsi:type="string">\Magento\Framework\CompiledInterception\Generator\CompiledInterceptor</item>

clear generated files and cache:

rm -rf generated/* && rm -rf var/cache/* && bin/magento cache:clean

generated interceptors instead of boilerplate that reads plugins config at runtime like this:

public function methodX($arg) {
    $pluginInfo = $this->pluginList->getNext($this->subjectType, 'methodX');
    if (!$pluginInfo) {
        return parent::methodX($arg);
    } else {
        return $this->___callPlugins('methodX', func_get_args(), $pluginInfo);
    }
}

should have a code built using config like:

public function methodX($arg) {
    switch(getCurrentScope()){
        case 'frontend':
            $this->_get_example_plugin()->beforeMethodX($this, $arg);
            $this->_get_another_plugin()->beforeMethodX($this, $arg);
            $result = $this->_get_around_plugin()->aroundMethodX($this, function($arg){
                return parent::methodX($arg);
            });
            return $this->_get_after_plugin()->afterMethodX($this, $result);
        case 'adminhtml':
            // ...
        default:
            return parent::methodX($arg);
    }
}

Contribution checklist (*)

• [*] Pull request has a meaningful description of its purpose
• [*] All commits are accompanied by meaningful commit messages
• [*] All new or changed code is covered with unit/integration tests (if applicable)
• All automated tests passed successfully (all builds on Travis CI are green)

Resolved issues:

1. resolves [Issue] compiled interceptors module #30311: compiled interceptors module

[m2-assistant]

Hi @fsw. Thank you for your contribution
Here is some useful tips how you can test your changes using Magento test environment.
Add the comment under your pull request to deploy test or vanilla Magento instance:

• @magento-engcom-team give me test instance - deploy test instance based on PR changes
• @magento-engcom-team give me 2.3-develop instance - deploy vanilla Magento instance

For more details, please, review the Magento Contributor Assistant documentation

[magento-cicd2]

All committers have signed the CLA.

[orlangur]

Hi @fsw,

We believe it changes performance and debugging magento significantly

Do you have some benchmarks to share with us?

[IvanChepurnyi]

@fsw This is a very nice module, do but did you consider compiling interceptors per area and specifying those on di for Magento to use, instead of using static calls to "getCurrentScope"?

[fsw]

@orlangur, on a standard instance, simple ApacheBench tests are showing about ~10% gain on a request, I will create some more detailed benchmark for you.

@IvanChepurnyi, actually this was the original idea, but:

• we wanted this to be 100% backward compatible and there were issues with class per scope approach: scopes can be switched at runtime when interception should be working, compiled mode classmap would have to be hacked to use different class depending on scope etc. So it was simply easier to implement this way.
• in a lot of cases plugins are exactly the same for all scopes so they are being grouped in a single switch case so we end up having 6 times less classes and code in opcache in exchange for this one static switch.

However I and not 100% sure used approach is better, do you think there are other upsides other than removing the static switch?

[IvanChepurnyi]

@fsw switch opcache argument sounds reasonable, but maybe you somehow could receive this current scope without doing this static call, like maybe pass it in the constructor, so DI container will automatically supply it for you?

[fsw]

@IvanChepurnyi this should be easily doable as code generator uses reflection anyway so should be easy to override parent constructor. I will check how this would affect performance.

[fsw]

@orlangur, trivial benchmark in developer mode:

bin/magento cache:clean > /dev/null
rm -rf generated/* && rm -rf var/cache/*
echo "warmup"
curl -so /dev/null -w '%{time_total}\n' $url
echo "cached"
ab -n 100 -c 4 $url | grep "Time per request:"

request		Interceptor (ms)	CompiledInterceptor (ms)	gain
home page	warmup request	15555	31342	-101.49%
	time per request	176.439	166.831	5.45%
	time per request (4 concurrent requests)	44.11	41.708	5.45%
listing	warmup request	14212	30557	-115.01%
	time per request	1813.871	1661.836	8.38%
	time per request (4 concurrent requests)	453.468	415.459	8.38%
product page	warmup request	15720	32143	-104.47%
	time per request	177.388	172.602	2.70%
	time per request (4 concurrent requests)	44.347	43.15	2.70%
admin login page	warmup request	16730	31671	-89.31%
	time per request	439.592	422.958	3.78%
	time per request (4 concurrent requests)	109.898	105.74	3.78%

You can see that first request with generated code purged renders a lot slower as interceptors now read config to generate, but in exchange following requests are faster. This obviously depends on number of plugins and should be better in production mode where DI takes far less time.

[fsw]

@IvanChepurnyi I have followed your suggestion and it seems to be a bit faster and code is much nicer indeed.
examples of generated interceptors:

https://github.com/fsw/magento2/tree/2.3-develop/lib/internal/Magento/Framework/CompiledInterception/Test/Unit/CompiledInterceptor/_out_interceptors

[IvanChepurnyi]

@fsw Looks nice! But I just noticed, seems like you also use object manager directly for plugin instantiation, so maybe it makes sense to supply it as well as a dependency?

[fsw]

@IvanChepurnyi already tried that :) but:

1. this would not be 100% backward compatible as currently plugins are "lazy loaded" as well and we can not be sure what happens in constructors. Here, we are doing stuff exactly in same order.
2. when I tried that (passing plugins in constructor) I got a lot of "circular dependency" issues as plugins were depending on objects that were intercepted and depended on other plugins etc.

[IvanChepurnyi]

@fsw I mean passing object manager for instantiation of plugins later. Sure for performance reasons pre-instantiation of plugins does not make sense.

[fsw]

@IvanChepurnyi oh yes, that makes sense. done.

Also I have 2 questions for general public:

1. Is slower generation time on first request a big problem? If so we can think about modifying class loader somehow so it would check file mtimes to cover most cases so developer would not have to purge var/generated/* each time plugin is added.

2. Is adding this as a separate generator with an option for a fallback a good approach? Because if this is 99.99% compatible (except edge cases like someone having plugins on config loaders) other approach would be to replace default interceptor generator with this?

[IvanChepurnyi]

@kandy good point.
@fsw how about creating a file watcher command line tool that will automatically clear generated code if plugin for that class is added in di.xml? I think it will greatly increase DevExperience, but I think it can be done a separate tool outside of this pull request, as it requires a dev to install an inotify extension, but it is worth it. Also, it is possible to use output from this tool instead of notifying, as it is much faster: https://docs.rs/crate/unison-fsmonitor/0.1.0

[kandy]

I think it makes sense to include your code directly to Interceptor library.
Also, we can implement a proxy that will select one of implementation depending on the mode. It may answer your question.

Also if this code will work in production mode we can make additional optimization like replace

$this->____om->get(\Magento\Framework\CompiledInterception\Test\Unit\Custom\Module\Model\ItemPlugin\Advanced::class);

with

new \Magento\Framework\ClassReplacedByPrreferance(
 $this->____om->get(\Dep1),
 $this->____om->get(\De2),
 $this->____om->get(\Dep3),
)

[fsw]

@kandy I think plugin instances are shared between interceptors, plus I think there is a bit more logic in DI when it creates constructor arguments (constants/virtual types etc) so I am not sure if removing this single step of DI would be worth it.

[kandy]

@fsw, It's hard to predict the performance impact now, and in any case, I don't recommend doing it in the scope of this PR

[kandy]

I believe that PR still has failed builds
@magento run all tests

[magento-automated-testing]

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please re-request them if they don't show in a reasonable amount of time.

[kandy]

@magento run all tests

[magento-automated-testing]

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please re-request them if they don't show in a reasonable amount of time.

[ihor-sviziev]

As discussed with @sidolov in Slack, he should ping the team working on finalizing this PR and write some updates here.

[ihor-sviziev]

As we discussed with @sidolov in Slack - he'll try to allocate time next week to go through this PR and figure out what's left.

[fsw]

@ihor-sviziev @sidolov this version and module version have diverged a bit but maybe it will be useful for you to take a look at fixes to module version to see if they are useful here:
https://github.com/creatuity/magento2-interceptors/pulls?q=is%3Apr+

[engcom-Hotel]

Hello @fsw,

Thanks for the contribution!

Currently, we are not processing the PRs targeted for 2.5-develop. Hence we are moving this PR on hold for now.

We will pick this PR in the future whenever we start working on 2.5-develop.

Thanks

[renttek]

As far as I can see, there is not even a timeline for 2.5, the next release in 2024 is 2.4.7.
So the earliest possible release after this (according the the plan) would be somewhere 2025.

So my honest (yes or no) question: will this PR ever happen? (it's been 4+ years now)

[loid345]

As far as I can see, there is not even a timeline for 2.5, the next release in 2024 is 2.4.7. So the earliest possible release after this (according the the plan) would be somewhere 2025.

So my honest (yes or no) question: will this PR ever happen? (it's been 4+ years now)

This will probably happen when shopware 8 is released and everyone will forget about magento)

[sinhaparul]

We are currently processing PRs targeted to 2.4-develop only. So, removing this from the community picked backlog for now.

[memegento-dev]

you were supposed to fight evil not join it

[Den4ik]

@sinhaparul I believe that we can update PR with changes for Magento 2.4 from https://github.com/creatuity/magento2-interceptors and move forward. What do you think?

[Aquive]

Any updates?