Filters by attribute groups and feature - improve perfomances

[kgleizes]

Questions	Answers
Description?	Performance improvement - filters by attributes group and features
Type?	improvement
BC breaks?	no
Deprecations?	no
Fixed ticket?	No ticket done.
How to test?	See testing explanations behind.
Sponsor company	202 ecommerce

We were looking for a solution with several filters on attributes group making very slow requests.

Details about the shop :
224 items in table attribute_group
4256 items in attribute
69 categories
2603 products
40244 lines in product_attribute

The main issue was on two filters by attribute group with not a lot of elements :

The main problem on the module's request is the count, and the fact to add where arguments based on attribute
group on the count request. For example, this request take 14 sec :

SELECT SQL_NO_CACHE pac.id_attribute, COUNT(DISTINCT p.id_product) c FROM (SELECT p.id_product, p.id_manufacturer, SUM(sa.quantity) as quantity, p.condition, p.weight, p.price, psales.quantity as sales, p.on_sale, p.date_add, cp.position FROM ps_product p LEFT JOIN ps_product_attribute pa ON (p.id_product = pa.id_product) LEFT JOIN ps_product_attribute_combination pac ON (pa.id_product_attribute = pac.id_product_attribute) LEFT JOIN ps_stock_available sa ON (p.id_product = sa.id_product AND IFNULL(pac.id_product_attribute, 0) = sa.id_product_attribute AND sa.id_shop = 1  AND sa.id_shop_group = 0 ) LEFT JOIN ps_product_sale psales ON (psales.id_product = p.id_product) INNER JOIN ps_category_product cp ON (p.id_product = cp.id_product) INNER JOIN ps_product_shop ps ON (p.id_product = ps.id_product AND ps.id_shop = 1 AND ps.active = TRUE) INNER JOIN ps_category c ON (cp.id_category = c.id_category AND c.active=1) LEFT JOIN ps_category_group cg ON (cg.id_category = c.id_category) WHERE ps.id_shop='1' AND ps.visibility IN ('both', 'catalog') AND cg.id_group='1' AND c.nleft>=71 AND c.nright<=84 GROUP BY p.id_product) p LEFT JOIN ps_product_attribute pa ON (p.id_product = pa.id_product) LEFT JOIN ps_product_attribute_combination pac ON (pa.id_product_attribute = pac.id_product_attribute) INNER JOIN ps_attribute a ON (a.id_attribute = pac.id_attribute) WHERE ((a.id_attribute_group=54)) GROUP BY pac.id_attribute

Whereas the same requests without the WHERE condition take 117ms :

SELECT SQL_NO_CACHE pac.id_attribute, COUNT(DISTINCT p.id_product) c FROM (SELECT p.id_product, p.id_manufacturer, SUM(sa.quantity) as quantity, p.condition, p.weight, p.price, psales.quantity as sales, p.on_sale, p.date_add, cp.position FROM ps_product p LEFT JOIN ps_product_attribute pa ON (p.id_product = pa.id_product) LEFT JOIN ps_product_attribute_combination pac ON (pa.id_product_attribute = pac.id_product_attribute) LEFT JOIN ps_stock_available sa ON (p.id_product = sa.id_product AND IFNULL(pac.id_product_attribute, 0) = sa.id_product_attribute AND sa.id_shop = 1  AND sa.id_shop_group = 0 ) LEFT JOIN ps_product_sale psales ON (psales.id_product = p.id_product) INNER JOIN ps_category_product cp ON (p.id_product = cp.id_product) INNER JOIN ps_product_shop ps ON (p.id_product = ps.id_product AND ps.id_shop = 1 AND ps.active = TRUE) INNER JOIN ps_category c ON (cp.id_category = c.id_category AND c.active=1) LEFT JOIN ps_category_group cg ON (cg.id_category = c.id_category) WHERE ps.id_shop='1' AND ps.visibility IN ('both', 'catalog') AND cg.id_group='1' AND c.nleft>=71 AND c.nright<=84 GROUP BY p.id_product) p LEFT JOIN ps_product_attribute pa ON (p.id_product = pa.id_product) LEFT JOIN ps_product_attribute_combination pac ON (pa.id_product_attribute = pac.id_product_attribute) GROUP BY pac.id_attribute

So the main point of this update is to remove the WHERE conditions in the SQL requests, and filter in PHP
code the results to get exactly the same results in final filters content.

Instead of launching for each filter the same attribute group results, it will be launched only one time for process.
This means that previously in my example, we had nearly the same request launched 9 times, with only the
WHERE condition to filter attribute group different.

Now, that's only one request to get results so that a big win of time.
The same process was reported on the features request, to win again a lot of time.

In my example, i had these filters :
9 filters by attribute group : 9 requests with two very slows = 28 sec
6 filters by feature : 6 requests going between 85ms to 125ms = 700 ms

After this update, with the same configuration :
9 filters by attribute group : 1 request with 120ms
6 filters by feature : 1 request with 107 ms

The more filters a store has, the greater the gain in performance.

Before update :

After update :

I have checked the final content (filter results) before and after and the result is the same.

It could have a very lack of performance (~15 to 20ms) in the case of only one filter of any kind is set.
So it could be discussed to have this gain as a configuration.

I never saw a configuration of filters with only one kind of filter (always three or four by kind on the minimum).

Don't hesitate to give me a feedback or feelings on this update.

Thank you.

[ps-jarvis]

Hello @kgleizes!

This is your first pull request on ps_facetedsearch repository of the PrestaShop project.

Thank you, and welcome to this Open Source community!

[bogdanbeniaminb]

I have tested this against version 4.0.1. The final result (filters content) is not the same. I get some filters that should not be there in comparison with the old code, and when applying those filters I get a "no results" page - so those filters are clearly wrong. This happens when there are multiple filters selected at once (probably the PHP code cannot process all the combinations). @kgleizes

[kgleizes]

Hello @bogdanbeniaminb thank you for this feedback.

I will take a look this week. Could you please give me some details about your tests ?

I tried a lot of combinations on my side before making the pull request and the results was the same that before.
On the first display of the page (without filters ) and applying multiple filters.

Thank you.

[kgleizes]

Hello @bogdanbeniaminb, thank you for this.

I made a lot of tests for fixing this point.

At this time, the best solution for me was to let the previous process when filters are selected.

So now we will have :

• When the count of products in filters is calculated without selected filters :
  ==> Performances improved

• When the count of products is calculated with selected filters :
  ==> Same as before

I will try to find a way to improve with the same logic when there are selected filters too.

At this step, that's already a big improvement when category page is displayed for the first time without cache.

Thank you for your feedbacks.