fix for #38315 issue (Missing indexing of required columns in catalog_product_entity_int table) #38316

rostilos · 2023-12-25T19:32:25Z

Description (*)

Added indexing of entity_id column to avoid cases of "hang" of select query in mysql ( full description of the problem is in linked issue )

For example, after edits, the SELECT query specified in the issue are successful and in a fraction of a second

Related Pull Requests

https://github.com/magento-gl/magento2ee/pull/271

Fixed Issues (if relevant)

Fixes Missing indexing of required columns in catalog_product_entity_int table #38315

Manual testing scenarios (*)

1.A relatively large catalog is needed.
In a test environment, it is reproduced on a test catalog generated with the
bin/magento setup:perf:generate-fixtures setup/performance-toolkit/profiles/ce/extra_large.xml
2. Start the reindex process
bin/magento indexer:reset
bin/magento indexer:reindex catalogsearch_fulltext
3. Successful reindex

Questions or comments

Contribution checklist (*)

Pull request has a meaningful description of its purpose
All commits are accompanied by meaningful commit messages
All new or changed code is covered with unit/integration tests (if applicable)
README.md files for modified modules are updated and included in the pull request if any README.md predefined sections require an update
All automated tests passed successfully (all builds are green)

…catalog_product_entity_int table)

m2-assistant · 2023-12-25T19:32:29Z

Hi @rostilos. Thank you for your contribution!
Here are some useful tips on how you can test your changes using Magento test environment.

Add the comment under your pull request to deploy test or vanilla Magento instance:

@magento give me test instance - deploy test instance based on PR changes
@magento give me 2.4-develop instance - deploy vanilla Magento instance

❗ Automated tests can be triggered manually with an appropriate comment:

@magento run all tests - run or re-run all required tests against the PR changes
@magento run <test-build(s)> - run or re-run specific test build(s)
For example: @magento run Unit Tests

<test-build(s)> is a comma-separated list of build names.

Allowed build names are:

Database Compare
Functional Tests CE
Functional Tests EE
Functional Tests B2B
Integration Tests
Magento Health Index
Sample Data Tests CE
Sample Data Tests EE
Sample Data Tests B2B
Static Tests
Unit Tests
WebAPI Tests
Semantic Version Checker

You can find more information about the builds here
ℹ️ Run only required test builds during development. Run all test builds before sending your pull request for review.

For more details, review the Code Contributions documentation.
Join Magento Community Engineering Slack and ask your questions in #github channel.

rostilos · 2023-12-25T19:35:45Z

@magento run all tests

magento-automated-testing · 2023-12-25T19:35:55Z

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please message the #magento-devops slack channel if they don't show in a reasonable amount of time and a representative will look into any issues.

rostilos · 2023-12-27T08:22:32Z

@magento run all tests

magento-automated-testing · 2023-12-27T08:22:38Z

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please message the #magento-devops slack channel if they don't show in a reasonable amount of time and a representative will look into any issues.

app/code/Magento/Catalog/etc/db_schema.xml

ihor-sviziev · 2023-12-27T11:41:21Z

According to failed tests, it works fine for Magento Open Source but won't work for Magento Commerce / B2B (conflicts with the Content Staging feature).
Do you think you'll be able to make it compatible?

rostilos · 2023-12-27T12:19:08Z

Yeah, I'll try to fix that
But to be honest, don't have much experience with B2B and EE versions

Co-authored-by: Ihor Sviziev <[email protected]>

rostilos · 2023-12-29T00:47:59Z

@magento run Database Compare

magento-automated-testing · 2023-12-29T00:48:12Z

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please message the #magento-devops slack channel if they don't show in a reasonable amount of time and a representative will look into any issues.

rostilos · 2023-12-29T08:37:55Z

@ihor-sviziev
Proposal for a fix.
I can't make compatibility with B2B and EE modules through edits in this repository. But, if edits are made, maybe you could make edits for DB schema in magento-catalog-staging module ?

<index referenceId="CATALOG_PRODUCT_ENTITY_INT_ATTRIBUTE_ID_ENTITY_ID" disabled="1"/>
<index referenceId="CATALOG_PRODUCT_ENTITY_INT_ATTRIBUTE_ID_ROW_ID" indexType="btree">
     <column name="attribute_id"/>
     <column name="row_id"/>
</index>

ihor-sviziev · 2024-01-03T07:45:05Z

@engcom-Hotel maybe you can help with fixing b2b/ee part?

tdgroot · 2024-01-30T08:21:01Z

I'm in favor of this PR, but don't we also need to keep the standalone ATTRIBUTE_ID index? Surely some functionality somewhere depends on that index to exist, so to me this seems like a regression.

engcom-Echo · 2024-01-31T05:33:40Z

@magento run all tests

Den4ik

Hi @engcom-Hotel
Looks like comment #38316 (comment) from @tdgroot were missed.
I'm agree with it and update looks like regression. We should add new index instead of update exist one.

engcom-Hotel · 2025-03-26T07:47:49Z

Valid point made by @tdgroot here #38316 (comment).

Thanks @Den4ik for highlighting this. Moving this PR in Changes Requested bucket for the changes. @rostilos please do the needful for the same.

Thanks

rostilos · 2025-03-26T11:29:11Z

@engcom-Hotel I agree with the comments above. Ok, will add this soon and test it from my end

engcom-Hotel · 2025-04-01T13:37:31Z

Thanks @rostilos, please let us know when are done with the changes. Meanwhile moving this PR as 'draft'.

rostilos · 2025-04-03T10:27:33Z

@engcom-Hotel added requested edits

Also, I remind you that the corresponding changes should be added to the EE version
#38316 (comment)

rostilos · 2025-04-03T10:29:12Z

@magento run all tests

Den4ik · 2025-04-07T21:12:03Z

@magento run all tests

Den4ik

@engcom-Hotel Changes looks good. Please assist with updates for EE and B2B

engcom-Charlie · 2025-04-08T06:05:42Z

@magento run all tests

engcom-Charlie · 2025-04-08T08:44:21Z

@magento run Functional Tests B2B, Functional Tests EE, Static Tests, Unit Tests

engcom-Charlie · 2025-04-08T11:52:53Z

Hi @rostilos,

Thank you for your contribution!

@ihor-sviziev and @Den4ik thank you for the review.

The mentioned changes have been already taken care in the attached Related PR https://github.com/magento-gl/magento2ee/pull/271, please find the changes below and the functional EE test is green.

I have fixed the static test failure in the same above EE PR.

I also dont see any need for these approval JIRAs as the SVC and Database compare are working fine now.

The Functional B2B failure is not consistent in recent 2 builds and it seems to be flaky.

Run 1:
https://public-results-storage-prod.magento-testing-service.engineering/reports/magento/magento2/pull/38316/207e4253d63539f4eb5a9164056517cf/Functional/allure-report-b2b/index.html#categories

Run 2:
https://public-results-storage-prod.magento-testing-service.engineering/reports/magento/magento2/pull/38316/da1c806f1b255e289be017ddc771cb18/Functional/allure-report-b2b/index.html#categories

@ihor-sviziev requesting you for review once so that we can able to proceed further on this PR. Please let me know if anything else is needed.

engcom-Charlie · 2025-04-09T05:19:43Z

@magento run Static Tests

engcom-Charlie · 2025-04-09T05:27:29Z

Thank you @ihor-sviziev for the quick check and approval.

QA has been done as mentioned here. Hence moving the PR to Extended Testing to look into the static failure.

engcom-Charlie · 2025-04-09T08:48:19Z

Static test is green now and B2B is flaky as mentioned above. Hence moving this PR to Merge in Progress.

fix for magento#38315 issue (Missing indexing of required columns in …

cec2444

…catalog_product_entity_int table)

m2-github-services added Partner: Perspective Team partners-contribution Pull Request is created by Magento Partner labels Dec 25, 2023

m2-community-project bot added the Progress: pending review label Dec 25, 2023

Merge branch '2.4-develop' into fix-for-issue-38315

f99fd4e

engcom-Hotel added the Priority: P2 A defect with this priority could have functionality issues which are not to expectations. label Dec 26, 2023

ihor-sviziev requested changes Dec 27, 2023

View reviewed changes

app/code/Magento/Catalog/etc/db_schema.xml Show resolved Hide resolved

m2-community-project bot assigned ihor-sviziev Dec 27, 2023

m2-community-project bot added Progress: needs update and removed Progress: pending review labels Dec 27, 2023

ihor-sviziev added the Area: Performance label Dec 27, 2023

rostilos and others added 2 commits December 28, 2023 18:26

Update app/code/Magento/Catalog/etc/db_schema.xml

5e332f5

Co-authored-by: Ihor Sviziev <[email protected]>

db_schema_whitelist updates for Magento_Catalog

4638b35

rostilos requested a review from ihor-sviziev December 29, 2023 14:46

ihor-sviziev added the help wanted label Jan 3, 2024

sinhaparul added the Project: Community Picked PRs upvoted by the community label Jan 25, 2024

Merge branch '2.4-develop' into fix-for-issue-38315

b2dee17

ihor-sviziev removed their request for review February 19, 2025 11:04

Den4ik requested changes Mar 25, 2025

View reviewed changes

engcom-Hotel moved this from Pending Approval to Changes Requested in Community Dashboard Mar 26, 2025

engcom-Hotel added the Progress: needs update label Mar 26, 2025

engcom-Hotel marked this pull request as draft April 1, 2025 13:37

magento#38315 fixes ( keep prev. index to prevent regression )

bdfe463

rostilos marked this pull request as ready for review April 3, 2025 10:27

Merge branch '2.4-develop' into fix-for-issue-38315

cd6c81d

engcom-Charlie self-assigned this Apr 7, 2025

Merge branch '2.4-develop' into fix-for-issue-38315

b83f427

Den4ik approved these changes Apr 7, 2025

View reviewed changes

engcom-Charlie requested a review from ihor-sviziev April 8, 2025 11:52

engcom-Charlie moved this from Changes Requested to Pending Review in Community Dashboard Apr 8, 2025

ihor-sviziev approved these changes Apr 8, 2025

View reviewed changes

engcom-Charlie moved this from Pending Review to Extended testing (optional) in Community Dashboard Apr 9, 2025

engcom-Charlie moved this from Extended testing (optional) to Merge in Progress in Community Dashboard Apr 9, 2025

magento-devops-reposync-svc merged commit 0ac260f into magento:2.4-develop Apr 25, 2025
11 of 12 checks passed

ct-prd-projects-boards-automation bot moved this from Merge in Progress to Recently Merged in Community Dashboard Apr 25, 2025

fix for #38315 issue (Missing indexing of required columns in catalog_product_entity_int table) #38316

fix for #38315 issue (Missing indexing of required columns in catalog_product_entity_int table) #38316

Uh oh!

Conversation

rostilos commented Dec 25, 2023 • edited by engcom-Echo Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description (*)

Related Pull Requests

Fixed Issues (if relevant)

Manual testing scenarios (*)

Questions or comments

Contribution checklist (*)

Uh oh!

m2-assistant bot commented Dec 25, 2023

Uh oh!

rostilos commented Dec 25, 2023

Uh oh!

magento-automated-testing bot commented Dec 25, 2023

Uh oh!

rostilos commented Dec 27, 2023

Uh oh!

magento-automated-testing bot commented Dec 27, 2023

Uh oh!

Uh oh!

ihor-sviziev commented Dec 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rostilos commented Dec 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rostilos commented Dec 29, 2023

Uh oh!

magento-automated-testing bot commented Dec 29, 2023

Uh oh!

rostilos commented Dec 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ihor-sviziev commented Jan 3, 2024

Uh oh!

tdgroot commented Jan 30, 2024

Uh oh!

engcom-Echo commented Jan 31, 2024

Uh oh!

Den4ik left a comment

Choose a reason for hiding this comment

Uh oh!

engcom-Hotel commented Mar 26, 2025

Uh oh!

rostilos commented Mar 26, 2025

Uh oh!

engcom-Hotel commented Apr 1, 2025

Uh oh!

rostilos commented Apr 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rostilos commented Apr 3, 2025

Uh oh!

Den4ik commented Apr 7, 2025

Uh oh!

Den4ik left a comment

Choose a reason for hiding this comment

Uh oh!

engcom-Charlie commented Apr 8, 2025

Uh oh!

engcom-Charlie commented Apr 8, 2025

Uh oh!

engcom-Charlie commented Apr 8, 2025

Uh oh!

engcom-Charlie commented Apr 9, 2025

Uh oh!

engcom-Charlie commented Apr 9, 2025

Uh oh!

engcom-Charlie commented Apr 9, 2025

Uh oh!

Uh oh!

Uh oh!

rostilos commented Dec 25, 2023 •

edited by engcom-Echo

Loading

ihor-sviziev commented Dec 27, 2023 •

edited

Loading

rostilos commented Dec 27, 2023 •

edited

Loading

rostilos commented Dec 29, 2023 •

edited

Loading

rostilos commented Apr 3, 2025 •

edited

Loading