optimize complexity of filter out unwanted recognizers from O(n*m ) to O(n)

[rgupta2508]

Reopen this PR #1508
Creating map of supported entity and recognizers and filtering out based on entity name using kay .

Code refactor.

Improve complexity of filter out unwanted recognizers from O(n*m )to O(n)

In current code for loop is running inside for loop that makes complexity to O(n*m). after this change there is only one loop by using one extra variable.
where
m = total number of all_possible_recognizers
n= total supported entities which we want to include .

Change Description

Describe your changes

Issue reference

This PR fixes issue #XX

Checklist

• I have reviewed the contribution guidelines
• I have signed the CLA (if required)
• My code includes unit tests
• All unit tests and lint checks pass locally
• My PR contains documentation updates / additions if required

[rgupta2508]

@microsoft-github-policy-service agree

[omri374]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[Copilot]

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

[github-advanced-security]

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

[github-advanced-security]

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

[omri374]

@rgupta2508 after some thinking, we've decided to close this PR. We appreciate the change, but in the tradeoff between computational efficiency and code complexity, we've decided that it's a bit risky. The number of combinations here (entities, languages, recognizers) is not high therefore we don't expect a significant performance boost here. We would be happy to be corrected otherwise if you did any analysis on this.

[rgupta2508]

Hi @omri374,
Thanks for review.
We s a walmart team, having lot of recognizers (multiple entity in one recognizers) in different language.
We are using this for each llm calls( more then 20 million call a day). so each request having different recognizers/entity(based on the usecase) in different language .
We observed significant improvement wrt time to take run for these data. and our llm prompt response time also improved .

[omri374]

Hi @rgupta2508

Thanks for this input! We will continue reviewing this and get back to you.

[rgupta2508]

Hi @omri374 Any updates regarding this PR?

[omri374]

hi @rgupta2508, apologies for the delay

[omri374]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[Copilot]

Copilot reviewed 1 out of 1 changed files in this pull request and generated no comments.

[omri374]

Can you please make this private + add type hints?

[omri374]

Furthermore, there are a few linting issues as can be seen here: https://dev.azure.com/csedevil/Presidio/_build/results?buildId=16970&view=logs&jobId=248a81d7-74ff-5424-a6bb-5f1f442ace41&j=248a81d7-74ff-5424-a6bb-5f1f442ace41&t=b4a88934-08b9-5ac5-8cd9-057bdd5531eb

Consider using ruff to test