Skip to content

Improve API consumption of _handle_bulk_insert_op #4091

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 27, 2025

Conversation

mr0re1
Copy link
Collaborator

@mr0re1 mr0re1 commented May 9, 2025

  • Instead of doing globalOperations.aggregateList perform series (usually just 1) of zonal list requests - to stop hitting HeavyWeightReadRequestsPerMinutePerProject limits;
  • Do not perform server-side filtering on non-indexed fields, instead do it client-side to avoid ListRequestsFilterCostOverheadPerMinutePerProjectPerRegion limits;
  • Incorporate createTime range and user (both are indexed) into server-side filtering.

New approach consumes 0 - filtered_list_cost_overhead, while old one was effectively churring (on server side) through all operations.

Latency is smaller but not significantly (~2 times improvement) - no thorough sampling has been done.

See: https://cloud.google.com/compute/api-quota#api-rate-limits

@mr0re1 mr0re1 requested review from samskillman and a team as code owners May 9, 2025 04:51
@mr0re1 mr0re1 added the release-chore To not include into release notes label May 9, 2025
@mr0re1 mr0re1 changed the title Improve API consumption while _handle_bulk_insert_op Improve API consumption of _handle_bulk_insert_op May 9, 2025
Copy link
Contributor

@tpdownes tpdownes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm concerned we are missing something in the case when multiple allowed zones are set by the user.

@tpdownes tpdownes assigned mr0re1 and unassigned tpdownes May 14, 2025
@tpdownes tpdownes self-requested a review May 14, 2025 18:42
@tpdownes tpdownes assigned tpdownes and unassigned mr0re1 May 14, 2025
@mr0re1 mr0re1 assigned mr0re1 and unassigned tpdownes May 14, 2025
@mr0re1 mr0re1 enabled auto-merge May 27, 2025 21:53
@mr0re1 mr0re1 merged commit 7c5bd5b into GoogleCloudPlatform:develop May 27, 2025
11 of 63 checks passed
@mr0re1 mr0re1 deleted the bi_trim branch May 27, 2025 22:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release-chore To not include into release notes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants