feat: Replace aiohttp.ClientSession with AlloyDBAdminAsyncClient #416

rhatgadkar-goog · 2025-01-27T19:24:53Z

This PR replaces calls to the AlloyDB admin API that use aiohttp.ClientSession to use the google.cloud.alloydb package's AlloyDBAdminAsyncClient instead.

The dependencies for this change were added in #428.

Did a performance test of this change by creating a script to create 500 connections. I ran the script twice: when using the AlloyDBAdminAsyncClient and when using aiohttp.ClientSession. Got similar results:

Using AlloyDBAdminAsyncClient:
Elapsed time: 1483.8013787269592 seconds

Using aiohttp.ClientSession:
Elapsed time: 1478.003143787384 seconds

Because the performance is very similar, we can use the AlloyDBAdminAsyncClient.

In the error case, for example, when passing an invalid instance URI, it was verified that the AlloyDBAdminAsyncClient returns a proper error message:

google.api_core.exceptions.NotFound: 404 Resource 'projects/jovial-evening-444518-a1/locations/us-east4/clusters/my-cluster-invalid/instances/my-cluster-primary' was not found

rhatgadkar-goog · 2025-01-27T19:26:26Z

noxfile.py

@@ -126,7 +126,7 @@ def cover(session):
 def default(session, path):
    # Install all test dependencies, then install this package in-place.
    session.install("-r", "requirements-test.txt")
-    session.install("-e", ".")
+    session.install(".")


Getting the following error if using session.install("-e", "."):

ImportError while loading conftest '/usr/local/google/home/rhatgadkar/alloydb-python-connector/tests/unit/conftest.py'. tests/unit/conftest.py:22: in <module> from mocks import FakeAlloyDBClient tests/unit/mocks.py:33: in <module> from google.cloud.alloydb.connector.connection_info import ConnectionInfo E ModuleNotFoundError: No module named 'google.cloud.alloydb.connector'

Maybe there’s a conflict when importing the google-cloud-alloydb-connector package when the google-cloud-alloydb package is installed. But when changing to session.install("."), this error doesn’t occur anymore.

rhatgadkar-goog · 2025-01-27T19:29:22Z

When trying to connect to an instance using this PR’s change, am getting the following error:

google.api_core.exceptions.RetryError: Timeout of 60.0s exceeded, last exception: 503 DNS resolution failed for https://alloydb.googleapis.com: C-ares status is not ARES_SUCCESS qtype=A name=https
 is_balancer=0: Domain name not found

Followed the work-around in telepresenceio/telepresence#1871 to use "native" GRPC DNS resolver, but still getting similar error:

google.api_core.exceptions.RetryError: Timeout of 60.0s exceeded, last exception: 503 DNS resolution failed for https://alloydb.googleapis.com: UNKNOWN: getaddrinfo("https://alloydb.googleapis.com"): Servname not supported for ai_socktype (-8)

Need to figure out how to resolve this error. Maybe it's related to the options we're passing when constructing AlloyDBAdminAsyncClient.

rhatgadkar-goog · 2025-01-28T17:23:36Z

When trying to connect to an instance using this PR’s change, am getting the following error:
google.api_core.exceptions.RetryError: Timeout of 60.0s exceeded, last exception: 503 DNS resolution failed for https://alloydb.googleapis.com: C-ares status is not ARES_SUCCESS qtype=A name=https
 is_balancer=0: Domain name not found
Followed the work-around in telepresenceio/telepresence#1871 to use "native" GRPC DNS resolver, but still getting similar error:
google.api_core.exceptions.RetryError: Timeout of 60.0s exceeded, last exception: 503 DNS resolution failed for https://alloydb.googleapis.com: UNKNOWN: getaddrinfo("https://alloydb.googleapis.com"): Servname not supported for ai_socktype (-8)
Need to figure out how to resolve this error. Maybe it's related to the options we're passing when constructing AlloyDBAdminAsyncClient.

Issue is now fixed in latest commit. When passing in the API endpoint in the ClientOptions object, the "https://" prefix needs to be removed.

rhatgadkar-goog · 2025-02-21T20:15:09Z

mypy.ini

Adding these because was getting an error when importing google.cloud.alloydb_v1beta and google.api_core.* libraries. Followed the approach here: https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-library-stubs-or-py-typed-marker

rhatgadkar-goog · 2025-02-21T20:16:49Z

tests/unit/test_packaging.py

-    google_cloud_alloydb = tmp_path / "google" / "cloud" / "alloydb"
-    google_cloud_alloydb.mkdir()
-    google_cloud_alloydb.joinpath("othermod.py").write_text("")
-    env = dict(os.environ, PYTHONPATH=str(tmp_path))


Removing this part of the test because it was failing. I think it fails because google.cloud.alloydb is not a namespace package anymore. When the othermod.py file is created in tmp_path, I think it doesn't look inside tmp_path when calling python -m google.cloud.alloydb.othermod, and it instead looks inside the directory where google.cloud.alloydb was installed.

Should this be a separate PR?

Yes, I've put this in a separate PR in #428.

rhatgadkar-goog · 2025-02-21T23:10:47Z

I did a performance test by creating a script to create 500 connections: connect.py.zip

I ran the script twice: when using the AlloyDBAdminAsyncClient and when using aiohttp.ClientSession. Got similar results:

Using AlloyDBAdminAsyncClient:
Elapsed time: 1483.8013787269592 seconds

Using aiohttp.ClientSession:
Elapsed time: 1478.003143787384 seconds

rhatgadkar-goog · 2025-03-10T23:57:26Z

Proper error message was returned when entering an invalid resource URI:

google.api_core.exceptions.NotFound: 404 Resource 'projects/jovial-evening-444518-a1/locations/us-east4/clusters/my-cluster-invalid/instances/my-cluster-primary' was not found

Similarly, an error is correctly returned when entering an invalid username:

pg8000.exceptions.DatabaseError: {'S': 'FATAL', 'V': 'FATAL', 'C': '28P01', 'M': 'password authentication failed for user "postgres2"', 'F': 'auth.c', 'L': '375', 'R': 'auth_failed'}

rhatgadkar-goog · 2025-03-12T01:06:58Z

noxfile.py

+        "xml",
+        "-o",
+        "sponge_log.xml",
+    )


Adding this change because was getting the following error in Windows unit tests:

Run nox -s unit-3.13 nox > Running session unit-3.13 nox > Creating virtual environment (virtualenv) using python.exe in .nox\unit-3-13 nox > python -m pip install -r requirements-test.txt nox > python -m pip install . nox > python -m pip install -r requirements.txt nox > pytest --cov=google.cloud.alloydb.connector -v --cov-config=.coveragerc --cov-report= --cov-fail-under=0 --junitxml=sponge_log.xml 'tests\unit' ImportError while loading conftest 'D:\a\alloydb-python-connector\alloydb-python-connector\tests\unit\conftest.py'. tests\unit\conftest.py:21: in <module> from mocks import FakeAlloyDBClient tests\unit\mocks.py:26: in <module> from cryptography import x509 .nox\unit-3-13\Lib\site-packages\cryptography\x509\__init__.py:7: in <module> from cryptography.x509 import certificate_transparency, verification .nox\unit-3-13\Lib\site-packages\cryptography\x509\certificate_transparency.py:8: in <module> from cryptography.hazmat.bindings._rust import x509 as rust_x509 E ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process nox > Command pytest --cov=google.cloud.alloydb.connector -v --cov-config=.coveragerc --cov-report= --cov-fail-under=0 --junitxml=sponge_log.xml 'tests\unit' failed with exit code 4 nox > Session unit-3.13 failed. Error: Process completed with exit code 1.

Followed the workaround in pytest-dev/pytest-cov#614 to solve this issue. It states that the module is being initialized twice because of the --cov parameter in pytest. So it suggests to use the coverage run command instead with the --include parameter.

Let's leave a comment in the code so future maintainers don't need to rediscover this.

Done. I've added the comment in #428

enocom · 2025-03-12T18:36:36Z

google/cloud/alloydb/connector/async_connector.py

@@ -235,5 +235,3 @@ async def close(self) -> None:
        """Helper function to cancel RefreshAheadCaches' tasks
        and close client."""
        await asyncio.gather(*[cache.close() for cache in self._cache.values()])
-        if self._client:


Is this no longer necessary? I would expect (at least from Go), that the client needs to be closed.

The client doesn't have a close method: https://github.com/googleapis/google-cloud-python/blob/main/packages/google-cloud-alloydb/google/cloud/alloydb_v1beta/services/alloy_db_admin/async_client.py

enocom · 2025-03-12T18:37:56Z

noxfile.py

+        "xml",
+        "-o",
+        "sponge_log.xml",
+    )


Let's leave a comment in the code so future maintainers don't need to rediscover this.

enocom · 2025-03-12T18:39:03Z

tests/unit/test_packaging.py

-    google_cloud_alloydb = tmp_path / "google" / "cloud" / "alloydb"
-    google_cloud_alloydb.mkdir()
-    google_cloud_alloydb.joinpath("othermod.py").write_text("")
-    env = dict(os.environ, PYTHONPATH=str(tmp_path))


Should this be a separate PR?

jackwotherspoon · 2025-03-12T18:45:47Z

Proper error message was returned when entering an invalid resource URI:

google.api_core.exceptions.NotFound: 404 Resource 'projects/jovial-evening-444518-a1/locations/us-east4/clusters/my-cluster-invalid/instances/my-cluster-primary' was not found

I would maybe also test 403 error for not having client permissions? That is an important one.

rhatgadkar-goog · 2025-03-17T17:45:47Z

Proper error message was returned when entering an invalid resource URI:
google.api_core.exceptions.NotFound: 404 Resource 'projects/jovial-evening-444518-a1/locations/us-east4/clusters/my-cluster-invalid/instances/my-cluster-primary' was not found
I would maybe also test 403 error for not having client permissions? That is an important one.

403 error message is returned a proper error message too:

google.api_core.exceptions.PermissionDenied: 403 Permission 'alloydb.clusters.generateClientCertificate' denied on 'projects/jovial-evening-444518-a1/locations/us-east4/clusters/my-cluster'

jackwotherspoon · 2025-03-17T18:10:36Z

403 error message is returned a proper error message too:

google.api_core.exceptions.PermissionDenied: 403 Permission 'alloydb.clusters.generateClientCertificate' denied on 'projects/jovial-evening-444518-a1/locations/us-east4/clusters/my-cluster'

This is not an actionable error message, if this the actual error AlloyDB rest API is returning then you may want to file an internal bug to improve error message. Error message should point at missing permission or missing API enablement etc so customer can act on it.

rhatgadkar-goog · 2025-03-17T18:21:58Z

403 error message is returned a proper error message too:
google.api_core.exceptions.PermissionDenied: 403 Permission 'alloydb.clusters.generateClientCertificate' denied on 'projects/jovial-evening-444518-a1/locations/us-east4/clusters/my-cluster'
This is not an actionable error message, if this the actual error AlloyDB rest API is returning then you may want to file an internal bug to improve error message. Error message should point at missing permission or missing API enablement etc so customer can act on it.

alloydb.clusters.generateClientCertificate is the missing permission. We're getting this error, because I removed roles/alloydb.client from the service account. This permission is part of roles/alloydb.client.

…kaging.py

These tests were for #400. But those changes are being removed, so we can remove those tests.

rhatgadkar-goog linked an issue Jan 27, 2025 that may be closed by this pull request

Switch to generated AlloyDBAdminAsyncClient over aiohttp.ClientSession #221

Open

rhatgadkar-goog commented Jan 27, 2025

View reviewed changes

rhatgadkar-goog force-pushed the 221-switch-to-generated-alloydbadminasyncclient-over-aiohttpclientsession branch from 44d084b to 197816e Compare February 21, 2025 19:40

rhatgadkar-goog commented Feb 21, 2025

View reviewed changes

rhatgadkar-goog force-pushed the 221-switch-to-generated-alloydbadminasyncclient-over-aiohttpclientsession branch 3 times, most recently from bbcea6b to bb1285a Compare March 12, 2025 00:26

rhatgadkar-goog commented Mar 12, 2025

View reviewed changes

rhatgadkar-goog changed the title ~~WIP: replace aiohttp.ClientSession with AlloyDBAdminAsyncClient~~ Replace aiohttp.ClientSession with AlloyDBAdminAsyncClient Mar 12, 2025

rhatgadkar-goog marked this pull request as ready for review March 12, 2025 18:27

rhatgadkar-goog requested a review from a team as a code owner March 12, 2025 18:27

blunderbuss-gcf bot assigned enocom Mar 12, 2025

rhatgadkar-goog changed the title ~~Replace aiohttp.ClientSession with AlloyDBAdminAsyncClient~~ feat: Replace aiohttp.ClientSession with AlloyDBAdminAsyncClient Mar 12, 2025

rhatgadkar-goog assigned jackwotherspoon Mar 12, 2025

enocom reviewed Mar 12, 2025

View reviewed changes

rhatgadkar-goog mentioned this pull request Mar 17, 2025

chore: add dependencies for AlloyDBAdminAsyncClient library #428

Merged

enocom self-requested a review March 17, 2025 19:44

enocom approved these changes Mar 17, 2025

View reviewed changes

rhatgadkar-goog requested a review from jackwotherspoon March 17, 2025 19:53

rhatgadkar-goog added 2 commits March 17, 2025 19:56

WIP: replace aiohttp.ClientSession with AlloyDBAdminAsyncClient

8fdc7d7

Fix '503 DNS resolution failed' error

72a4814

rhatgadkar-goog added 9 commits March 17, 2025 19:56

Fix unit tests

2bcd203

Set user_agent as member variable, remove failing subtest in test_pac…

2d1493a

…kaging.py

fix lint errors

65e3397

Remove close() from AlloyDBClient because it is a no-op

c498957

Remove unneeded tests

364c5b1

These tests were for #400. But those changes are being removed, so we can remove those tests.

Remove unused import to retryerror and rename to import as

32cb0b5

Don't import cryptography.x509

edf3372

downgrade cryptography version

a3e3bda

Remove --cov parameter in pytest

a3800ba

rhatgadkar-goog force-pushed the 221-switch-to-generated-alloydbadminasyncclient-over-aiohttpclientsession branch from 530a338 to a3800ba Compare March 17, 2025 20:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Replace aiohttp.ClientSession with AlloyDBAdminAsyncClient #416

feat: Replace aiohttp.ClientSession with AlloyDBAdminAsyncClient #416

rhatgadkar-goog commented Jan 27, 2025 •

edited

Loading

rhatgadkar-goog Jan 27, 2025 •

edited

Loading

rhatgadkar-goog commented Jan 27, 2025

rhatgadkar-goog commented Jan 28, 2025

rhatgadkar-goog Feb 21, 2025

rhatgadkar-goog Feb 21, 2025 •

edited

Loading

enocom Mar 12, 2025

rhatgadkar-goog Mar 17, 2025

rhatgadkar-goog commented Feb 21, 2025 •

edited

Loading

rhatgadkar-goog commented Mar 10, 2025

rhatgadkar-goog Mar 12, 2025 •

edited

Loading

enocom Mar 12, 2025

rhatgadkar-goog Mar 17, 2025

enocom Mar 12, 2025

rhatgadkar-goog Mar 17, 2025 •

edited

Loading

enocom Mar 12, 2025

enocom Mar 12, 2025

jackwotherspoon commented Mar 12, 2025

rhatgadkar-goog commented Mar 17, 2025

jackwotherspoon commented Mar 17, 2025

rhatgadkar-goog commented Mar 17, 2025

feat: Replace aiohttp.ClientSession with AlloyDBAdminAsyncClient #416

Are you sure you want to change the base?

feat: Replace aiohttp.ClientSession with AlloyDBAdminAsyncClient #416

Conversation

rhatgadkar-goog commented Jan 27, 2025 • edited Loading

rhatgadkar-goog Jan 27, 2025 • edited Loading

Choose a reason for hiding this comment

rhatgadkar-goog commented Jan 27, 2025

rhatgadkar-goog commented Jan 28, 2025

Choose a reason for hiding this comment

rhatgadkar-goog Feb 21, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rhatgadkar-goog commented Feb 21, 2025 • edited Loading

rhatgadkar-goog commented Mar 10, 2025

rhatgadkar-goog Mar 12, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rhatgadkar-goog Mar 17, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jackwotherspoon commented Mar 12, 2025

rhatgadkar-goog commented Mar 17, 2025

jackwotherspoon commented Mar 17, 2025

rhatgadkar-goog commented Mar 17, 2025

rhatgadkar-goog commented Jan 27, 2025 •

edited

Loading

rhatgadkar-goog Jan 27, 2025 •

edited

Loading

rhatgadkar-goog Feb 21, 2025 •

edited

Loading

rhatgadkar-goog commented Feb 21, 2025 •

edited

Loading

rhatgadkar-goog Mar 12, 2025 •

edited

Loading

rhatgadkar-goog Mar 17, 2025 •

edited

Loading