AwsWrapperDataSource should implement Closable

[ivankhoosty]

Describe the bug

We are using the driver in multiregion setup and when the cluster moves from active to passive, it’s not possible to shutdown the AWS datasource. All the monitoring threads continue to run and try to connect to db , so even if we close the internal hikari pools, they get started again.

Expected Behavior

All internal hikari pools are closed and all driver threads are stopped

What plugins are used? What other connection properties were set?

Iam, efm2

Current Behavior

All threads continue to run, internal hikari pools remain open

Reproduction Steps

Not possible to close the datasource

Possible Solution

No response

Additional Information/Context

No response

The AWS Advanced JDBC Driver version used

2.6.0

JDK version used

21

Operating System and version

Linux

[sergiyvamz]

Hi @ivankhoosty

Thank you for raising this issue. The driver maintains different monitoring threads for monitoring purposes, and its standard behavior is to close these monitoring threads and their associated monitoring connections after a period of inactivity. For example, the Host Monitoring Plugin (efm2) stops monitoring a particular node after 10 minutes.

aws-advanced-jdbc-wrapper/wrapper/src/main/java/software/amazon/jdbc/plugin/efm2/HostMonitorServiceImpl.java

Line 44 in 054938b

public static final AwsWrapperProperty MONITOR_DISPOSAL_TIME_MS =

You can configure this inactivity disposal time using the corresponding configuration parameter.
https://github.com/aws/aws-advanced-jdbc-wrapper/blob/main/docs/using-the-jdbc-driver/using-plugins/UsingTheHostMonitoringPlugin.md

Similar logic is implemented for other monitoring threads and plugins.

With the internal connection pool configuration, it appears that closing an existing pool is an overlooked scenario. The driver does not provide any method to close a specific internal connection pool. You may attempt to call HikariPooledConnectionProvider.releaseResources(), but this might cause disruption to your application since it closes connection pools for both active and inactive clusters. You'll need to store a reference to HikariPooledConnectionProvider instance when your application initializes an internal connection pool to be able to clean it up when needed. Also, ConnectionProviderManager.releaseResources() can be called to achieve the same results.
https://github.com/aws/aws-advanced-jdbc-wrapper/blob/main/examples/AWSDriverExample/src/main/java/software/amazon/InternalConnectionPoolPasswordWarning.java

Alternatively, you may consider calling Driver.clearCaches() to stop all monitoring threads and clean up all internal driver caches.

Implementing a Closeable interface for the AwsWrapperDataSource class is definitely a possible solution. However, the main question is determining the desired behavior. There might be other instances of the AwsWrapperDataSource class held by the user application, so I assume that calling close() on one instance should not affect the others. In the case of internal connection pools, closing an instance of AwsWrapperDataSource should not close the associated connection pool(s) since they may be in use by other instances of AwsWrapperDataSource. Implementing Closeable does not appear to be an effective solution to this issue.

We will continue working on this issue. I will keep you informed of our progress.

[ivankhoosty]

thank you @sergiyvamz for the very detailed response.

For some context, we use the AWS driver with RDS Aurora global db with 2 regions. The driver is documented to only work in single region, so we came up with a replaceable/swappable datasource shim that changes the datasource if the region changes state from active to passive or the other way around. For active region, we use aws driver with all the plugins, and for passive region we just use hikari datasource with aws driver only using iam plugin. When we switch from active to passive, we need to "shutdown/close" the aws driver completely. I found HikariPoolsHolder.closeAllPools() as I am familiar with that part of the code, but it was not enough. With your help above, I just tried Driver.releaseResources() and it does exactly what I need - complete shutdown of all threads and pools.

Clearly this might not be an acceptable solution for anyone using multiple aws datasources in the same application, and implementing tracking which pools each aws datasources uses in order to implement Closable requires more work from your side.

PS the reason we need to shutdown the driver in passive region is because the topology pulls the writer node from db which is actually in another region, and then constructs url like node_from_active_region.url_from_passive_region, which obviously doesn't exist.