Merge pull request alfa-jpn#11 from hawknewton/upstream_safe_disconnect

Provide a disconnect only option that is safe to use with transactions (with changes)

Author: alfa-jpn

README.md

@@ -22,15 +22,16 @@ Or install it yourself as:
     $ gem install mysql2-aurora
 
 ## Usage
-This gem extends Mysql2::Client. You can use `aurora_max_retry` option.
+This gem extends Mysql2::Client. You can use `aurora_max_retry` and `aurora_disconnect_on_readonly` options.
 
 ```ruby
 Mysql2::Client.new(
   host:             'localhost',
   username:         'root',
   password:         'change_me',
   reconnect:        true,
-  aurora_max_retry: 5
+  aurora_max_retry: 5,
+  aurora_disconnect_on_readonly: true
 )
 ```
 
@@ -44,8 +45,39 @@ development:
   password:         change_me
   reconnect:        true
   aurora_max_retry: 5
+  aurora_disconnect_on_readonly: true
+
 ```
 
+There are essentially two methods for handling and RDS Aurora failover. When there is an Aurora RDS failover event the primary writable server can change it's role to become a read_only replica. This can happen without active database connections droppping.
+This leaves the connection in a state where writes will fail, but the application belives it's connected to a writeable server. Writes will now perpetually fail until the database connection is closed and re-established connecting back to the new primary.
+
+To provide automatic recovery from this method you can use either a graceful retry, or an immediate disconnection option.
+
+### Retry
+
+Setting aurora_max_retry, mysql2 will not disconnect and automatically attempt re-connection to the database when a read_only error message is encountered.
+This has the benefit that to the application the error is transparent and the query will be re-run against the new primary when the connection succeeds.
+
+It is however not safe to use with transactions
+
+Consider:
+
+* Transaction is started on the primary server A
+* Failover event occurs, A is now readonly
+* Application issues a write statement, read_only exception is thrown
+* mysql2-aurora gem handles this by reconnecting transparently to the new primary B
+* Aplication continues issuing writes however on a new connection in auto-commit mode, no new transaction was started
+
+The application remains un-aware it is now operating outside of a transaction, this can leave data in an inconcistent state, and issuing a ROLLBACK, or COMMIT will not have the expected outcome.
+
+### Immediate disconnect
+
+Setting aurora_disconnect_on_readonly to true, will cause mysql2 to close the connection to the database on read_only exception. The original exception will be thrown up the stack to the application.
+With the database connection disconnected, the next statement will hit the disconnected error and the application can handle this as it would normally when been disconnected from the database.
+
+This is safe with transactions.
+
 ## Contributing
 
 Bug reports and pull requests are welcome on GitHub at https://github.com/alfa-jpn/mysql2-aurora.

lib/mysql2/aurora.rb

@@ -13,9 +13,11 @@ class Client
       # @note [Override] with reconnect options
       # @param [Hash] opts Options
       # @option opts [Integer] aurora_max_retry Max retry count, when failover. (Default: 5)
+      # @option opts [Bool] aurora_disconnect_on_readonly, when readonly exception hit terminate the connection (Default: false)
       def initialize(opts)
-        @opts      = Mysql2::Util.key_hash_as_symbols(opts)
+        @opts = Mysql2::Util.key_hash_as_symbols(opts)
         @max_retry = @opts.delete(:aurora_max_retry) || 5
+        @disconnect_only = @opts.delete(:aurora_disconnect_on_readonly) || false
         reconnect!
       end
 
@@ -27,19 +29,23 @@ def query(*args)
         begin
           client.query(*args)
         rescue Mysql2::Error => e
+          raise e unless e.message&.include?('--read-only')
+
           try_count += 1
 
-          if e.message&.include?('--read-only') && try_count <= @max_retry
+          if @disconnect_only
+            warn '[mysql2-aurora] Database is readonly, Aurora failover event likely occured, closing database connection'
+            disconnect!
+          elsif try_count <= @max_retry
             retry_interval_seconds = [1.5 * (try_count - 1), 10].min
 
             warn "[mysql2-aurora] Database is readonly. Retry after #{retry_interval_seconds}seconds"
             sleep retry_interval_seconds
             reconnect!
-
             retry
-          else
-            raise e
           end
+
+          raise e
         end
       end
 
@@ -48,16 +54,19 @@ def query(*args)
       def reconnect!
         query_options = (@client&.query_options&.dup || {})
 
-        begin
-          @client&.close
-        rescue StandardError
-          nil
-        end
+        disconnect!
 
         @client = Mysql2::Aurora::ORIGINAL_CLIENT_CLASS.new(@opts)
         @client.query_options.merge!(query_options)
       end
 
+      # Close connection to database server
+      def disconnect!
+        @client&.close
+      rescue StandardError
+        nil
+      end
+
       # Delegate method call to client.
       # @param [String] name  Method name
       # @param [Array]  args  Method arguments

spec/mysql2/aurora_spec.rb

@@ -1,13 +1,16 @@
 RSpec.describe Mysql2::Aurora::Client do
   let :client do
     Mysql2::Client.new(
-      host:             ENV['TEST_DB_HOST'],
-      username:         ENV['TEST_DB_USER'],
-      password:         ENV['TEST_DB_PASS'],
-      aurora_max_retry: 10
+      host:                          ENV['TEST_DB_HOST'],
+      username:                      ENV['TEST_DB_USER'],
+      password:                      ENV['TEST_DB_PASS'],
+      aurora_max_retry:              10,
+      aurora_disconnect_on_readonly: aurora_disconnect_on_readonly
     )
   end
 
+  let(:aurora_disconnect_on_readonly) { false }
+
   describe 'Mysql2::Aurora::VERSION' do
     subject do
       Mysql2::Aurora::VERSION
@@ -49,6 +52,26 @@
   end
 
   describe '#query' do
+    context 'When aurora_disconnect_on_readonly is true' do
+      let(:aurora_disconnect_on_readonly) { true }
+
+      before :each do
+        allow(client).to receive(:warn)
+        allow(client.client).to receive(:query).and_raise(Mysql2::Error, 'ERROR 1290 (HY000): The MySQL server is running with the --read-only option so it cannot execute this statement')
+      end
+
+      subject do
+        client.query('SELECT CURRENT_USER() AS user')
+      end
+
+      describe '#query' do
+        it 'disconnects immediately' do
+          expect(client).to receive(:disconnect!)
+          expect { subject }.to raise_error(Mysql2::Error)
+        end
+      end
+    end
+
     subject do
       client.query('SELECT CURRENT_USER() AS user')
     end