ActiveRecord connection sharing: connection is returned to the pool

[kolen]

When ActiveRecord integration is enabled, ConnAdapter is constructed with connection taken from ActiveRecord pool:

https://github.com/QueueClassic/queue_classic/blob/4260d8963ddef91bdb9ae390d93e811521b83350/lib/queue_classic.rb#L57

However, according to ActiveRecord documentation, it's not a correct way to do it, as the connection will be returned to the pool by Action Pack at the end of request:

Simply use ActiveRecord::Base.connection as with Active Record 2.1 and earlier (pre-connection-pooling). Eventually, when you're done with the connection(s) and wish it to be returned to the pool, you call ActiveRecord::Base.clear_active_connections!. This will be the default behavior for Active Record when used in conjunction with Action Pack's request handling cycle.

As ConnAdapter is not going to release its connection, ActiveRecord::Base.connection_pool.checkout should be used.

This might cause PG::ConnectionBad: connection is closed problem described here: https://github.com/QueueClassic/queue_classic/issues/302#issuecomment-491861292, which I have regularly too. (Also there were old issues from the time ActiveRecord integration was not supported: #141, #134, #96). Pool may close connections returned to it, and it may close connection that appears to be shared between QC::ConnAdapter and the pool after that accidental connection release.

Proposed change is:

--- a/lib/queue_classic.rb
+++ b/lib/queue_classic.rb
@@ -59,7 +59,7 @@
   def self.default_conn_adapter
     return @conn_adapter if defined?(@conn_adapter) && @conn_adapter
     if rails_connection_sharing_enabled?
-      @conn_adapter = ConnAdapter.new(ActiveRecord::Base.connection.raw_connection)
+      @conn_adapter = ConnAdapter.new(ActiveRecord::Base.connection_pool.checkout.raw_connection)
     else
       @conn_adapter = ConnAdapter.new
     end

However, I don't know yet how to make reproducible test case, as it involves pool state.

[kolen]

Activerecord pool may close connections in e.g. flush.

Closed connection state is distinct from just disconnected (i.e. by TCP connection reset). Closed connection can't be reconnected by reset method used by ConnAdapter:

> conn.close
=> nil
> conn.exec "select 1 from ads;"
PG::ConnectionBad: connection is closed
> conn.reset
PG::ConnectionBad: connection is closed

That's why once this error happens, it persists for the lifetime of ConnAdapter (and therefore of queue).

[kolen]

Note that it only causes problems with QC.enqueue from Rails web app. It does not cause problems with worker, as long as tasks don't do ActiveRecord::Base.clear_active_connections!.

[kolen]

Seems that just replacing ActiveRecord::Base.connection.raw_connection with ActiveRecord::Base.connection_pool.checkout.raw_connection, as I proposed, will not work reliably: ActiveRecord has "reaping" feature that returns checked out connections to pool.

It decides to recover connection back if thread that checked out the connection is no longer alive. In queue_classic default queue's connection is effectively global:

• default_conn_adapter is thread-local memoized value:

  https://github.com/QueueClassic/queue_classic/blob/4260d8963ddef91bdb9ae390d93e811521b83350/lib/queue_classic.rb#L53-L54

• but default_queue is global:

  https://github.com/QueueClassic/queue_classic/blob/4260d8963ddef91bdb9ae390d93e811521b83350/lib/queue_classic/config.rb#L31-L33

• and Queue instance memoizes connection adapter (and connection adapter holds connection):

  https://github.com/QueueClassic/queue_classic/blob/4260d8963ddef91bdb9ae390d93e811521b83350/lib/queue_classic/queue.rb#L21-L23

So, when AR pool reaps connections, if thread in which QC was touched for the first time is dead, the connection will be stolen from ConnAdapter and will become shared with AR pool.

[kolen]

Probably better way to fix this would be to leave ActiveRecord::Base.connection.raw_connection untouched, but to remove memoization of @conn_adapter in Queue. However, it also has setter conn_adapter=, and keeping API compatibility will be trickier.

This will work for non-activerecord connections too, as conn adapter is already memoized in thread-local in QC.default_conn_adapter: no need to memoize it further in Queue.

(However, for ActiveRecord connections, ActiveRecord::Base.connection.raw_connection should never be memoized)