elasticsearch-ruby HTTPClient::KeepAliveDisconnected when the node request is timing out

I am initializing my connection like

Elasticsearch::Client.new hosts: ['https://<ip-of-node1>:9200', 'https://<ip-of-node2>:9200']. with following options - {reload_on_failure: true, reload_connections: true}.

Whenever there is request timeout on any of these nodes, I am getting the following traceback - Traceback (most recent call last): from faraday (0.9.2) lib/faraday/connection.rb:377:in run_request' from faraday (0.9.2) lib/faraday/rack_builder.rb:139:in build_response' from faraday (0.9.2) lib/faraday/adapter/httpclient.rb:36:in call' from httpclient (2.7.2) lib/httpclient.rb:858:in request' from httpclient (2.7.2) lib/httpclient.rb:1016:in do_request' from httpclient (2.7.2) lib/httpclient.rb:1127:in protect_keep_alive_disconnected' from httpclient (2.7.2) lib/httpclient.rb:1134:in rescue in protect_keep_alive_disconnected' from httpclient (2.7.2) lib/httpclient.rb:1021:in block in do_request' from httpclient (2.7.2) lib/httpclient.rb:1241:in do_get_block' from httpclient (2.7.2) lib/httpclient.rb:1294:in do_get_header' from httpclient (2.7.2) lib/httpclient/session.rb:547:in get_header' from httpclient (2.7.2) lib/httpclient/session.rb:771:in read_header' from httpclient (2.7.2) lib/httpclient/session.rb:788:in parse_header' from /opt/ruby-2.7.1/lib/ruby/2.7.0/timeout.rb:105:in timeout' from /opt/ruby-2.7.1/lib/ruby/2.7.0/timeout.rb:95:in block in timeout' from httpclient (2.7.2) lib/httpclient/session.rb:795:in block in parse_header' HTTPClient::KeepAliveDisconnected (HTTPClient::KeepAliveDisconnected: )

I see that in elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/http/faraday.rb, we have defined

def host_unreachable_exceptions [::Faraday::Error::ConnectionFailed, ::Faraday::Error::TimeoutError] end

then why is TimeoutError is not reloading connection, instead giving the above exception.

To verify this, I did a curl on the same node and the request did timeout but from the gem, i am facing some issues

Sep 17 '20 18:09 kushal-ladha

Came across this issue - https://github.com/elastic/elasticsearch-ruby/commit/68350f9f978badef57930aec35441f4d975f65bf

monkey-patched the 6.8.0 gem with the following commit but I still see the same error.

This only happens when I use reload_connections/reload_on_failure option while creating the connection. Whenever one of the host goes down, the request timeouts and the see the message -

` [2020-09-19T09:41:16.806831 #155817] ERROR -- : [Faraday::TimeoutError] Connection timed out - connect(2) for "10.1.14.130" port 9200 (10.1.14.130:9200) {:scheme=>"https", :user=>nil, :password=><REDACTED>, :host=>"x.x.x.x", :path=>"", :port=>9200, :protocol=>"https"} W, [2020-09-19T09:41:16.806888 #155817] WARN -- : [Faraday::TimeoutError] Reloading connections (attempt 1 of 3) - true /mnt/apps/.gembundle/ruby/2.7.0/gems/json-1.8.6/lib/json/common.rb:155: warning: Using the last argument as keyword parameters is deprecated I, [2020-09-19T09:41:16.845696 #155817] INFO -- : GET https://x.x.x.x:9200/_nodes/http [status:200, request:0.038s, query:n/a] D, [2020-09-19T09:41:16.845748 #155817] DEBUG -- : < {"_nodes":{"total":16,"successful":16,"failed":0},"cluster_name":"elasticsearch_cluster","nodes":{"xtxVW568SMaZAEKO9BX9sQ":........ .....

/mnt/apps/.gembundle/ruby/2.7.0/gems/faraday-0.9.2/lib/faraday/rack_builder.rb:55: warning: Capturing the given block using Proc.new is deprecated; use &block instead /mnt/apps/.gembundle/ruby/2.7.0/gems/faraday-0.9.2/lib/faraday/options.rb:62: warning: Capturing the given block using Proc.new is deprecated; use &block instead /mnt/apps/.gembundle/ruby/2.7.0/gems/faraday-0.9.2/lib/faraday/rack_builder.rb:55: warning: Capturing the given block using Proc.new is deprecated; use &block instead /mnt/apps/.gembundle/ruby/2.7.0/gems/faraday-0.9.2/lib/faraday/options.rb:62: warning: Capturing the given block using Proc.new is deprecated; use &block instead /mnt/apps/.gembundle/ruby/2.7.0/gems/faraday-0.9.2/lib/faraday/rack_builder.rb:55: warning: Capturing the given block using Proc.new is deprecated; use &block instead /mnt/apps/.gembundle/ruby/2.7.0/gems/faraday-0.9.2/lib/faraday/options.rb:62: warning: Capturing the given block using Proc.new is deprecated; use &block instead /mnt/apps/.gembundle/ruby/2.7.0/gems/faraday-0.9.2/lib/faraday/rack_builder.rb:55: warning: Capturing the given block using Proc.new is deprecated; use &block instead /mnt/apps/.gembundle/ruby/2.7.0/gems/faraday-0.9.2/lib/faraday/options.rb:62: warning: Capturing the given block using Proc.new is deprecated; use &block instead /mnt/apps/.gembundle/ruby/2.7.0/gems/faraday-0.9.2/lib/faraday/rack_builder.rb:55: warning: Capturing the given block using Proc.new is deprecated; use &block instead /mnt/apps/.gembundle/ruby/2.7.0/gems/faraday-0.9.2/lib/faraday/options.rb:62: warning: Capturing the given block using Proc.new is deprecated; use &block instead /mnt/apps/.gembundle/ruby/2.7.0/gems/faraday-0.9.2/lib/faraday/rack_builder.rb:55: warning: Capturing the given block using Proc.new is deprecated; use &block instead /mnt/apps/.gembundle/ruby/2.7.0/gems/faraday-0.9.2/lib/faraday/options.rb:62: warning: Capturing the given block using Proc.new is deprecated; use &block instead /mnt/apps/.gembundle/ruby/2.7.0/gems/faraday-0.9.2/lib/faraday/rack_builder.rb:55: warning: Capturing the given block using Proc.new is deprecated; use &block instead /mnt/apps/.gembundle/ruby/2.7.0/gems/faraday-0.9.2/lib/faraday/options.rb:62: warning: Capturing the given block using Proc.new is deprecated; use &block instead /mnt/apps/.gembundle/ruby/2.7.0/gems/faraday-0.9.2/lib/faraday/rack_builder.rb:55: warning: Capturing the given block using Proc.new is deprecated; use &block instead /mnt/apps/.gembundle/ruby/2.7.0/gems/faraday-0.9.2/lib/faraday/options.rb:62: warning: Capturing the given block using Proc.new is deprecated; use &block instead /mnt/apps/.gembundle/ruby/2.7.0/gems/faraday-0.9.2/lib/faraday/rack_builder.rb:55: warning: Capturing the given block using Proc.new is deprecated; use &block instead /mnt/apps/.gembundle/ruby/2.7.0/gems/faraday-0.9.2/lib/faraday/options.rb:62: warning: Capturing the given block using Proc.new is deprecated; use &block instead /mnt/apps/.gembundle/ruby/2.7.0/gems/faraday-0.9.2/lib/faraday/rack_builder.rb:55: warning: Capturing the given block using Proc.new is deprecated; use &block instead /mnt/apps/.gembundle/ruby/2.7.0/gems/faraday-0.9.2/lib/faraday/options.rb:62: warning: Capturing the given block using Proc.new is deprecated; use &block instead /mnt/apps/.gembundle/ruby/2.7.0/gems/faraday-0.9.2/lib/faraday/rack_builder.rb:55: warning: Capturing the given block using Proc.new is deprecated; use &block instead /mnt/apps/.gembundle/ruby/2.7.0/gems/faraday-0.9.2/lib/faraday/options.rb:62: warning: Capturing the given block using Proc.new is deprecated; use &block instead /mnt/apps/.gembundle/ruby/2.7.0/gems/faraday-0.9.2/lib/faraday/rack_builder.rb:55: warning: Capturing the given block using Proc.new is deprecated; use &block instead /mnt/apps/.gembundle/ruby/2.7.0/gems/faraday-0.9.2/lib/faraday/options.rb:62: warning: Capturing the given block using Proc.new is deprecated; use &block instead /mnt/apps/.gembundle/ruby/2.7.0/gems/faraday-0.9.2/lib/faraday/rack_builder.rb:55: warning: Capturing the given block using Proc.new is deprecated; use &block instead /mnt/apps/.gembundle/ruby/2.7.0/gems/faraday-0.9.2/lib/faraday/options.rb:62: warning: Capturing the given block using Proc.new is deprecated; use &block instead /mnt/apps/.gembundle/ruby/2.7.0/gems/faraday-0.9.2/lib/faraday/rack_builder.rb:55: warning: Capturing the given block using Proc.new is deprecated; use &block instead /mnt/apps/.gembundle/ruby/2.7.0/gems/faraday-0.9.2/lib/faraday/options.rb:62: warning: Capturing the given block using Proc.new is deprecated; use &block instead /mnt/apps/.gembundle/ruby/2.7.0/gems/faraday-0.9.2/lib/faraday/rack_builder.rb:55: warning: Capturing the given block using Proc.new is deprecated; use &block instead /mnt/apps/.gembundle/ruby/2.7.0/gems/faraday-0.9.2/lib/faraday/options.rb:62: warning: Capturing the given block using Proc.new is deprecated; use &block instead /mnt/apps/.gembundle/ruby/2.7.0/gems/faraday-0.9.2/lib/faraday/rack_builder.rb:55: warning: Capturing the given block using Proc.new is deprecated; use &block instead /mnt/apps/.gembundle/ruby/2.7.0/gems/faraday-0.9.2/lib/faraday/options.rb:62: warning: Capturing the given block using Proc.new is deprecated; use &block instead F, [2020-09-19T09:41:16.852591 #155817] FATAL -- : Custom[HTTPClient::KeepAliveDisconnected] HTTPClient::KeepAliveDisconnected: ({:id=>"7Lv2UFAcSd2tqjxK5VCDHQ", :name=>"name", :version=>"6.8.3", :host=>"x.x.x.x", :port=>"9200", :roles=>["data", "ingest"], :attributes=>{"aws_availability_zone"=>"us-east-1d", "ml.machine_memory"=>"64215576576", "ml.max_open_jobs"=>"20", "xpack.installed"=>"true", "ml.enabled"=>"true"}, :protocol=>"http"}) Traceback (most recent call last): 16: from faraday (0.9.2) lib/faraday/connection.rb:377:in run_request' 15: from faraday (0.9.2) lib/faraday/rack_builder.rb:139:in build_response' 14: from faraday (0.9.2) lib/faraday/adapter/httpclient.rb:36:in call' 13: from httpclient (2.7.2) lib/httpclient.rb:858:in request' 12: from httpclient (2.7.2) lib/httpclient.rb:1016:in do_request' 11: from httpclient (2.7.2) lib/httpclient.rb:1127:in protect_keep_alive_disconnected' 10: from httpclient (2.7.2) lib/httpclient.rb:1134:in rescue in protect_keep_alive_disconnected' 9: from httpclient (2.7.2) lib/httpclient.rb:1021:in block in do_request' 8: from httpclient (2.7.2) lib/httpclient.rb:1241:in do_get_block' 7: from httpclient (2.7.2) lib/httpclient.rb:1294:in do_get_header' 6: from httpclient (2.7.2) lib/httpclient/session.rb:547:in get_header' 5: from httpclient (2.7.2) lib/httpclient/session.rb:771:in read_header' 4: from httpclient (2.7.2) lib/httpclient/session.rb:788:in parse_header' 3: from /opt/ruby-2.7.1/lib/ruby/2.7.0/timeout.rb:105:in timeout' 2: from /opt/ruby-2.7.1/lib/ruby/2.7.0/timeout.rb:95:in block in timeout' 1: from httpclient (2.7.2) lib/httpclient/session.rb:795:in block in parse_header' HTTPClient::KeepAliveDisconnected (HTTPClient::KeepAliveDisconnected: )

Sep 19 '20 09:09 kushal-ladha

It seems like we're experiencing the same issue, once one of the 3 nodes is down, we get these timeout errors and every request to ElasticSearch will fail.

How would one configure the client to still make requests to the other nodes, while keeping the broken one out of the pool?

Mar 12 '21 14:03 matsimitsu

We are facing the same issue as well. one of the node going down causes all requests to elastic search timeout!

Nov 07 '22 08:11 tsrivishnu

I'm going to try the advanced options from here: https://www.elastic.co/guide/en/elasticsearch/client/ruby-api/7.17/advanced-config.html

I cannot reproduce a host being unresponsive so I have to wait until that happens in our production cluster. What I can confirm is that if the host is completely down, these advanced config options are working correctly and connect to a different host

Nov 07 '22 10:11 tsrivishnu

In my case the issue was that I was not passing the protocol in scheme options while setting up the connection, so it worked initially for some request and when it use to reload the connection(because of the reload_connections and reload_on_failure) option, it was taking the default scheme: 'http' instead of 'https'. Took me a while to understand the issue but finally resolved it by passing scheme during the connection creation itself.

Nov 07 '22 10:11 kushal-ladha