Cortex icon indicating copy to clipboard operation
Cortex copied to clipboard

POST /api/maintenance/migrate throws socketException

Open BrijJhala opened this issue 5 years ago • 1 comments

Once cortex is installed from binary, we run /api/maintenance/migrate to bootstrap cortex index and super admin

Request Type

Bug

Work Environment

Openshift 4.3

Question Answer
OS version (server) Redhat
OS version (client) Linux version 4.18.0-193.14.3.el8_2.x86_64 ([email protected]) (gcc version 8.3.1 20191121 (Red Hat 8.3.1-5) (GCC)
Cortex version / git hash 3.0.1
Package Type From binary downloaded from hive cortex project supporting elastic 6.8.8

Problem Description

Steps to Reproduce

  1. Once cortex image is installed,
  2. We run python script using HTTPSConnection.request for POST /api/maintenance/migrate 3 it throws below error

Possible Solutions

Remedy explained here : https://github.com/sksamuel/elastic4s/issues/1433

Complementary information

[info] c.s.e.h.ElasticClient$ - Creating HTTP client on https://tis-elastic-es-http:9200 [info] c.s.e.h.ElasticClient$ - Creating HTTP client on https://tis-elastic-es-http:9200 [info] c.s.e.h.ElasticClient$ - Creating HTTP client on https://tis-elastic-es-http:9200 [info] o.e.s.MigrationSrv - Create a new empty database [info] o.e.s.MigrationSrv - Migrate database from version 0, add operations for version 2 [info] o.e.s.MigrationSrv - Migrate database from version 0, add operations for version 3 [info] o.e.s.MigrationSrv - Migrate database from version 0, add operations for version 4 [error] o.e.s.MigrationSrv - Migration fail com.sksamuel.elastic4s.http.JavaClientExceptionWrapper: java.net.SocketTimeoutException at com.sksamuel.elastic4s.http.ElasticsearchJavaRestClient$$anon$1.onFailure(ElasticsearchJavaRestClient.scala:63) at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:850) at org.elasticsearch.client.RestClient$1.retryIfPossible(RestClient.java:588) at org.elasticsearch.client.RestClient$1.failed(RestClient.java:567) at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:134) at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.failed(AbstractClientExchangeHandler.java:419) at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:375) at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92) at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39) at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175) Caused by: java.net.SocketTimeoutException: null at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:375) at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92) at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39) at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175) at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:263) at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:492) at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:213) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)

[info] o.t.c.s.ErrorHandler - POST /api/maintenance/migrate returned 500 com.sksamuel.elastic4s.http.JavaClientExceptionWrapper: java.net.SocketTimeoutException at com.sksamuel.elastic4s.http.ElasticsearchJavaRestClient$$anon$1.onFailure(ElasticsearchJavaRestClient.scala:63) at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:850) at org.elasticsearch.client.RestClient$1.retryIfPossible(RestClient.java:588) at org.elasticsearch.client.RestClient$1.failed(RestClient.java:567) at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:134) at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.failed(AbstractClientExchangeHandler.java:419) at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:375) at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92) at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39) at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)

BrijJhala avatar Oct 04 '20 23:10 BrijJhala

@nadouani any pointer here. what could be potentially issue ? its sporadic but occurs periodically. appreciate your help !

BrijJhala avatar Oct 09 '20 18:10 BrijJhala