POST /api/maintenance/migrate throws socketException
Once cortex is installed from binary, we run /api/maintenance/migrate to bootstrap cortex index and super admin
Request Type
Bug
Work Environment
Openshift 4.3
| Question | Answer |
|---|---|
| OS version (server) | Redhat |
| OS version (client) | Linux version 4.18.0-193.14.3.el8_2.x86_64 ([email protected]) (gcc version 8.3.1 20191121 (Red Hat 8.3.1-5) (GCC) |
| Cortex version / git hash | 3.0.1 |
| Package Type | From binary downloaded from hive cortex project supporting elastic 6.8.8 |
Problem Description
Steps to Reproduce
- Once cortex image is installed,
- We run python script using HTTPSConnection.request for POST /api/maintenance/migrate 3 it throws below error
Possible Solutions
Remedy explained here : https://github.com/sksamuel/elastic4s/issues/1433
Complementary information
[info] c.s.e.h.ElasticClient$ - Creating HTTP client on https://tis-elastic-es-http:9200 [info] c.s.e.h.ElasticClient$ - Creating HTTP client on https://tis-elastic-es-http:9200 [info] c.s.e.h.ElasticClient$ - Creating HTTP client on https://tis-elastic-es-http:9200 [info] o.e.s.MigrationSrv - Create a new empty database [info] o.e.s.MigrationSrv - Migrate database from version 0, add operations for version 2 [info] o.e.s.MigrationSrv - Migrate database from version 0, add operations for version 3 [info] o.e.s.MigrationSrv - Migrate database from version 0, add operations for version 4 [error] o.e.s.MigrationSrv - Migration fail com.sksamuel.elastic4s.http.JavaClientExceptionWrapper: java.net.SocketTimeoutException at com.sksamuel.elastic4s.http.ElasticsearchJavaRestClient$$anon$1.onFailure(ElasticsearchJavaRestClient.scala:63) at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:850) at org.elasticsearch.client.RestClient$1.retryIfPossible(RestClient.java:588) at org.elasticsearch.client.RestClient$1.failed(RestClient.java:567) at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:134) at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.failed(AbstractClientExchangeHandler.java:419) at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:375) at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92) at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39) at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175) Caused by: java.net.SocketTimeoutException: null at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:375) at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92) at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39) at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175) at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:263) at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:492) at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:213) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
[info] o.t.c.s.ErrorHandler - POST /api/maintenance/migrate returned 500 com.sksamuel.elastic4s.http.JavaClientExceptionWrapper: java.net.SocketTimeoutException at com.sksamuel.elastic4s.http.ElasticsearchJavaRestClient$$anon$1.onFailure(ElasticsearchJavaRestClient.scala:63) at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:850) at org.elasticsearch.client.RestClient$1.retryIfPossible(RestClient.java:588) at org.elasticsearch.client.RestClient$1.failed(RestClient.java:567) at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:134) at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.failed(AbstractClientExchangeHandler.java:419) at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:375) at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92) at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39) at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
@nadouani any pointer here. what could be potentially issue ? its sporadic but occurs periodically. appreciate your help !