MapStore2 icon indicating copy to clipboard operation
MapStore2 copied to clipboard
verified publisher icon geosolutions-it

Problems with proxy on certain servers

Open offtherailz opened this issue 3 years ago • 7 comments

Description

When proxing certain servers, the proxy fails, with empty response. This happens only in certain installations. Actually we tested with:

  • DEV
  • previous stable-binary (2022.01.02)

How to reproduce

  • Open mapstore
  • Load the map attached map.zip

Expected Result The layers should be visible

Current Result The layers have errors

  • [x] Not browser related
Browser info (use this site: https://www.whatsmybrowser.org/ for non expert users)
Browser Affected Version
map.zip

|Internet Explorer| | |Edge| | |Chrome| | |Firefox| | |Safari| |

Other useful information

Logged error

Error in log:

[http-nio-8080-exec-6] ERROR it.geosolutions.httpproxy.HTTPProxy - Error executing HTTP method
java.net.SocketException: Connection reset
	at java.base/java.net.SocketInputStream.read(SocketInputStream.java:186)
	at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
	at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:478)
	at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:472)
	at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)
	at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1454)
	at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1065)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at it.geosolutions.httpproxy.HTTPProxy.executeProxyRequest(HTTPProxy.java:704)
	at it.geosolutions.httpproxy.HTTPProxy.doGet(HTTPProxy.java:343)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:670)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:779)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1789)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.base/java.lang.Thread.run(Thread.java:829)

In the initial investigation we noticed that tomcat update solved the problem, but this was valid only for tomcat 8. Updating to latest tomcat 9 didn't solved anything.

Details:

Moving the war in a newer tomcat (8.5.82) solved the issue.

Replicate on MapStore binary 2022.01.02 (tomcat 8.5.69) and on our installations server.

image

Instead updating to apache-tomcat-9.0.68 didn't resolved the issue.

offtherailz avatar Oct 12 '22 15:10 offtherailz

@randomorder investigating a bit more with @offtherailz it seems we are having a problem with the tomcat version. Is it possible to update it in DEV as indicated in description for a final test in DEV?

tdipisa avatar Oct 13 '22 16:10 tdipisa

would not be a problem but in dev we have tomcat 9.0.65. Can you confirm you have the same problem in DEV?

randomorder avatar Oct 13 '22 16:10 randomorder

@randomorder

would not be a problem but in dev we have tomcat 9.0.65. Can you confirm you have the same problem in DEV?

yes it is not effectively possible to add a catalog source https://sdigeo-free.austrocontrol.at/geoserver/free/wms in DEV (http_proxy 1.3.0) while it is possible in c125 DEV instance where we are using http_proxy 1.1.1.

@offtherailz, it can also be a problems of the http_proxy version? Can we maybe to a try using in MS DEV the v1.1.1?

tdipisa avatar Oct 14 '22 09:10 tdipisa

I re-cecked. The problems seems to happen also on binary of MapStore 2022.02.01 (I can see the same behavior, but in this case the error is not logged). Connection reset can happen for many reasons, I think it needs a debug.

offtherailz avatar Oct 14 '22 10:10 offtherailz

@taba90 it would be good if you can do a quick debug to see what is happening. Then we will schedule what need to be done accordingly.

@offtherailz it would e good anyway to provide in a new comment all the details related to checks you have made so far by reporting http_proxy and tomcat versions used, where it works and where it doesn't. This is important to let @taba90 to better understand the problem and better focus it for an effective debug session.

tdipisa avatar Oct 14 '22 10:10 tdipisa

@tdipisa I've done a quick check and according to the documentation found "java.net.SocketException: Connection reset " happens if the server terminated the connection or the connection is lost due to network issues, so usually seems that is an exception that if raised on the client side is because the socket was closed by the server before the client could finish to read. There might be however other reasons that might involve the client side as well. It would be anyway usefull to enable logging of the httpclient library used by the proxy in order to have a deeper look. I can check this better monday

taba90 avatar Oct 14 '22 13:10 taba90

@taba90 ok, thank you so much. Keep me updated then.

tdipisa avatar Oct 14 '22 14:10 tdipisa

I need to configure debug logging level for the apache proxy library used by the http proxy and check the full logs then. @randomorder can I have access to one of the machines with the failing instances in order to set the logging configuration and restart the proxy?

taba90 avatar Oct 18 '22 06:10 taba90

@randomorder can you please provide what requested above?

tdipisa avatar Oct 18 '22 12:10 tdipisa

@taba90 after the austrocontrol project update we experienced the same problem. Switching back to the http_proxy 1.1.1 solved the problem.

@offtherailz can you please assist @taba90 for the above?

tdipisa avatar Oct 19 '22 16:10 tdipisa

Just discovered a different strange behavior here where are we using v1.3.0. Connecting to:

https://sdigeo-free.austrocontrol.at/geoserver/free/wms

I got the following error (HTTP 500):

<!doctype html><html lang="en"><head><title>HTTP Status 500 – Internal Server Error</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 500 – Internal Server Error</h1><hr class="line" /><p><b>Type</b> Exception Report</p><p><b>Message</b> Servlet.init() for servlet [HttpProxy] threw exception</p><p><b>Description</b> The server encountered an unexpected condition that prevented it from fulfilling the request.</p><p><b>Exception</b></p><pre>javax.servlet.ServletException: Servlet.init() for servlet [HttpProxy] threw exception
	org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
	org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
	org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
	org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
	org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
	org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
	org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:890)
	org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1789)
	org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
	org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
	org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	java.base&#47;java.lang.Thread.run(Thread.java:829)
</pre><p><b>Root Cause</b></p><pre>java.lang.NoClassDefFoundError: Could not initialize class org.apache.http.impl.conn.ManagedHttpClientConnectionFactory
	org.apache.http.impl.conn.PoolingHttpClientConnectionManager$InternalConnectionFactory.&lt;init&gt;(PoolingHttpClientConnectionManager.java:620)
	org.apache.http.impl.conn.PoolingHttpClientConnectionManager.&lt;init&gt;(PoolingHttpClientConnectionManager.java:181)
	org.apache.http.impl.conn.PoolingHttpClientConnectionManager.&lt;init&gt;(PoolingHttpClientConnectionManager.java:165)
	org.apache.http.impl.conn.PoolingHttpClientConnectionManager.&lt;init&gt;(PoolingHttpClientConnectionManager.java:156)
	org.apache.http.impl.conn.PoolingHttpClientConnectionManager.&lt;init&gt;(PoolingHttpClientConnectionManager.java:132)
	org.apache.http.impl.conn.PoolingHttpClientConnectionManager.&lt;init&gt;(PoolingHttpClientConnectionManager.java:123)
	it.geosolutions.httpproxy.HTTPProxy.init(HTTPProxy.java:135)
	org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
	org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
	org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
	org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
	org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
	org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
	org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:890)
	org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1789)
	org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
	org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
	org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	java.base&#47;java.lang.Thread.run(Thread.java:829)
</pre><p><b>Note</b> The full stack trace of the root cause is available in the server logs.</p><hr class="line" /><h3>Apache Tomcat/9.0.65</h3></body></html>

No problem connecting to our GS stable.

tdipisa avatar Oct 20 '22 07:10 tdipisa

still need devops here @tdipisa ?

randomorder avatar Oct 21 '22 10:10 randomorder

not for now @randomorder, still waiting @taba90 to look at this. See above.

tdipisa avatar Oct 21 '22 10:10 tdipisa

talked with @offtherailz, working on investigating this currently.

taba90 avatar Oct 24 '22 08:10 taba90

@tdipisa I did not come to anything. Steps taken so far:

  1. Comparing the raw httprequests using Fiddler of failing proxy, successful proxy and chrome and they all seems the same.
  2. debugging source code of the http-client library to see where exactly the failures is: when reading the headers from geoserver. To proceed with the investigation I would need to check the log on geoserver side after the request fails. Currently I'm blocked on this.

taba90 avatar Oct 25 '22 09:10 taba90

@taba90 let's move this in blocked status for now since we don't have the GS logs from https://sdigeo-free.austrocontrol.at/geoserver/free/wms

tdipisa avatar Oct 25 '22 10:10 tdipisa

@taba90 I've this error below here where 1.4-SNAPSHOT is used

<!doctype html><html lang="en"><head><title>HTTP Status 500 – Internal Server Error</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 500 – Internal Server Error</h1><hr class="line" /><p><b>Type</b> Exception Report</p><p><b>Message</b> Servlet.init() for servlet [HttpProxy] threw exception</p><p><b>Description</b> The server encountered an unexpected condition that prevented it from fulfilling the request.</p><p><b>Exception</b></p><pre>javax.servlet.ServletException: Servlet.init() for servlet [HttpProxy] threw exception
	org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:544)
	org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
	org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:698)
	org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:364)
	org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:526)
	org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
	org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:831)
	org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1651)
	org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	java.lang.Thread.run(Thread.java:748)
</pre><p><b>Root Cause</b></p><pre>java.lang.NoClassDefFoundError: Could not initialize class org.apache.http.conn.ssl.SSLConnectionSocketFactory
	org.apache.http.impl.conn.PoolingHttpClientConnectionManager.getDefaultRegistry(PoolingHttpClientConnectionManager.java:116)
	org.apache.http.impl.conn.PoolingHttpClientConnectionManager.&lt;init&gt;(PoolingHttpClientConnectionManager.java:123)
	it.geosolutions.httpproxy.HTTPProxy.init(HTTPProxy.java:135)
	org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:544)
	org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
	org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:698)
	org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:364)
	org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:526)
	org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
	org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:831)
	org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1651)
	org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	java.lang.Thread.run(Thread.java:748)
</pre><p><b>Note</b> The full stack trace of the root cause is available in the server logs.</p><hr class="line" /><h3>Apache Tomcat/8.5.69</h3></body></html>

Nothing relevant reported by the GS log with VERBOSE level. If you want we can schedule a call to look together.

tdipisa avatar Oct 26 '22 14:10 tdipisa

@tdipisa It seems some dependency issue. I will check depedencies version for http client in the C040 downstream project.

taba90 avatar Oct 26 '22 14:10 taba90

moving in blocked for now. A review of the overall MV dependency management is suggested for this.

tdipisa avatar Nov 02 '22 16:11 tdipisa