HttpServerHandler did not close back-end tcp socket in some case cause dead tcp socket between proxy and backend service

[steven-xing]

I run into this problem when running membrane proxy 4.1 standalone in front of a service running on tomcat 8.

The issue can be reproduced in following way:

Run a SOAPUI client, access the SOAP service(running on tomcat 8) through the proxy, the response will be about 2.5MB in size, after the client start receiving data and before the data transfer finish, kill the SOAPUI client, this will cause a dead tcp socket between the proxy and the SOAP service(on tomcat).

Linux "ss -nt" commands shows:

State      Recv-Q    Send-Q                Local Address:Port            Peer Address:Port
ESTAB   94622          0                      20.168.1.32:35057             20.168.1.65:8443
ESTAB   0              637056                20.168.1.65:8443               20.168.1.32:35057 

And this dead socket will be there forever until the proxy or the tomcat get restarted.

To me the cause of this problem may be in the HttpServerHandler, if there is an exception happen(such as IO exception between a client the proxy), at code line 112 of HttpServerHandler: process();

at that time, the boundConnection is NULL (since line 97 to 100 set it to NULL), when reach finally section(line 156), since the local variable boundConnection is NULL, so will not try to close it, and current code does not check the exchange.getTargetConnection().

This will leave the connection (exchange.getTargetConnection()) keep open and will not be closed.

since I have no control on client behavior, thus if any client not close the tcp connection to proxy gracefully, it could trigger this problem, currently I have a running system which have this problem, and the count of above dead sockets increase about 50-100 per-week, and it could cause run out tcp socket between the proxy and service at some point.

I have tried all different settings on both proxy side and tomcat side, such as "KeepAlive" etc, non of them can avoid this problem.

If more detail of this problem is required, I can provide.

[rrayst]

Thanks for the reproduction instructions. These will help the investigation. I'll try to get to this in the next few days. Please ping back, if there should be no further response.

[steven-xing]

Any update on this issue? If more detail or data required, let me know. Thanks

[symle]

@steven-xing could you send me the wsdl did you send binary data? I try to resolve this issue if it not resolved.

[steven-xing]

@symle No, it is a direct call through the proxy to the backend soap API, we did not do any REST-SOAP conversion. The response is a big SOAP response which contains about 5-10MB data. For the wsdl, it is a very old service which is running on axis 1.3, and the doc style is rpc-encoded.

I do not see it is useful to send you the whole wsdl file.

To reproduce the problem, I think you can run a server behind a proxy, then run a client just do a http request to retrieve a large file or large amount of data which give you enough time to kill the client before the transfer finish.

Thanks.

[symle]

On second try I regenerate the problem, Now I try to solve it.