getodk
Socket closed when pushing 3k+ submissions to Central
Software versions
Briefcase v1.17.4, macOS 10.15.4, Java 11
Problem description
2020-10-02 10:37:18,044 [ForkJoinPool-5-worker-3] INFO o.a.http.impl.execchain.RetryExec - I/O exception (java.net.SocketException) caught when processing request to {s}->https://test.central.getodk.org:443: Socket closed
2020-10-02 10:37:18,044 [ForkJoinPool-5-worker-3] INFO o.a.http.impl.execchain.RetryExec - Retrying request to {s}->https://test.central.getodk.org:443
2020-10-02 10:37:18,044 [ForkJoinPool-5-worker-1] ERROR o.o.briefcase.reused.job.JobsRunner - Error running Job
java.io.UncheckedIOException: java.net.SocketException: Socket closed
at org.opendatakit.briefcase.reused.http.CommonsHttp.uncheckedExecute(CommonsHttp.java:149)
at org.opendatakit.briefcase.reused.http.CommonsHttp.execute(CommonsHttp.java:96)
at org.opendatakit.briefcase.push.central.PushToCentral.pushSubmission(PushToCentral.java:270)
at org.opendatakit.briefcase.push.central.PushToCentral.lambda$null$13(PushToCentral.java:160)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.stream.ReferencePipeline$11$1.accept(ReferencePipeline.java:373)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)
at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.execLocalTasks(ForkJoinPool.java:1040)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1058)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: java.net.SocketException: Socket closed
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:339)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
Caused by: java.net.SocketException: Socket closed
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at org.apache.http.client.fluent.Request.internalExecute(Request.java:173)
at org.apache.http.client.fluent.Executor.execute(Executor.java:262)
at org.opendatakit.briefcase.reused.http.CommonsHttp.uncheckedExecute(CommonsHttp.java:142)
... 16 common frames omitted
Steps to reproduce the problem
Try to push 3k+ submissions to a Central server. This has happened to me several times and I have to close Briefcase and restart the transfer.
Expected behavior
No crash.
Other information
I haven't started doing any exploration. Maybe it's a concurrency issue? Or could the server be taking too long to respond at some point? It may very well just be intermittent and have nothing to do with the number of submissions sent.
It seems that with the same data set it failed twice at 4113 submissions sent. 4105 with the same form definition but different set of submissions. Sent several batches of ~500 forms and then an attempt to push 3000 failed at 1114. It's starting to feel like a resource management issue on the Briefcase end of things.
@getodk/testers here's a form and submissions that I was getting the failure for. It would be great to try on Windows and get a sense of whether it might be Mac-only. I'm on macOS 10.15.4, Java 11.
I was able to push to Central without any problems, none crash was visible on Windows 10.
Confirming that on master as of right now (dd75f13) I'm still getting the same crash. Latest attempt was at 4402 submissions.