playframework
/count endpoint barfs under load
I am doing some load testing with the starter example. So for example
ab -n 100000 http://localhost:9000/count
This is ApacheBench, Version 2.3 <$Revision: 1757674 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests
Server Software:
Server Hostname: localhost
Server Port: 9000
Document Path: /count
Document Length: 5 bytes
Concurrency Level: 1
Time taken for tests: 206.992 seconds
Complete requests: 100000
Failed requests: 49210
(Connect: 0, Receive: 0, Length: 49210, Exceptions: 0)
Total transferred: 40149210 bytes
HTML transferred: 549210 bytes
Requests per second: 483.11 [#/sec] (mean)
Time per request: 2.070 [ms] (mean)
Time per request: 2.070 [ms] (mean, across all concurrent requests)
Transfer rate: 189.42 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 148.9 0 19570
Processing: 0 1 8.4 0 1024
Waiting: 0 1 8.0 0 1024
Total: 0 2 149.1 1 19571
Percentage of the requests served within a certain time (ms)
50% 1
66% 1
75% 1
80% 1
90% 1
95% 1
98% 1
99% 1
100% 19571 (longest request)
It appears I can throw the /count endpoint off by setting a higher concurrency
ab -c12 -n 100000 http://localhost:9000/count
This is ApacheBench, Version 2.3 <$Revision: 1757674 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
Completed 10000 requests
apr_pollset_poll: The timeout specified has expired (70007)
Total of 16387 requests completed
This always happens at around 16k requests, even at a concurrency of only 2
Os is MacOs Sierra, Scala 2.12.2 (installed via brew), 2.6.x branch of this repo
Nothing too dramatic but still a bit of a surprise how quickly threading can barf, even with Scala :)
out of curiosity I tested this too with similar results
ab -c12 -n 100000 http://localhost:9000/count
This is ApacheBench, Version 2.3 <$Revision: 1757674 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
Completed 10000 requests
apr_pollset_poll: The timeout specified has expired (70007)
Total of 16405 requests completed
Then decided to run it under two shells while hitting /count via my browser.
first...
...
Benchmarking localhost (be patient)
Completed 10000 requests
apr_pollset_poll: The timeout specified has expired (70007)
Total of 13774 requests completed
second...
...
Benchmarking localhost (be patient)
apr_pollset_poll: The timeout specified has expired (70007)
Total of 2626 requests completed
The browser requests returned the updated counts caused by the ab commands and continued to work after the ab commands barfed. Surprising the ab commands failed but heartening that concurrent access with Chrome worked fine.
True, good thing the browser still gets data. I just found it strange that the concurrency causes some sort of deadlock (I would assume ... even with your two ab scripts running you get until around 16k), I thought this whole actor thing was to avoid exactly this....
This always happens at around 16k requests, even at a concurrency of only 2
Please see the performance test framework results at https://playframework.github.io/prune/ -- this is far more likely to be a consequence of doing a load test on MacOs Sierra, with the load testing framework on the machine itself, using apache bench.
Try on a standalone Linux machine and use gatling or wrk: see https://developer.lightbend.com/guides/play-rest-api/appendix.html#load-testing for details.
