httpillage
httpillage copied to clipboard
Refactor to use Net::HTTP
A performance issue was discovered that seems to be related to the use of Mechanize.
Initial tests indicate that raw Net::HTTP GET queries are about 10x faster than leveraging Mechanize's get method. In these test cases the post request resulted in a server error causing significant latency. These tests should be taken with a grain of salt.
Tests with 5 threads and 100 requests on ruby 2.2.3p173 resulted in the following runtimes:
- Net::HTTP.get - < 1 second
- Mechanize Get - 6 seconds
- Net::HTTP.post - 19 seconds
- Mechanize Post - 23 seconds
Tests with 5 threads and 100 requests on jruby 9.0.1.0 (2.2.2) resulted in the following runtimes:
- Net::HTTP.get - 2 seconds
- Mechanize Get - 31 seconds
- Net::HTTP.post - 98 seconds
- Mechanize Post - 123 seconds