async-http-faraday icon indicating copy to clipboard operation
async-http-faraday copied to clipboard

Using Faraday in_parallel without immediately raising Faraday::TimeoutError

Open d11wtq opened this issue 8 months ago • 1 comments

We're not currently using Async in our project, though we are looking to switch some code that was previously using Faraday with the :typhoeus adapter to instead use the :async_http adapter.

One feature we were relying on was the ability to run a batch of requests in parallel without raising exceptions if one of the requests hits a timeout. Say 20 requests are run in parallel and one of them times out, we still want the 19 successful responses (and we can retry the one timeout separately).

When I simulate this situation with the :async_http adapter, what I'm seeing is that the Async::TimeoutError is handled, then re-raised as Faraday::TimeoutError, even when running in parallel mode. This means that the whole in_parallel block raises an exception and therefore the other responses cannot be reached.

What typhoeus does instead is it puts two values into the response env:

  • env[:typhoeus_connection_failed] (can be understood as Faraday::ConnectionFailed with the details found in env[:typhoeus_return_message])
  • env[:typhoeus_timed_out] (can be understood as Faraday::TimeoutError)

Is there any way to keep the same capability of firing off 20 requests and letting some of them timeout or hit connection issues without failing all the others?

d11wtq avatar May 09 '25 11:05 d11wtq