crawly
crawly copied to clipboard
fix: request_options middleware overriding spider options
Currently, any options set in the Crawly.Request will get overriden by the RequestOptions middleware. This PR fixes that to allow merging of both request options.
Crawly.Request.new(
"http://example.com",
headers,
# the body option here will get overriden by the middleware from the config
body: body
)
config :crawly,
middlewares: [
{Crawly.Middlewares.RequestOptions, proxy: {'https://proxy.com', 3128}}
],