requests icon indicating copy to clipboard operation
requests copied to clipboard

Add support for HTTPS proxies.

Open jalopezsilva opened this issue 5 years ago • 5 comments

Starting with 1.26 urllib3 now supports HTTPS proxies. To enable the support two changes are needed:

  • An additional proxy_kwargs argument that can be passed from the session. This dictionary will be used to pass any arguments needed to the underlying adapter. The parameter is optional.
  • The use_forwarding_for_https mode requires us to send the absolute URI when enabled.

The resulting API is very similar except it takes an additional parameter when needed. An example:

    session = requests.Session()
    proxies = {"http": "https://proxy.example", "https": "https://proxy.example"}

    proxies_kwargs = {
        "proxy_ssl_context": proxy_ssl_context(),
    }

    response = session.get(
       "https://www.google.com", proxies=proxies, proxies_kwargs=proxies_kwargs
    )

jalopezsilva avatar Nov 19 '20 19:11 jalopezsilva

Requests is under feature freeze which means we will not add any additional keyword arguments to the public API like this.

sigmavirus24 avatar Nov 19 '20 23:11 sigmavirus24

@sigmavirus24 thanks for the feedback! Is adding an additional keyword argument to the public API the only concern? I've been reviewing the options available and there's no easy way around this unless we modify the proxy_manager_for method in HTTPAdapter

Could we add a proxy_kwargs attribute to HTTPAdapter? Rather than passing the proxy_kwargs dictionary through multiple top level APIs we could expose it like this:

s = requests.Session()
s.adapters['https://'].proxy_kwargs = { ... }

That minimizes some of the changes needed. Would that work?

Facebook needs these changes upstreamed so I would love it if we can find a path forward here. I can create our own HTTPAdapter internally but t it leaves the requests community without being able to benefit from the work I put in to expand the HTTPS proxy support in urllib3.

jalopezsilva avatar Nov 25 '20 00:11 jalopezsilva

@nateprewitt Are we still planning on doing this for 2.26.0? It doesn't look like it's moved since being added to the milestone.

sethmlarson avatar Jul 07 '21 13:07 sethmlarson

I popped it out of 2.26.0 so we can get a release unblocked. I'll follow up with @jalopezsilva on getting this in the next minor release.

nateprewitt avatar Jul 07 '21 16:07 nateprewitt

That would be amazing @nateprewitt. Let me know which changes are required.

jalopezsilva avatar Jul 07 '21 17:07 jalopezsilva