[bug] getting "HTTP/1.1 501 Not Implemented"

[elyran]

Describe the bug

hi, i like the idea of your proxy, but, getting HTTP/1.1 501 Not Implemented for curl -v --proxy http://localhost:8080 -k https://worldtimeapi.org/api/timezone/Asia/Jerusalem. what am i missing? how can i make it work?

Logs

user1@user1-pc:~/Downloads$ curl -v --proxy http://localhost:8080 -k https://worldtimeapi.org/api/timezone/Asia/Jerusalem
*   Trying 127.0.0.1:8080...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8080 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to worldtimeapi.org:443
> CONNECT worldtimeapi.org:443 HTTP/1.1
> Host: worldtimeapi.org:443
> User-Agent: curl/7.68.0
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 501 Not Implemented
< 
* Received HTTP code 501 from proxy after CONNECT
* CONNECT phase completed!
* Closing connection 0
curl: (56) Received HTTP code 501 from proxy after CONNECT

Additional information

see Reproduction url for code

Reproduction

https://jsfiddle.net/g2fk1vph/

[callumgare]

I believe it's not a bug but rather a feature that this library doesn't (yet?) support. There's different methods to proxy HTTP requests, it can be done using the HTTP method CONNECT (https://en.wikipedia.org/wiki/HTTP_tunnel) or it can be done by supplying the full url instead of just a path in the HTTP request (https://en.wikipedia.org/wiki/Proxy_server#Web_proxy_servers). It seems this library supports the latter method but not the former.

# Start the proxy server
node -e 'const {HttpCachingProxy} = require("@loopback/http-caching-proxy"); const proxy = new HttpCachingProxy({cachePath: "/tmp/.proxy-cache", port: 3000}); proxy.start().then(() => console.log(`Proxy started at ${proxy.url}`))'

# Send a request 
curl -v --request-target 'https://worldtimeapi.org/api/timezone/Asia/Jerusalem' -H "Host: worldtimeapi.org" http://localhost:3000

Personally I'd love to see this library support the HTTP tunnel proxy method (that's what I was looking for when I came across this library) so maybe this ticket can be turned from a bug ticket into a feature request?

[callumgare]

Oh, I've just realised that I think the HTTP tunnel method would be largely useless since for HTTPS requests the proxy would just be transparently forwarding encrypted HTTPS traffic and thus would not be able to cache any content.

[elyran]

I also vote for a feature request. Thank you, appreciate you trying to help out

[callumgare]

@elyran I came up with an ugly workaround which may or may not work in your situation. It's detailed in the snippets of code at the end of this post: https://github.com/apify/got-scraping/issues/140#issuecomment-2126865698