Host header not respected when calling upstream server

[tjsousa]

I tried to use the embedded plug strategy inside a Phoenix app and run into an issue where the client set Host header wouldn't be respected in the upstream request.

I digged into it and found this to be caused by https://github.com/benoitc/hackney/issues/114, as :hackney only looks for the Host header before overriding it. Since the headers are being set from a Plug connection, they're lower-cased by default.

I tried HTTPotion as a client and the headers seem to be respected. Any thoughts on the best approach to overcome this issue?

[slogsdon]

@tjsousa Thanks for the report! I would say that this should be handled in HTTPoison and/or Hackney, but there's an unknown amount of time to get that fixed upstream. Saying that, we could most likely handle this inside of ReverseProxy, but that's not the most ideal/optimal solution.

To be honest, I'd be up for switching to HTTPotion/ibrowse if it's a better fit. I just used HTTPoison because it's what I've always used.

Thoughts?

[tjsousa]

I agree. I opened an issue in HTTPoison for it: https://github.com/edgurgel/httpoison/issues/100, but seems like solving it upstream is the way to go.

I also have a fix in place for ReverseProxy where I'm basically capitalizing all headers before sending them to HTTPoison, but I feel like it's okay to wait.

Being too new to Elixir to make the HTTPotion vs HTTPoison call, but I've heard performance bottlenecks found in the former motivated the development of the latter. I must say I'm interested in an use-case for transparent proxying where performance is critical. Have you tried this code under load? How does it behave?

Thank you so much for your quick reply and feedback.

[slogsdon]

I haven't completed any performance benchmarks on this project before, but this project will probably never win any competitions against reverse proxies written with more low-level code. That said, here are the hypothetical limiting factors:

• The web server used by Plug, in this case Cowboy. ReverseProxy can only serve as fast as Cowboy can serve.
• The cache store/retrieval. A cache hit with a slow cache store/retrieval will result in a slower proxy response.
• The web client/runner combo. Most of this will be the client, but there will need to be some translation of the client response to the proxy response sent to the end-user
• Network speeds between the proxy and the upstream. This includes the call stack when used as a plug in another Plug-based application or the other way around.
• The upstream. A cache miss with a slow upstream will result in a slower proxy response.

In order to maximize performance of ReverseProxy as a reverse proxy (as the project stands today), these would be ideal:

• Limit time spent within ReverseProxy's control. Simplifying necessary glue code around the bits and pieces would go a long way here.
• Efficiently select upstream when more than one is available. Nothing has really been done here other than picking the first in the list.
• Allow various web server to serve proxy requests. Plug handles this for us.
• Allow various cache stores and retrievers. Caching isn't available now but will probably ship with either a file-based or in-memory solution as a starting point (neither are really ideal).
• Allow various web clients/runners. I don't think one solution will be a best fit for all use cases.

I'm by no means an expert in creating reverse proxies, so this list probably isn't 100% complete/valid but is at least a proper starting point.

I see a project like ReverseProxy being beneficial by leveraging the BEAM as a distributed reverse proxy. This is a half-baked idea of mine, but it could potentially provide consistency, availability, and partition tolerance that outweighs any performance penalties.