envoyproxy
Dynamic host determination for plaintext HTTP encapsulation in CONNECT
Title: *Dynamically extracting the plaintext HTTP host/authority and encapsulating in CONNECT *
Description:
I have followed the example found in https://github.com/envoyproxy/envoy/blob/main/configs/encapsulate_http_in_http2_connect.yaml for setting up HTTP encapsulating in CONNECT. The example uses a fixed hostname of host.com:443 and I want to dynamically determine it from the :authority of the plaintext HTTP connection.
I have read about and see that with https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/network/tcp_proxy/v3/tcp_proxy.proto#extensions-filters-network-tcp-proxy-v3-tcpproxy-tunnelingconfig the hostname can be a command operator. In fact, I have an example which handles encrypted traffic, TLS inspector and %REQUESTED_SERVER_NAME% which works perfectly. But I need to also handle plaintext HTTP traffic and encapsulate that in CONNECT.
The suggestion in https://github.com/envoyproxy/envoy/issues/30674 seems to be to use dynamic metadata and propagate this to the internal listener and then use this metadata to set the hostname. https://www.envoyproxy.io/docs/envoy/latest/configuration/other_features/internal_listener#internal-upstream-transport allows for the exchange of the filter state from the downstream listener to the internal listener through a user space socket. I have configured https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/transport_sockets/internal_upstream/v3/internal_upstream.proto#envoy-v3-api-msg-extensions-transport-sockets-internal-upstream-v3-internalupstreamtransport and wrapped a https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/transport_sockets/raw_buffer/v3/raw_buffer.proto transport socket. My hope was that I could use something like the envoy.network.upstream_server_name metadata key and set it in an HTTP connection manager to the value of %REQ(:AUTHORITY)% and then set the hostname to something like %DYNAMIC_METADATA(envoy.network:upstream_server_name)%:443. This simply doesn't work for me and I cannot seem to figure out how to get this to work.
- In setting up https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/transport_sockets/internal_upstream/v3/internal_upstream.proto#envoy-v3-api-msg-extensions-transport-sockets-internal-upstream-v3-internalupstreamtransport I'm passing through metadata for the
envoy.networkkey for bothhostandclustermetadata kinds. - I did somewhat try to use https://github.com/envoyproxy/envoy/pull/29655#issue-1899144846 as a reference for how to configure the
envoy.transport_sockets.internal_upstream.
I've hit a wall as I cannot seem to figure out how to carry %REQ(:AUTHORITY)% to the internal listener so it can be used as the hostname. Any hints would be welcome.
Relevant Links:
- https://github.com/envoyproxy/envoy/blob/main/configs/encapsulate_http_in_http2_connect.yaml
- https://github.com/envoyproxy/envoy/issues/30674
Here's a config for a setup from the http://localhost:9901/config_dump endpoint (with some irrelevant data removed). I'm mostly confused at why setting the metadata for envoy.network.upstream_server_name isn't working and/or isn't being passed to the internal listener:
(I'm aware of the Basic auth string in the JSON as it is simply base64 encoded user1:password1.)
{
"configs": [
{
"@type": "type.googleapis.com/envoy.admin.v3.ClustersConfigDump",
"version_info": "1717190533136161000",
"static_clusters": [
{
"cluster": {
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "ext_authz_cluster",
"type": "STATIC",
"load_assignment": {
"cluster_name": "ext_authz_cluster",
"endpoints": [
{
"lb_endpoints": [
{
"endpoint": {
"address": {
"socket_address": {
"address": "127.0.0.1",
"port_value": 18001
}
}
}
}
]
}
]
},
"typed_extension_protocol_options": {
"envoy.extensions.upstreams.http.v3.HttpProtocolOptions": {
"@type": "type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions",
"explicit_http_config": {
"http2_protocol_options": {}
}
}
}
},
"last_updated": "2024-05-31T21:22:13.280Z"
},
{
"cluster": {
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "xds_cluster",
"type": "STATIC",
"load_assignment": {
"cluster_name": "xds_cluster",
"endpoints": [
{
"lb_endpoints": [
{
"endpoint": {
"address": {
"socket_address": {
"address": "127.0.0.1",
"port_value": 18000
}
}
}
}
]
}
]
},
"typed_extension_protocol_options": {
"envoy.extensions.upstreams.http.v3.HttpProtocolOptions": {
"@type": "type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions",
"explicit_http_config": {
"http2_protocol_options": {}
}
}
}
},
"last_updated": "2024-05-31T21:22:13.280Z"
}
],
"dynamic_active_clusters": [
{
"version_info": "1717190533136161000",
"cluster": {
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "connect_upstream_cluster",
"type": "LOGICAL_DNS",
"connect_timeout": "5s",
"circuit_breakers": {
"thresholds": [
{
"max_connections": 10000,
"max_pending_requests": 10000,
"max_requests": 10000,
"max_retries": 3
}
]
},
"dns_lookup_family": "V4_ONLY",
"load_assignment": {
"cluster_name": "connect_upstream_cluster",
"endpoints": [
{
"lb_endpoints": [
{
"endpoint": {
"address": {
"socket_address": {
"address": "localhost",
"port_value": 10443
}
},
"hostname": "localhost"
}
}
]
}
]
},
"typed_extension_protocol_options": {
"envoy.extensions.upstreams.http.v3.HttpProtocolOptions": {
"@type": "type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions",
"use_downstream_protocol_config": {
"http_protocol_options": {}
}
}
},
"respect_dns_ttl": true
},
"last_updated": "2024-05-31T21:22:13.286Z"
},
{
"version_info": "1717190533136161000",
"cluster": {
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "encap_0.0.0.0:16080",
"connect_timeout": "5s",
"circuit_breakers": {
"thresholds": [
{
"max_connections": 10000,
"max_pending_requests": 10000,
"max_requests": 10000,
"max_retries": 3
}
]
},
"transport_socket": {
"name": "envoy.transport_sockets.internal_upstream",
"typed_config": {
"@type": "type.googleapis.com/envoy.extensions.transport_sockets.internal_upstream.v3.InternalUpstreamTransport",
"passthrough_metadata": [
{
"kind": {
"cluster": {}
},
"name": "envoy.network"
},
{
"kind": {
"host": {}
},
"name": "envoy.network"
}
],
"transport_socket": {
"name": "envoy.transport_sockets.raw_buffer",
"typed_config": {
"@type": "type.googleapis.com/envoy.extensions.transport_sockets.raw_buffer.v3.RawBuffer"
}
}
}
},
"load_assignment": {
"cluster_name": "encap_0.0.0.0:16080",
"endpoints": [
{
"lb_endpoints": [
{
"endpoint": {
"address": {
"envoy_internal_address": {
"server_listener_name": "encap_0.0.0.0:16080"
}
}
}
}
]
}
]
},
"respect_dns_ttl": true
},
"last_updated": "2024-05-31T21:22:13.287Z"
}
]
},
{
"@type": "type.googleapis.com/envoy.admin.v3.ListenersConfigDump",
"version_info": "1717190533136161000",
"dynamic_listeners": [
{
"name": "tcp_0.0.0.0:16443",
"active_state": {
"version_info": "1717190533136161000",
"listener": {
"@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "tcp_0.0.0.0:16443",
"address": {
"socket_address": {
"address": "0.0.0.0",
"port_value": 16443
}
},
"filter_chains": [
{
"filters": [
{
"name": "envoy.tcp_proxy",
"typed_config": {
"@type": "type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy",
"stat_prefix": "tcp",
"cluster": "connect_upstream_cluster",
"access_log": [
{
"name": "envoy.access_loggers.file",
"typed_config": {
"@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog",
"path": "/dev/stdout",
"log_format": {
"json_format": {
"upstreamRemoteAddress": "%UPSTREAM_REMOTE_ADDRESS%",
"bytesReceived": "%BYTES_RECEIVED%",
"bytesSent": "%BYTES_SENT%",
"downstreamRemoteAddress": "%DOWNSTREAM_REMOTE_ADDRESS%",
"duration": "%DURATION%",
"requestedServerName": "%REQUESTED_SERVER_NAME%",
"startTime": "%START_TIME%",
"responseFlags": "%RESPONSE_FLAGS%",
"upstreamCluster": "%UPSTREAM_CLUSTER%"
}
}
}
}
],
"tunneling_config": {
"hostname": "%REQUESTED_SERVER_NAME%:443",
"headers_to_add": [
{
"header": {
"key": "Proxy-Authorization",
"value": "Basic dXNlcjE6cGFzc3dvcmQx"
}
}
]
}
}
}
]
}
],
"listener_filters": [
{
"name": "envoy.filters.listener.tls_inspector",
"typed_config": {
"@type": "type.googleapis.com/envoy.extensions.filters.listener.tls_inspector.v3.TlsInspector"
}
}
],
"enable_reuse_port": false
},
"last_updated": "2024-05-31T21:22:13.290Z"
}
},
{
"name": "tcp_0.0.0.0:16080",
"active_state": {
"version_info": "1717190533136161000",
"listener": {
"@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "tcp_0.0.0.0:16080",
"address": {
"socket_address": {
"address": "0.0.0.0",
"port_value": 16080
}
},
"filter_chains": [
{
"filter_chain_match": {
"application_protocols": [
"http/1.1",
"h2c",
"http/1.0",
"h3",
"h2"
]
},
"filters": [
{
"name": "envoy.filters.network.http_connection_manager",
"typed_config": {
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
"stat_prefix": "http",
"route_config": {
"name": "http",
"virtual_hosts": [
{
"name": "default",
"domains": [
"*"
],
"routes": [
{
"match": {
"prefix": "/"
},
"route": {
"cluster": "encap_0.0.0.0:16080"
}
}
]
}
]
},
"http_filters": [
{
"name": "envoy.filters.http.set_filter_state",
"typed_config": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.set_filter_state.v3.Config",
"on_request_headers": [
{
"object_key": "envoy.network.upstream_server_name",
"format_string": {
"text_format_source": {
"inline_string": "%REQ(:AUTHORITY)%"
}
},
"shared_with_upstream": "TRANSITIVE"
}
]
}
},
{
"name": "envoy.filters.http.router",
"typed_config": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router"
}
}
],
"http2_protocol_options": {
"allow_connect": true
},
"access_log": [
{
"name": "envoy.access_loggers.file",
"typed_config": {
"@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog",
"path": "/dev/stdout",
"log_format": {
"json_format": {
"protocol": "%PROTOCOL%",
"responseCodeDetails": "%RESPONSE_CODE_DETAILS%",
"requestedServerName": "%REQUESTED_SERVER_NAME%",
"upstreamCluster": "%UPSTREAM_CLUSTER%",
"path": "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%",
"downstreamRemoteAddress": "%DOWNSTREAM_REMOTE_ADDRESS%",
"requestMethod": "%REQ(:METHOD)%",
"responseCode": "%RESPONSE_CODE%",
"bytesReceived": "%BYTES_RECEIVED%",
"duration": "%DURATION%",
"responseFlags": "%RESPONSE_FLAGS%",
"requestId": "%REQ(X-REQUEST-ID)%",
"startTime": "%START_TIME%",
"upstreamRemoteAddress": "%UPSTREAM_REMOTE_ADDRESS%",
"bytesSent": "%BYTES_SENT%",
"upstreamProtocol": "%UPSTREAM_PROTOCOL%",
"requestAuthority": "%REQ(:AUTHORITY)%"
}
}
}
}
],
"use_remote_address": true,
"upgrade_configs": [
{
"upgrade_type": "CONNECT"
},
{
"upgrade_type": "websocket",
"filters": [
{
"name": "envoy.filters.http.set_filter_state",
"typed_config": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.set_filter_state.v3.Config",
"on_request_headers": [
{
"object_key": "envoy.network.upstream_server_name",
"format_string": {
"text_format_source": {
"inline_string": "%REQ(:AUTHORITY)%"
}
},
"shared_with_upstream": "TRANSITIVE"
}
]
}
},
{
"name": "envoy.filters.http.router",
"typed_config": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router"
}
}
]
}
],
"append_x_forwarded_port": true
}
}
]
}
],
"listener_filters": [
{
"name": "envoy.filters.listener.http_inspector",
"typed_config": {
"@type": "type.googleapis.com/envoy.extensions.filters.listener.http_inspector.v3.HttpInspector"
}
}
],
"enable_reuse_port": false
},
"last_updated": "2024-05-31T21:22:13.292Z"
}
},
{
"name": "encap_0.0.0.0:16080",
"active_state": {
"version_info": "1717190533136161000",
"listener": {
"@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "encap_0.0.0.0:16080",
"filter_chains": [
{
"filters": [
{
"name": "envoy.tcp_proxy",
"typed_config": {
"@type": "type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy",
"stat_prefix": "tcp",
"cluster": "connect_upstream_cluster",
"access_log": [
{
"name": "envoy.access_loggers.file",
"typed_config": {
"@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog",
"path": "/dev/stdout",
"log_format": {
"json_format": {
"upstreamCluster": "%UPSTREAM_CLUSTER%",
"responseFlags": "%RESPONSE_FLAGS%",
"bytesSent": "%BYTES_SENT%",
"downstreamRemoteAddress": "%DOWNSTREAM_REMOTE_ADDRESS%",
"duration": "%DURATION%",
"requestedServerName": "%REQUESTED_SERVER_NAME%",
"startTime": "%START_TIME%",
"upstreamRemoteAddress": "%UPSTREAM_REMOTE_ADDRESS%",
"bytesReceived": "%BYTES_RECEIVED%"
}
}
}
}
],
"tunneling_config": {
"hostname": "%DYNAMIC_METADATA(envoy.network:upstream_server_name)%:80",
"headers_to_add": [
{
"header": {
"key": "Proxy-Authorization",
"value": "Basic dXNlcjE6cGFzc3dvcmQx"
}
}
]
}
}
}
]
}
],
"internal_listener": {}
},
"last_updated": "2024-05-31T21:22:13.292Z"
}
}
]
},
{
"@type": "type.googleapis.com/envoy.admin.v3.ScopedRoutesConfigDump"
},
{
"@type": "type.googleapis.com/envoy.admin.v3.RoutesConfigDump",
"static_route_configs": [
{
"route_config": {
"@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"name": "http",
"virtual_hosts": [
{
"name": "default",
"domains": [
"*"
],
"routes": [
{
"match": {
"prefix": "/"
},
"route": {
"cluster": "encap_0.0.0.0:16080"
}
}
]
}
]
},
"last_updated": "2024-05-31T21:22:13.291Z"
}
]
},
{
"@type": "type.googleapis.com/envoy.admin.v3.SecretsConfigDump"
}
]
}
I should also point out that the tunnelling for TLS traffic on 16443 is working properly. The issue is with listener on 16080 as that is the one intended to handle the plaintext HTTP encapsulation in a CONNECT.
This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
I'd still like help if possible. I'm not able to tag it as "help wanted" on my own.
This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.
"hostname": "%DYNAMIC_METADATA(envoy.network:upstream_server_name)%:80
should not it be network.upstream_server_name instead of network:upstream_server_name?
Another thing I am not sure is that you are setting filter state and querying dynamic metadata.
@marc-barry PTAL at https://github.com/envoyproxy/envoy/pull/35559
This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.
This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.