envoyproxy
Listener is not created when there are multiple multiple listeners having same protocols and ports but different hostname
Description:
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
name: infra-gateway
spec:
gatewayClassName: infra-gateway
listeners:
- name: grpc-https
hostname: 'grpc.example.com'
protocol: HTTPS
port: 443
allowedRoutes:
namespaces:
from: Same
tls:
certificateRefs:
- group: ""
kind: Secret
name: infra-gateway-grpc-tls
mode: Terminate
- name: wildcard-https
hostname: '*.example.com'
protocol: HTTPS
port: 443
allowedRoutes:
namespaces:
from: Same
tls:
certificateRefs:
- group: ""
kind: Secret
name: infra-gateway-wildcard-tls
Having the above configuration should create two different listeners. However it create only one listener grpc-https.
Repro steps:
Apply the above configuration and then run egctl and look for the wildcard listener. It won't be in place.
Environment:
envoy-gateway v0.6.0
A configuration like that it should be possible as recommended by cert-manager https://cert-manager.io/docs/usage/gateway/#two-listeners-with-the-same-secret-name
Also from upstream config https://gateway-api.sigs.k8s.io/v1alpha2/guides/tls/ it should be possible.
This issue has been automatically marked as stale because it has not had activity in the last 30 days.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This issue has been automatically marked as stale because it has not had activity in the last 30 days.
I'm having similar issue, but in my case problem is multiple TLS listeners with other mode and hostnames:
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
labels:
argocd.argoproj.io/instance: cluster-config
name: default
namespace: envoy-gateway-system
spec:
gatewayClassName: envoy-gateway
listeners:
- allowedRoutes:
namespaces:
from: All
name: http-0
port: 80
protocol: HTTP
- allowedRoutes:
namespaces:
from: All
name: tls-pass-0
port: 443
protocol: TLS
tls:
mode: Passthrough
- allowedRoutes:
namespaces:
from: All
hostname: '*.p.<redacted>'
name: https-0
port: 443
protocol: HTTPS
tls:
certificateRefs:
- group: ''
kind: Secret
name: <redacted>
mode: Terminate
- allowedRoutes:
namespaces:
from: All
hostname: ssh.<redacted>
name: tls-term-0
port: 443
protocol: TLS
tls:
certificateRefs:
- group: ''
kind: Secret
name: <redacted>
mode: Terminate
tls-pass-0 is accepted, https-0 is accepted and tls-term-0 is not accepted, probably when adding more tls-terms or even httpses those won't be accepted too even with other hostnames
Status:
status:
addresses:
- type: IPAddress
value: 192.168.64.209
conditions:
- lastTransitionTime: '2024-03-28T11:32:57Z'
message: The Gateway has been scheduled by Envoy Gateway
observedGeneration: 5
reason: Accepted
status: 'True'
type: Accepted
- lastTransitionTime: '2024-03-28T11:32:57Z'
message: 'Address assigned to the Gateway, 1/1 envoy Deployment replicas available'
observedGeneration: 5
reason: Programmed
status: 'True'
type: Programmed
listeners:
- attachedRoutes: 3
conditions:
- lastTransitionTime: '2024-03-28T11:32:57Z'
message: Sending translated listener configuration to the data plane
observedGeneration: 5
reason: Programmed
status: 'True'
type: Programmed
- lastTransitionTime: '2024-03-28T11:32:57Z'
message: Listener has been successfully translated
observedGeneration: 5
reason: Accepted
status: 'True'
type: Accepted
- lastTransitionTime: '2024-03-28T11:32:57Z'
message: Listener references have been resolved
observedGeneration: 5
reason: ResolvedRefs
status: 'True'
type: ResolvedRefs
name: http-0
supportedKinds:
- group: gateway.networking.k8s.io
kind: HTTPRoute
- group: gateway.networking.k8s.io
kind: GRPCRoute
- attachedRoutes: 3
conditions:
- lastTransitionTime: '2024-03-28T11:32:57Z'
message: Sending translated listener configuration to the data plane
observedGeneration: 5
reason: Programmed
status: 'True'
type: Programmed
- lastTransitionTime: '2024-03-28T11:32:57Z'
message: Listener has been successfully translated
observedGeneration: 5
reason: Accepted
status: 'True'
type: Accepted
- lastTransitionTime: '2024-03-28T11:32:57Z'
message: Listener references have been resolved
observedGeneration: 5
reason: ResolvedRefs
status: 'True'
type: ResolvedRefs
name: tls-pass-0
supportedKinds:
- group: gateway.networking.k8s.io
kind: TLSRoute
- attachedRoutes: 0
conditions:
- lastTransitionTime: '2024-03-28T11:32:57Z'
message: Sending translated listener configuration to the data plane
observedGeneration: 5
reason: Programmed
status: 'True'
type: Programmed
- lastTransitionTime: '2024-03-28T11:32:57Z'
message: Listener has been successfully translated
observedGeneration: 5
reason: Accepted
status: 'True'
type: Accepted
- lastTransitionTime: '2024-03-28T11:32:57Z'
message: Listener references have been resolved
observedGeneration: 5
reason: ResolvedRefs
status: 'True'
type: ResolvedRefs
name: https-0
supportedKinds:
- group: gateway.networking.k8s.io
kind: HTTPRoute
- group: gateway.networking.k8s.io
kind: GRPCRoute
- attachedRoutes: 1
conditions:
- lastTransitionTime: '2024-03-28T11:32:57Z'
message: Only one TCP/TLS listener is allowed in a given port
observedGeneration: 5
reason: ProtocolConflict
status: 'True'
type: Conflicted
- lastTransitionTime: '2024-03-28T11:32:57Z'
message: 'Listener is invalid, see other Conditions for details.'
observedGeneration: 5
reason: Invalid
status: 'False'
type: Programmed
- lastTransitionTime: '2024-03-28T11:32:57Z'
message: Listener references have been resolved
observedGeneration: 5
reason: ResolvedRefs
status: 'True'
type: ResolvedRefs
name: tls-term-0
supportedKinds:
- group: gateway.networking.k8s.io
kind: TCPRoute
Seems like the same issue like here: https://github.com/envoyproxy/gateway/issues/2964 I will have some time now to debug this
I have tested it against the gateway with multiple listeners
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
name: infra-gateway
spec:
gatewayClassName: eg-internal
listeners:
- name: company-https
hostname: '*.company.com'
protocol: HTTPS
port: 443
allowedRoutes:
namespaces:
from: Same
tls:
certificateRefs:
- group: ""
kind: Secret
name: company-tls
namespace: envoy-gateway-system
mode: Terminate
- name: example-https
hostname: www.example.com
protocol: HTTPS
port: 443
allowedRoutes:
namespaces:
from: Same
tls:
certificateRefs:
- group: ""
kind: Secret
name: example-tls
namespace: envoy-gateway-system
mode: Terminate
Without merged gateways, that has another bug related to infra provisioning, I could successfully test out this gateway with two different services. I have checked that XDS IR is translated properly to match different apps.
Company listener:
curl -v -HHost:'*.company.com' --resolve "sub.company.com:443:172.18.255.200" \
--cacert rsa-cert-wildcard.pem https://'sub.company.com'/get
* Added sub.company.com:443:172.18.255.200 to DNS cache
* Hostname sub.company.com was found in DNS cache
* Trying 172.18.255.200:443...
* Connected to sub.company.com (172.18.255.200) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* CAfile: rsa-cert-wildcard.pem
* CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server accepted h2
* Server certificate:
* subject: CN=Test Inc
* start date: Mar 19 16:04:47 2024 GMT
* expire date: Mar 19 16:04:47 2025 GMT
* subjectAltName: host "sub.company.com" matched cert's "*.company.com"
* issuer: CN=Test Inc
* SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://sub.company.com/get
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: *.company.com]
* [HTTP/2] [1] [:path: /get]
* [HTTP/2] [1] [user-agent: curl/8.4.0]
* [HTTP/2] [1] [accept: */*]
> GET /get HTTP/2
> Host:*.company.com
> User-Agent: curl/8.4.0
> Accept: */*
>
< HTTP/2 200
< content-type: application/json
< x-content-type-options: nosniff
< date: Fri, 29 Mar 2024 15:13:34 GMT
< content-length: 474
<
{
"path": "/get",
"host": "*.company.com",
"method": "GET",
"proto": "HTTP/1.1",
"headers": {
"Accept": [
"*/*"
],
"User-Agent": [
"curl/8.4.0"
],
"X-Envoy-Internal": [
"true"
],
"X-Forwarded-For": [
"172.18.0.3"
],
"X-Forwarded-Proto": [
"https"
],
"X-Request-Id": [
"da1adb99-00a7-401e-bc19-be3d0a1f5203"
]
},
"namespace": "envoy-gateway-system",
"ingress": "",
"service": "",
"pod": "company-service-78d6759b89-kjp67"
* Connection #0 to host sub.company.com left intact
}
Example listener:
curl -v -HHost:www.example.com --resolve "www.example.com:443:172.18.255.200" \ ✔ 16:13:34
--cacert www.example.com.crt https://www.example.com/get
* Added www.example.com:443:172.18.255.200 to DNS cache
* Hostname www.example.com was found in DNS cache
* Trying 172.18.255.200:443...
* Connected to www.example.com (172.18.255.200) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* CAfile: www.example.com.crt
* CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server accepted h2
* Server certificate:
* subject: CN=www.example.com; O=example organization
* start date: Mar 19 16:13:34 2024 GMT
* expire date: Mar 19 16:13:34 2025 GMT
* common name: www.example.com (matched)
* issuer: O=example Inc.; CN=example.com
* SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://www.example.com/get
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: www.example.com]
* [HTTP/2] [1] [:path: /get]
* [HTTP/2] [1] [user-agent: curl/8.4.0]
* [HTTP/2] [1] [accept: */*]
> GET /get HTTP/2
> Host:www.example.com
> User-Agent: curl/8.4.0
> Accept: */*
>
< HTTP/2 200
< content-type: application/json
< x-content-type-options: nosniff
< date: Fri, 29 Mar 2024 15:14:23 GMT
< content-length: 476
<
{
"path": "/get",
"host": "www.example.com",
"method": "GET",
"proto": "HTTP/1.1",
"headers": {
"Accept": [
"*/*"
],
"User-Agent": [
"curl/8.4.0"
],
"X-Envoy-Internal": [
"true"
],
"X-Forwarded-For": [
"172.18.0.3"
],
"X-Forwarded-Proto": [
"https"
],
"X-Request-Id": [
"47eea4cf-afb7-4336-9088-5998b8ab3ddc"
]
},
"namespace": "envoy-gateway-system",
"ingress": "",
"service": "",
"pod": "example-service-55694dc865-wvn6c"
* Connection #0 to host www.example.com left intact
}
Please note that for the Infra IR there will be only one Infra IR listener for the multiple Gateway listeners configured with the same port value, as it is directly matched to k8s envoy infrastructure service port. We are working on adding unified port naming for the Envoy infra and probably could consider to somehow extend Infra IR output to give more valuable information to the users on top, when dealing with such scenarios.
This issue has been automatically marked as stale because it has not had activity in the last 30 days.
This issue has been automatically marked as stale because it has not had activity in the last 30 days.
hey, I concluded my debugging here https://github.com/envoyproxy/gateway/issues/1861#issuecomment-2032093712 I couldn't reproduce this issue, even with the same protocol and port but a different hostname, I could match proper backend service. The outcome of it would be to find out if we can serve IR listeners on different ports in the service when they are duplicated
there are 2 configs provided by users 1)
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
name: infra-gateway
spec:
gatewayClassName: infra-gateway
listeners:
- name: grpc-https
hostname: 'grpc.example.com'
protocol: HTTPS
port: 443
allowedRoutes:
namespaces:
from: Same
tls:
certificateRefs:
- group: ""
kind: Secret
name: infra-gateway-grpc-tls
mode: Terminate
- name: wildcard-https
hostname: '*.example.com'
protocol: HTTPS
port: 443
allowedRoutes:
namespaces:
from: Same
tls:
certificateRefs:
- group: ""
kind: Secret
name: infra-gateway-wildcard-tls
and
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
labels:
argocd.argoproj.io/instance: cluster-config
name: default
namespace: envoy-gateway-system
spec:
gatewayClassName: envoy-gateway
listeners:
- allowedRoutes:
namespaces:
from: All
name: http-0
port: 80
protocol: HTTP
- allowedRoutes:
namespaces:
from: All
name: tls-pass-0
port: 443
protocol: TLS
tls:
mode: Passthrough
- allowedRoutes:
namespaces:
from: All
hostname: '*.p.<redacted>'
name: https-0
port: 443
protocol: HTTPS
tls:
certificateRefs:
- group: ''
kind: Secret
name: <redacted>
mode: Terminate
- allowedRoutes:
namespaces:
from: All
hostname: ssh.<redacted>
name: tls-term-0
port: 443
protocol: TLS
tls:
certificateRefs:
- group: ''
kind: Secret
name: <redacted>
mode: Terminate
hey @cnvergence are we unable to repro 1. or 2. or both ?
hey @arkodg, I tried both of them, like I said the traffic passed in my case properly to the correct backend. But it might be that other folks want to have those listeners created as separate ports in the Kubernetes service, we don't support it currently, as we share ports (using 443 for all of them). IMO, this is still something we could take a look at and create new ports on the service when there is a duplicate between listeners under the same gateway, I think Contour has something working for this case.
@cnvergence did you try on v0.0.0-latest ? I think we made some refactors that improved/removed these constraints
https://github.com/envoyproxy/gateway/pull/3271
conditions:
- lastTransitionTime: '2024-03-28T11:32:57Z'
message: Only one TCP/TLS listener is allowed in a given port
observedGeneration: 5
reason: ProtocolConflict
status: 'True'
type: Conflicted
- lastTransitionTime: '2024-03-28T11:32:57Z'
message: 'Listener is invalid, see other Conditions for details.'
observedGeneration: 5
reason: Invalid
status: 'False'
type: Programmed
@JuniorJPDJ can you try with v0.0.0-latest to confirm that the issue you are facing is fixed ? the v1.1 minor release should be out in the coming weeks
@arkodg it looks like updating to latest version of v0.0.0-latest fixed the issue.
I haven't tried attaching any routes yet, but listener is accepted now.
awesome thanks for confirming @JuniorJPDJ, there should be a stable image with the fix out soon