Backend and BackendTLSPolicy stops working after v1.5.6

[th0th]

Description: I have a Backend + BackendTLSPolicy config working on v1.5.6. I use it for proxying an external URL from a subpath of an HTTPRoute. It stops working as I upgrade to v1.6.0.

Repro steps:

Backend:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: Backend
metadata:
  name: backend
spec:
  endpoints:
    - fqdn:
        hostname: httpshost.tld
        port: 443

BackendTLSPolicy:

apiVersion: gateway.networking.k8s.io/v1alpha3
kind: BackendTLSPolicy
metadata:
  name: backendtlspolicy
spec:
  targetRefs:
    - group: gateway.envoyproxy.io
      kind: Backend
      name: backend
  validation:
    hostname: httpshost.tld
    wellKnownCACertificates: System

HTTPRoute:

...
    - backendRefs:
        - group: gateway.envoyproxy.io
          kind: Backend
          name: backend
          port: 443
      filters:
        - type: URLRewrite
          urlRewrite:
            hostname: httpshost.tld
            path:
              replaceFullPath: /
              type: ReplaceFullPath
      matches:
        - path:
            type: PathPrefix
            value: /some-path
...

[zirain]

can you share the config_dump before and after?

[th0th]

Here is the diff outputs of egctl config envoy-proxy cluster between v1.5.6 and v1.6.0:

$ diff 1.6.0.json 1.5.6.json

93c93,100
<                       "loadBalancingWeight": 1
---
>                       "loadBalancingWeight": 1,
>                       "metadata": {
>                         "filterMetadata": {
>                           "envoy.transport_socket_match": {
>                             "name": "httproute/webgazer-development/webgazer-frontend/rule/1/tls/0"
>                           }
>                         }
>                       }
146a154,189
>             "transportSocketMatches": [
>               {
>                 "match": {
>                   "name": "httproute/webgazer-development/webgazer-frontend/rule/1/tls/0"
>                 },
>                 "name": "httproute/webgazer-development/webgazer-frontend/rule/1/tls/0",
>                 "transportSocket": {
>                   "name": "envoy.transport_sockets.tls",
>                   "typedConfig": {
>                     "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext",
>                     "commonTlsContext": {
>                       "combinedValidationContext": {
>                         "defaultValidationContext": {
>                           "matchTypedSubjectAltNames": [
>                             {
>                               "matcher": {
>                                 "exact": "httpshost.tld"
>                               },
>                               "sanType": "DNS"
>                             }
>                           ]
>                         },
>                         "validationContextSdsSecretConfig": {
>                           "name": "webgazer-unius-webhooks/webgazer-development-ca",
>                           "sdsConfig": {
>                             "ads": {},
>                             "resourceApiVersion": "V3"
>                           }
>                         }
>                       }
>                     },
>                     "sni": "httpshost.tld"
>                   }
>                 }
>               }
>             ],
149,150c192,193
<           "lastUpdated": "2025-12-10T00:32:34.635Z",
<           "versionInfo": "4a3e17044e84fccbe1c5b8cfc66827b448bd4fff1ee9c8515cd897ddd2eaaeac"
---
>           "lastUpdated": "2025-12-10T00:37:00.132Z",
>           "versionInfo": "9b89a340ead864ec7ce750f129ed0210a437ab212f14ceac9868d98526a6feac"
347c390
<       "versionInfo": "10"
---
>       "versionInfo": "1"

[arkodg]

what does the SAN on the server cert look like ? you can use something like openssl x509 -in cert.pem -noout -text

[th0th]

what does the SAN on the server cert look like ? you can use something like openssl x509 -in cert.pem -noout -text

Do you mean the server that is running the external URL? It has common name, and a wild-card subdomain alternative name matching the host on Backend here. It is a bit all over the place because I redacted some of the information I know. Would it help if I created a reproducible example, and shared the resource yamls?

[arkodg]

cc @zhaohuabing @guydc

[DerrickMartinez]

We're seeing the same thing. Adding insecureSkipVerify will fix it but no longer able to validate normally. You can test with the following example:

apiVersion: gateway.networking.k8s.io/v1alpha3
kind: BackendTLSPolicy
metadata:
  name: host-backend-tls-sendgrid
spec:
  targetRefs:
  - group: gateway.envoyproxy.io
    kind: Backend
    name: my.host.com-sendgrid
  validation:
    hostname: sendgrid.net
    wellKnownCACertificates: System
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: Backend
metadata:
  name: my.host.com-sendgrid
spec:
  endpoints:
  - fqdn:
      hostname: sendgrid.net
      port: 443
  type: Endpoints