Using Flagger with gateway api removes all custom filters and routing rules in HTTPRoute resource

[aravindhbw]

When creating a canary resource using Flagger, the canary deployment process creates a new HTTPRoute resource that overrides the existing httpbin HTTPRoute, causing the loss of all custom filters and routing rules.

Deployment

apiVersion: v1
kind: Service
metadata:
  name: httpbin
  labels:
    app: httpbin
    service: httpbin
spec:
  ports:
  - name: http
    port: 8000
    targetPort: 8080
  selector:
    app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
      version: v1
  template:
    metadata:
      labels:
        app: httpbin
        version: v1
    spec:
      serviceAccountName: httpbin
      containers:
      - image: docker.io/mccutchen/go-httpbin:v2.15.0
        imagePullPolicy: IfNotPresent
        name: httpbin
        ports:
        - containerPort: 8080
        env:
        - name: HTTPBIN_RESPONSE_HEADERS
          value: '{"X-Service": "httpbin", "X-Version": "v1", "X-Backend": "stable"}'

HTTPRoute

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: httpbin
spec:
  parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: apps
  hostnames:
    - "www.example.com"
  rules:
    # Rule 1: Exact path matching for health checks
    # curl --verbose --header "Host: www.example.com" http://$GATEWAY_HOST/health
    - matches:
        - path:
            type: Exact
            value: /health
      backendRefs:
        - group: ""
          kind: Service
          name: httpbin
          port: 8000
          weight: 100
      filters:
        - type: URLRewrite
          urlRewrite:
            path:
              type: ReplaceFullPath
              replaceFullPath: /status/200

    # Rule 2: Path prefix matching for httpbin (existing functionality)
    # curl --verbose --header "Host: www.example.com" http://$GATEWAY_HOST/httpbin/get
    - matches:
        - path:
            type: PathPrefix
            value: /httpbin/
      backendRefs:
        - group: ""
          kind: Service
          name: httpbin
          port: 8000
          weight: 100
      filters:
        - type: URLRewrite
          urlRewrite:
            path:
              type: ReplacePrefixMatch
              replacePrefixMatch: /

    # Rule 3: Method-based routing - GET requests to httpbin
    # curl --verbose --header "Host: www.example.com" http://$GATEWAY_HOST/api/get
    - matches:
        - path:
            type: PathPrefix
            value: /api/
          method: GET
      backendRefs:
        - group: ""
          kind: Service
          name: httpbin
          port: 8000
          weight: 100
      filters:
        - type: URLRewrite
          urlRewrite:
            path:
              type: ReplacePrefixMatch
              replacePrefixMatch: /

    # Rule 4: Query parameter routing - headers service
    # curl --verbose --header "Host: www.example.com" "http://$GATEWAY_HOST/query-test?service=httpbin"
    - matches:
        - path:
            type: PathPrefix
            value: /query-test
          queryParams:
            - name: service
              value: httpbin
      backendRefs:
        - group: ""
          kind: Service
          name: httpbin
          port: 8000
          weight: 100
      filters:
        - type: URLRewrite
          urlRewrite:
            path:
              type: ReplaceFullPath
              replaceFullPath: /headers
        - type: RequestHeaderModifier
          requestHeaderModifier:
            add:
              - name: X-Route-Type
                value: "query-param-routing"
              - name: X-Debug-Service
                value: "httpbin"

    # Rule 5: Header-based routing - API version
    # curl --verbose --header "Host: www.example.com" --header "X-API-Version: v2" http://$GATEWAY_HOST/header-test
    - matches:
        - path:
            type: PathPrefix
            value: /header-test
          headers:
            - name: X-API-Version
              value: v2
      backendRefs:
        - group: ""
          kind: Service
          name: httpbin
          port: 8000
          weight: 100
      filters:
        - type: URLRewrite
          urlRewrite:
            path:
              type: ReplaceFullPath
              replaceFullPath: /headers
        - type: RequestHeaderModifier
          requestHeaderModifier:
            add:
              - name: X-Route-Type
                value: "header-routing"
              - name: X-API-Version-Matched
                value: "v2"

    # Rule 6: Default fallback route
    # curl --verbose --header "Host: www.example.com" http://$GATEWAY_HOST/get
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - group: ""
          kind: Service
          name: httpbin
          port: 8000
          weight: 100
      filters:
        - type: RequestHeaderModifier
          requestHeaderModifier:
            add:
              - name: X-Route-Type
                value: "default"

Canary

---
# Simple Flagger Canary for httpbin with Gateway API
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: httpbin-canary
  namespace: default
spec:
  skipAnalysis: false
  provider: gatewayapi:v1
  # Target deployment
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: httpbin
  # Service port
  service:
    name: httpbin
    port: 8000
    targetPort: 8080
    # Gateway API configuration
    gatewayRefs:
    - name: apps
      namespace: default
    hosts:
    - www.example.com
  # Canary analysis configuration
  analysis:
    # Schedule interval (default 60s)
    interval: 30s
    # Max number of failed metric checks before rollback
    threshold: 2
    # Max traffic percentage routed to canary
    maxWeight: 50
    # Canary increment step
    stepWeight: 25

The result is that the existing httpbin HTTPRoute is updated with the required setup for canary but the routing rules and filters are removed and only a default rule to match path prefix "/" is created.

Expected behavior:

Preserve existing HTTPRoute rules and only modify the backend references/weights

[aravindhbw]

Looking at the docs here, there is some support for header manipulation and uri rewrites but it seems to be global at the Canary level, so it would apply to all routing rules in the HTTPRoute resource generated by Canary. We have use-cases where we want filters for request modification per routing rule and I am not sure if that is supported yet in Flagger.