flagger icon indicating copy to clipboard operation
flagger copied to clipboard
verified publisher icon fluxcd

Can virtual service created by flagger integrate with my own virtual service, thanks.

Open johnzheng1975 opened this issue 3 years ago • 6 comments

Describe the feature

Can virtual service created by flagger integrate with my own virtual service, thanks.

What problem are you trying to solve? Putting virtual service and CORS into canary is a bit complex. Can I create my own virtual service/ CORS under gateway still? And flagger only create percentage virtual service/ destinationrule for this service?

Can they work together well? Thus will bring more flexible.

Thanks a lot.

### Proposed solution Can this works?

Canary.yaml

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  labels:
    app.kubernetes.io/managed-by: Helm
    helm.toolkit.fluxcd.io/name: golang-try01-quickstart
    helm.toolkit.fluxcd.io/namespace: onecloud-dev
  name: golang-try01-quickstart
  namespace: onecloud-dev
spec:
  analysis:
    interval: 60s
    maxWeight: 90
    metrics:
    - interval: 30s
      name: request-success-rate-onecloud
      thresholdRange:
        min: 99
    stepWeight: 5
    threshold: 2
  autoscalerRef:
    apiVersion: autoscaling/v2beta1
    kind: HorizontalPodAutoscaler
    name: golang-try01-quickstart
  progressDeadlineSeconds: 1200
  service:
    port: 80
    portDiscovery: true
    targetPort: 8080
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: golang-try01-quickstart

My own public virtual service, FYI. (will be more complex later, put it outside is more flexible)

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  labels:
    app.kubernetes.io/managed-by: Helm
    helm.toolkit.fluxcd.io/name: golang-try01-quickstart
    helm.toolkit.fluxcd.io/namespace: onecloud-dev
  name: golang-try01-quickstart-apigee
  namespace: onecloud-dev
spec:
  gateways:
  - istio-system/apigee-gateway
  hosts:
  - golang-try01-quickstart-onecloud-dev.api.non-prod-uw2.xxxxx.io
  http:
  - match:
    - uri:
        regex: ^/try01/.+$
    route:
    - destination:
        host: golang-try01-quickstart
        port:
          number: 80

The canary created by canary is:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: golang-try01-quickstart
  namespace: onecloud-dev
spec:
  gateways:
  - mesh
  hosts:
  - golang-try01-quickstart
  http:
  - route:
    - destination:
        host: golang-try01-quickstart-primary
      weight: 100
    - destination:
        host: golang-try01-quickstart-canary
      weight: 0

The destination rule created by canary is:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: golang-try01-quickstart-canary
  namespace: onecloud-dev
spec:
  host: golang-try01-quickstart-canary
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: golang-try01-quickstart-primary
  namespace: onecloud-dev
spec:
  host: golang-try01-quickstart-primary

The service created by canary

(newenv):  $ k get virtualservice -A | grep try01
onecloud-dev       golang-try01-quickstart                          ["mesh"]                           ["golang-try01-quickstart"]                                                                
onecloud-dev       golang-try01-quickstart-apigee                   ["istio-system/apigee-gateway"]    ["golang-try01-quickstart-onecloud-dev.api.non-prod-uw2.hponecloud.io"] 
onecloud-dev       golang-try01-quickstart-private                  ["istio-system/private-gateway"]   ["golang-try01-quickstart-onecloud-dev.priv.non-prod-uw2.hponecloud.io"]

What do you want to happen? Add any considered drawbacks. When I call the domain form outside

curl golang-try01-quickstart-onecloud-dev.api.non-prod-uw2.xxxxx.io/try01/health

Expect result: The load should go to canary pods also, based on weight

Actual result: However, currently all of them go into primary pods.

My question: Is possible for flagger to support my requirements? Thanks very much

johnzheng1975 avatar Apr 18 '22 19:04 johnzheng1975

Using a delegate virtual service is going to be the best path forward.

Create a top level virtual service (your public one, and then delegate to the virtual service from the canary).

knechtionscoding avatar Apr 18 '22 20:04 knechtionscoding

@KnechtionsCoding , thanks a lot Can you kindly point out what is the problem in my upper configure? Form https://github.com/fluxcd/flagger/issues/274#issuecomment-523093368, seems this should works. but not sure what is the problem in my configure.

johnzheng1975 avatar Apr 18 '22 20:04 johnzheng1975

Your canary object should look something like:

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: podinfo-istio
spec:
  provider: istio
  # deployment reference
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: podinfo-istio
  autoscalerRef:
    apiVersion: autoscaling/v2beta2
    kind: HorizontalPodAutoscaler
    name: podinfo-istio
  service:
    # ClusterIP port number
    port: 9898
    # container port number or name (optional)
    targetPort: http
    portDiscovery: true
    delegation: true
    trafficPolicy:
      tls:
        mode: ISTIO_MUTUAL
    # # Istio retry policy (optional)
    retries:
      attempts: 3
      perTryTimeout: 1s
      retryOn: "gateway-error,connect-failure,refused-stream"
  ...

Notice the part around delegate and no mesh/hosts.

Then your public virtual service can look something like, with all the options you'd like:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: public vs
spec:
  gateways:
    - <ingress-gateway>
    - mesh
  hosts:
    - <external-url>
    - podinfo-istio
    - podinfo-istio.demo
    - podinfo-istio.demo.svc.cluster.local
    - podinfo-istio-canary
    - podinfo-istio-canary.demo
    - podinfo-istio-canary.demo.svc.cluster.local
  http:
  - match:
    - uri:
        prefix: /
    name: "application"
    delegate:
      name: podinfo-istio # <-- Should route to pod info istio
      namespace: demo

knechtionscoding avatar Apr 18 '22 21:04 knechtionscoding

Still not work.

  • If call with root@nginx:/# curl golang-try01-quickstart.onecloud-dev/try01/version?1111111, the load can enter canary pods.
  • If call from outside, return 503. Here is my public virtual service
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  annotations:
    meta.helm.sh/release-name: golang-try01-quickstart
    meta.helm.sh/release-namespace: onecloud-dev
  creationTimestamp: "2022-04-13T02:29:34Z"
  generation: 2
  labels:
    app.kubernetes.io/managed-by: Helm
    helm.toolkit.fluxcd.io/name: golang-try01-quickstart
    helm.toolkit.fluxcd.io/namespace: onecloud-dev
  name: golang-try01-quickstart-apigee
  namespace: onecloud-dev
  resourceVersion: "9142765"
  uid: 04ca9e67-a118-441d-b1a0-d4ae23f2f9a0
spec:
  gateways:
  - istio-system/apigee-gateway
  - mesh
  hosts:
  - golang-try01-quickstart-onecloud-dev.api.non-prod-uw2.hponecloud.io
  - golang-try01-quickstart
  - golang-try01-quickstart.onecloud-dev
  - golang-try01-quickstart.onecloud-dev.svc.cluster.local
  - golang-try01-quickstart-canary
  - golang-try01-quickstart-canary.onecloud-dev
  - golang-try01-quickstart-canary.onecloud-dev.svc.cluster.local
  http:
  - match:
    - uri:
        prefix: /try01/
    delegate:
      name: golang-try01-quickstart
      namespace: onecloud-dev

Here is my canary:

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  annotations:
    meta.helm.sh/release-name: golang-try01-quickstart
    meta.helm.sh/release-namespace: onecloud-dev
  creationTimestamp: "2022-04-10T21:38:44Z"
  generation: 6
  labels:
    app.kubernetes.io/managed-by: Helm
    helm.toolkit.fluxcd.io/name: golang-try01-quickstart
    helm.toolkit.fluxcd.io/namespace: onecloud-dev
  name: golang-try01-quickstart
  namespace: onecloud-dev
  resourceVersion: "9221083"
  uid: ce561735-c174-4d55-92f2-036046271c5f
spec:
  analysis:
    interval: 60s
    maxWeight: 90
    metrics:
    - interval: 30s
      name: request-success-rate-onecloud
      thresholdRange:
        min: 99
    stepWeight: 5
    threshold: 2
    webhooks:
    - metadata:
        gitUrl: ssh://[email protected]/onecloud/golang-try01-quickstart.git
        project: onecloud-dev
        region: uw2
        release: v0.0.25
        service: golang-try01-quickstart
        stack: dev
      name: devex
      type: post-rollout
      url: http://finch.infra/devex
  autoscalerRef:
    apiVersion: autoscaling/v2beta1
    kind: HorizontalPodAutoscaler
    name: golang-try01-quickstart
  progressDeadlineSeconds: 1200
  service:
    port: 80
    portDiscovery: true
    targetPort: 8080
    delegation: true
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: golang-try01-quickstart

Here is virtualservice generated by canary:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: golang-try01-quickstart
  namespace: onecloud-dev
... ...
spec:
  hosts: []
  http:
  - route:
    - destination:
        host: golang-try01-quickstart-primary
      weight: 35
    - destination:
        host: golang-try01-quickstart-canary
      weight: 65

johnzheng1975 avatar Apr 18 '22 22:04 johnzheng1975

@stefanprodan Do you have some suggestion for this? Thanks a lot

johnzheng1975 avatar Apr 19 '22 00:04 johnzheng1975

I tried successfully with your official example https://docs.flagger.app/faq Note that there are some error for your official docs - I can update your example with a PR later. Additional question, does delegate support multiple ports in a service -- 80 for restful and 3000 for grpc on a single service, thanks.

johnzheng1975 avatar Apr 20 '22 08:04 johnzheng1975

@johnzheng1975 Do you find solution about delegate support multiple ports ?

jnac1984 avatar Nov 16 '22 14:11 jnac1984