charts add "How to set up an Ingress?" to docs

Hello

I'm trying to setup some airflow servers on separate on the same clusters I'm trying to have the following setup :

One ingress-nginx for my cluster
Multiple Airflow servers on separate namespaces (e.g dev, prod)
Generate through your charts the ingress setup

Is this possible ?

I already tried to use one ingress fer namespace and setup some ingress class name but I fall into some issue related to #425.

I'm not really used to Ingress setups, so please tell me if you have any lead to a better solution :)

Thanks in advance !

EDIT My environment :

kubectl

kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"8b5a19147530eaac9476b0ab82980b4088bbc1b2", GitTreeState:"clean", BuildDate:"2021-09-15T21:38:50Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"0b17c6315e806a66d507e77760a5d60ab5cccfd8", GitTreeState:"clean", BuildDate:"2021-08-30T01:42:22Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}

helm

helm version
version.BuildInfo{Version:"v3.7.0", GitCommit:"eeac83883cb4014fe60267ec6373570374ce770b", GitTreeState:"clean", GoVersion:"go1.16.8"}

Airflow: 1.10.15

Oct 16 '21 18:10 LvffY

To be more precise, I already a working setup with one ingress on a specific cluster and one airflow server across all my cluster.

To get that working, I use the following commands :

One airflow server across namespaces

Click to expand

Install nginx

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx --create-namespace --namespace ingress

Configure cert-manager

helm repo add jetstack https://charts.jetstack.io
helm repo update
helm upgrade --install cert-manager --create-namespace --namespace cert-manager --version v1.5.4 jetstack/cert-manager --set installCRDs=true

Install issuer and certificate.

Based on the following files

Issuer.yaml

---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: my_email
    privateKeySecretRef:
      name: letsencrypt-account-key
    solvers:
    - http01:
       ingress:
         class: nginx

Certificate.yaml

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: airflow-certificate
spec:
  secretName: airflow-certificate
  issuerRef:
    kind: Issuer
    name: letsencrypt
  dnsNames:
    - my_dns

kubectl apply -n airflow-ns -f issuer.yaml
kubectl apply -n airflow-ns -f certificate.yaml

Ensure my IP has a DNS (based on my Azure cloud setup)

IP=$(kubectl -n ingress get service ingress-nginx-controller -o jsonpath={.status.loadBalancer.ingress..ip})

DNSNAME="my_dns"

PUBLICIPID=$(az network public-ip list --query "[?ipAddress!=null]|[?contains(ipAddress, '$IP')].[id]" --output tsv)

az network public-ip update --ids $PUBLICIPID --dns-name $DNSNAME

Install airflow

helm upgrade --install airflow airflow-stable/airflow --namespace airflow-ns --version "8.5.2" --values values.yaml

Especially, the ingress section, is setup like

ingress:
  ## if we should deploy Ingress resources
  ##
  enabled: true

  ## the `apiVersion` to use for Ingress resources
  ## - for Kubernetes 1.19 and later: "networking.k8s.io/v1"
  ## - for Kubernetes 1.18 and before: "networking.k8s.io/v1beta1"
  ##
  apiVersion: networking.k8s.io/v1

  ## configs for the Ingress of the web Service
  ##
  web:
    ## annotations for the web Ingress
    ##
    annotations:
      kubernetes.io/ingress.class: nginx
      cert-manager.io/cluster-issuer: letsencrypt

    ## additional labels for the web Ingress
    ##
    labels: {}

    ## the path for the web Ingress
    ## - [WARNING] do NOT include the trailing slash (for root, set an empty string)
    ##
    ## ____ EXAMPLE _______________
    ##   # webserver URL: http://example.com/airflow
    ##   path: "/airflow"
    ##
    path: ""

    ## the hostname for the web Ingress
    ##
    host: "my_dns"

    ## configs for web Ingress TLS
    ##
    tls:
      ## enable TLS termination for the web Ingress
      ##
      enabled: true

      ## the name of a pre-created Secret containing a TLS private key and certificate
      ##
      secretName: "airflow-certificate"

    ## http paths to add to the web Ingress before the default path
    ##
    ## ____ EXAMPLE _______________
    ##   precedingPaths:
    ##     - path: "/*"
    ##       serviceName: "my-service"
    ##       servicePort: "port-name"
    ##
    precedingPaths: []

    ## http paths to add to the web Ingress after the default path
    ##
    ## ____ EXAMPLE _______________
    ##   succeedingPaths:
    ##     - path: "/extra-service"
    ##       serviceName: "my-service"
    ##       servicePort: "port-name"
    ##
    succeedingPaths: []

  ## configs for the Ingress of the flower Service
  ##
  flower:
    ## annotations for the flower Ingress
    ##
    annotations: {}

    ## additional labels for the flower Ingress
    ##
    labels: {}

    ## the path for the flower Ingress
    ## - [WARNING] do NOT include the trailing slash (for root, set an empty string)
    ##
    ## ____ EXAMPLE _______________
    ##   # flower URL: http://example.com/airflow/flower
    ##   path: "/airflow/flower"
    ##
    path: ""

    ## the hostname for the flower Ingress
    ##
    host: ""

    ## configs for flower Ingress TLS
    ##
    tls:
      ## enable TLS termination for the flower Ingress
      ##
      enabled: false

      ## the name of a pre-created Secret containing a TLS private key and certificate
      ##
      secretName: ""

    ## http paths to add to the flower Ingress before the default path
    ##
    ## ____ EXAMPLE _______________
    ##   precedingPaths:
    ##     - path: "/*"
    ##       serviceName: "my-service"
    ##       servicePort: "port-name"
    ##
    precedingPaths: []

    ## http paths to add to the flower Ingress after the default path
    ##
    ## ____ EXAMPLE _______________
    ##   succeedingPaths:
    ##     - path: "/extra-service"
    ##       serviceName: "my-service"
    ##       servicePort: "port-name"
    ##
    succeedingPaths: []

Result

This setup works fine, when going to my DNS address I have my Airflow UI and when inpecting the logs of my ingress controller, I can find the following lines ingress_ok

One airflow server by namespace

Click to expand

Install nginx

Based on this tutorial :

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm upgrade --install airflow-dem-soa-1-ingress ingress-nginx/ingress-nginx --namespace airflow-ns--set controller.ingressClassResource.name=my-class-name --set controller.ingressClassResource.controllerValue="k8s.io/my-class-name" --set controller.ingressClassResource.enabled=true --set controller.ingressClassByName=true

Configure cert-manager

helm repo add jetstack https://charts.jetstack.io
helm repo update
helm upgrade --install cert-manager --create-namespace --namespace cert-manager --version v1.5.4 jetstack/cert-manager --set installCRDs=true

Install issuer and certificate.

Based on the following files

Issuer.yaml

---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: my-class-name-issuer
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: my_email
    privateKeySecretRef:
      name: my-class-name-letsencrypt-account-key
    solvers:
    - http01:
       ingress:
         class: my-class-name

Certificate.yaml

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: my-class-name-certificate
spec:
  secretName: my-class-name-certificate
  issuerRef:
    kind: Issuer
    name: my-class-name-issuer
  dnsNames:
    - my_dns

kubectl apply -n airflow-ns -f issuer.yaml
kubectl apply -n airflow-ns -f certificate.yaml

Ensure my IP has a DNS (based on my Azure cloud setup)

IP=$(kubectl -n ingress get service ingress-nginx-controller -o jsonpath={.status.loadBalancer.ingress..ip})

DNSNAME="my_dns"

PUBLICIPID=$(az network public-ip list --query "[?ipAddress!=null]|[?contains(ipAddress, '$IP')].[id]" --output tsv)

az network public-ip update --ids $PUBLICIPID --dns-name $DNSNAME

Install airflow

helm upgrade --install airflow airflow-stable/airflow --namespace airflow-ns --version "8.5.2" --values values.yaml

Especially, the ingress section, is setup like

ingress:
  ## if we should deploy Ingress resources
  ##
  enabled: true

  ## the `apiVersion` to use for Ingress resources
  ## - for Kubernetes 1.19 and later: "networking.k8s.io/v1"
  ## - for Kubernetes 1.18 and before: "networking.k8s.io/v1beta1"
  ##
  apiVersion: networking.k8s.io/v1

  ## configs for the Ingress of the web Service
  ##
  web:
    ## annotations for the web Ingress
    ##
    annotations:
      kubernetes.io/ingress.class: my-class-name
      cert-manager.io/issuer: my-class-name-issuer

    ## additional labels for the web Ingress
    ##
    labels: {}

    ## the path for the web Ingress
    ## - [WARNING] do NOT include the trailing slash (for root, set an empty string)
    ##
    ## ____ EXAMPLE _______________
    ##   # webserver URL: http://example.com/airflow
    ##   path: "/airflow"
    ##
    path: ""

    ## the hostname for the web Ingress
    ##
    host: "my_dns"

    ## configs for web Ingress TLS
    ##
    tls:
      ## enable TLS termination for the web Ingress
      ##
      enabled: true

      ## the name of a pre-created Secret containing a TLS private key and certificate
      ##
      secretName: "my-class-name-certificate"

    ## http paths to add to the web Ingress before the default path
    ##
    ## ____ EXAMPLE _______________
    ##   precedingPaths:
    ##     - path: "/*"
    ##       serviceName: "my-service"
    ##       servicePort: "port-name"
    ##
    precedingPaths: []

    ## http paths to add to the web Ingress after the default path
    ##
    ## ____ EXAMPLE _______________
    ##   succeedingPaths:
    ##     - path: "/extra-service"
    ##       serviceName: "my-service"
    ##       servicePort: "port-name"
    ##
    succeedingPaths: []

  ## configs for the Ingress of the flower Service
  ##
  flower:
    ## annotations for the flower Ingress
    ##
    annotations: {}

    ## additional labels for the flower Ingress
    ##
    labels: {}

    ## the path for the flower Ingress
    ## - [WARNING] do NOT include the trailing slash (for root, set an empty string)
    ##
    ## ____ EXAMPLE _______________
    ##   # flower URL: http://example.com/airflow/flower
    ##   path: "/airflow/flower"
    ##
    path: ""

    ## the hostname for the flower Ingress
    ##
    host: ""

    ## configs for flower Ingress TLS
    ##
    tls:
      ## enable TLS termination for the flower Ingress
      ##
      enabled: false

      ## the name of a pre-created Secret containing a TLS private key and certificate
      ##
      secretName: ""

    ## http paths to add to the flower Ingress before the default path
    ##
    ## ____ EXAMPLE _______________
    ##   precedingPaths:
    ##     - path: "/*"
    ##       serviceName: "my-service"
    ##       servicePort: "port-name"
    ##
    precedingPaths: []

    ## http paths to add to the flower Ingress after the default path
    ##
    ## ____ EXAMPLE _______________
    ##   succeedingPaths:
    ##     - path: "/extra-service"
    ##       serviceName: "my-service"
    ##       servicePort: "port-name"
    ##
    succeedingPaths: []

Result

This setup doesn't work, when going to my DNS address I have my Airflow UI but noted as "unsecure" with no certificates found. Moreover, when going to my ingress controller logs, I have the following lines : ingress_ko

After some research, I fall into the official troubleshooting ingress-nginx and is marked as because "use of deprecated ingress class annotation".

I really think this is related to #425/#427/#426.

Am I right ? If so, do you ave any workaround and/or date to review #426 ?

Thanks in advance !

Oct 17 '21 18:10 LvffY

I seem to have a manual workaround. I just disable charts ingress creation and create one manually.

One airflow server by namespace

Click to expand

Install nginx

Based on this tutorial :

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm upgrade --install airflow-dem-soa-1-ingress ingress-nginx/ingress-nginx --namespace airflow-ns--set controller.ingressClassResource.name=my-class-name --set controller.ingressClassResource.controllerValue="k8s.io/my-class-name" --set controller.ingressClassResource.enabled=true --set controller.ingressClassByName=true

Configure cert-manager

helm repo add jetstack https://charts.jetstack.io
helm repo update
helm upgrade --install cert-manager --create-namespace --namespace cert-manager --version v1.5.4 jetstack/cert-manager --set installCRDs=true

Install issuer and certificate.

Based on the following files

Issuer.yaml

---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: my-class-name-issuer
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: my_email
    privateKeySecretRef:
      name: my-class-name-letsencrypt-account-key
    solvers:
    - http01:
       ingress:
         class: my-class-name

Certificate.yaml

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: my-class-name-certificate
spec:
  secretName: my-class-name-certificate
  issuerRef:
    kind: Issuer
    name: my-class-name-issuer
  dnsNames:
    - my_dns

kubectl apply -n airflow-ns -f issuer.yaml
kubectl apply -n airflow-ns -f certificate.yaml

Ensure my IP has a DNS (based on my Azure cloud setup)

IP=$(kubectl -n ingress get service ingress-nginx-controller -o jsonpath={.status.loadBalancer.ingress..ip})

DNSNAME="my_dns"

PUBLICIPID=$(az network public-ip list --query "[?ipAddress!=null]|[?contains(ipAddress, '$IP')].[id]" --output tsv)

az network public-ip update --ids $PUBLICIPID --dns-name $DNSNAME

Install airflow

helm upgrade --install airflow airflow-stable/airflow --namespace airflow-ns --version "8.5.2" --values values.yaml

Especially, the ingress section, is setup like

ingress:
  ## if we should deploy Ingress resources
  ##
  enabled: false

  ## the `apiVersion` to use for Ingress resources
  ## - for Kubernetes 1.19 and later: "networking.k8s.io/v1"
  ## - for Kubernetes 1.18 and before: "networking.k8s.io/v1beta1"
  ##
  apiVersion: networking.k8s.io/v1

  ## configs for the Ingress of the web Service
  ##
  web:
    ## annotations for the web Ingress
    ##
    annotations: {}

    ## additional labels for the web Ingress
    ##
    labels: {}

    ## the path for the web Ingress
    ## - [WARNING] do NOT include the trailing slash (for root, set an empty string)
    ##
    ## ____ EXAMPLE _______________
    ##   # webserver URL: http://example.com/airflow
    ##   path: "/airflow"
    ##
    path: ""

    ## the hostname for the web Ingress
    ##
    host: ""

    ## configs for web Ingress TLS
    ##
    tls:
      ## enable TLS termination for the web Ingress
      ##
      enabled: false

      ## the name of a pre-created Secret containing a TLS private key and certificate
      ##
      secretName: ""

    ## http paths to add to the web Ingress before the default path
    ##
    ## ____ EXAMPLE _______________
    ##   precedingPaths:
    ##     - path: "/*"
    ##       serviceName: "my-service"
    ##       servicePort: "port-name"
    ##
    precedingPaths: []

    ## http paths to add to the web Ingress after the default path
    ##
    ## ____ EXAMPLE _______________
    ##   succeedingPaths:
    ##     - path: "/extra-service"
    ##       serviceName: "my-service"
    ##       servicePort: "port-name"
    ##
    succeedingPaths: []

  ## configs for the Ingress of the flower Service
  ##
  flower:
    ## annotations for the flower Ingress
    ##
    annotations: {}

    ## additional labels for the flower Ingress
    ##
    labels: {}

    ## the path for the flower Ingress
    ## - [WARNING] do NOT include the trailing slash (for root, set an empty string)
    ##
    ## ____ EXAMPLE _______________
    ##   # flower URL: http://example.com/airflow/flower
    ##   path: "/airflow/flower"
    ##
    path: ""

    ## the hostname for the flower Ingress
    ##
    host: ""

    ## configs for flower Ingress TLS
    ##
    tls:
      ## enable TLS termination for the flower Ingress
      ##
      enabled: false

      ## the name of a pre-created Secret containing a TLS private key and certificate
      ##
      secretName: ""

    ## http paths to add to the flower Ingress before the default path
    ##
    ## ____ EXAMPLE _______________
    ##   precedingPaths:
    ##     - path: "/*"
    ##       serviceName: "my-service"
    ##       servicePort: "port-name"
    ##
    precedingPaths: []

    ## http paths to add to the flower Ingress after the default path
    ##
    ## ____ EXAMPLE _______________
    ##   succeedingPaths:
    ##     - path: "/extra-service"
    ##       serviceName: "my-service"
    ##       servicePort: "port-name"
    ##
    succeedingPaths: []

Install ingress

Based on the file ingress.yaml

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-class-name-ingress
  annotations:
    cert-manager.io/issuer: my-class-name-issuer
spec:
  ingressClassName: "my-class-name"
  tls:
    - hosts:
        - my_dns
      secretName: my-class-name-certificate
  rules:
    - host: my_dns
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: airflow-web
                port:
                  number: 8080

Result

This setup is working 🥳 : when going to my DNS address I have my Airflow UI and noted as *secure*

It would be very handful that this feature is implemented directly in the charts, I could probably contribute if needed :)

Oct 18 '21 07:10 LvffY

@LvffY great news you fixed your issue!

regarding the ingressClassName issue, we will try and merge something that resolves #427 ASAP
Do you want to update the "How to set up an Ingress?" docs with what you have learned?
- For example, splitting it into "options" (like some of the other docs), the "options" could be:
  - "using embedded values"
  - "using cert-manager"
  - "using custom ingresses"

Oct 25 '21 03:10 thesuperzapper

@thesuperzapper Why not, I'll try to pass some time before the end of the week.

I'll submit a PR.

Oct 25 '21 09:10 LvffY

@thesuperzapper I have a first PR, please tell me if anything can be improved (I'll make the technical improvement e.g squash afterwards).

See #448

Oct 25 '21 12:10 LvffY