Istio setup and best practices

[christian-roggia]

This ticket addresses the setup with Istio + Envoy in a Kubernetes environment. This ticket provides a working example of an Istio setup and could be potentially adapted to any Envoy-based gateway such as Ambassador.

The version of Istio used is the latest available: 1.8.1.

Related ticket https://github.com/ory/oathkeeper/issues/263.

The following EnvoyFilter is working with a setup of ory Hydra (1.8.5) + Kratos (0.5.5) + Oathkeeper (0.38.5).

I would like to request a review by the ory team and ask whether they could verify that all the headers used by oathkeeper have been whitelisted.

Hopefully, this YAML will be useful and save some time and many headaches for all those who need to set up Istio as an edge proxy.

Our setup is working on both GKE and AWS with 2 ingresses (one secured with IAP and one public) which points to 2 Istio ingress gateways (NodePort) which authenticate and authorize requests via oathkeeper (EnvoyFilter).

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: oathkeeper
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: GATEWAY
        listener:
          filterChain:
            filter:
              name: "envoy.filters.network.http_connection_manager"
              subFilter:
                name: "envoy.filters.http.router"
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.http.ext_authz
          typed_config:
            '@type': type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
            failure_mode_allow: false
            http_service:
              path_prefix: /decisions
              server_uri:
                uri: http://oathkeeper-api.oathkeeper.svc.cluster.local:4456
                cluster: outbound|4456||oathkeeper-api.oathkeeper.svc.cluster.local
                timeout: 10s
              authorization_request:
                allowed_headers:
                  patterns:
                  - exact: accept
                  - exact: authorization
                  - exact: cookie
                  - exact: content-type
                  - exact: x-forwarded-for
                  - exact: x-forwarded-proto
                  - exact: x-forwarded-host
              authorization_response:
                allowed_upstream_headers:
                  patterns:
                  - exact: authorization

[christian-roggia]

This ticket could be (and probably should be) merged into the documentation.

[aeneasr]

Would absolutely love having that in the docs, maybe as a guide?

[sebastianvoss]

Thanks @christian-roggia, this was really helpful to set it up in Istio.

Is there also a way to set something like X-User header based on the identity returned by oathkeeper?

[christian-roggia]

Thanks @christian-roggia, this was really helpful to set it up in Istio.

Is there also a way to set something like X-User header based on the identity returned by oathkeeper?

I believe you only need to whitelist the headers you would like to forward to the upstream service.

You can do so by adding the headers you need to the allowed_upstream_headers list. I think that in oathkeeper you can set them through a mutator rule.

[sebastianvoss]

@christian-roggia, you are right. It works a you described it. Thanks a lot!

[rauanmayemir]

Do you need to allow extra headers for tracing to work? I want oathkeeper filter request to be included as a span, i.e a part of the main request trace.

[christian-roggia]

I haven't tried it yet, but you can enable tracing on istio itself and probably you will need to whitelist some headers for oathkeeper to propagate distributed traces, I guess it also depends on the tracing platform you are using.. let me know if you manage to do it!

[rauanmayemir]

@christian-roggia I added tracing headers in authorization_request.allowed_headers, but I'm not sure. I think it works as it is without adding anything there. I will know more later.

[rajsenthil]

Thanks for the post @christian-roggia. I am looking for similar setup and this post provide a solution. Can you please confirm, for EnvoyFilter to work, whether ory Hydra is required here? Can the Envoy filter work with oathkeeper + kratos to intercept the request for a valid session cookie exists or not? Thanks again.

[christian-roggia]

I confirm that ory/hydra isn't required for the EnvoyFilter to work. You only need ory/oathkeeper as it provides the decision endpoint, you could technically deploy it alone with Istio/Envoy and nothing else in the cluster.

TL;DR: Kratos + Oathkeeper with Istio and this EnvoyFilter work just fine.

[batistein]

@christian-roggia Thank you very much for your example!! Do you know how to reduce the scope of the filter so it's not coupled on the gateway? We have a lot of other services running under the same gateway, where the oathkeeper shouldn't be used... I know this is an istio thing but I didn't found anything related to this topic. The only thing I saw was the newly introduced External Authorization from istio... as I understand the above example could be translated to this object as well right?

[christian-roggia]

You can use the workloadSelector to target only specific Pods, you just need to use your own labels instead of the labels of the ingressgateway, and yes, it should be possible to translate to the new External Authorization object but I haven't worked on that yet. Any example is welcome though!

[batistein]

Thanks for the quick response @christian-roggia I tried it with a custom label which is on every pod I want to secure. Unfortunately, it seems like that the oathkeeper is not authorizing and bypassed. Do I also need to change the spec.configPatches.match.context=Gateway to something else? Thank you very much!

[christian-roggia]

Yes, you are right. I believe that changing to SIDECAR_INBOUND could be a step in the right direction?

[nickjn92]

Later versions of Istio now offer an easier way to plugin external authorization: https://istio.io/latest/docs/tasks/security/authorization/authz-custom/

[github-actions[bot]]

Hello contributors!

I am marking this issue as stale as it has not received any engagement from the community or maintainers for a year. That does not imply that the issue has no merit! If you feel strongly about this issue

• open a PR referencing and resolving the issue;
• leave a comment on it and discuss ideas on how you could contribute towards resolving it;
• leave a comment and describe in detail why this issue is critical for your use case;
• open a new issue with updated details and a plan for resolving the issue.

Throughout its lifetime, Ory has received over 10.000 issues and PRs. To sustain that growth, we need to prioritize and focus on issues that are important to the community. A good indication of importance, and thus priority, is activity on a topic.

Unfortunately, burnout has become a topic of concern amongst open-source projects.

It can lead to severe personal and health issues as well as opening catastrophic attack vectors.

The motivation for this automation is to help prioritize issues in the backlog and not ignore, reject, or belittle anyone.

If this issue was marked as stale erroneously you can exempt it by adding the backlog label, assigning someone, or setting a milestone for it.

Thank you for your understanding and to anyone who participated in the conversation! And as written above, please do participate in the conversation if this topic is important to you!

Thank you 🙏✌️