[Gitlab] Refresh token is invalid or has already been claimed by another client

[kfirfer]

Preflight Checklist

• [X] I agree to follow the Code of Conduct that this project adheres to.
• [X] I have searched the issue tracker for an issue that matches the one I want to file, without success.
• [X] I am not looking for support or already pursued the available support channels without success.

Version

2.33.0

Storage Type

In-memory

Installation Type

Official Helm chart

Actual Behavior

After 1 hour, Im getting

$ kubectl get po
Unable to connect to the server: failed to refresh token: oauth2: cannot fetch token: 400 Bad Request
Response: {"error":"invalid_request","error_description":"Refresh token is invalid or has already been claimed by another client."}

And I need to re-authenticate

Configuration

config:
  issuer: https://dex.xxx.xxx
  storage:
    type: kubernetes
    config:
      inCluster: true
  expiry:
    idTokens: "1h"
    refreshTokens:
      reuseInterval: "3s"
      absoluteLifetime: "39600h" # 1650 days
      validIfNotUsedFor: "2160h" # 90 days
  oauth2:
    responseTypes: ["code", "token", "id_token"]
    skipApprovalScreen: true
  connectors:
    - type: gitlab
      id: gitlab
      name: GitLab
      config:
        baseURL: https://gitlab.com
        clientID: xxxxx
        clientSecret: xxxx
        redirectURI: https://dex.xxx.xxx/callback
        useLoginAsID: false

Logs

Idk if its relating time="2022-07-30T14:14:50Z" level=error msg="failed to rotate keys: keys already rotated by another server instance"

[nabokihms]

Hello, @kfirfer. There are two related issues.

1. As of the v2.33 release, the Gitlab connector uses refresh tokens to make the authentication pipeline works for Gitlab >= 15. Gitlab issues a new refresh token for each refresh. This feature is called refresh token rotation and protects refresh tokens from stealing. Thus, users must send a new refresh token for each refresh request. However, Dex tries to refresh token for each concurrent request with a stale refresh token for some of them, which can invalidate the token. Related issue: https://github.com/dexidp/dex/issues/2547 (I have a solution and will open a PR this week).

2. The second problem is a kubectl problem. It cannot handle refresh token rotation because of the bug https://github.com/kubernetes/kubernetes/issues/78494. Currently, the only solution I can propose is to use the plugin for the kubectl - kubelogin. This plugin does not utilize the native OIDC authentication mechanism and is not affected by the bug.

[shencan]

Has the issue been fixed？ I had the same problem

[nabokihms]

@shencan could you please provide more info? Logs during the error will be helpful 🙏

[shencan]

dex log ime="2023-11-28T02:05:09Z" level=error msg="refresh token with id zmg7wpixm3tvnge2sdkbelt5v claimed twice"

[shencan]

The following errors often occur after a period of time

k get node

E1206 11:37:24.151289   55468 memcache.go:265] couldn't get current server API group list: Get "https://xxxxxyl4.ap-east-1.eks.amazonaws.com/api?timeout=32s": failed to refresh token: oauth2: "invalid_request" "Refresh token is invalid or has already been claimed by another client."

E1206 11:37:25.183706   55468 memcache.go:265] couldn't get current server API group list: Get "https://xxxxxx.yl4.ap-east-1.eks.amazonaws.com/api?timeout=32s": failed to refresh token: oauth2: "invalid_request" "Refresh token is invalid or has already been claimed by another client."

E1206 11:37:26.240523   55468 memcache.go:265] couldn't get current server API group list: Get "https://xxxxxxx.yl4.ap-east-1.eks.amazonaws.com/api?timeout=32s": failed to refresh token: oauth2: "invalid_request" "Refresh token is invalid or has already been claimed by another client."

E1206 11:37:27.398184   55468 memcache.go:265] couldn't get current server API group list: Get "https://xxxxxx.yl4.ap-east-1.eks.amazonaws.com/api?timeout=32s": failed to refresh token: oauth2: "invalid_request" "Refresh token is invalid or has already been claimed by another client."

E1206 11:37:28.395145   55468 memcache.go:265] couldn't get current server API group list: Get "https://xxxxxx.ap-east-1.eks.amazonaws.com/api?timeout=32s": failed to refresh token: oauth2: "invalid_request" "Refresh token is invalid or has already been claimed by another client."

Unable to connect to the server: failed to refresh token: oauth2: "invalid_request" "Refresh token is invalid or has already been claimed by another client."

my dex config

connectors:
  - type: ldap
    id: ldap
    name: LDAP
    config:
      host: abcefg:389
      insecureNoSSL: true
      bindDN: xxxxxx
      bindPW: xxxxxx
      usernamePrompt: "LDAP User Email"
      userSearch:
        baseDN: xxxxxxx
        filter: "xxxxxx"
        username: uid
        idAttr: uid
        emailAttr: mail
        nameAttr: uid
        preferredUsernameAttr: uid    
      groupSearch:
        baseDN: xxxxxx
        filter: "(objectClass=groupOfNames)"
        userMatchers:
        - userAttr: DN
          groupAttr: member
        nameAttr: cn
  - type: gitlab
    id: gitlab
    name: GitLab
    config:
      baseURL: https://xxxxxxxxx
      clientID: '{{ .Env.GITLAB_CLIENT_ID }}'
      clientSecret: '{{ .Env.GITLAB_CLIENT_SECRET }}'
      redirectURI: https://dex.xxxxxxx.com/callback  
issuer: https://dex.xxxxxxx.com
oauth2:
  responseTypes:
  - code
  - token
  - id_token
  skipApprovalScreen: true
expiry:
  idTokens: 5m
  refreshTokens:
    absoluteLifetime: 876000h
    reuseInterval: 1m
    validIfNotUsedFor: 43920h
logger:
  level: "debug" 
staticClients:
- id: eksdemo-dexclient
  name: eksdemo dex demo
  redirectURIs:
  - https://login.xxxxxx.com/callback/eksdemo
  secret: xxxxxxx
- id: eks-study-dexclient
  name: eks-study dex demo
  redirectURIs:
  - https://login.xxxxxxx.com/callback/eks-study
  secret: xxxxxxxxx
storage:
  config:
    inCluster: true
  type: kubernetes

my dex pod error log

time="2023-12-06T03:37:19Z" level=error msg="refresh token with id bl2lyz3akol5eqdlewuwk56yr claimed twice"
time="2023-12-06T03:37:20Z" level=error msg="refresh token with id bl2lyz3akol5eqdlewuwk56yr claimed twice"
time="2023-12-06T03:37:24Z" level=error msg="refresh token with id bl2lyz3akol5eqdlewuwk56yr claimed twice"
time="2023-12-06T03:37:24Z" level=error msg="refresh token with id bl2lyz3akol5eqdlewuwk56yr claimed twice"
time="2023-12-06T03:37:25Z" level=error msg="refresh token with id bl2lyz3akol5eqdlewuwk56yr claimed twice"
time="2023-12-06T03:37:25Z" level=error msg="refresh token with id bl2lyz3akol5eqdlewuwk56yr claimed twice"
time="2023-12-06T03:37:26Z" level=error msg="refresh token with id bl2lyz3akol5eqdlewuwk56yr claimed twice"
time="2023-12-06T03:37:26Z" level=error msg="refresh token with id bl2lyz3akol5eqdlewuwk56yr claimed twice"
time="2023-12-06T03:37:27Z" level=error msg="refresh token with id bl2lyz3akol5eqdlewuwk56yr claimed twice"
time="2023-12-06T03:37:27Z" level=error msg="refresh token with id bl2lyz3akol5eqdlewuwk56yr claimed twice"
time="2023-12-06T03:37:28Z" level=error msg="refresh token with id bl2lyz3akol5eqdlewuwk56yr claimed twice"
time="2023-12-06T03:37:28Z" level=error msg="refresh token with id bl2lyz3akol5eqdlewuwk56yr claimed twice"