Linkerd Tap doesn't seem to work with EKS Access Entries authentication

[multimac]

What is the issue?

I was trying to use linkerd viz tap to debug an internal issue, but kept getting tap authorization failed. After looking at the documentation page in the error message (https://linkerd.io/2.16/tasks/securing-linkerd-tap/), it seemed like the problem might be in Linkerd itself as kubectl can-i works

How can it be reproduced?

I believe this might need an EKS cluster configured with EKS Access Entries for authentication :/

Logs, error output, etc

$ kubectl auth can-i watch deployments.v1alpha1.tap.linkerd.io/[xxx] --namespace [xxx] --subresource tap yes $ linkerd viz tap deployment/[xxx] --namespace [xxx] HTTP error, status Code [403] (unexpected API response: {"error":"tap authorization failed (not authorized to access deployments.tap.linkerd.io), visit https://linkerd.io/tap-rbac for more information"})

output of linkerd check -o short

linkerd-identity
----------------
× trust anchors are using supported crypto algorithm
    Invalid trustAnchors:
        * 498364598153846236906649535816328601770266843336 One Model Linkerd Root CA must use P-256 curve for public key, instead P-384 was used
    see https://linkerd.io/2/checks/#l5d-identity-trustAnchors-use-supported-crypto for hints

linkerd-jaeger
--------------
‼ jaeger extension proxies are up-to-date
    some proxies are not running the current version:
        * collector-9cb9f47cb-2p4dn (edge-24.10.1)
        * collector-9cb9f47cb-s8zg2 (edge-24.10.1)
        * jaeger-injector-759b7ff6fb-5xcmz (edge-24.10.1)
        * jaeger-injector-759b7ff6fb-q46tg (edge-24.10.1)
    see https://linkerd.io/2/checks/#l5d-jaeger-proxy-cp-version for hints
‼ jaeger extension proxies and cli versions match
    collector-9cb9f47cb-2p4dn running edge-24.10.1 but cli running edge-24.10.2
    see https://linkerd.io/2/checks/#l5d-jaeger-proxy-cli-version for hints

linkerd-viz
-----------
‼ viz extension proxies are up-to-date
    some proxies are not running the current version:
        * alertmanager-alertmanager-0 (edge-24.10.1)
        * metrics-api-fd554c577-8htqz (edge-24.10.1)
        * metrics-api-fd554c577-ftj2d (edge-24.10.1)
        * prometheus-prometheus-0 (edge-24.10.1)
        * tap-5579f7499d-2v4dg (edge-24.10.1)
        * tap-5579f7499d-5jf5j (edge-24.10.1)
        * tap-5579f7499d-bmdfg (edge-24.10.1)
        * tap-injector-5bfbc5b658-fqr7c (edge-24.10.1)
        * tap-injector-5bfbc5b658-kvz6s (edge-24.10.1)
        * tap-injector-5bfbc5b658-mrlmg (edge-24.10.1)
        * web-694c5b64d8-hs76l (edge-24.10.1)
        * web-694c5b64d8-kkqtp (edge-24.10.1)
    see https://linkerd.io/2/checks/#l5d-viz-proxy-cp-version for hints
‼ viz extension proxies and cli versions match
    alertmanager-alertmanager-0 running edge-24.10.1 but cli running edge-24.10.2
    see https://linkerd.io/2/checks/#l5d-viz-proxy-cli-version for hints

Status check results are ×

Environment

Kubernetes Version: v1.30 Cluster Environment: AWS EKS Host OS: Amazon Bottlerocket Linkerd version: Client version: edge-24.10.2 / Server version: edge-24.10.1

Possible solution

I've made a commit which I will push up in a PR shortly, but I suspect the issue may be because the SubjectAccessReview done by the Tap controller doesn't pass in any of the "extra" user attributes

When looking at the audit logs generated by our Kubernetes control plane and comparing the linkerd viz tap vs. kubectl can-i, I can see that kubectl can-i is passing some additional "extra" fields that seem relevant to the EKS Access Entries authentication.

My commit updates ResourceAuthzForUser in linkerd/pkg/k8s/authz.go to take in the list of extra attributes, retrieved via the X-Remote-Extras- HTTP header

Additional context

No response

Would you like to work on fixing this bug?

yes

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[jseiser]

We have this same issue.

Kubernetes Version: v1.31
Cluster Environment: AWS EKS
Host OS: Amazon Bottlerocket
linkerd version
Client version: edge-25.3.2
Server version: edge-25.1.2

We can not tap in the UI, or on the CLI

> kubectl auth can-i watch pods.tap.linkerd.io --all-namespaces --as system:serviceaccount:linkerd-viz:web
yes

 linkerd viz tap -n linkerd deploy/linkerd-controller --as $(whoami)
Cannot connect to Linkerd Viz: namespaces is forbidden: User "justin" cannot list resource "namespaces" in API group "" at the cluster scope
Validate the install with: linkerd viz check

check's are fine

> linkerd viz check
linkerd-viz
-----------
√ linkerd-viz Namespace exists
√ can initialize the client
√ linkerd-viz ClusterRoles exist
√ linkerd-viz ClusterRoleBindings exist
√ tap API server has valid cert
√ tap API server cert is valid for at least 60 days
√ tap API service is running
√ linkerd-viz pods are injected
√ viz extension pods are running
√ viz extension proxies are healthy
‼ viz extension proxies are up-to-date
    some proxies are not running the current version:
        * metrics-api-55c95c4f49-4ds2t (edge-25.1.2)
        * metrics-api-55c95c4f49-g44p2 (edge-25.1.2)
        * prometheus-6d4d66c58d-585m6 (edge-25.1.2)
        * tap-7b958b9784-qf5cb (edge-25.1.2)
        * tap-7b958b9784-skknm (edge-25.1.2)
        * tap-injector-769865c97-d4ngv (edge-25.1.2)
        * tap-injector-769865c97-r2srb (edge-25.1.2)
        * web-6cfd8f5bb7-9rg47 (edge-25.1.2)
        * web-6cfd8f5bb7-lh58f (edge-25.1.2)
    see https://linkerd.io/2/checks/#l5d-viz-proxy-cp-version for hints
‼ viz extension proxies and cli versions match
    metrics-api-55c95c4f49-4ds2t running edge-25.1.2 but cli running edge-25.3.2
    see https://linkerd.io/2/checks/#l5d-viz-proxy-cli-version for hints
√ prometheus is installed and configured correctly
√ viz extension self-check

Status check results are √

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[jseiser]

This is still an issue, we can not tap our workloads.

>  linkerd viz tap -n linkerd deploy/linkerd-controller --as $(whoami)
Cannot connect to Linkerd Viz: namespaces is forbidden: User "justin" cannot list resource "namespaces" in API group "" at the cluster scope
Validate the install with: linkerd viz check

[Tolsto]

Doesn't work for me on EKS with access entries authentication as well.