spire-tutorials icon indicating copy to clipboard operation
spire-tutorials copied to clipboard
verified publisher icon spiffe

Workload attestation fails in k8s/quickstart

Open donaldh opened this issue 3 years ago • 5 comments

I followed the K8S quickstart guide https://spiffe.io/docs/latest/try/getting-started-k8s/ but the final step failed:

$ kubectl exec -it $(kubectl get pods -o=jsonpath='{.items[0].metadata.name}' \
   -l app=client)  -- /bin/sh
/opt/spire # /opt/spire/bin/spire-agent api fetch -socketPath /run/spire/sockets/agent.sock
rpc error: code = PermissionDenied desc = no identity issued

From the spire-agent logs it seems like there are only unix selectors available but the registration entry uses k8s selectors:

time="2022-04-22T09:14:17Z" level=debug msg="PID attested to have selectors" pid=5532 selectors="[type:\"unix\" value:\"uid:0\" type:\"unix\" value:\"user:root\" type:\"unix\" value:\"gid:0\" type:\"unix\" value:\"group:root\" type:\"unix\" value:\"supplementary_gid:1\" type:\"unix\" value:\"supplementary_group:bin\" type:\"unix\" value:\"supplementary_gid:2\" type:\"unix\" value:\"supplementary_group:daemon\" type:\"unix\" value:\"supplementary_gid:3\" type:\"unix\" value:\"supplementary_group:sys\" type:\"unix\" value:\"supplementary_gid:4\" type:\"unix\" value:\"supplementary_group:adm\" type:\"unix\" value:\"supplementary_gid:6\" type:\"unix\" value:\"supplementary_group:disk\" type:\"unix\" value:\"supplementary_gid:10\" type:\"unix\" value:\"supplementary_group:wheel\" type:\"unix\" value:\"supplementary_gid:11\" type:\"unix\" value:\"supplementary_group:floppy\" type:\"unix\" value:\"supplementary_gid:20\" type:\"unix\" value:\"supplementary_group:dialout\" type:\"unix\" value:\"supplementary_gid:26\" type:\"unix\" value:\"supplementary_group:tape\" type:\"unix\" value:\"supplementary_gid:27\" type:\"unix\" value:\"supplementary_group:video\"]" subsystem_name=workload_attestor
time="2022-04-22T09:14:17Z" level=error msg="No identity issued" method=FetchX509SVID pid=5532 registered=false service=WorkloadAPI subsystem_name=endpoints

donaldh avatar Apr 22 '22 09:04 donaldh

Were there any logs above those two lines you've pasted, particularly any from the k8s workload attestor plugin?

azdagron avatar Apr 25 '22 21:04 azdagron

Here's the full agent log:

time="2022-04-25T22:04:43Z" level=warning msg="Current umask 0022 is too permissive; setting umask 0027"
time="2022-04-25T22:04:43Z" level=info msg="Starting agent with data directory: \"/run/spire\""
time="2022-04-25T22:04:43Z" level=info msg="Plugin loaded" external=false plugin_name=memory plugin_type=KeyManager subsystem_name=catalog
time="2022-04-25T22:04:43Z" level=info msg="Plugin loaded" external=false plugin_name=k8s plugin_type=WorkloadAttestor subsystem_name=catalog
time="2022-04-25T22:04:43Z" level=info msg="Plugin loaded" external=false plugin_name=unix plugin_type=WorkloadAttestor subsystem_name=catalog
time="2022-04-25T22:04:43Z" level=info msg="Plugin loaded" external=false plugin_name=k8s_sat plugin_type=NodeAttestor subsystem_name=catalog
time="2022-04-25T22:04:43Z" level=info msg="Bundle loaded" subsystem_name=attestor trust_domain_id="spiffe://example.org"
time="2022-04-25T22:04:43Z" level=debug msg="No pre-existing agent SVID found. Will perform node attestation" path=/run/spire/agent_svid.der subsystem_name=attestor
time="2022-04-25T22:04:43Z" level=info msg="SVID is not found. Starting node attestation" subsystem_name=attestor trust_domain_id="spiffe://example.org"
time="2022-04-25T22:04:43Z" level=info msg="Node attestation was successful" spiffe_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/686784f9-5648-4e87-b504-e649095c25f6" subsystem_name=attestor trust_domain_id="spiffe://example.org"
time="2022-04-25T22:04:43Z" level=info msg="Starting Workload and SDS APIs" subsystem_name=endpoints
time="2022-04-25T22:04:44Z" level=debug msg="Starting checker" name=agent subsystem_name=health
time="2022-04-25T22:04:44Z" level=info msg="Serving health checks" address="0.0.0.0:8080" subsystem_name=health
time="2022-04-25T22:04:48Z" level=debug msg="Entry created" entry=b86228a7-e69c-433d-85e4-f5165ab6ddfd selectors_added=3 spiffe_id="spiffe://example.org/ns/spire/sa/spire-agent" subsystem_name=cache_manager
time="2022-04-25T22:04:48Z" level=debug msg="Entry created" entry=ebc866a1-c02b-439f-bc90-7d189ab55899 selectors_added=2 spiffe_id="spiffe://example.org/ns/default/sa/default" subsystem_name=cache_manager
time="2022-04-25T22:04:48Z" level=debug msg="Renewing stale entries" count=2 limit=500 subsystem_name=manager
time="2022-04-25T22:04:48Z" level=info msg="Renewing X509-SVID" spiffe_id="spiffe://example.org/ns/spire/sa/spire-agent" subsystem_name=manager
time="2022-04-25T22:04:48Z" level=info msg="Renewing X509-SVID" spiffe_id="spiffe://example.org/ns/default/sa/default" subsystem_name=manager
time="2022-04-25T22:04:48Z" level=debug msg="SVID updated" entry=b86228a7-e69c-433d-85e4-f5165ab6ddfd spiffe_id="spiffe://example.org/ns/spire/sa/spire-agent" subsystem_name=cache_manager
time="2022-04-25T22:04:48Z" level=debug msg="SVID updated" entry=ebc866a1-c02b-439f-bc90-7d189ab55899 spiffe_id="spiffe://example.org/ns/default/sa/default" subsystem_name=cache_manager
time="2022-04-25T22:05:08Z" level=debug msg="PID attested to have selectors" pid=2290693 selectors="[type:\"unix\" value:\"uid:0\" type:\"unix\" value:\"user:root\" type:\"unix\" value:\"gid:0\" type:\"unix\" value:\"group:root\" type:\"unix\" value:\"supplementary_gid:1\" type:\"unix\" value:\"supplementary_group:bin\" type:\"unix\" value:\"supplementary_gid:2\" type:\"unix\" value:\"supplementary_group:daemon\" type:\"unix\" value:\"supplementary_gid:3\" type:\"unix\" value:\"supplementary_group:sys\" type:\"unix\" value:\"supplementary_gid:4\" type:\"unix\" value:\"supplementary_group:adm\" type:\"unix\" value:\"supplementary_gid:6\" type:\"unix\" value:\"supplementary_group:disk\" type:\"unix\" value:\"supplementary_gid:10\" type:\"unix\" value:\"supplementary_group:wheel\" type:\"unix\" value:\"supplementary_gid:11\" type:\"unix\" value:\"supplementary_group:floppy\" type:\"unix\" value:\"supplementary_gid:20\" type:\"unix\" value:\"supplementary_group:dialout\" type:\"unix\" value:\"supplementary_gid:26\" type:\"unix\" value:\"supplementary_group:tape\" type:\"unix\" value:\"supplementary_gid:27\" type:\"unix\" value:\"supplementary_group:video\"]" subsystem_name=workload_attestor
time="2022-04-25T22:05:08Z" level=error msg="No identity issued" method=FetchX509SVID pid=2290693 registered=false service=WorkloadAPI subsystem_name=endpoints

donaldh avatar Apr 25 '22 22:04 donaldh

I can confirm that an entry with a unix selector works:

kubectl exec -n spire spire-server-0 -- \
    /opt/spire/bin/spire-server entry create \
    -spiffeID spiffe://example.org/unix/zero \
    -parentID spiffe://example.org/ns/spire/sa/spire-agent \
    -selector unix:uid:0
Entry ID         : b696877c-4ed6-4daf-91cc-8a1b3f4ff7da
SPIFFE ID        : spiffe://example.org/unix/zero
Parent ID        : spiffe://example.org/ns/spire/sa/spire-agent
Revision         : 0
TTL              : default
Selector         : unix:uid:0

/opt/spire # /opt/spire/bin/spire-agent api fetch -socketPath /run/spire/sockets/agent.sock
Received 1 svid after 4.101928ms

SPIFFE ID:		spiffe://example.org/unix/zero
SVID Valid After:	2022-04-25 22:17:08 +0000 UTC
SVID Valid Until:	2022-04-25 23:17:18 +0000 UTC
CA #1 Valid After:	2022-04-24 02:57:32 +0000 UTC
CA #1 Valid Until:	2022-04-25 02:57:42 +0000 UTC
CA #2 Valid After:	2022-04-24 14:57:32 +0000 UTC
CA #2 Valid Until:	2022-04-25 14:57:42 +0000 UTC
CA #3 Valid After:	2022-04-25 02:57:32 +0000 UTC
CA #3 Valid Until:	2022-04-26 02:57:42 +0000 UTC
CA #4 Valid After:	2022-04-25 14:57:32 +0000 UTC
CA #4 Valid Until:	2022-04-26 14:57:42 +0000 UTC

donaldh avatar Apr 25 '22 22:04 donaldh

I have the same problem /opt/spire # /opt/spire/bin/spire-agent api fetch -socketPath /run/spire/sockets/agent.sock rpc error: code = PermissionDenied desc = no identity issued

the spire-agent logs is : time="2022-05-24T08:59:19Z" level=error msg="Failed to collect all selectors for PID" error="workload attestor \"k8s\" failed: rpc error: code = Internal desc = workloadattestor(k8s): unable to perform request: Get \"https://127.0.0.1:10250/pods\": dial tcp 127.0.0.1:10250: connect: connection refused" pid=2639581 subsystem_name=workload_attestor time="2022-05-24T08:59:19Z" level=debug msg="PID attested to have selectors" pid=2639581 selectors="[type:\"unix\" value:\"uid:0\" type:\"unix\" value:\"user:root\" type:\"unix\" value:\"gid:0\" type:\"unix\" value:\"group:root\" type:\"unix\" value:\"supplementary_gid:0\" type:\"unix\" value:\"supplementary_group:root\" type:\"unix\" value:\"supplementary_gid:1\" type:\"unix\" value:\"supplementary_group:bin\" type:\"unix\" value:\"supplementary_gid:2\" type:\"unix\" value:\"supplementary_group:daemon\" type:\"unix\" value:\"supplementary_gid:3\" type:\"unix\" value:\"supplementary_group:sys\" type:\"unix\" value:\"supplementary_gid:4\" type:\"unix\" value:\"supplementary_group:adm\" type:\"unix\" value:\"supplementary_gid:6\" type:\"unix\" value:\"supplementary_group:disk\" type:\"unix\" value:\"supplementary_gid:10\" type:\"unix\" value:\"supplementary_group:wheel\" type:\"unix\" value:\"supplementary_gid:11\" type:\"unix\" value:\"supplementary_group:floppy\" type:\"unix\" value:\"supplementary_gid:20\" type:\"unix\" value:\"supplementary_group:dialout\" type:\"unix\" value:\"supplementary_gid:26\" type:\"unix\" value:\"supplementary_group:tape\" type:\"unix\" value:\"supplementary_gid:27\" type:\"unix\" value:\"supplementary_group:video\"]" subsystem_name=workload_attestor time="2022-05-24T08:59:19Z" level=error msg="No identity issued" method=FetchX509SVID pid=2639581 registered=false service=WorkloadAPI subsystem_name=endpoints

how did you solve this problem?? @donaldh ,can you help me

sys0613 avatar May 24 '22 12:05 sys0613

I change something ,It run success . kubectl exec -n spire spire-server-0 -- \ /opt/spire/bin/spire-server entry create \ -spiffeID spiffe://example.org/ns/spire/sa/spire-agent \ -selector k8s_sat:cluster:demo-cluster \ #change here -selector k8s_sat:agent_ns:spire \ -selector k8s_sat:agent_sa:spire-agent \ -node

change the clusterName "demo-cluster",It run success : " -selector k8s_sat:cluster:demo-cluster "

sys0613 avatar Jun 20 '22 23:06 sys0613