Getting "failed to check whether build ID symbol exists error" for parca-agent

[dnoland1]

Deployed parca into my v1.21.9 cluster using:

kubectl create namespace parca
kubectl apply -f https://github.com/parca-dev/parca/releases/download/v0.8.1/kubernetes-manifest.yaml
kubectl apply -f https://github.com/parca-dev/parca-agent/releases/download/v0.5.0/kubernetes-manifest.yaml

No profiles are generated and parca-agent repeatedly has the errors:

level=warn ts=2022-03-17T20:43:04.616549742Z caller=debuginfo.go:154 component=debuginfoextractor msg="failed to check whether build ID symbol exists" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: lookup parca.parca.svc.cluster.local: no such host\""
level=error ts=2022-03-17T20:43:09.063871255Z caller=write_client.go:85 msg="Write client failed to send profiles" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: lookup parca.parca.svc.cluster.local: no such host\""

The DNS name appears to work fine when using nslookup:

# nslookup parca.parca.svc.cluster.local 10.43.0.10
Server:        10.43.0.10
Address:    10.43.0.10#53

Name:    parca.parca.svc.cluster.local
Address: 10.43.23.110

[kakkoyun]

Hello @dnoland1, thanks for reporting the issue.

It seems odd. It looks like agent failed connecting to parca-server. Rather, I would say it has connected when it's started its lifecycle but failed to send data on later stages. Could you please share more information on your setup?

[dnoland1]

@kakkoyun I'm running a v1.21.9 RKE cluster on SLES 15 SP3 nodes on AWS. I've tried two other distributions - K3s and RKE2 and experienced the same problem. However, if I try Ubuntu 20.04.4, it works fine. So I suspect it's related to something in the OS. I'm using the public AMI ami-064b1c0dca9a64b20 in us-west-2 which is a vanilla SLES. I've been discussing on discord as well.

[kakkoyun]

@dnoland1 Thanks for the detailed information. I have also seen the discussions on the Discord. We'll look into it.

[dnoland1]

Thanks so much! If you need more information or anything I can do to help, please let me know.

[kakkoyun]

@dnoland1 Have you had a chance to test the latest stable versions of Parca and Parca Agent? Does the error persists?

[dnoland1]

@kakkoyun I tried again, but could not get far. After running:

kubectl create namespace parca
kubectl apply -f https://github.com/parca-dev/parca/releases/download/v0.13.0/kubernetes-manifest.yaml

The parca pod was crashing:

NAME                     READY   STATUS             RESTARTS     AGE
parca-7f87f45f9d-7n8bt   0/1     CrashLoopBackOff   1 (6s ago)   16s

Logs:

$ kubectl logs -n parca parca-7f87f45f9d-7n8bt
ooooooooo.
`888   `Y88.
 888   .d88'  .oooo.   oooo d8b  .ooooo.   .oooo.
 888ooo88P'  `P  )88b  `888""8P d88' `"Y8 `P  )88b
 888          .oP"888   888     888        .oP"888
 888         d8(  888   888     888   .o8 d8(  888
o888o        `Y888""8o d888b    `Y8bod8P' `Y888""8o



level=error name=parca ts=2022-11-07T22:47:44.664594609Z caller=parca.go:138 msg="failed to read config" path=/var/parca/parca.yaml
level=error name=parca ts=2022-11-07T22:47:44.66467657Z caller=main.go:59 msg="Program exited with error" err="parsing YAML file /var/parca/parca.yaml: yaml: unmarshal errors:\n  line 1: field debug_info not found in type config.Config"

[kakkoyun]

Thanks for trying :) This is an issue we need to fix on the server-side.