prometheus-engine icon indicating copy to clipboard operation
prometheus-engine copied to clipboard

Service account transmission issue on non-GKE clusters

Open rgaume-delfingen opened this issue 3 years ago • 5 comments

Hello, In that link you assert that you just need to provide a service account for non-GKE clusters: https://cloud.google.com/stackdriver/docs/managed-prometheus/setup-managed I Have done that, and as the documentation said, I only updated the operator config. The operator pod works well and create two more pods : collector and rules-evaluator. At that point, the two newly created pods are looking for the default credentials (that does not exist on this cluster since its not a GKE one). Since you documentation ends here, I expected that the operator pod to transmit its service account for the two new ones but it does not.

level=error ts=[...] caller=main.go:129 msg="Creating a Cloud Monitoring Exporter failed" err="create metric client: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.

At this point either :

  • The documentation is not complete and I need to update another kind;
  • There is a bug since the service account is here and used by the first pod.

I will try to read your manifests and find the correct kind to configure those pods if there is one. Else, I will try to inject my service account via a volume, but it seems like a patch to an issue in my spirit.

Thanks for your help!

rgaume-delfingen avatar May 16 '22 10:05 rgaume-delfingen

Hello,

If I can make my issue more precise do not hesitate :)

rgaume-delfingen avatar May 31 '22 13:05 rgaume-delfingen

Hi! A couple of quick questions:

  1. Which pod is emitting that authentication error -- Is it the collector, the rules-evaluator, or both?
  2. Do you see this flag --export.credentials-file= present in the manifest for the collector, or the rule-evaluator (depending on where you are seeing the issue)?

realschwa avatar May 31 '22 16:05 realschwa

@rgaume-delfingen gentle ping to see if you still need help here -- let me know 🙂

realschwa avatar Jun 15 '22 15:06 realschwa

Hello sorry ! I still have the issue but we are busy migrating our GKE cluster to gmp ! I will reply to your answer before the end of this week. Sorry for the inconvenience

rgaume-delfingen avatar Jun 15 '22 18:06 rgaume-delfingen

@realschwa I'm seeing this issue now. In my case:

  1. the collector is emitting the error;
  2. I do not have that flag

pbnsilva avatar Sep 26 '22 09:09 pbnsilva

Hi @pbnsilva,

Did you follow the steps for providing credentials explicitly? This should pass your credentials through to the collectors to authenticate to the API.

pintohutch avatar Dec 15 '22 01:12 pintohutch

I'm going to close this issue as not reproducible for now. Feel free to re-open if the issues persists and we can help debug.

pintohutch avatar Jan 20 '23 21:01 pintohutch