GCP: VM SA created with official terraform module is missing permissions

[heilerich]

Issue description

We are currently trying to deploy constellation in an enterprise GCP account with imposes some restrictions using organization policies. One of those is the unavailability of a default Service Account for Compute Instances which constellation previously relied upon to discover peer instances during bootstrapping (and possibly other things which I am not aware of). This was supposed to be fixed with #3656 in release v2.22.0.

We tried a fresh install using that release. The installation process was still hanging with the bootstrapper service never coming online. The journal on the control plane machines contained the following error:

{
  "time": "<removed>",
  "level": "ERROR",
  "source": {
    "function": "github.com/edgelesssys/constellation/v2/disk-mapper/internal/rejoinclient.(*RejoinClient).Start",
    "file": "disk-mapper/internal/rejoinclient/rejoinclient.go",
    "line": 82
  },
  "msg": "Failed to get control-plane endpoints",
  "rejoinClient": {
    "error": "retrieving load balancer endpoint from cloud provider: getting regional forwarding rule: error listing global forwarding rules with UID <removed>: googleapi: Error 403: Required 'compute.forwardingRules.list' permission for 'projects/<removed>'"
  }
}

Currently the official terraform module sets the following permissions for the VM service account

https://github.com/edgelesssys/constellation/blob/4a93870d1d37fbba1b648b5dbe4c181807489645/terraform/infrastructure/iam/gcp/main.tf#L84-L90

This includes compute.globalForwardingRules.list, but not compute.forwardingRules.list which the error in the bootstrapper log is asking for.

After manually granting the service account the compute.forwardingRules.list permission the constellation bootstrapping process successfully proceeded to starting the control plane processes.

Steps to reproduce the behavior

• Create a GCP Project and disable the default service account
• Booststrap constellation using the terraform module and provider

I assume this has been working in your test setup, so I am wondering what might be different in our environment compared your test setup. In order for the official terraform module to work in our GCP organization the additional permission would have to be added.

AFAIK the organization policies active in our environment are the default policy package that comes with an account the uses the Sovereign Cloud offering by T-Systems. We are not trying to install constellation in a sovereign cloud enabled GCP project though. It is just in the same GCP organization as sovereign cloud projects and therefore inherits some of the security policies set on the organization level.

Version

v2.22.0

Constellation Config

No response

[msanft]

Hey @heilerich.

first of all, thanks for this thorough and well-written report. This is what I love to see as an engineer!

I assume this has been working in your test setup, so I am wondering what might be different in our environment compared your test setup.

It did (and does) indeed. I wonder if the policies in the sovereign cloud offerings differ from "regular" ones, or if this is just caused by our project being quite old by now, thus not getting (potentially) updated policies. I'll try to find out about it and will report back here in case of any noteworthy findings.

In any case, we'll add the missing permission to our IAM service account.

Thanks a lot for this report!

Cheers, Moritz

[heilerich]

I think we found the difference and it has nothing to do with the organization policies. We are using the internal_load_balancer option of the official TF module.

https://github.com/edgelesssys/constellation/blob/a5a7cec11bb8acb9df753bcce0470346a5edd6ca/terraform/infrastructure/gcp/modules/internal_load_balancer/main.tf#L60 https://github.com/edgelesssys/constellation/blob/a5a7cec11bb8acb9df753bcce0470346a5edd6ca/terraform/infrastructure/gcp/modules/loadbalancer/main.tf#L58

So we don't have GlobalForwardingRules, but we do have (regional) ForwardingRules. This is why we need the compute.forwardingRules.list permission, but not the compute.globalForwardingRules.list permission. I guess your test setup does not cover the internal loadbalancer configuration?

[msanft]

Thanks for the update! We do test that configuration on a weekly basis, but the tests failed for a different reason (not product-related) since we've built the VM SA feature, leaving the issue go unnoticed.

So that issue should have been found if everything went normally.

Cheers, Moritz