Support for IAM-based signing for GCS Bucket blobs

[benglewis]

Current state:

Currently, gcsfs does not natively support generating signed URLs using IAM-based credentials provided by GCP Workload Identity or other non-private key credentials. This is a limitation when running on environments such as Google Kubernetes Engine (GKE) with Workload Identity, where only google.auth.compute_engine.credentials.Credentials are available (which is what IAM-based credentials which are provided via Workload Identity Federation and the recommended approach: GCP Best Practices for Workload Identity). These credentials do not include a private key, which is required for the current signing functionality in gcsfs.

Use case:

In environments like GKE with Workload Identity:

• Using service account key files is discouraged for security reasons.
• Credentials provided via Workload Identity rely on token-based authentication and require the use of the signBlob API to generate signed URLs.

Adding support for IAM-based signing would enable secure and native integration with GCP for generating signed URLs in such environments.

Proposed Solution:

Implement an alternative signing mechanism in gcsfs that leverages the IAM Service Account Credentials API to generate signed URLs. This could be achieved by: 1. Detecting the type of credentials in use (e.g., google.auth.compute_engine.credentials.Credentials). 2. Using the signBlob API to sign requests when private key-based credentials are unavailable.

Example of Desired Behavior:

The sign() method in gcsfs could use IAM credentials to generate signed URLs transparently, even in environments where private keys are unavailable. For example:

import gcsfs
fs = gcsfs.GCSFileSystem()
signed_url = fs.sign("bucket-name/object-name", expiration=3600)
print(signed_url)  # Uses IAM-based signing if private key is not present

Notes:

• Proper configuration is needed to ensure the correct IAM permissions (e.g., roles/iam.serviceAccountTokenCreator) are in place.

Finally: I would be happy to open a PR if it is relevant.

[martindurant]

Sorry I didn't respond to this earlier. I would be happy to see the signing method you suggest submitted as a PR. I'm not sure if it would be possible to test this via our CI, though.