"gcloud alpha storage cp" command fails because of missing "google-crc32c" binary

[fleroux514]

TL;DR

gcloud alpha storage cp command fails because of missing google-crc32c binary

Expected behavior Able to use this workflow:

      - name: Authenticate to Google
        id: 'auth'
        uses: 'google-github-actions/[email protected]'
        with:
          token_format: 'access_token'
          workload_identity_provider: ${{ secrets.GCP__GHA__WORKLOAD_IDENTITY_PROVIDER }}
          service_account: ${{ env.SERVICE_ACCOUNT }}
          create_credentials_file: true
      - name: Set up Cloud SDK
        uses: google-github-actions/[email protected]
      - name: Download archive from Google Storage
        run: |
          gcloud auth login --brief --cred-file="${{ steps.auth.outputs.credentials_file_path }}"
          gcloud --quiet alpha storage cp gs://test.txt ./downloads

Observed behavior

Copying gs://test.txt to file://./downloads/test.txt
ERROR: This download was skipped since the google-crc32c
binary is not installed, and Python hash computation will likely
throttle performance. You can change this by installing the binary
by running "/usr/bin/python3 -m pip install google-crc32c --target /opt/hostedtoolcache/gcloud/365.0.1/x64/lib/third_party" or
modifying the "storage/check_hashes" config setting.

Additional information Running the workflow on ubuntu-latest

[sethvargo]

Looks like we might need to install some packages. https://cloud.google.com/storage/docs/gsutil/addlhelp/CRC32CandInstallingcrcmod

[fleroux514]

@sethvargo All the packages listed in your link seem to be installed in a workflow running on ubuntu-latest.

The current workaround is to add this line prior to the 'gcloud alpha storage' command.

sudo pip install google-crc32c==1.3.0 --target /usr/lib/google-cloud-sdk/lib/third_party

[sethvargo]

Hi @fleroux514

I just tried to reproduce this and it seems to work for me:

steps:
- uses: 'actions/checkout@v2'

- id: 'auth'
  uses: 'google-github-actions/auth@v0'
  with:
    workload_identity_provider: 'projects/416438230019/locations/global/workloadIdentityPools/my-pool/providers/github'
    service_account: '[email protected]'

- id: 'setup-gcloud'
  uses: 'google-github-actions/setup-gcloud@v0'
  with:
    install_components: 'alpha'

- id: 'download'
  run: |-
    gcloud alpha storage cp gs://sv-actions-test/file.txt ./
    ls -alh .

How large is the file?

[sethvargo]

I was able to reproduce with a 2GB file, so it's not all files; just files over a certain size.

Unfortunately we don't really control the base image, and I don't think it's safe for us to pre-install this software (especially since this is only in an alpha component). @bharathkkb @averikitsch thoughts?

My personal preference would be to document this requirement, add an example, and link to https://cloud.google.com/storage/docs/gsutil/addlhelp/CRC32CandInstallingcrcmod.

[bharathkkb]

@sethvargo adding an example rather than preinstalling makes sense. I suspect as the command group matures this maybe handled by gcloud. /cc @verbanicm since Mike is compiling a list of useful examples

[sethvargo]

The latest versions of gcloud storage do not require crcmod.