terraform icon indicating copy to clipboard operation
terraform copied to clipboard
verified publisher icon hashicorp

Error: Failed to get existing workspaces: containers.Client#ListBlobs: Failure sending request: StatusCode=0 -- Original Error: Get "https://storage.blob.core.windows.net/container?comp=list&prefix=terraform.tfstateenv%3A&restype=container": EOF

Open rod2198 opened this issue 2 years ago • 3 comments

Terraform Version

Terraform v1.6.1 on windows_amd64

Terraform Configuration Files

terraform {
  required_providers {
    azurerm = {
      source = "hashicorp/azurerm"
      version = "3.79.0"
    }
  }
  backend "azurerm" {
    resource_group_name  = "rg_name"
    storage_account_name = "storage"
    container_name            = "container"
    key                                = "terraform.tfstate"
  }
}

Debug Output

2023-11-07T14:52:25.778Z [TRACE] statemgr.Filesystem: reading initial snapshot from terraform.tfstate
2023-11-07T14:52:25.778Z [TRACE] statemgr.Filesystem: snapshot file has nil snapshot, but that's okay
2023-11-07T14:52:25.778Z [TRACE] statemgr.Filesystem: read nil snapshot
2023-11-07T14:52:25.778Z [TRACE] Meta.Backend: ignoring local "default" workspace because its state is empty
2023-11-07T14:52:25.784Z [DEBUG] New state was assigned lineage "xxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxx"
2023-11-07T14:52:25.784Z [DEBUG] Azure Backend Request: 
GET /container?comp=list&prefix=terraform.tfstateenv%3A&restype=container HTTP/1.1
Host: storage.blob.core.windows.net
User-Agent: HashiCorp Terraform/1.6.1 (+https://www.terraform.io)
Content-Type: application/xml; charset=utf-8
X-Ms-Date: Tue, 07 Nov 2023 14:52:25 GMT
X-Ms-Version: 2018-11-09
Accept-Encoding: gzip
2023-11-07T14:54:36.671Z [DEBUG] Request to https://storage.blob.core.windows.net/container?comp=list&prefix=terraform.tfstateenv%3A&restype=container completed with no response
2023-11-07T14:55:06.674Z [DEBUG] Azure Backend Request: 
GET /container?comp=list&prefix=terraform.tfstateenv%3A&restype=container HTTP/1.1
Host: storage.blob.core.windows.net
User-Agent: HashiCorp Terraform/1.6.1 (+https://www.terraform.io)
Content-Type: application/xml; charset=utf-8
X-Ms-Date: Tue, 07 Nov 2023 14:52:25 GMT
X-Ms-Version: 2018-11-09
Accept-Encoding: gzip

Expected Behavior

Hi everyone,

I kindly ask for your support with the problem I am currently experiencing. Basically I need to configure the backend in my terraform code but currently from the client where I'm trying it's not possible and I get the error I indicated. The client is an agent pool used for a pipeline on Azure Devops, so I take advantage of the pre-built modules that are already present.

Actual Behavior

I have this issue during the terraform init process:

terraform.exe init -backend-config=storage_account_name=storage -backend-config=container_name=container -backend-config=key=terraform.tfstate -backend-config=resource_group_name=rg -backend-config=subscription_id=xxxxxx-xxxx-xxxxx-xxxxxx-xxxxxxx -backend-config=tenant_id=xxxxx-xxxxx-xxxx-xxxxx-xxxxxx -backend-config=client_id=*** -backend-config=client_secret=***

Error: Failed to get existing workspaces: containers.Client#ListBlobs: Failure sending request: StatusCode=0 -- Original Error: Get "https://storage.blob.core.windows.net/container?comp=list&prefix=terraform.tfstateenv%3A&restype=container": EOF

Steps to Reproduce

  1. terraform init -backend-config

Additional Context

No response

References

No response

rod2198 avatar Nov 07 '23 15:11 rod2198

I just came across this error. Here are some things I noticed/tried while trying to fix the issue. Hopefully this helps someone.

My deployment is also running in Devops. I have found that the same debug message that rod2198 had also appears for me on terraform init, plan, workspace, and apply (probably any command that needs to access the remote storage). From watching the logs I can see that it hangs for a very long time on these requests. Sometimes it retries and succeeds with a 200 response but even if it gets through each command, the time it takes is upward of 30 mins. If it gets "completed with no response" enough times then the command fails with the same error that rod2198 got. I have found that this issue gets progressively worse the more times you attempt to run the pipeline/commands. My pipeline is running on a self hosted Linux agent.

I found that this issue does not happen if the commands are run locally or if not using remote storage. I have checked that the Azure CLI token has not expired, I have checked that the DNS for the storage account resolves. I have checked the firewall and storage account for failures or denies (there are none). The service principle we use to authenticate has the correct permissions to access the storage account.

This issue varies in severity. sometimes it only returns "completed with no response" once and then the rest of the pipeline succeeds without issue and other times every command hangs and timesout until eventually it hits the max retries and errors.

Perks-of-Being-a-Cauliflower avatar Jan 19 '24 04:01 Perks-of-Being-a-Cauliflower

I have the same issue in terraform 1.7.1, same checks that @Perks-of-Being-a-Cauliflower mentioned are done.

Run terraform init -backend-config="key=portal/prod/portal-infra/terraform.tfstate" -reconfigure
2024-01-25T10:23:45.425Z [INFO]  Terraform version: 1.7.1
2024-01-25T10:23:45.426Z [DEBUG] using github.com/hashicorp/go-tfe v1.41.0
2024-01-25T10:23:45.426Z [DEBUG] using github.com/hashicorp/hcl/v2 v2.19.1
2024-01-25T10:23:45.426Z [DEBUG] using github.com/hashicorp/terraform-svchost v0.1.1
2024-01-25T10:23:45.426Z [DEBUG] using github.com/zclconf/go-cty v1.14.1
2024-01-25T10:23:45.426Z [INFO]  Go runtime version: go1.21.5
2024-01-25T10:23:45.426Z [INFO]  CLI args: []string{"terraform", "init", "-backend-config=key=portal/prod/portal-infra/terraform.tfstate", "-reconfigure"}
2024-01-25T10:23:45.426Z [DEBUG] Attempting to open CLI config file: /home/docker/.terraformrc
2024-01-25T10:23:45.426Z [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2024-01-25T10:23:45.426Z [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2024-01-25T10:23:45.426Z [DEBUG] ignoring non-existing provider search directory /home/docker/.terraform.d/plugins
2024-01-25T10:23:45.426Z [DEBUG] ignoring non-existing provider search directory /home/docker/.local/share/terraform/plugins
2024-01-25T10:23:45.426Z [DEBUG] ignoring non-existing provider search directory /usr/local/share/terraform/plugins
2024-01-25T10:23:45.426Z [DEBUG] ignoring non-existing provider search directory /usr/share/terraform/plugins
2024-01-25T10:23:45.432Z [INFO]  CLI command args: []string{"init", "-backend-config=key=portal/prod/portal-infra/terraform.tfstate", "-reconfigure"}

Initializing the backend...
2024-01-25T10:23:45.468Z [DEBUG] New state was assigned lineage "3c1ca212-5cf4-17be-3ccd-e2aa328fc412"
2024-01-25T10:23:45.469Z [DEBUG] checking for provisioner in "."
2024-01-25T10:23:45.470Z [DEBUG] checking for provisioner in "/home/docker/actions-runner/_work/_temp/c27432b6-08db-46cd-923a-9c0bad6b620a"
2024-01-25T10:23:45.471Z [INFO]  Testing if Service Principal / Client Certificate is applicable for Authentication..
2024-01-25T10:23:45.471Z [INFO]  Testing if Multi Tenant Service Principal / Client Secret is applicable for Authentication..
2024-01-25T10:23:45.471Z [INFO]  Testing if Service Principal / Client Secret is applicable for Authentication..
2024-01-25T10:23:45.471Z [INFO]  Testing if OIDC is applicable for Authentication..
2024-01-25T10:23:45.471Z [INFO]  Using OIDC for Authentication
2024-01-25T10:23:45.471Z [INFO]  Getting OAuth config for endpoint https://login.microsoftonline.com/ with  tenant ***
2024-01-25T10:23:45.473Z [DEBUG] Obtaining an MSAL / Microsoft Graph token for Resource Manager..
2024-01-25T10:23:45.475Z [DEBUG] New state was assigned lineage "17814ded-cbf7-e921-8333-aa8b0e533eb3"
2024-01-25T10:23:45.475Z [DEBUG] Building the Container Client from an Access Token (using user credentials)
2024-01-25T10:23:46.239Z [DEBUG] Azure Backend Request: 
POST /subscriptions/***/resourceGroups/rg-iasp-euw-terraformBackend/providers/Microsoft.Storage/storageAccounts/<storageaccountname>/listKeys?api-version=2021-01-01 HTTP/1.1
Host: management.azure.com
User-Agent: HashiCorp Terraform/1.7.1 (+https://www.terraform.io)
Content-Length: 0
X-Ms-Authorization-Auxiliary: 
Accept-Encoding: gzip
2024-01-25T10:23:46.444Z [DEBUG] Azure Backend Response for https://management.azure.com/subscriptions/***/resourceGroups/rg-iasp-euw-terraformBackend/providers/Microsoft.Storage/storageAccounts/<storageaccountname>/listKeys?api-version=2021-01-01: 
HTTP/2.0 200 OK
Content-Length: 288
Cache-Control: no-cache
Content-Type: application/json
Date: Thu, 25 Jan 2024 10:23:45 GMT
Expires: -1
Pragma: no-cache
Server: Microsoft-Azure-Storage-Resource-Provider/1.0,Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
X-Ms-Correlation-Request-Id: b4547ff4-c8aa-46b1-826c-5ab999fdaaf7
X-Ms-Ratelimit-Remaining-Subscription-Resource-Requests: 11999
X-Ms-Request-Id: 0a1f3c40-66f5-40f5-8172-0e639393bacf
X-Ms-Routing-Request-Id: WESTEUROPE:20240125T102346Z:b4547ff4-c8aa-46b1-826c-5ab999fdaaf7

{"keys":[{"keyName":"key1","value":"****************","permissions":"FULL"},{"keyName":"key2","value":"****************","permissions":"FULL"}]}
2024-01-25T10:23:46.445Z [DEBUG] Azure Backend Request: 
GET /tfstate?comp=list&prefix=portal%2Fprod%2Fportal-infra%2Fterraform.tfstateenv%3A&restype=container HTTP/1.1
Host: <storageaccountname>.blob.core.windows.net
User-Agent: HashiCorp Terraform/1.7.1 (+https://www.terraform.io)
Content-Type: application/xml; charset=utf-8
X-Ms-Date: Thu, 25 Jan 2024 10:23:46 GMT
X-Ms-Version: 2018-11-09
Accept-Encoding: gzip
2024-01-25T10:39:37.968Z [DEBUG] Request to https://<storageaccountname>.blob.core.windows.net/tfstate?comp=list&prefix=portal%2Fprod%2Fportal-infra%2Fterraform.tfstateenv%3A&restype=container completed with no response
2024-01-25T10:40:07.971Z [DEBUG] Azure Backend Request: 
GET /tfstate?comp=list&prefix=portal%2Fprod%2Fportal-infra%2Fterraform.tfstateenv%3A&restype=container HTTP/1.1
Host: <storageaccountname>.blob.core.windows.net
User-Agent: HashiCorp Terraform/1.7.1 (+https://www.terraform.io)
Content-Type: application/xml; charset=utf-8
X-Ms-Date: Thu, 25 Jan 2024 10:23:46 GMT
X-Ms-Version: 2018-11-09
Accept-Encoding: gzip
2024-01-25T10:56:01.010Z [DEBUG] Request to https://<storageaccountname>.blob.core.windows.net/tfstate?comp=list&prefix=portal%2Fprod%2Fportal-infra%2Fterraform.tfstateenv%3A&restype=container completed with no response
2024-01-25T10:57:01.016Z [DEBUG] Azure Backend Request: 
GET /tfstate?comp=list&prefix=portal%2Fprod%2Fportal-infra%2Fterraform.tfstateenv%3A&restype=container HTTP/1.1
Host: <storageaccountname>.blob.core.windows.net
User-Agent: HashiCorp Terraform/1.7.1 (+https://www.terraform.io)
Content-Type: application/xml; charset=utf-8
X-Ms-Date: Thu, 25 Jan 2024 10:23:46 GMT
X-Ms-Version: 2018-11-09
Accept-Encoding: gzip
2024-01-25T11:12:52.720Z [DEBUG] Request to https://<storageaccountname>.blob.core.windows.net/tfstate?comp=list&prefix=portal%2Fprod%2Fportal-infra%2Fterraform.tfstateenv%3A&restype=container completed with no response
2024-01-25T11:14:52.721Z [DEBUG] Azure Backend Request: 
GET /tfstate?comp=list&prefix=portal%2Fprod%2Fportal-infra%2Fterraform.tfstateenv%3A&restype=container HTTP/1.1
Host: <storageaccountname>.blob.core.windows.net
User-Agent: HashiCorp Terraform/1.7.1 (+https://www.terraform.io)
Content-Type: application/xml; charset=utf-8
X-Ms-Date: Thu, 25 Jan 2024 10:23:46 GMT
X-Ms-Version: 2018-11-09
Accept-Encoding: gzip
2024-01-25T11:30:45.872Z [DEBUG] Request to https://<storageaccountname>.blob.core.windows.net/tfstate?comp=list&prefix=portal%2Fprod%2Fportal-infra%2Fterraform.tfstateenv%3A&restype=container completed with no response
Initializing modules...
2024-01-25T11:34:45.879Z [DEBUG] Module installer: begin RG
2024-01-25T11:34:45.880Z [DEBUG] Module installer: RG installed at modules/rg
2024-01-25T11:34:45.880Z [DEBUG] Module installer: begin subnet_app
- RG in modules/rg
2024-01-25T11:34:45.883Z [DEBUG] Module installer: subnet_app installed at modules/subnet
- subnet_app in modules/subnet
╷
│ Error: Failed to get existing workspaces: containers.Client#ListBlobs: Failure sending request: StatusCode=0 -- Original Error: Get "https://<storageaccountname>.blob.core.windows.net/tfstate?comp=list&prefix=portal%2Fprod%2Fportal-infra%2Fterraform.tfstateenv%3A&restype=container": read tcp 10.233.134.164:47904->20.157.146.104:443: read: connection timed out
│ 
│ 
╵

Error: Process completed with exit code 1.

Niahm avatar Jan 25 '24 14:01 Niahm

Guys i solved the issue!

We solved the issue, basically it was necessary to configure the peering between Devops agent VM Vnet and the storage private endpoint Vnet and modify the NSG associated with the subnet to allow 443 traffic on the container.

I hope that this can help you.

rod2198 avatar Jan 25 '24 14:01 rod2198