google-cloud-cpp icon indicating copy to clipboard operation
google-cloud-cpp copied to clipboard
verified publisher icon googleapis

Bad Gateway errors running integration test

Open coryan opened this issue 3 years ago • 12 comments

This was on Kokoro + Windows. This could be a problem in the environment (just Kokoro and/or Windows resolving the wrong host). It is possible that gRPC (or our code), does not discard what is probably an unusable socket after a 502 error.

https://source.cloud.google.com/results/invocations/72aee9e2-779c-4194-b754-a7a8ccb766ff

2022-02-08T18:35:39.111405000Z [DEBUG] <6640> Commit() >> status=UNAVAILABLE: 502:Bad Gateway (C:\b\rvommcsc\execroot\com_github_googleapis_google_cloud_cpp\google/cloud/internal/log_wrapper.h:89)


2022-02-08T18:35:39.216968800Z [DEBUG] <6640> Commit() << google.spanner.v1.CommitRequest { session: "projects/cloud-cpp-testing-resources/instances/test-instance-5/databases/db-2022-02-08-481maaq2usgavf9c/sessions/AN4G3x8rgh4TUYiYdMjD4U4pTSzb9xnAXSkVSXwXznybrzUUKeG-eEtWt8EelQ" transaction_id: "AN4G3x-pbnZVtUyBoXohbUKO8jSzMV3gGnbPTw" mutations { insert { table: "DataTypes" columns: "Id" columns: "JsonValue" values { values { string_value: "Id-0" } values { string_value: "[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[null]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]" } } } } request_options { } } (C:\b\rvommcsc\execroot\com_github_googleapis_google_cloud_cpp\google/cloud/internal/log_wrapper.h:86)


2022-02-08T18:35:45.710956700Z [DEBUG] <6640> Commit() >> status=ABORTED: Aborted due to transient fault (C:\b\rvommcsc\execroot\com_github_googleapis_google_cloud_cpp\google/cloud/internal/log_wrapper.h:89)


2022-02-08T18:35:46.054285900Z [DEBUG] <6640> BeginTransaction() << google.spanner.v1.BeginTransactionRequest { session: "projects/cloud-cpp-testing-resources/instances/test-instance-5/databases/db-2022-02-08-481maaq2usgavf9c/sessions/AN4G3x8rgh4TUYiYdMjD4U4pTSzb9xnAXSkVSXwXznybrzUUKeG-eEtWt8EelQ" options { read_write { } } request_options { } } (C:\b\rvommcsc\execroot\com_github_googleapis_google_cloud_cpp\google/cloud/internal/log_wrapper.h:86)


2022-02-08T18:35:46.132042300Z [DEBUG] <6640> BeginTransaction() >> response=google.spanner.v1.Transaction { id: "AN4G3x-zcM93tgG_emDlJjxl2ToRrpYBVKEKeg" } (C:\b\rvommcsc\execroot\com_github_googleapis_google_cloud_cpp\google/cloud/internal/log_wrapper.h:91)


2022-02-08T18:35:46.132358300Z [DEBUG] <6640> Commit() << google.spanner.v1.CommitRequest { session: "projects/cloud-cpp-testing-resources/instances/test-instance-5/databases/db-2022-02-08-481maaq2usgavf9c/sessions/AN4G3x8rgh4TUYiYdMjD4U4pTSzb9xnAXSkVSXwXznybrzUUKeG-eEtWt8EelQ" transaction_id: "AN4G3x-zcM93tgG_emDlJjxl2ToRrpYBVKEKeg" mutations { insert { table: "DataTypes" columns: "Id" columns: "JsonValue" values { values { string_value: "Id-0" } values { string_value: "[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[null]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]" } } } } request_options { } } (C:\b\rvommcsc\execroot\com_github_googleapis_google_cloud_cpp\google/cloud/internal/log_wrapper.h:86)


2022-02-08T18:40:48.679830400Z [DEBUG] <6640> Commit() >> status=UNAVAILABLE: 502:Bad Gateway (C:\b\rvommcsc\execroot\com_github_googleapis_google_cloud_cpp\google/cloud/internal/log_wrapper.h:89)


2022-02-08T18:40:48.822204900Z [DEBUG] <6640> Commit() << google.spanner.v1.CommitRequest { session: "projects/cloud-cpp-testing-resources/instances/test-instance-5/databases/db-2022-02-08-481maaq2usgavf9c/sessions/AN4G3x8rgh4TUYiYdMjD4U4pTSzb9xnAXSkVSXwXznybrzUUKeG-eEtWt8EelQ" transaction_id: "AN4G3x-zcM93tgG_emDlJjxl2ToRrpYBVKEKeg" mutations { insert { table: "DataTypes" columns: "Id" columns: "JsonValue" values { values { string_value: "Id-0" } values { string_value: "[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[null]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]" } } } } request_options { } } (C:\b\rvommcsc\execroot\com_github_googleapis_google_cloud_cpp\google/cloud/internal/log_wrapper.h:86)


2022-02-08T18:40:55.271720800Z [DEBUG] <6640> Commit() >> status=ABORTED: Aborted due to transient fault (C:\b\rvommcsc\execroot\com_github_googleapis_google_cloud_cpp\google/cloud/internal/log_wrapper.h:89)

coryan avatar Feb 08 '22 19:02 coryan

https://source.cloud.google.com/results/invocations/caf2931a-fe4c-4496-816e-ffc19a847a7b

coryan avatar Feb 09 '22 01:02 coryan

https://source.cloud.google.com/results/invocations/8632bc7a-fe86-4442-bb17-6d20ec5cdb28

coryan avatar Feb 09 '22 11:02 coryan

https://source.cloud.google.com/results/invocations/4abece1b-0e1c-4f57-9433-9b077938446c

coryan avatar Feb 09 '22 12:02 coryan

https://source.cloud.google.com/results/invocations/66ec7626-1497-4702-939e-b45ecbbd4029

coryan avatar Feb 09 '22 12:02 coryan

No repeats in 90d (and we think these were Kokoro related). Closing.

coryan avatar May 20 '22 00:05 coryan

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/b2810edf-3a53-48c7-9dfd-7771c938778c;tab=detail?project=cloud-cpp-testing-resources
  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/dfb7815aa2d80deec9f1dda3b526b24b8ede7242/fedora-35-asan/log-b2810edf-3a53-48c7-9dfd-7771c938778c.txt
2022-06-07T18:28:14.733885824Z [DEBUG] <139930538013696> GenerateAccessToken() << google.iam.credentials.v1.GenerateAccessTokenRequest { name: "projects/-/serviceAccounts/iam-t...<truncated>..." scope: "https://www.googleapis.com/auth/...<truncated>..." lifetime { seconds: 60 } } (./google/cloud/internal/log_wrapper.h:89)
2022-06-07T18:28:30.662757007Z [DEBUG] <139930538013696> GenerateAccessToken() >> status=UNAVAILABLE: 502:Bad Gateway (./google/cloud/internal/log_wrapper.h:92)
terminate called after throwing an instance of 'std::runtime_error'
  what():  Error in non-idempotent operation GenerateAccessToken: 502:Bad Gateway

dbolduc avatar Jun 07 '22 19:06 dbolduc

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/9851987a-c5b7-4d48-a622-f8a1afb6d938;tab=detail?project=cloud-cpp-testing-resources
  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/ee484a2ef86d3e851602430ff72c29899b61141e/fedora-35-coverage/log-9851987a-c5b7-4d48-a622-f8a1afb6d938.txt

coryan avatar Jun 24 '22 12:06 coryan

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/7f262d8b-4145-4e3a-ac0a-c1a3956c8211;tab=detail?project=cloud-cpp-testing-resources
  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/e9cbcd6cb429a8516446b59e04b3beef816a1636/fedora-35-asan/log-7f262d8b-4145-4e3a-ac0a-c1a3956c8211.txt

devbww avatar Jul 02 '22 23:07 devbww

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/46754a1d-5dd6-4346-aa87-bfa31fe521e0;tab=detail?project=cloud-cpp-testing-resources
  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/eba6d7f39d1c572ec7359c72b009d25c83cd1fc3/fedora-36-quickstart-production/log-46754a1d-5dd6-4346-aa87-bfa31fe521e0.txt

devbww avatar Jul 27 '22 22:07 devbww

I wonder if that request is retryable

coryan avatar Jul 27 '22 22:07 coryan

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/c2474f81-6117-4c31-914e-d1576b49631f;tab=detail?project=cloud-cpp-testing-resources
  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/fe748ba37d785a106ea8e72cfb28a620701f4c00/fedora-msan-msan/log-c2474f81-6117-4c31-914e-d1576b49631f.txt
[ RUN      ] BigQueryReadIntegrationTest.SplitReadStreamProtoSuccess
google/cloud/bigquery/integration_tests/bigquery_read_integration_test.cc:244: Failure
Value of: session_response_2
Expected: code is equal to OK and message is anything
  Actual: 136-byte object <80-56 01-00 A0-70 00-00 00-00 00-00 02-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 ... 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00>, whose status is UNAVAILABLE: Error in non-idempotent operation CreateReadSession: 502:Bad Gateway, with a code that isn't equal to OK, but a message that is anything

dbolduc avatar Aug 03 '22 18:08 dbolduc

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/0b6e539f-4f1e-4d7c-8281-3bd79a7265b5;tab=detail?project=cloud-cpp-testing-resources
  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/2eb854399c0dd2b9ef8c19bafcfcff3832d627c3/fedora-36-libcxx/log-0b6e539f-4f1e-4d7c-8281-3bd79a7265b5.txt

BigQueryReadIntegrationTest SplitReadStreamProtoSuccess again

dbolduc avatar Aug 11 '22 00:08 dbolduc

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/99dc4387-6e9c-4328-a8f2-f9c25a80d2b5;tab=detail?project=cloud-cpp-testing-resources
  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/3942b25db6d47c8d421aa827ff54d28fd5229eb9/fedora-36-integration-daily/log-99dc4387-6e9c-4328-a8f2-f9c25a80d2b5.txt

bigtable table admin integration tests

dbolduc avatar Oct 07 '22 16:10 dbolduc

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/f5034eea-76f4-439c-b2a8-658d2a8ede53;tab=detail?project=cloud-cpp-testing-resources
  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/5fcdc0dd9ad88ab76dfc2fdd08e35ea501a2885e/demo-debian-buster-demo-install/log-f5034eea-76f4-439c-b2a8-658d2a8ede53.txt

BigQuery quickstart

coryan avatar Dec 02 '22 00:12 coryan

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/734d940b-6e98-4305-9b18-61e50191e42b;tab=detail?project=cloud-cpp-testing-resources
  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/ff0202c9264a333286fa1fecb7ff4962b26ccf66/demo-alpine-stable-demo-install/log-734d940b-6e98-4305-9b18-61e50191e42b.txt

Also BigQuery quickstart but a different build

coryan avatar Dec 02 '22 00:12 coryan

Closing in favor of #10413

coryan avatar Dec 09 '22 16:12 coryan