How to retry upon RPC_DEADLINE_TOO_SHORT?

[lezhnev74]

We see that some requests return erroneous responses like this one:

Response
-------
Headers: {
    "request-id": "kAd93dduv2eyfPBYF1RHNg",
    "date": "Tue, 21 Jun 2022 11:01:11 GMT",
    "alt-svc": "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\""
}

Fault
-------
Status code: 3
Details: Request contains an invalid argument.
Failure: {"errors":[{"errorCode":{"requestError":"RPC_DEADLINE_TOO_SHORT"},"message":"The request timed out because the specified deadline was too short to complete the request. Please set a larger deadline and retry."}],"requestId":"kAd93dduv2eyfPBYF1RHNg"}

As you can see the error is related to a small timeout, however the status code in the response is 3 which corresponds to INVALID_ARGUMENT code (is it a correct code for this error?).

So how to correctly configure RetrySettings so they handle this case? It contains retryableCodes key, but what code we must include in there to handle the above problem?

[PierrickVoulet]

Hi @lezhnev74 - Did you have the chance to check whether it is an expected behavior with the Google Ads API support? This looks wrong to me.

[lezhnev74]

I thought to start here, but let me contact them over email and get back here afterwards.

[AnashOommen]

Out of curiosity,

1. what timeout are you setting (if any)?
2. which methods generate this error?
3. if you are setting timeouts, why are you setting them?

On Tue, Jun 21, 2022, 12:21 PM Dmitry Lezhnev @.***> wrote:

I thought to start here, but let me contact them over email and get back here afterwards.

— Reply to this email directly, view it on GitHub https://github.com/googleads/google-ads-php/issues/817#issuecomment-1161975400, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABCOJCCFWM2NZCUVMWSPADTVQHTW3ANCNFSM5ZMJIHIQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[lezhnev74]

@AnashOommen yup, here is our current retry settings:

[lezhnev74]

I think the settings are never applied as the response status code (3) is not among retryable codes. I think this package correctly decodes grpc messages and we can trust status codes we see in logs.

[AnashOommen]

Also,

1. What methods are generating this error?
2. Is there a particular reason why you are setting the 50000ms as initial deadline? Is this due to a platform restriction where the code is deployed, or is this more like a good engineering practice?

On Wed, Jun 22, 2022, 2:19 AM Dmitry Lezhnev @.***> wrote:

I think the settings are never applied as the response status code (3) is not among retryable codes. I think this package correctly decodes grpc messages and we can trust status codes we see in logs.

— Reply to this email directly, view it on GitHub https://github.com/googleads/google-ads-php/issues/817#issuecomment-1162691259, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABCOJCFYW5IPHJIE5E5OWE3VQKWA3ANCNFSM5ZMJIHIQ . You are receiving this because you were mentioned.Message ID: @.***>

[lezhnev74]

Ok, the discussion over email led nowhere, they think this is a client-level issue, not API itself.

What methods are generating this error?

I see in logs that only Search requests give this error. This is a sample query that return the error:

SELECT 
                metrics.impressions,
                metrics.clicks,
                metrics.cost_micros,
                metrics.conversions                
FROM customer
WHERE segments.date BETWEEN 20210604 AND 20210813

Is there a particular reason why you are setting the 50000ms as initial deadline

We have been using these settings since 2019, if you can suggest a better defaults I'd update our code.

[lezhnev74]

Again, the default numbers you see above only work if the return code is among retryableCodes, right? Since it returns code 3, the retry never happens. That is how I see the case.

[PierrickVoulet]

Again, the default numbers you see above only work if the return code is among retryableCodes, right? Since it returns code 3, the retry never happens. That is how I see the case.

The API status code is 3 => INVALID_ARGUMENT even though the only error listed in the details is RPC_DEADLINE_TOO_SHORT. I believe this behavior is related to Google Ads API so the Google Ads API support should be able to confirm whether it works as expected and why.

The API status code 3 => INVALID_ARGUMENT could be added to the retryableCodes so that the retry is triggered when such an error occurs. Did you have the chance to try this yet? Any issues if you do that?

[lezhnev74]

I believe this behavior is related to Google Ads API so the Google Ads API support should be able to confirm whether it works as expected and why

This is a circle, they led me here, you lead me there. I will ping them again and refer to this thread.

The API status code 3 => INVALID_ARGUMENT could be added to the retryableCodes

Yes, that is a "workaround" which feels wrong. Invalid argument is a nonretryable failure by nature, right?

[lezhnev74]

Over email they responded: Yes, I can confirm that this is the possible behaviour of the API. To address this, you may continue posting on GitHub.. Since API response is quite vague, the retrying logic should also check requestError along with status code. So in the case of this problem the codes would be:

• status code 3 (invalid argument)
• request error: 33 (RPC_DEADLINE_TOO_SHORT)

AFAIK, this package can't do that and we'll have to implement it somehow in our app

[PierrickVoulet]

Over email they responded: Yes, I can confirm that this is the possible behaviour of the API. To address this, you may continue posting on GitHub..

Thanks for confirming.

Since API response is quite vague, the retrying logic should also check requestError along with status code. So in the case of this problem the codes would be:

• status code 3 (invalid argument)
• request error: 33 (RPC_DEADLINE_TOO_SHORT) AFAIK, this package can't do that and we'll have to implement it somehow in our app

Correct, this is not supported. It would be challenging to add support for such a behavior because the retry mechanism is implemented at the gax-php level and it is not aware of the API-specific errors such as RequestError. RPC_DEADLINE_TOO_SHORT .

I suspect the root cause for this issue is that you set initialRpcTimeoutMillis to 50000 which may be too short of a deadline for this particular GoogleAds.Search API request. Could you try using something higher? When confirmed, I will circle back with the API team as they may be able to provide more details on how to always set a valid value.

[lezhnev74]

I suspect the root cause for this issue is that you set initialRpcTimeoutMillis to 50000 which may be too short of a deadline for this particular GoogleAds.Search API request. Could you try using something higher?

Yeah, good point. Let me start with tweaking the numbers. I'll come back in a few weeks to confirm if that helped. Because the error is quite random I might need more time to see if that works.

[PierrickVoulet]

I just confirmed that retry settings are all optional. If not set, default values are set for you.

@lezhnev74 The best solution may be to remove unnecessary settings including initialRpcTimeoutMillis altogether.

[adrianlaraperez88]

Hi, Google Ads Team I have the same issue in V11,

const CLIENT_TIMEOUT_MILLIS = 20 * 60 * 1000;

Error: `------- Method Name: /google.ads.googleads.v11.services.GoogleAdsService/Search Host: googleads.googleapis.com Headers: { "x-goog-api-client": "gl-php/7.4.22 gccl/15.1.0 gapic/15.1.0 gax/1.13.0 grpc/1.25.0 rest/1.13.0 pb/+n", "x-goog-request-params": "customer_id=XXXXXXXX", "developer-token": "REDACTED", "login-customer-id": "XXXXXXXXX" } Request: {"customerId":"1206260014","query":"SELECT ad_group_criterion.criterion_id,\r\n\t\t\tad_group_criterion.keyword.text,\r\n\t\t\tad_group.name,\r\n\t\t\tad_group_criterion.status, \r\n\t\t\tad_group_criterion.keyword.match_type, \r\n\t\t\tcampaign.id, \r\n\t\t\tsegments.date,\r\n\t\t\tad_group_criterion.quality_info.quality_score, \r\n\t\t\tad_group_criterion.quality_info.post_click_quality_score, \r\n\t\t\tad_group_criterion.quality_info.creative_quality_score,\r\n\t\t\tad_group_criterion.position_estimates.top_of_page_cpc_micros, \r\n\t\t\tad_group_criterion.position_estimates.first_page_cpc_micros, \r\n\t\t\tad_group_criterion.position_estimates.first_position_cpc_micros,\r\n\t\t\tmetrics.clicks, \r\n\t\t\tmetrics.impressions,\r\n\t\t\tmetrics.engagements, \r\n\t\t\tmetrics.interactions,\r\n\t\t\tmetrics.cost_micros,\r\n\t\t\tad_group_criterion.effective_cpc_bid_micros, \r\n\t\t\tmetrics.average_cost,\r\n\t\t\tmetrics.average_cpc,\r\n\t\t\tmetrics.search_exact_match_impression_share, \r\n\t\t\tmetrics.search_impression_share,\r\n\t\t\tmetrics.search_rank_lost_impression_share, \r\n\t\t\tmetrics.conversions,\r\n\t\t\tmetrics.all_conversions,\r\n\t\t\tmetrics.view_through_conversions, \r\n\t\t\tcampaign.name \r\n\t\t\tFROM keyword_view \r\n\t\t\tWHERE ad_group_criterion.status IN ('PAUSED','ENABLED') \r\n\t\t\tAND ad_group_criterion.negative = false \r\n\t\t\tAND campaign.status IN ('ENABLED','PAUSED') \r\n\t\t\tAND segments.date DURING YESTERDAY"}

Response

Headers: { "request-id": "QHpmE3VpaG4owaDd-XmRTQ", "date": "Fri, 22 Jul 2022 01:31:21 GMT", "alt-svc": "h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"" }

Fault

Status code: 3 Details: Request contains an invalid argument. Failure: {"errors":[{"errorCode":{"requestError":"RPC_DEADLINE_TOO_SHORT"},"message":"The request timed out because the specified deadline was too short to complete the request. Please set a larger deadline and retry."}],"requestId":"QHpmE3VpaG4owaDd-XmRTQ"} `

Query $query ="SELECT ad_group_criterion.criterion_id, ad_group_criterion.keyword.text, ad_group.name, ad_group_criterion.status, ad_group_criterion.keyword.match_type, campaign.id, segments.date, ad_group_criterion.quality_info.quality_score, ad_group_criterion.quality_info.post_click_quality_score, ad_group_criterion.quality_info.creative_quality_score, ad_group_criterion.position_estimates.top_of_page_cpc_micros, ad_group_criterion.position_estimates.first_page_cpc_micros, ad_group_criterion.position_estimates.first_position_cpc_micros, metrics.clicks, metrics.impressions, metrics.engagements, metrics.interactions, metrics.cost_micros, ad_group_criterion.effective_cpc_bid_micros, metrics.average_cost, metrics.average_cpc, metrics.search_exact_match_impression_share, metrics.search_impression_share, metrics.search_rank_lost_impression_share, metrics.conversions, metrics.all_conversions, metrics.view_through_conversions, campaign.name FROM keyword_view WHERE ad_group_criterion.status IN ('PAUSED','ENABLED') AND ad_group_criterion.negative = false AND campaign.status IN ('ENABLED','PAUSED') AND segments.date DURING YESTERDAY";

[PierrickVoulet]

@adrianlaraperez88 Did you try to increase the timeout?

If you have a hard 20 second limit please contact the Google Ads API support team to check how you may optimize your API requests to meet your time constraints.

[adrianlaraperez88]

@PierrickVoulet I make increase timeout same you do it in example and working fine with little Accounts, but with the big account or DateRange more than Yesterday send us an error.

[PierrickVoulet]

@PierrickVoulet I make increase timeout same you do it in example and working fine with little Accounts, but with the big account or DateRange more than Yesterday send us an error.

The timeouts you are using are too small for the requests to complete. You have two options:

1. Increase the timeouts.
2. Optimize your API requests to meet your time constraints, the Google Ads API support team can help you with this.

The example illustrates how to override the default behavior of the client library. The default behavior is just fine for the vast majority of use cases. Could you explain why you need to override it?

[adrianlaraperez88]

@PierrickVoulet I sent msg to the Support team, and made a test up 20 min to 120 min and continued with the same issue.

[PierrickVoulet]

@PierrickVoulet I sent msg to the Support team, and made a test up 20 min to 120 min and continued with the same issue.

Do you get the issue RPC_DEADLINE_TOO_SHORT if you do not override the client timeouts? Here is an example of how it looks like:

public static function runExample(GoogleAdsClient $googleAdsClient, int $customerId)
 {
    $response = $googleAdsClient->getGoogleAdsServiceClient()->search(
        $customerId,
        'SELECT ... FROM ...'
    );
    
    foreach ($response->iterateAllElements() as $googleAdsRow) {
        // Do something.
    }
}

[adrianlaraperez88]

Hi @PierrickVoulet, this was my last Test,

const CLIENT_TIMEOUT_MILLIS = 200 * 60 * 1000;
public function getKeywordPerformace($range = 'YESTERDAY'){
			$query ="SELECT ad_group_criterion.criterion_id,
			ad_group_criterion.keyword.text,
			ad_group.name,
			ad_group_criterion.status, 
			ad_group_criterion.keyword.match_type, 
			campaign.id, 
			segments.date,
			ad_group_criterion.quality_info.quality_score, 
			ad_group_criterion.quality_info.post_click_quality_score, 
			ad_group_criterion.quality_info.creative_quality_score,
			ad_group_criterion.position_estimates.top_of_page_cpc_micros, 
			ad_group_criterion.position_estimates.first_page_cpc_micros, 
			ad_group_criterion.position_estimates.first_position_cpc_micros,
			metrics.clicks, 
			metrics.impressions,
			metrics.engagements, 
			metrics.interactions,
			metrics.cost_micros,
			ad_group_criterion.effective_cpc_bid_micros, 
			metrics.average_cost,
			metrics.average_cpc,
			metrics.search_exact_match_impression_share, 
			metrics.search_impression_share,
			metrics.search_rank_lost_impression_share, 
			metrics.conversions,
			metrics.all_conversions,
			metrics.view_through_conversions, 
			campaign.name 
			FROM keyword_view 
			WHERE ad_group_criterion.status IN ('PAUSED','ENABLED') 
			AND ad_group_criterion.negative = false 
			AND campaign.status IN ('ENABLED','PAUSED')
			AND segments.date DURING ".$range;
			
			$settings =[ 'pageSize' => 100,'retrySettings' => [
                        // Sets the timeout that is used for the first try to one tenth of the
                        // maximum accumulative timeout of the call.
                        // Note: This overrides the default value and can lead to
                        // RequestError.RPC_DEADLINE_TOO_SHORT errors when too small. We recommend
                        // to do it only if necessary.
                        'initialRpcTimeoutMillis' => self::CLIENT_TIMEOUT_MILLIS ,
                        // Sets the maximum timeout that can be used for any given try to one fifth
                        // of the maximum accumulative timeout of the call (two times greater than
                        // the timeout that is used for the first try).
                        'maxRpcTimeoutMillis' => self::CLIENT_TIMEOUT_MILLIS 
						
                    ]];
	
			$response = $this->service->getGoogleAdsServiceClient()->search($this->customerId, $query, $settings);
		$report = array();
		foreach ($response->iterateAllElements() as $googleAdsRow) {
			$campaign = $googleAdsRow->getCampaign();
			$metrics = $googleAdsRow->getMetrics();
			$adGroupCriterion = $googleAdsRow->getAdGroupCriterion();
			
			$segments = $googleAdsRow->getSegments();
			$adGroup = $googleAdsRow->getAdGroup();
			$positionEstimates = $adGroupCriterion->getPositionEstimates();
			$qualityInfo = $adGroupCriterion->getQualityInfo();
			..
			..
			```

[PierrickVoulet]

Please try replacing

$response = $this->service->getGoogleAdsServiceClient()->search($this->customerId, $query, $settings)

with

$response = $this->service->getGoogleAdsServiceClient()->search($this->customerId, $query).

[adrianlaraperez88]

@PierrickVoulet same result `[2022-07-28T02:04:49.823469-04:00] google-ads.WARNING: Request made: Host: "googleads.googleapis.com", Method: "/google.ads.googleads.v11.services.GoogleAdsService/Search", CustomerId: xxxx014, RequestId: "5q7kthTHRdL1Z5Msp4f_Tg", IsFault: 1, FaultMessage: "["The request timed out because the specified deadline was too short to complete the request. Set a larger deadline and retry."]"
[2022-07-28T02:04:49.825815-04:00] google-ads.NOTICE: Request

Method Name: /google.ads.googleads.v11.services.GoogleAdsService/Search Host: googleads.googleapis.com Headers: { "x-goog-api-client": "gl-php/7.4.22 gccl/15.1.0 gapic/15.1.0 gax/1.13.0 grpc/1.25.0 rest/1.13.0 pb/+n", "x-goog-request-params": "customer_id=xxxxxxx014", "developer-token": "REDACTED", "login-customer-id": "xxxxxxx" } Request: {"customerId":"xxxxxxxx014","query":"SELECT ad_group_criterion.criterion_id,\r\n\t\t\tad_group_criterion.keyword.text,\r\n\t\t\tad_group.name,\r\n\t\t\tad_group_criterion.status, \r\n\t\t\tad_group_criterion.keyword.match_type, \r\n\t\t\tcampaign.id, \r\n\t\t\tsegments.date,\r\n\t\t\tad_group_criterion.quality_info.quality_score, \r\n\t\t\tad_group_criterion.quality_info.post_click_quality_score, \r\n\t\t\tad_group_criterion.quality_info.creative_quality_score,\r\n\t\t\tad_group_criterion.position_estimates.top_of_page_cpc_micros, \r\n\t\t\tad_group_criterion.position_estimates.first_page_cpc_micros, \r\n\t\t\tad_group_criterion.position_estimates.first_position_cpc_micros,\r\n\t\t\tmetrics.clicks, \r\n\t\t\tmetrics.impressions,\r\n\t\t\tmetrics.engagements, \r\n\t\t\tmetrics.interactions,\r\n\t\t\tmetrics.cost_micros,\r\n\t\t\tad_group_criterion.effective_cpc_bid_micros, \r\n\t\t\tmetrics.average_cost,\r\n\t\t\tmetrics.average_cpc,\r\n\t\t\tmetrics.search_exact_match_impression_share, \r\n\t\t\tmetrics.search_impression_share,\r\n\t\t\tmetrics.search_rank_lost_impression_share, \r\n\t\t\tmetrics.conversions,\r\n\t\t\tmetrics.all_conversions,\r\n\t\t\tmetrics.view_through_conversions, \r\n\t\t\tcampaign.name \r\n\t\t\tFROM keyword_view \r\n\t\t\tWHERE ad_group_criterion.status IN ('PAUSED','ENABLED') \r\n\t\t\tAND ad_group_criterion.negative = false \r\n\t\t\tAND campaign.status IN ('ENABLED','PAUSED')\r\n\t\t\tAND segments.date DURING LAST_BUSINESS_WEEK"}

Response

Headers: { "request-id": "5q7kthTHRdL1Z5Msp4f_Tg", "date": "Thu, 28 Jul 2022 06:04:47 GMT", "alt-svc": "h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"" }

Fault

Status code: 3 Details: Request contains an invalid argument. Failure: {"errors":[{"errorCode":{"requestError":"RPC_DEADLINE_TOO_SHORT"},"message":"The request timed out because the specified deadline was too short to complete the request. Set a larger deadline and retry."}],`

[PierrickVoulet]

I tried on my side and do no reproduce the error. Could double check and provide the latest version of the code you execute?

Here is what I use on my side:

$query="SELECT ad_group_criterion.criterion_id,
ad_group_criterion.keyword.text,
ad_group.name,
ad_group_criterion.status, 
ad_group_criterion.keyword.match_type, 
campaign.id, 
segments.date,
ad_group_criterion.quality_info.quality_score, 
ad_group_criterion.quality_info.post_click_quality_score, 
ad_group_criterion.quality_info.creative_quality_score,
ad_group_criterion.position_estimates.top_of_page_cpc_micros, 
ad_group_criterion.position_estimates.first_page_cpc_micros, 
ad_group_criterion.position_estimates.first_position_cpc_micros,
metrics.clicks, 
metrics.impressions,
metrics.engagements, 
metrics.interactions,
metrics.cost_micros,
ad_group_criterion.effective_cpc_bid_micros, 
metrics.average_cost,
metrics.average_cpc,
metrics.search_exact_match_impression_share, 
metrics.search_impression_share,
metrics.search_rank_lost_impression_share, 
metrics.conversions,
metrics.all_conversions,
metrics.view_through_conversions, 
campaign.name 
FROM keyword_view 
WHERE ad_group_criterion.status IN ('PAUSED','ENABLED') 
AND ad_group_criterion.negative = false 
AND campaign.status IN ('ENABLED','PAUSED')
AND segments.date DURING YESTERDAY";

$googleAdsClient = (new GoogleAdsClientBuilder())
    ->fromFile()
    ->withOAuth2Credential((new OAuth2TokenBuilder())->fromFile()->build())
    ->build();

$stream = $googleAdsClient->getGoogleAdsServiceClient()->search($customerId, $query);

foreach ($stream->iterateAllElements() as $googleAdsRow) {
    printf(
        "Campaign with ID %d and name '%s' was found.%s",
        $googleAdsRow->getCampaign()->getId(),
        $googleAdsRow->getCampaign()->getName(),
        PHP_EOL
    );
}

[adrianlaraperez88]

@PierrickVoulet yeah I have the same code but maybe a difference is the Date range and # of keywords, a little accounts this code working perf but big accounts send me an error

[PierrickVoulet]

@PierrickVoulet yeah I have the same code but maybe a difference is the Date range and # of keywords, a little accounts this code working perf but big accounts send me an error

I forgot to mention that I tested using the same customer and date range as you did with the request 5q7kthTHRdL1Z5Msp4f_Tg but did not get any errors. Could you run it again with no $settings and provide some request IDs that return the RPC_DEADLINE_TOO_SHORT error?

[adrianlaraperez88]

Hi @PierrickVoulet, please check this request-id "request-id": "qJB0TzWw2VRULPCj2mF8wQ",

[PierrickVoulet]

Hi @PierrickVoulet, please check this request-id "request-id": "qJB0TzWw2VRULPCj2mF8wQ",

Thank you for the additional example. I ran the exact same request for the exact same customer as you did:

$query = "SELECT ad_group_criterion.criterion_id, ad_group_criterion.keyword.text, ad_group.name, ad_group_criterion.status,  ad_group_criterion.keyword.match_type,  campaign.id,  segments.date, ad_group_criterion.quality_info.quality_score,  ad_group_criterion.quality_info.post_click_quality_score,  ad_group_criterion.quality_info.creative_quality_score, ad_group_criterion.position_estimates.top_of_page_cpc_micros,  ad_group_criterion.position_estimates.first_page_cpc_micros,  ad_group_criterion.position_estimates.first_position_cpc_micros, metrics.clicks,  metrics.impressions, metrics.engagements,  metrics.interactions, metrics.cost_micros, ad_group_criterion.effective_cpc_bid_micros,  metrics.average_cost, metrics.average_cpc, metrics.search_exact_match_impression_share,  metrics.search_impression_share, metrics.search_rank_lost_impression_share,  metrics.conversions, metrics.all_conversions, metrics.view_through_conversions,  campaign.name  FROM keyword_view  WHERE ad_group_criterion.status IN ('PAUSED','ENABLED')  AND ad_group_criterion.negative = false  AND campaign.status IN ('ENABLED','PAUSED') AND segments.date DURING THIS_MONTH"

1. I am getting the RPC_DEADLINE_TOO_SHORT when I set the client timeout to 1 second:

$response = $googleAdsServiceClient->search($customerId, $query, [
    'retrySettings' => [
        'initialRpcTimeoutMillis' => 1000,
        'maxRpcTimeoutMillis' => 1000
    ]
]);

2. But I am getting a successful response if I remove the optional settings:

$response = $googleAdsServiceClient->search($customerId, $query)

There is likely something wrong on your environment. Could you try running it again without the optional settings from a different environment?

[fiboknacky]

Closing due to inactivity.