google-search-results-ruby JSON::ParserError raised on Cloudflare 502 Bad Gateway

Hey there!

This isn't a super critical issue, but today we hit a few exceptions like the following when calling SerpApi with the gem and getting a 502 repsonse from Cloudflare:

JSON::ParserError: 767: unexpected token at '<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>cloudflare</center>
</body>
</html>
'

Here's the relevant part of the backtrace:

/app/vendor/ruby-2.6.10/lib/ruby/2.6.0/json/common.rb:156:in `parse'
/app/vendor/ruby-2.6.10/lib/ruby/2.6.0/json/common.rb:335:in `load'
/app/vendor/bundle/ruby/2.6.0/gems/google_search_results-2.2.0/lib/search/serp_api_search.rb:141:in `rescue in get_results'
/app/vendor/bundle/ruby/2.6.0/gems/google_search_results-2.2.0/lib/search/serp_api_search.rb:136:in `get_results'
/app/vendor/bundle/ruby/2.6.0/gems/google_search_results-2.2.0/lib/search/serp_api_search.rb:57:in `get_html'

The Cloudflare response is HTML, but the gem assumes the response will be JSON and tries to parse it:

if error = JSON.load(e.io.read)["error"]
  puts "server returns error for url: #{url}"
  raise error
else
  puts "fail: fetch url: #{url}"
  raise e
end

Aug 28 '23 18:08 jeremysmithco

@jeremysmithco Sorry for the delay in replying. I think this got buried.

Was this a one-time issue or recurring?

It it was recurring, can you send us details - full code and your IP address to look things up - to [email protected] ? Thank you!

Mar 03 '25 20:03 hartator

@hartator Hey, thanks for the reply! It's not frequent, but we still get them occasionally. Just checked our exception tracker and we've had a few in the past 2 months. Sometimes we get the Cloudflare bad gateway page (shared above) and sometimes we get a 500 html response that appears to be from the app itself:

JSON::ParserError: unexpected token

JSON::ParserError: unexpected token at '<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta name="description" content="">
    <meta name="author" content="">
    <meta name="apple-mobile-web-app-title" content="SerpApi">
    <meta name="application-name" content="SerpApi">
    <meta name="theme-color" content="#ffffff">

    <title>500 - Internal Server Error</title>

    <link rel="apple-touch-icon" sizes="180x180" href="/apple-touch-icon.png">
    <link rel="icon" type="image/png" sizes="32x32" href="/favicon-32x32.png">
    <link rel="icon" type="image/png" sizes="16x16" href="/favicon-16x16.png">
    <link rel="manifest" href="/manifest.json">
    <link rel="mask-icon" href="/safari-pinned-tab.svg" color="#5bbad5">

    <link rel="stylesheet" href="/maisonnette/dist/blue/assets/css/app.css" type="text/css"/>
    <link rel="stylesheet" href="/third_party/fontawesome/css/all.min.css?version=5.15.4">
    <link rel="stylesheet" href="/demo-redesign/css/error_v1.1.css">
  </head>
  <body>
    <a class="logo" href="/" title="Home"><img src="/img/SerpApi-gentle-white.png" alt="" /></a>
    <div class="wrap-error wrap-error-500">
      <div class="info">
        <h1><span>500</span> - Internal Server Error</h1>
        <p>Something went wrong on our end.</p>
        <div class="links">
          <p>While we're fixing it, try one of these links:</p>
          <ul>
            <li><i class="fas fa-home"></i><a href="/">Homepage</a></li>
            <li><i class="fab fa-google"></i><a href="/search-api">Google Search API</a></li>
            <li><img src="/demo-redesign/icons/static/bing.svg" alt=""><a href="/bing-search-api">Bing Search API</a></li>
            <li><i class="fas fa-code"></i><a href="/playground">Playground</a></li>
            <li class="support">You can also contact support:</li>
            <li><i class="fas fa-envelope"></i><a href="mailto:[email protected]">[email protected]</a></li>
          </ul>
        </div>
      </div>
      <div class="img">
        <img src="/demo-redesign/images/cat-500.png" alt="" />
      </div>
    </div>
  </body>
</html>
'

I'll email the details, thanks!

Mar 03 '25 21:03 jeremysmithco

Thanks for emailing us the details @jeremysmithco! And thank you also for your patience while we look into this.

Front

Mar 03 '25 22:03 schaferyan

@schaferyan Hey, sure thing, thanks! :)

Mar 03 '25 22:03 jeremysmithco

This issue is happening sporadically. I suggest to improve error handling in https://github.com/serpapi/google-search-results-ruby/blob/master/lib/search/serp_api_search.rb#L59

Status code to be checked
Additionally, if we expect a JSON object, we can make simple test of first character of string matching {. Then if not matches, raise SerpApiException.

Apr 29 '25 14:04 romanm-anyword

Hi @romanm-anyword, thank you very much for the suggestion! Our engineers will look into it and take your feedback into consideration while working on this.

Apr 30 '25 08:04 marm123