Add ``GoogleOther`` to spider list

Open Fighter456 opened this issue 1 year ago • 0 comments

66.249.68.4 - - [02/Aug/2024:22:25:38 +0200] "GET REDACTED HTTP/1.1" 200 168162 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.6478.182 Mobile Safari/537.36 (compatible; GoogleOther)" 66.249.68.6 - - [02/Aug/2024:22:25:13 +0200] "GET REDACTED HTTP/1.1" 200 51797 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.6478.182 Mobile Safari/537.36 (compatible; GoogleOther)"

Please visit https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers?hl=en#googleother for more information.

GoogleOther is the generic crawler that may be used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development.

Aug 02 '24 20:08 Fighter456