icrawler icon indicating copy to clipboard operation
icrawler copied to clipboard
verified publisher icon hellock

GoogleImageCrawler returns no results (TypeError: 'NoneType' object is not iterable)

Open syedowaisalishah opened this issue 5 months ago • 5 comments

Hi,

I’m trying to use GoogleImageCrawler with the following script:

from icrawler.builtin import GoogleImageCrawler
from icrawler import ImageDownloader
import os

# Custom image downloader without format filtering
class CustomDownloader(ImageDownloader):
    def _filter(self, task):
        return super()._filter(task)  # No extension/type filtering now

def download_images(plant_name, category, num_images=300):
    save_dir = os.path.join('dataset', category.replace(' ', '_'), plant_name.replace(' ', '_'))
    os.makedirs(save_dir, exist_ok=True)

    crawler = GoogleImageCrawler(
        downloader_cls=CustomDownloader,
        storage={'root_dir': save_dir}
    )

    crawler.crawl(
        keyword=f"{plant_name} plant images",
        max_num=num_images,
        min_size=(512, 512),
        file_idx_offset=0
    )

if __name__ == '__main__':
    plant_name = input("Enter plant name (e.g., 'Loropetalum 'Plum''): ").strip()
    category = input("Enter category (e.g., 'Small or Mass Planting Shrub'): ").strip()

    download_images(plant_name, category)
    print("✅ Done! Check your dataset folder.")

Error / Logs:

2025-08-21 14:47:15,712 - INFO - parser - parsing result page https://www.google.com/search?q=coconut+palm+plant+images&ijn=0&start=0&tbs=&tbm=isch
Exception in thread parser-001:
Traceback (most recent call last):
  ...
  File ".../icrawler/parser.py", line 93, in worker_exec
    for task in self.parse(response, **kwargs):
TypeError: 'NoneType' object is not iterable

Environment:

  • OS: Ubuntu 20.04
  • Python: 3.10
  • icrawler version: (please confirm with pip show icrawler)

Notes:

  • Baidu and Bing crawlers work fine.
  • Only GoogleImageCrawler fails with the above error.
  • Looks like Google changed its HTML/response format.

Is there any fix or workaround planned for Google support? Thanks!

syedowaisalishah avatar Aug 21 '25 13:08 syedowaisalishah

Hello, just recommend my repo: https://github.com/CharlesPikachu/imagedl which contains more reasonable search results extraction and help download google images with many different strategies.

CharlesPikachu avatar Oct 22 '25 12:10 CharlesPikachu

i was able to fix it by changing line 136 of icrawler/builtin/google.py ( https://github.com/hellock/icrawler/blob/master/icrawler/builtin/google.py#L136 ) to params = dict(q=keyword, ijn=int(i / 100), start=i, tbs=filter_str, udm=2) as well as changing line 86 of icrawler/parser.py ( https://github.com/hellock/icrawler/blob/master/icrawler/parser.py#L86 ) to response = self.session.get(url, timeout=req_timeout, headers={"user-agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36"})

meow6969 avatar Nov 23 '25 10:11 meow6969

i was able to fix it by changing line 136 of icrawler/builtin/google.py ( https://github.com/hellock/icrawler/blob/master/icrawler/builtin/google.py#L136 ) to params = dict(q=keyword, ijn=int(i / 100), start=i, tbs=filter_str, udm=2) as well as changing line 86 of icrawler/parser.py ( https://github.com/hellock/icrawler/blob/master/icrawler/parser.py#L86 ) to response = self.session.get(url, timeout=req_timeout, headers={"user-agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36"})

No, it s not working :( same error

Nov 23 11:36:41 photo001 gunicorn[445850]: 2025-11-23 11:36:41,645 - INFO - parser - parsing result page https://www.google.com/search?q=Reebok+RBK-009-CCC-05+Granatowy+NOSIZE&ijn=0&start=0&tbs=&udm=2
Nov 23 11:36:41 photo001 gunicorn[445850]: Exception in thread parser-001:
Nov 23 11:36:41 photo001 gunicorn[445850]: Traceback (most recent call last):
Nov 23 11:36:41 photo001 gunicorn[445850]:   File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
Nov 23 11:36:41 photo001 gunicorn[445850]:     self.run()
Nov 23 11:36:41 photo001 gunicorn[445850]:   File "/usr/lib/python3.12/threading.py", line 1010, in run
Nov 23 11:36:41 photo001 gunicorn[445850]:     self._target(*self._args, **self._kwargs)
Nov 23 11:36:41 photo001 gunicorn[445850]:   File "/home/azart_parser/azart_venv/lib/python3.12/site-packages/icrawler/parser.py", line 94, in worker_exec
Nov 23 11:36:41 photo001 gunicorn[445850]:     for task in self.parse(response, **kwargs):
Nov 23 11:36:41 photo001 gunicorn[445850]: TypeError: 'NoneType' object is not iterable
Nov 23 11:36:44 photo001 gunicorn[445852]: 2025-11-23 11:36:44,603 - INFO - icrawler.crawler - start crawling...
Nov 23 11:36:44 photo001 gunicorn[445852]: 2025-11-23 11:36:44,603 - INFO - icrawler.crawler - starting 1 feeder threads...
Nov 23 11:36:44 photo001 gunicorn[445852]: 2025-11-23 11:36:44,604 - INFO - feeder - thread feeder-001 exit
Nov 23 11:36:44 photo001 gunicorn[445852]: 2025-11-23 11:36:44,604 - INFO - icrawler.crawler - starting 1 parser threads...
Nov 23 11:36:44 photo001 gunicorn[445852]: 2025-11-23 11:36:44,604 - INFO - icrawler.crawler - starting 1 downloader threads...
Nov 23 11:36:45 photo001 gunicorn[445852]: 2025-11-23 11:36:45,015 - INFO - parser - parsing result page https://www.google.com/search?q=Reebok+RBK-009-CCC-05+Granatowy+NOSIZE&ijn=0&start=0&tbs=&udm=2
Nov 23 11:36:45 photo001 gunicorn[445852]: Exception in thread parser-001:
Nov 23 11:36:45 photo001 gunicorn[445852]: Traceback (most recent call last):
Nov 23 11:36:45 photo001 gunicorn[445852]:   File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
Nov 23 11:36:45 photo001 gunicorn[445852]:     self.run()
Nov 23 11:36:45 photo001 gunicorn[445852]:   File "/usr/lib/python3.12/threading.py", line 1010, in run
Nov 23 11:36:45 photo001 gunicorn[445852]:     self._target(*self._args, **self._kwargs)
Nov 23 11:36:45 photo001 gunicorn[445852]:   File "/home/azart_parser/azart_venv/lib/python3.12/site-packages/icrawler/parser.py", line 94, in worker_exec
Nov 23 11:36:45 photo001 gunicorn[445852]:     for task in self.parse(response, **kwargs):
Nov 23 11:36:45 photo001 gunicorn[445852]: TypeError: 'NoneType' object is not iterable
Nov 23 11:36:46 photo001 gunicorn[445850]: 2025-11-23 11:36:46,051 - INFO - downloader - no more download task for thread downloader-001
Nov 23 11:36:46 photo001 gunicorn[445850]: 2025-11-23 11:36:46,051 - INFO - downloader - thread downloader-001 exit
Nov 23 11:36:46 photo001 gunicorn[445850]: 2025-11-23 11:36:46,052 - INFO - icrawler.crawler - Crawling task done!
Nov 23 11:36:46 photo001 gunicorn[445850]: Processing article Gino Rossi OJ-82713 Beżowy ciemny NOSIZE, Q value:
Nov 23 11:36:46 photo001 gunicorn[445850]: Waiting 7.34 seconds before crawling...
Nov 23 11:36:46 photo001 gunicorn[445849]: 2025-11-23 11:36:46,461 - INFO - icrawler.crawler - start crawling...
Nov 23 11:36:46 photo001 gunicorn[445849]: 2025-11-23 11:36:46,461 - INFO - icrawler.crawler - starting 1 feeder threads...
Nov 23 11:36:46 photo001 gunicorn[445849]: 2025-11-23 11:36:46,462 - INFO - feeder - thread feeder-001 exit
Nov 23 11:36:46 photo001 gunicorn[445849]: 2025-11-23 11:36:46,462 - INFO - icrawler.crawler - starting 1 parser threads...
Nov 23 11:36:46 photo001 gunicorn[445849]: 2025-11-23 11:36:46,462 - INFO - icrawler.crawler - starting 1 downloader threads...
Nov 23 11:36:46 photo001 gunicorn[445849]: 2025-11-23 11:36:46,891 - INFO - parser - parsing result page https://www.google.com/search?q=Reebok+RBK-009-CCC-05+Granatowy+NOSIZE&ijn=0&start=0&tbs=&udm=2
Nov 23 11:36:46 photo001 gunicorn[445849]: Exception in thread parser-001:
Nov 23 11:36:46 photo001 gunicorn[445849]: Traceback (most recent call last):
Nov 23 11:36:46 photo001 gunicorn[445849]:   File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
Nov 23 11:36:46 photo001 gunicorn[445849]:     self.run()
Nov 23 11:36:46 photo001 gunicorn[445849]:   File "/usr/lib/python3.12/threading.py", line 1010, in run
Nov 23 11:36:46 photo001 gunicorn[445849]:     self._target(*self._args, **self._kwargs)
Nov 23 11:36:46 photo001 gunicorn[445849]:   File "/home/azart_parser/azart_venv/lib/python3.12/site-packages/icrawler/parser.py", line 94, in worker_exec
Nov 23 11:36:46 photo001 gunicorn[445849]:     for task in self.parse(response, **kwargs):
Nov 23 11:36:46 photo001 gunicorn[445849]: TypeError: 'NoneType' object is not iterable

ExsyDev avatar Nov 23 '25 11:11 ExsyDev

i was able to fix it by changing line 136 of icrawler/builtin/google.py ( https://github.com/hellock/icrawler/blob/master/icrawler/builtin/google.py#L136 ) to params = dict(q=keyword, ijn=int(i / 100), start=i, tbs=filter_str, udm=2) as well as changing line 86 of icrawler/parser.py ( https://github.com/hellock/icrawler/blob/master/icrawler/parser.py#L86 ) to response = self.session.get(url, timeout=req_timeout, headers={"user-agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36"})

No, it s not working :( same error

Nov 23 11:36:41 photo001 gunicorn[445850]: 2025-11-23 11:36:41,645 - INFO - parser - parsing result page https://www.google.com/search?q=Reebok+RBK-009-CCC-05+Granatowy+NOSIZE&ijn=0&start=0&tbs=&udm=2
Nov 23 11:36:41 photo001 gunicorn[445850]: Exception in thread parser-001:
Nov 23 11:36:41 photo001 gunicorn[445850]: Traceback (most recent call last):
Nov 23 11:36:41 photo001 gunicorn[445850]:   File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
Nov 23 11:36:41 photo001 gunicorn[445850]:     self.run()
Nov 23 11:36:41 photo001 gunicorn[445850]:   File "/usr/lib/python3.12/threading.py", line 1010, in run
Nov 23 11:36:41 photo001 gunicorn[445850]:     self._target(*self._args, **self._kwargs)
Nov 23 11:36:41 photo001 gunicorn[445850]:   File "/home/azart_parser/azart_venv/lib/python3.12/site-packages/icrawler/parser.py", line 94, in worker_exec
Nov 23 11:36:41 photo001 gunicorn[445850]:     for task in self.parse(response, **kwargs):
Nov 23 11:36:41 photo001 gunicorn[445850]: TypeError: 'NoneType' object is not iterable
Nov 23 11:36:44 photo001 gunicorn[445852]: 2025-11-23 11:36:44,603 - INFO - icrawler.crawler - start crawling...
Nov 23 11:36:44 photo001 gunicorn[445852]: 2025-11-23 11:36:44,603 - INFO - icrawler.crawler - starting 1 feeder threads...
Nov 23 11:36:44 photo001 gunicorn[445852]: 2025-11-23 11:36:44,604 - INFO - feeder - thread feeder-001 exit
Nov 23 11:36:44 photo001 gunicorn[445852]: 2025-11-23 11:36:44,604 - INFO - icrawler.crawler - starting 1 parser threads...
Nov 23 11:36:44 photo001 gunicorn[445852]: 2025-11-23 11:36:44,604 - INFO - icrawler.crawler - starting 1 downloader threads...
Nov 23 11:36:45 photo001 gunicorn[445852]: 2025-11-23 11:36:45,015 - INFO - parser - parsing result page https://www.google.com/search?q=Reebok+RBK-009-CCC-05+Granatowy+NOSIZE&ijn=0&start=0&tbs=&udm=2
Nov 23 11:36:45 photo001 gunicorn[445852]: Exception in thread parser-001:
Nov 23 11:36:45 photo001 gunicorn[445852]: Traceback (most recent call last):
Nov 23 11:36:45 photo001 gunicorn[445852]:   File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
Nov 23 11:36:45 photo001 gunicorn[445852]:     self.run()
Nov 23 11:36:45 photo001 gunicorn[445852]:   File "/usr/lib/python3.12/threading.py", line 1010, in run
Nov 23 11:36:45 photo001 gunicorn[445852]:     self._target(*self._args, **self._kwargs)
Nov 23 11:36:45 photo001 gunicorn[445852]:   File "/home/azart_parser/azart_venv/lib/python3.12/site-packages/icrawler/parser.py", line 94, in worker_exec
Nov 23 11:36:45 photo001 gunicorn[445852]:     for task in self.parse(response, **kwargs):
Nov 23 11:36:45 photo001 gunicorn[445852]: TypeError: 'NoneType' object is not iterable
Nov 23 11:36:46 photo001 gunicorn[445850]: 2025-11-23 11:36:46,051 - INFO - downloader - no more download task for thread downloader-001
Nov 23 11:36:46 photo001 gunicorn[445850]: 2025-11-23 11:36:46,051 - INFO - downloader - thread downloader-001 exit
Nov 23 11:36:46 photo001 gunicorn[445850]: 2025-11-23 11:36:46,052 - INFO - icrawler.crawler - Crawling task done!
Nov 23 11:36:46 photo001 gunicorn[445850]: Processing article Gino Rossi OJ-82713 Beżowy ciemny NOSIZE, Q value:
Nov 23 11:36:46 photo001 gunicorn[445850]: Waiting 7.34 seconds before crawling...
Nov 23 11:36:46 photo001 gunicorn[445849]: 2025-11-23 11:36:46,461 - INFO - icrawler.crawler - start crawling...
Nov 23 11:36:46 photo001 gunicorn[445849]: 2025-11-23 11:36:46,461 - INFO - icrawler.crawler - starting 1 feeder threads...
Nov 23 11:36:46 photo001 gunicorn[445849]: 2025-11-23 11:36:46,462 - INFO - feeder - thread feeder-001 exit
Nov 23 11:36:46 photo001 gunicorn[445849]: 2025-11-23 11:36:46,462 - INFO - icrawler.crawler - starting 1 parser threads...
Nov 23 11:36:46 photo001 gunicorn[445849]: 2025-11-23 11:36:46,462 - INFO - icrawler.crawler - starting 1 downloader threads...
Nov 23 11:36:46 photo001 gunicorn[445849]: 2025-11-23 11:36:46,891 - INFO - parser - parsing result page https://www.google.com/search?q=Reebok+RBK-009-CCC-05+Granatowy+NOSIZE&ijn=0&start=0&tbs=&udm=2
Nov 23 11:36:46 photo001 gunicorn[445849]: Exception in thread parser-001:
Nov 23 11:36:46 photo001 gunicorn[445849]: Traceback (most recent call last):
Nov 23 11:36:46 photo001 gunicorn[445849]:   File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
Nov 23 11:36:46 photo001 gunicorn[445849]:     self.run()
Nov 23 11:36:46 photo001 gunicorn[445849]:   File "/usr/lib/python3.12/threading.py", line 1010, in run
Nov 23 11:36:46 photo001 gunicorn[445849]:     self._target(*self._args, **self._kwargs)
Nov 23 11:36:46 photo001 gunicorn[445849]:   File "/home/azart_parser/azart_venv/lib/python3.12/site-packages/icrawler/parser.py", line 94, in worker_exec
Nov 23 11:36:46 photo001 gunicorn[445849]:     for task in self.parse(response, **kwargs):
Nov 23 11:36:46 photo001 gunicorn[445849]: TypeError: 'NoneType' object is not iterable

Could you please confirm which version of iCrawler are you using?

ZhiyuanChen avatar Dec 10 '25 02:12 ZhiyuanChen

Environment:

  • OS: Ubuntu 20.04
  • Python: 3.10
  • icrawler version: (please confirm with pip show icrawler)

Notes:

  • Baidu and Bing crawlers work fine.
  • Only GoogleImageCrawler fails with the above error.
  • Looks like Google changed its HTML/response format.

Is there any fix or workaround planned for Google support? Thanks!

Please confirm the iCrawler version.

ZhiyuanChen avatar Dec 10 '25 02:12 ZhiyuanChen