scrapydo

scrapydo copied to clipboard

I use code: ``` try: result = scrapydo.run_spider(MySpider, **kwargs) except Exception: result = None ``` ERROR:scrapy.core.scraper:Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/scrapy/utils/defer.py", line 102, in...

wsgggws

Provide better error handling

1

There are several cases that cause to fail silently and return nothing. Ideally, we should be able to propagate somehow the exceptions.

rmax

rmax

When using `scrapydo.fetch`, cookies are not persisted between calls. This happens because each call to `fetch` creates a crawler and an engine which instantiate all the related extensions and middlewares,...

rmax