CrawlerX icon indicating copy to clipboard operation
CrawlerX copied to clipboard
verified publisher icon leopardslab

Feature: Add some new crawlers for popular web pages

Open sajithaliyanage opened this issue 5 years ago • 1 comments

Add crawl spiders for the following or popular websites.

  • Youtube
  • Quora
  • Facebook
  • Reddit
  • GitHub

Currently implemented spiders can be found in - https://github.com/leopardslab/CrawlerX/tree/master/scrapy_app/scrapy_app/spiders

sajithaliyanage avatar Oct 05 '20 17:10 sajithaliyanage

@sajithaliyanage Is this issue still opened? I like to work on this

ffalpha avatar Feb 16 '21 11:02 ffalpha