status-crawler

status-crawler copied to clipboard

User Agent and Proxies

5

Hi there. I tested your spider and its working good. Two suggestion are: - UserAgents options, so you can change to whatever you want - Proxies use, so you can...

shtefcs

NPM package?

5

Any interest in publishing this as a CLI tool on NPM? I have have a Travis CI build running on a static site that checks for various errors, and I'd...

nybblr

--required-values spiders ALL

1

casperjs --start-url=http://www.proxymis.com --required-values=proxymis.com spider.js does spider links that does not contain the url like: 200 http://www.google-analytics.com/ga.js 200 http://fonts.gstatic.com/s/economica/v4/UK4l2VEpwjv3gdcwbwXE9InF5uFdDttMLvmWuJdhhgs.ttf 200 http://fonts.gstatic.com/s/economica/v4/jObgDQiPUtmACAaaK3pMG6CWcynf_cDxXwCLxiixG1c.ttf 200 http://fonts.gstatic.com/s/lato/v11/v0SdcGFAl2aezM9Vq_aFTQ.ttf 200 http://fonts.gstatic.com/s/lato/v11/nj47mAZe0mYUIySgfn0wpQ.ttf 200 http://connect.facebook.net/fr_FR/all.js#xfbml=1 Shouldn't it ONLY spider...

yarekc

Both command line params have no effect ... these settings are only read from config.json Bests

wirtsi