crawly

crawly copied to clipboard

Fixes syntax error on the example on the read me.

zemuldo

I've tried many tutorials and demos, but none of them have worked for me. Either it appears to run but I get no output file, or a puzzling `function Crawly.fetch/1...

dogweather

The tutorial uses the homebase site. I think the site layout has changed - for example, there's no body returned for this code in the tutorial: ``` iex(3)> response =...

james-s-w-clark

Generic `Pipeline crash` error

4

Hi, I'm getting a generic [pipeline crash](https://github.com/elixir-crawly/crawly/blame/60c5e1ab33915d4e6ec7652ca3b34793b5031077/lib/crawly/utils.ex#L96) error on the current release. It seems though that this [was solved](https://github.com/elixir-crawly/crawly/blame/master/lib/crawly/utils.ex#L107) on the `master` branch 3 months ago. Any plans to release...

thiagomajesk

[Feat] Support auto close spider when all requests finished

2

It seems that when all the workers’ requests list are empty, the crawler still cannot stop automatically. Although closespider_timeout can solve some scenarios, there is a new problem of ending...

EdmondFrank

Demo page not loading

1

It seems like the [demo page](http://crawlyui.com/) is not working correctly (The connection has timed out). Please see the screenshot below. 

braynm

Currently, any options set in the `Crawly.Request` will get overriden by the `RequestOptions` middleware. This PR fixes that to allow merging of both request options. ``` Crawly.Request.new( "http://example.com", headers, #...

suzdalnitski

Hello How could I develop a parametric spider and run it like Crawly.Engine.start_spider(MySpider, base_url, start_urls) Thanks in advance Matteo

matteoredaelli

https://github.com/elixir-crawly/crawly/issues/144

matteoredaelli

Remove URL fragment before storing in `Crawly.Middlewares.UniqueRequest`

3

Usually a fragment leads to the same page.

tanguilp