wtfloris
Allow filtering on WoningNet "loting" listings
It would be useful if we could filter the results of WoningNet to only show those with
res["PublicatieModel"] == "Lotingmodel"
for those of us not blessed with the necessary amount of waitlist years.
And while we're at it, perhaps also allow filtering on social and private sector listings?
I'm not sure if the best way to go about this would be to add a new filter (which would only be effective on the WoningNet scrapers) or to split the WoningNet agencies into lottery and non-lottery.
I'm willing to create a PR (though I'm not sure how to test it myself)
To be fair, I think adding more filters is a good idea - and it's okay that they're applicable for a subset of the agencies. But this would require a small refactor of the parsing. I would add an interface which is aware of the various rental properties, i.e implementers could possibly add more properties such as size etc. These could then be filtered on.
I've been hesitant to add more global filters because this adds a parameter for all websites. You can see how this grows exponentially: 10 websites with 4 parameters (url, address, price and city) is only 40 parsed parameters in total. Add 2 parameters (e.g. amount of rooms and total area), and you now only need 3 extra websites to double the total amount of parsed parameters to 78.
As websites get modified and APIs change without notice, it's a matter of time before your total amount of parsed parameters becomes large enough for something to break on a weekly or even daily basis. If people can't count on Hestia to work well on their favourite websites then they have to keep checking them manually, at which point Hestia is useless. So it comes down to a reliability vs. feature set tradeoff, where reliability is much more important in my opinion.
Additionally, unless we can parse these new parameters for 100% of listings on all websites, people are going to be left wondering why some homes do have the total area and some don't. And how do you handle homes that don't have it? Do you send it to people that have filters set because otherwise they might miss something? Or do you not send it because you don't want to send something incorrect? And whichever one you choose, people will think the filtering is broken (and complain about it).
tldr: I think it adds too much complexity and maintenance for too little gain, but please convince me if you disagree
That being said (to get back on topic on this specific question): I think adding a filter for loterij/waitlist homes on WoningNet is a good addition. The WoningNet websites are all the same so in practice this only adds a single parsed parameter. I'll write something for this and test it in the dev env.
I agree that we should not add too many filters in the bot since that would make scraping more complicated while also being more of a maintenance burden. It's really not that hard to just open up the link and quickly glance at the property if it's suited for your needs or not. Then there is the complication of categorizing the many different types of models, which should not break at any time.
But yes, we should probably split out social housing from the free market since the former usually requires a registration at a housing corporation with long waiting lists, which not everyone has or has interest in
