PropertyGuru-Scraper PropertyGuru scraping for another country

hey @DicksonC96 super nice work! Kinda new to this and trying to scrape data for singapore and would like to get some guidance. i assume the code structure would work exactly the same on the singapore site?

Jan 01 '23 12:01 tljoven

Thanks @tljoven. I suppose if they are using the same html structure for the SG's. Though you might need to change the API/URL parameters for your region. Hope it helps!

Jan 01 '23 13:01 DicksonC96

Hi @DicksonC96, how did you manage to bypass the Cloudflare captcha? I tried using your functions but it doesn't work. @tljoven did you manage to successfully scrape data from the Singapore website?

Mar 15 '23 03:03 benjamin-mak

@benjamin-mak yep

Mar 15 '23 06:03 tljoven

@tljoven what did you do to bypass cloudflare?

Mar 15 '23 06:03 benjamin-mak

I didnt have to O.o @benjamin-mak

Mar 15 '23 08:03 tljoven

Hi @DicksonC96, how did you manage to bypass the Cloudflare captcha? I tried using your functions but it doesn't work. @tljoven did you manage to successfully scrape data from the Singapore website?

I can't as well. Cloudscrapper couldn't bypass Level 2 Cloudflare captcha. Suspected that they upgraded their plan since 1st January 2023.

@tljoven mind to share which module u using? Still cloudscrapper?

Mar 15 '23 10:03 DicksonC96

wait why is there captcha? you log in to scrape?

Mar 16 '23 12:03 tljoven

Does it mean sg data is not available?

May 07 '23 09:05 Arvedek

Hi May I check if propertyguru.sg can still be scraped?

May 15 '23 13:05 Arvedek

@Arvedek I could not get cloudscrapper to work, so I used a third-party web scraper to assist me. However, it also does not fully work, I managed to get about 70-80% of the data only

May 15 '23 13:05 benjamin-mak

If you want, I can share with you the code and you can try it for yourself.

May 15 '23 13:05 benjamin-mak

If you want, I can share with you the code and you can try it for yourself.

That would be nice! Thanks

May 15 '23 14:05 Arvedek

Go to my profile and follow me on Linkedin, I'll share with you there

May 16 '23 06:05 benjamin-mak

Done> Go to my profile and follow me on Linkedin, I'll share with you there

May 16 '23 12:05 Arvedek

I'm also scraping for SG. The cloudscraper did work, but I got 403 frequetly after sending certain amount of requests. I wonder that is this the only problem existing in SG PropertyGuru? If it's not the only issue in SG version, may I know how did you manage to deal with it? edit: I figured out that the 403 reponses mean the cloudscraper didn't work. The 403 page would lately execute self invoking JavaScript and bring user to the anti-bot page. It seems there are almost no good off-line(not using any scraping online API) approaches to bypass cloudflare anti-bot now(I tried undetected_chromdriver). I have an alternaltive, is to pass anti-bot by human then copy the cloudflare token. Using the same IP the same User-Agent, it should provide about 30 ~ 60 mins to scrape semi-automatically.

Oct 30 '23 03:10 SungLuRent