French characters "à" are not converted correctly

[Jeremytijal]

Hi Sophie,

I noticed that some non-English (French) content end up not getting converted correctly. As a random example, try scraping this page: https://ensemblesurleterrain.bouyguestelecom.fr the result https://test-webscrapper.netlify.app

You'll see that the page is saved with a bunch of � characters in place of the "à"

This problem existed before the last PR. I thought it would be solved, but it was not.

[s0ph1e]

Hi @Jeremytijal 👋

Thank you for reporting an issue. Could you please check if that happens with the latest version 5.2.0? I'll be able to test it by myself later today or tomorrow

[Jeremytijal]

Hi @s0ph1e,

I tested it with the latest version 5.2.0 and I have the same problem.

[s0ph1e]

Yep, now I see it. The fix works when html response contains a charset in the content-type header. In this case there is no charset in the header and it saves it in binary as it was in previous versions.

Looks like it makes sense to use utf8 for html files if there was no charset. @phawxby WDYT (as the author of the latest fix)?

[phawxby]

I think it's risky to assume a charset unless we know for sure what it is. What if it's ANSI, ASCII, etc, we could break existing users. However tha then brings about a new issue, there's multiple ways to specify the charset of a file

1. On the response header, like we use now.
2. On a meta-tag for html
3. As a rule in CSS
4. And although deprecated, as an attribute.

I think we have 2 options.

1. Default to 'utf-8' for all text types, which could potentially break things for existing users, especially for those scraping older applications.
2. Add some basic additional rules which we switch to based on the content type. I think 2 & 3 above are going to cover 99% of use cases. I think this is the best approach personally.

The best place to do this is is here.

1. If the encoding is still binary after checking the headers then get the mime type of the response.
2. Add a switch based on mimetype for css/html.
3. Add 2 new functions, getMimeFromHtml and getMimeFromCss. a. HTML: Use cheerio to parse the response body and see if you find a <meta charset="utf-8"> b. CSS: The CSS spec is incredibly strict, you should just be able to do .includes('@charset "UTF-8"');

I would take a stab at it but I go on vacation in a few days and i'm intentionally not taking a laptop

[s0ph1e]

I like the idea of getting the encoding from html or css content, it sounds better than using utf8 as default and should not be difficult to do.

Most probably I will not have time to implement it in the near future so I really appreciate any help with it.

Thank you for your input and enjoy your vacation without laptop @phawxby 🏖️🌴