apify
Publish crawlee package to conda
It seems that conda does not have a convenient way of specifying optional dependencies - for example, crawlee[cli], ... As far as I saw, the somewhat used practice is to maintain separate packages. In this case, it would be crawlee-cli, crawlee-all, crawlee-playwright ...
It is not worth the maintenance from my point of view. I would suggest publishing crawlee[all] as crawlee to conda-forge and only add smaller optional packages if there is huge demand for it. I believe that currently, the conda users can still use smaller crawlee configurations through pip.
PR: About adding to conda-forge https://github.com/conda-forge/staged-recipes/pull/30504
Due to curl-cffi being available only on Linux in conda-forge , I made it Linux conditional to be able to build on all platforms. Users on other platforms who will want to use curl_impersonate will have to install curl-cffi on their own in some way. Since we have other clients, I do not consider this a major issue.
So on conda forge, the Crawlee package will be:
- crawlee[all] for Linux
- crawlee[all - curl-cffi] for MacOs and Windows
Also, I had to add conda_build_config to avoid Python 3.9, but we can later drop it once even conda-forge stops supporting it by default later this year.
Also to make it build on Windows I had to add one more conditional import: jinxed # [win]