crawl4ai
crawl4ai copied to clipboard
Feature request: Opt out mechanisms
Please make it easy for your users to implement the following opt out mechanisms:
- robots.txt
-
deviantarts
noaitag - the w3m TDM protocol (see #124)
With github actively recommending you and 21K+ stars you have a certain kind of responsibility. This is a matter of respect towards the people whose data your users rely on.
One line mentions in the docs I needed grep to find aren't enough.