JayBizzle
AI bots
I wonder whether this lib could/would add support to detect AI bots, so crawlers which are used to feed AI engines. I found a repo which lists most of them: https://github.com/ai-robots-txt/ai.robots.txt
maybe thats something which could be used for/with crawler-detect?
Hmmm, interesting suggestion.
Would you see our current isCrawler() method returning true for these AI bots, with a new additional method called something like isAiBot()?
Open to implementation suggestions here 👍
I think a separate method would be good, as I think some users might be interested in 'whatever' definition this lib had of a crawler before.
AI bots are a different class which might need different handling IMO
A crawler is a crawler or not? Regardless of what the crawler actually does with the responses. If users want to differentiate then they should look at https://github.com/VolkswAIgen/VolkswAIgen - but for the purpose of this library I would think that a bot from OpenAI or Claude is detected as crawler.
Agreed @beberlei -- I don't see a use case where it'd be necessary to treat an "AI" bot differently. Certainly if that were the case, I'd imagine that is beyond the scope of this library.
Yes. A bot is a bot. Proper pronouns for bots is next generations problem, not ours.
