There are a lot of interesting stats in this Cloudflare blog post about AI bots. I still worry that we might over-correct when blocking LLM training. For example, CCBot is mentioned, but Common Crawl goes back 15+ years and has a variety of non-AI uses.
@manton excuse my ignorance but what would be the downside of blocking ‘good’ bots?
Presumably search rankings but if you’re not fussed about that what else could happen?
@gregmorris Depends on the bot. Could be no direct impact, but I can imagine if a tool doesn't work, then a user might wonder why it's broken with just certain blogs.
