robots.txt
I always knew it was there but never really looked into it. The standard relies on voluntary compliance. So it doesn’t _really_ do anything? Seems like the ones you would want to block would be the same group to ignore it? 🤔
@Burk I think that’s unfortunately true; the spec was written by optimists relying on the honor system. I always set it up because of the sitemap link feature it has, and as an explicit statement re who I want indexing me, as a matter of course.
@Burk Even blocking on user agent or IP addresses doesn’t work, since AI companies just lie about those. We need legislation!