People worry about AI bots overloading servers, but in my experience the dumb requests for potential security holes are much worse. Bots trying to get into non-existent files like htpasswd, wp-admin.php, etc. Servers getting hammered today.

@manton Gotta use agents to protect against other agents!

@manton Well, these requests usually result in a 404, instead of content pages that might need to be rendered first. Also, on my server, this is the fastest way to get fail2banned for a couple of days. 😆

@manton I see. Can you catch the bogus paths (like htpasswd) on server / reverse proxy level? I use Nginx as reverse proxy, and return a 444 on all those paths (which essentially just rudely slams the door shut). Too many of them invokes fail2ban.

@shred Yes, currently I catch a bunch of the bogus paths at the proxy level, but many slip through. I essentially have a long hard-coded list of suspicious paths. It’s a bit messy. (I wonder if anyone maintains a directory of bad paths…)