@lizpixels Hmm, how would that work (can it?) in MB? Can we? @manton do you mind if I toss this in your direction? :)

@jtr robots.txt in a Micro.blog context is sparsely documented in Search engine indexing. An OpenAI-specific discussion with an example is over at the Help Center. I’m thinking about writing a plug-in to make this a little easier…

@jtr @sod Looks like @jdm made a plug-in that adds the ChatGPT bot to robots.txt. It’s in the plug-in directory called “No Index+”. It blocks all bots, though, so we probably should have a plug-in that only blocks GPTBot too.

@sod @manton of course it’s under my nose…
So my robot.txt currently has User-agent: * Disallow: / which means all crawlers are disallowed (I didn’t touch this file below, I thought these are on by default?). So from this, it’s pretty easy to add what @lizpixels suggested above, which will only disallow GPTBot.

I’m comfortable with messing with my theme to that degree, I can imagine not all folks here do though.

@sod that would be great if you did!

@ericmwalk @jtr I did! 😊 Custom Robots makes it a copy-n-paste job.

@manton oh that’s a good point, it’s easy enough to fork and make a specific one. If no one else has done it, I’ll make one exclusively for cgpt.

@manton OpenAI lists two bots in their documentation one for their crawler and another for GPT user plugins. Should the explicit version block both or just the cgpt crawler?