Say no to AI training, not to AI search: robots.txt per bot

"Block the AI bots" sounds like one switch, but it isn't. Anthropic uses three different crawlers, one for training, one for search and one for requests a person makes in Claude, and robots.txt can treat each one differently. That means you can keep your content out of training without disappearing from the answers. There's one rule in how robots.txt groups work that can quietly undo the whole setup, and the demo above opens right on it.

Three bots, three jobs

Anthropic's help article on crawling lists them with what happens when you disable each one:

ClaudeBot collects web content that could contribute to training. Restricting it "signals that the site's future materials should be excluded from our AI model training datasets".
Claude-SearchBot analyses content to improve search results. Blocking it "prevents our system from indexing your content for search optimization", which may reduce your visibility in user search results.
Claude-User fetches pages when someone asks Claude something. Blocking it prevents Claude from retrieving your content in response to a user query.

So if what you want is "don't train on my site, but do cite it", the file is short:

# Training: no. Search and user requests: yes.
User-agent: ClaudeBot
Disallow: /

User-agent: *
Disallow: /private/

That's the "No training, yes search" example in the demo. ClaudeBot has its own group and gets nothing. Claude-SearchBot and Claude-User aren't named, so they fall back to the * group and can read everything except /private/.

The same article has a few practical notes. The file has to go in the top-level directory of every subdomain you want to opt out. Blocking Anthropic's IP addresses instead "may not work correctly or persistently guarantee an opt-out", because it stops them from reading your robots.txt. They also support the non-standard Crawl-delay directive.

The trap: naming a bot makes it forget `*`

Now suppose you want Claude-SearchBot to focus on your blog and you add a small group for it. That's the "The * trap" example:

User-agent: *
Disallow: /private/

User-agent: ClaudeBot
Disallow: /

# Looks harmless...
User-agent: Claude-SearchBot
Allow: /blog/

Look at the /private/x column in the demo. Claude-SearchBot is now allowed there, and the cell gets an orange * badge. The reason is that a crawler follows one group only: the one that names it. If such a group exists, the * group is ignored completely, not merged in. Its only rule says Allow: /blog/, nothing in it matches /private/x, and a path that no rule matches is allowed.

This isn't a quirk of my demo. Google's open-source robots.txt parser, which its README describes as the production code Googlebot uses, has a test for exactly this: with Disallow: /x/ under * and Disallow: /y/ under FooBot, FooBot is allowed into /x/page. In the code, once a group for the agent has been seen, the matcher never looks at the global rules again.

The fix is boring: repeat in each named group the * rules that should still apply ("Trap fixed" in the demo).

User-agent: Claude-SearchBot
Allow: /blog/
Disallow: /private/

How a rule wins

Inside a group, order doesn't matter. Every Allow and Disallow whose pattern matches the path is a candidate, and the longest pattern wins. If an Allow and a Disallow are the same length, Allow wins. Patterns are anchored at the start of the path, * matches any sequence of characters, and $ only means "end of the URL" when it's the last character.

This is the matching function the demo uses, ported line by line from Google's robots.cc. It keeps a list of positions in the path that can still match the pattern read so far:

function rtMatches(path, pattern) {
  const n = path.length; let pos = [0];
  for (let k = 0; k < pattern.length; k++) {
    const c = pattern[k];
    if (c === '$' && k === pattern.length - 1) return pos[pos.length - 1] === n;
    if (c === '*') { const from = pos[0]; pos = []; for (let i = from; i <= n; i++) pos.push(i); }
    else { pos = pos.filter(p => p < n && path[p] === c).map(p => p + 1); if (!pos.length) return false; }
  }
  return true;
}

rtMatches('/fish.html', '/fish');  // true: rules are prefixes
rtMatches('/catfish', '/fish');    // false: anchored at the start
rtMatches('/x/y.html', '/*.html'); // true
rtMatches('/foo/bar/', '/foo/bar$'); // false: $ ends the URL

A rule's priority is simply the length of its pattern. To check that the port behaves like the original, I translated the assertions from the upstream test file (robots_test.cc, which says it tests against RFC 9309) to JavaScript: 151 checks on groups, case, longest match, encoding and special characters, plus the URL and escaping helpers. They all pass against the code shipped in the demo.

Gotchas

The user-agent token is case-insensitive, the path isn't. User-agent: claudebot applies to ClaudeBot, but Disallow: /Private/ doesn't block /private/.
Only letters, - and _ count in the token. Google's parser stops at anything else, so User-agent: Foo Bar is read as Foo. The demo's last row lets you type a token and flags invalid ones like Googlebot/2.1.
Several User-agent lines in a row share one group. That's the tidy way to give Claude-SearchBot and Claude-User the same rules.
An empty Disallow: allows everything for that group, and it still counts as a named group, so the * rules are still ignored.
robots.txt is a request, not a lock. It works when the crawler honours it, which Anthropic says its bots do. It doesn't protect private content.

What to keep an eye on

Your own named groups. Every time someone adds a User-agent: block for a specific bot, check which * rules it just dropped. The demo is a quick way to see it.
New tokens. Vendors split and rename crawlers. Read their own documentation for the exact names rather than copying a list from someone else's robots.txt.
This site. richi.codes currently has a single User-agent: * group with Allow: / (the "richi.codes today" example), so every bot that respects the file can read everything.