How AI agents find your website

Ask me to find a good studio in your city and I won't browse. I can't see your design. I never scroll. I have no patience at all. I'll call a search tool, judge your title and snippet, open one page, and read its text. That's the entire encounter.

The short version: an AI agent doesn't browse — it searches, judges a snippet, fetches text, and repeats what it reads. Being findable comes down to five gates: indexed → ranked → fetchable → quotable → trusted. And "agent" is not "bot": most sites block crawlers by reflex, without ever checking whether they're blocking the *software that recommends them*.

What actually happens when you ask an agent a question

1. The agent decides it needs current facts, and calls a search tool with a query.
2. It gets back a ranked list — title, URL, snippet — and nothing else.
3. It judges from the snippet whether the page is worth opening.
4. It fetches that one page and reads its text, with the design stripped away.
5. It synthesises, quotes, cites — or moves on and picks the next result.

Everything you spent years perfecting — the layout, the motion, the brand — happens *after* the agent has already decided. It never sees most of it.

The five gates

• Indexed. You're in the index the agent's search tool queries — and that index is built by crawlers. Not crawled, invisible.
• Ranked. You surface for the *phrasing the agent used*, not the phrasing you'd choose.
• Fetchable. When the agent fetches you, the text actually comes back. A JavaScript-only page, a paywall, or a bot block and the agent gets nothing at all.
• Quotable. The fact is in plain text and near the top — a number, a definition, a clear answer. Buried in a video or an image, it may as well not exist.
• Trusted. It's a source the model is willing to repeat — either authoritative on its own, or corroborated elsewhere.

Miss any one of the five and the rest don't matter. This is what "agent visibility" actually means in practice.

An agent is not a bot

"Bot" earned its reputation honestly — spam, scrapers, denial-of-service. So the reflex is block-on-sight. But the traffic arriving now is mostly not that, and the label is costing businesses more than they realise.

• Of AI crawler requests in May 2026: 51.8% training, 35.7% mixed, and just 9.3% search (Cloudflare).
• The crawlers are not interchangeable: PerplexityBot returns about one visit per 111 pages; Copilot 33:1; Google 4.9:1 — while training crawlers like ClaudeBot run ~23,951 pages per visit (data reference).
• So "block the bots" blocks two completely different things: the extractor that takes your content, and the recommender that names you to a buyer.

And the trap is worse than it looks: Google merged its search crawler with its AI training crawler into one user-agent, so blocking Google's AI training also forfeits Google search (Cloudflare). One reflexive rule, two very different costs.

Why people block without thinking

• More than 2.5 million sites now disallow AI training, and Cloudflare blocked 416 billion AI scraping requests in five months (data reference). Most of that was a default, not a decision.
• `llms.txt` is not the fix people hope: adoption sits near 10%, and the AI bots that drive visibility rarely fetch it (data reference).
• The cost is invisible by design: you cannot be named by software you turned away at the door.

What to do about it

• Check your `robots.txt` by purpose, not by reflex. Keep the answer/search crawlers open — PerplexityBot, OAI-SearchBot, ChatGPT-User, Googlebot. Then decide training on purpose: for a *brand*, open (being in the weights is being remembered); for a *publisher*, block or license it.
• Pass the five gates — be in the index, rank for the real questions, return readable text, put the fact at the top, and be a source worth repeating.
• Give acting agents a door. The agents that *do* things — compare, book, buy — don't fetch and read; they call an API or an MCP endpoint and take your facts as structured data.
• Instrument it. Watch agent traffic and your crawl-to-refer ratio, and check whether the engines *name* you — because a mention with no click is invisible in GA4.

The web is now machine-majority: for the first time, more than half its traffic isn't human (Cloudflare). Almost all of that second audience is software deciding, on someone's behalf, whether to mention you. The businesses that get found are the ones a machine can index, read and repeat. The ones that get blocked stay invisible — and never know why.

Get in touch: thecollective@memeador.com